AWS Certified Machine Learning - Specialty Exam

A I M 3 0 8 Build accurate training datasets with Amazon SageMaker Ground Truth

Warren Barkley Vikram Madan Kevin Dela Rosa GM, Augmented AI Product Lead ML Engineer, Perception Amazon Web Services Amazon Web Services Snap

• Massive scale: ML models need large, labeled datasets

• High accuracy: ML models depend on accurately labeled data

As a result, building the training dataset takes up to 80% of a data scientist’s time Amazon SageMaker Ground Truth

Build highly accurate training datasets Reduce data labeling costs by up to 70%

▪ Built-in features to improve label accuracy ▪ Automated data labeling capability ▪ Access to multiple workforces: Amazon Mechanical Turk and third-party vendors ▪ Option to bring your own workers and secure handling of data ▪ Tight integration with Amazon SageMaker pipeline More accurate and efficient data labeling

Active learning An accurate training model is trained from dataset is ready for use human-labeled data

Human-labeled data is then sent back to retrain and improve the machine learning model Built-in data labeling workflows

Image classification Bounding boxes Semantic segmentation

Custom

Text classification Named entity recognition Custom data labeling workflows

Learn more @ https://amzn.to/2OsREAk Human workforce options © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Labeling job creation © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Scan is powered by AI

Machine learning & computer vision are at the core of the scan: • Image classification • Object detection • Semantic segmentation • Content-based information retrieval • Ranking Etc. Machine learning – bird’s-eye view

Dataset Model Evaluation Deployment collection training Machine learning – bird’s-eye view

Dataset Model Evaluation Deployment collection training “Data is the new oil”

ImageNet ImageNet Many large-scale public CNN datasets are available; for data label example: • ImageNet (1M images ) • Open Images (9M images) New output Pre-trained • Places (10M images) New task layer for CNN data target Data is great …

Getting to state of the art

{ "label": "Hand", "score": 0.99794453 }  Your application’s data is different

{ "label": "Hand", "score": 0.9703855 }  New problem

We have relevant images we can learn from, but no labels Solution Amazon SageMaker Ground Truth

What we liked: • In AWS • We’re already using some AWS solutions in our training workflows (Amazon SageMaker training, Amazon S3), so it’s easy to point to data • Speed • We can get images labeled on-demand quickly • Flexibility to leverage public or private workforces to label data • Ability to kick off labeling jobs programmatically or via UI Integrating with Amazon SageMaker Ground Truth

Get target smInput := sagemaker.CreateLabelingJobInput{ images HumanTaskConfig: &humanTaskConfig,

InputConfig: &inputConfig,

LabelAttributeName: &attrName, Pre-processing LabelCategoryConfigS3Uri: &job.CategoryUri,

LabelingJobName: &fullJobName,

OutputConfig: &outputConfig,

Submit Ground RoleArn: roleArn, Truth job }

client.CreateLabelingJob(&smInput) Extract labels

Data labeling pipeline (Kubeflow)

Integrating with Amazon SageMaker Ground Truth

Get target Labeled images image list

Create TF Pre-processing record

Submit Ground SageMaker Truth job training

Extract Extract labels model file BigQuery

Data labeling pipeline Model training pipeline

Learn more @ https://bit.ly/35GzfH4 Impact

• We can gather labels for 1000s of images in hours • Incorporating new data improves our predictions • Our models do the right thing now

No longer labeled as hand { "label": "Hand", "score": 0.06862099 }  Tips for getting high-quality labels

Provide clear instructions • Show images of good and bad examples • Bootstrap this by running a small test set and gathering common mistakes

Opt for the smallest label set possible • If you have multiple potential labels, narrow down the field instead of exposing all possible labels

• Evaluate and improve your labels

• Make labeling easier for your labelers

• Use multiplicity to improve accuracy

• Measure accuracy & throughput of labelers

• Label only what you need to Evaluate and improve your labels

☐ Yes ☑ No

Learn more @ https://amzn.to/364LQFh Make labeling easier for your labelers

Learn more @ https://amzn.to/33IbiyL Auto-segment uses Deep Extreme Cut (DEXTR) algorithm Use multiplicity to improve accuracy

https://amzn.to/2N9PrsD Measure accuracy and throughput of labelers

Raw worker responses emitted to S3

{ "answers": [{"answerContent": { "crowd-classifier":{"label":"Athlete"}}, Response from worker 1 "submissionTime":"2019-10-16T03:25:56.656Z", "workerId":"private.us-west-2.2fa5a9d73ef73ba0", "workerMetadata": { "identityData": { "identityProviderType":"Cognito", "issuer":"https://cognito-idp.us-west-2.amazonaws.com/us-west-2_K2Rl3SHuq", "sub":"c9a8f4a4-ed4a-4dad-a722-8532d0d6016e“ } }

}, {"answerContent": { "crowd-classifier":{"label":"Animal"}}, Response from worker 2 "submissionTime":"2019-10-16T03:27:31.048Z", "workerId":"private.us-west-2.7dcbcca1ce3117d8", "workerMetadata": { "identityData": { "identityProviderType":"Cognito", "issuer":"https://cognito-idp.us-west-2.amazonaws.com/us-west-2_K2Rl3SHuq", "sub":"7eb0d3bc-2da5-4244-b14f-d9ec6ffe2e17“ } } }] } Measure accuracy and throughput of labelers

Amazon CloudWatch Logs & metrics for worker throughput

{ "worker_id": "cd449a289e129409", "cognito_user_pool_id": "us-east-2_IpicJXXXX", "cognito_sub_id": "d6947aeb-0650-447a-ab5d-894db61017fd", "task_accepted_time": "Wed Aug 14 16:00:59 UTC 2019", "task_submitted_time": "Wed Aug 14 16:01:04 UTC 2019", "task_returned_time": "", "workteam_arn": "arn:aws:sagemaker:us-east-2:############:workteam/private-crowd/Sample-labeling-team", "labeling_job_arn": "arn:aws:sagemaker:us-east-2:############:labeling-job/metrics-demo", "work_requester_account_id": "############", "job_reference_code": "############", "job_type": "Private", "event_type": "TasksSubmitted", "event_timestamp": "1565798464"

}

Learn more @ https://amzn.to/34N2eZQ Label only what you need to

“Not all data is created equal”

Check out the full blog @ https://amzn.to/2VZnBDv Label only what you need to

Check out the full blog @ https://amzn.to/2VZnBDv Learn ML with AWS Training and Certification The same training that our own developers use, now available on demand

Role-based ML learning paths for developers, data scientists, data platform engineers, and business decision makers

70+ free digital ML courses from AWS experts let you learn from real-world challenges tackled at AWS

Validate expertise with the AWS Certified Machine Learning - Specialty exam

Visit https://aws.training/machinelearning