ML Fridays Rapidly Build State of the Art Image Classification and Detection Models

ML Fridays Rapidly build state of the art image classification and detection models

Sumir Kumar/Arun Lokanatha Kumar Solution Architects Date : 30-04-2021

• A brief background of Computer Vision

• Introduction to Amazon Rekognition Custom Labels

• Walkthrough of Rekognition Custom Labels workflow

• Demo with multiple use cases

• Evaluating improving model performance.

• Customer Examples

• Wrap-up and Q&A

AI Services

VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS

AMAZON AMAZON AMAZON AMAZON AMAZON AMAZON AMAZON AMAZON AMAZON AMAZON REKOGNITION REKOGNITION TEXTRACT POLLY TRANSCRIBE TRANSLATE COMPREHEND LEX FORECAST PERSONALIZE IMAGE VIDEO & COMPREHEND MEDICAL ML Services

Amazon SageMaker Ground Truth Notebooks Algorithms + Marketplace Reinforcement Learning Training Optimization Deployment Hosting

ML Frameworks + Infrastructure

FRAMEWORKS INTERFACES INFRASTRUCTURE

E C 2 P 3 E C 2 G 4 FPGAS D L E L A S T I C E L A S T I C GREENGRASS ELASTIC INFERENTIA & P 3 D N E C 2 C 5 CONTAINERS CONTAINER KUBERNETES INFERENCE & A M I s SERVICE SERVICE

Labels (Object, Scenes Face Search and Activities)

Unsafe Image and Face Detection Video Detection and Analysis

Text in Image Celebrity Recognition

Person Pathing Real-time video analysis

Helmet 97.4% Outdoors 99.7% Objects Nature 99.7% Scenes

Bounding boxes for common objects Person 99.6% Confidence and hierarchy Snowboarding returned per label 99.6% (e.g., Snow  Nature)

Snow 99.6%

Customization “Amazon Rekognition Labels is not specific enough for what I need for my business”

Amazon Rekognition Machine Part Machine Part Machine Part Labels

Deep Machine Tens of thousands of Several weeks Learning Expertise labeled images to months to (hard to find) (expensive and time complete consuming)

Scaling to peak Error-prone and needs can be subjective leading to expensive inconsistencies

Customized Image Analysis to easily detect objects and scenes you define as most relevant to your domain

Guided Experience Train and evaluate Easy-to-use fully to create labeled with no coding and managed API images no ML experience

Less training data Customized to your From tens to hundreds of images, business need instead of tens of thousands

Custom image analysis in hours Customize and use model in No ML expertise required hours, instead of several weeks or months

Business users create ML models Intuitive UI. Zero lines of code. Fully managed by AWS

Single Label Classification Multi-Label Classification Object detection “Which class of objects does “Which combination of classes “What object is present, this picture represent” are represented in this image” and where is it?”

“Premade foods” Buster Arthur

“Produce” “Mountains, Dusk”

“Premade foods” Buster Arthur

“Produce” “Mountains, Dusk”

Required Resources (Time, Compute, Data)

DetectLabels recognizes common classes of objects and scenes. Other Amazon Rekognition APIs offer specialized capability for content moderation, faces, or text Custom Labels provides scalable API to custom models that recognizes specific objects and scenes, each potentially different Picture by Hay Kranen / PD, Wikipedia

Model generalization: Pre-trained models that transfer to new tasks

Model selection: Train a set of models, choose best performing model

Specific use case Train and Choose best Custom (objects, scenes) Optimize performing model model

• Label data by hand in the Amazon Rekognition Custom Label Console • Add images by dragging and dropping them into your dataset gallery view. • You’re limited to uploading 30 images at one time.

• Import images from Amazon S3 bucket • Use images from an existing S3 bucket by entering the S3 folder location. • Automatically add labels based on your folder names.

• Import images labeled by Amazon SageMaker Ground Truth • Provide the location of your .manifest file. • Also supports other .manifest files not created in Ground Truth, as long as they are formatted properly.

APIs Description

CreateProject Creates your project Create your Model CreateProjectVersion Trains your model

StartProjectVersion Starts your inference end point

Using your model DetectCustomLabels Calls your custom model

StopProjectVersion Stops your inference end point

DescribeProjects Describes your project Control Plane APIs DescribeProjectVersions Describes your model

CreateProjectVersion (Train Custom Labels)

Where your output goes Specify your model

"OutputConfig": "ProjectArn": "string" { "S3Bucket": "string", "S3KeyPrefix": "string” }

Training Dataset Details Testing Dataset Details

"TrainingData": { "TestingData": { "Assets": [ "Assets": [ { { "GroundTruthManifest": { "GroundTruthManifest": { "S3Object": { "S3Object": { "Bucket": "string", "Bucket": "string", "Name": "string", "Name": "string", "Version": "string”}}}]} "Version": "string" }}}]}

• DetectCustomLabels aws rekognition detect-custom-labels --project-version-arn "model_arn"\ --image '{"S3Object":{"Bucket":"bucket","Name":"image"}}'\ --min-confidence 70

{ "ProjectVersionArn": "string", "Image":{ "S3Object":{ "Bucket":"string", "Name":"string", { "Version":"string" "CustomLabels": [ } { }, "Name": "MyLogo", "MinConfidence": 90, "Confidence": 77.7729721069336, "MaxLabels": 10, "Geometry": { } "BoundingBox": { "Width": 0.198987677693367, "Height": 0.31296101212501526, "Left": 0.07924537360668182, DetectCustomLabels "Top": 0.4037395715713501 } } } ] }

Start and stop the model endpoint

Assign inference units • Priced at inference unit per hour • Supports 5 transactions per second (TPS)

“We need to find the oral history interviews we have, which were shot with golden record awards in the background”

Damaged Infected Healthy cucumber leaves cucumber leaf cucumber leaf

“Find Arthur and his friends in every video and image”

Arthur Buster Buster

• Train models with only bounding box or image-level labels for best performance • Increase train and test samples as variability of your object and images increases

Precision is the fraction of correct predictions (true positives) over all model predictions (true and false positives) at the assumed threshold for an individual label.

Recall is the fraction of your test set labels that were predicted correctly above the assumed threshold.

Model performance is an aggregate measure that takes into account both precision and recall over all labels (for example, F1 score or average precision)

The assumed threshold for each label is the value above which a prediction is counted as a true or false positive. It is set based on your test dataset.

It is important to note that Rekognition Custom Labels does not return “accuracy”

• It instead displays F1 score • F1 score is calculated as the average of precision and recall

• Models with high precision will report a low number of false positives • You want your home security system to be very precise; when in doubt assume ‘no’ • If it is unsure, it will chose not to apply any label

• Models with high recall will report a low number of false negatives • You want your COVID-19 test to have high recall; when in doubt assume ‘yes’ • If it is unsure, it will chose not to apply too many labels

Increase Accuracy Increase throughput Decrease Cost

• Balance your dataset better • Shrink the size of your images • Don’t reinvent the wheel (if Rekognition labels already • Increase resolution of training • Use classification model instead does something, use that images of detection model instead)

• Increase number of training • Consider using raw bytes instead • Consider batching images if you images of S3 cannot use one full inference unit • Try classification model versus detection model • Increase throughput of model to require less • Ensure bounding boxes are drawn inference units precisely

• Smart Camera Tagging • Identify specific items of interest in a camera feed, like a vehicle or animal • Detect items or patterns (like downed power lines) from satellite imagery

• Document Classification • Particularly helpful in cases where images are visually rich and text light (think ID documents)

• Logo/Character Recognition In Media & Social Media • Identify logos or characters in media for easy searching and tagging

• Manufacturing Automation • Uses Rekognition Custom Labels to detect defects • Also utilize Amazon Rekognition DetectText to identify serial numbers

Less training data Customized to your From tens to hundreds of images, business need instead of tens of thousands

Custom image analysis in hours Customize and use model in No ML expertise required hours, instead of several weeks or months

Business users create ML models Intuitive UI. Zero lines of code. Fully managed by AWS

© 2020, Amazon Web Services, Inc. or its Affiliates. © 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved | 44 Getting Started https://rekognition-immersionday.workshop.aws/custom_labels.html

To start and stop and test from webapp use below solution:- https://github.com/aws-samples/amazon-rekognition-custom-labels- demo

Create larger model datasets using model assistance:- https://github.com/aws-samples/amazon-rekognition-custom-labels- feedback-solution

Continuous model improvement workflow:- https://github.com/aws-samples/amazon-rekognition-custom-labels- a2i-automated-continuous-model-improvement

Can contact: [email protected]