Media Discovery and Compliance with Amazon Rekognition

A I M 2 3 2 Media discovery and compliance with Amazon Rekognition

Venkatesh Bagaria Simon Eldridge Principal Product Manager - Technical Chief Product Officer Amazon Rekognition SDVI Corp

1. What is Amazon Rekognition

2. Media customers and use cases

3. Getting started on your media applications

4. Optimizing media supply chains with SDVI The AWS ML Stack Broadest and most complete set of Machine Learning capabilities

AI SERVICES

VISION SPEECH TEXT SEARCH NEW CHATBOTS PERSONALIZATION FORECASTING FRAUD NEW DEVELOPMENT NEW CONTACT CENTERS

Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Amazon Rekognition Polly Transcribe Comprehend Translate Textract Kendra Lex Personalize Forecast Fraud Detector CodeGuru Connect +Medical +Medical with Contact Lens

ML SERVICES

SageMaker Studio IDE NEW Amazon SageMaker Ground Truth ML SageMaker data labeling Marketplace Neo Built-in SageMaker SageMaker Model SageMaker Model SageMaker algorithms Notebooks NEW Experiments NEW tuning Autopilot NEW hosting Model Monitor NEW

ML FRAMEWORKS & INFRASTRUCTURE

NEW NEW

Deep Learning GPUs & Elastic Inferentia FPGA NEW AMIs & Containers CPUs Inference Amazon Rekognition Image and Video

Object, Scene and Activity Content Moderation Text

Face Detection and Analysis Face Search Celebrity Recognition

Live Stream Video Pathing Amazon Rekognition Custom Labels

Customized Image Analysis to easily detect objects and scenes you define as most relevant to your domain Amazon Rekognition Custom Labels

Guided experience to create labeled images

Train and evaluate with no coding and no ML experience

Easy-to-use fully managed API Amazon Rekognition customers © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Rekognition Video for Media Media use cases for Amazon Rekognition

Compliance Detect potentially inappropriate content to avoid issues in global markets, and to & Moderation increase brand safety for advertisers Quality Control

Discovery & Search Automate the creation of a rich metadata index (objects, people, scenes, activities, spoken words) to easily find and monetize your content Monetization

Marketing Contextual Make advertising relevant to the context of the content. Avoid brand damage Advertising through unintended associations. & Content Advertising Understand content performance based on metadata and user preferences Performance

Amazon Confidential Compliance and Quality Control Human review is tedious, non-scalable and error prone Compliance and Quality Control AI services reduce the amount of content that needs human review

Amazon Rekognition Video Custom Labels Amazon Transcribe Human review Compliance and Quality Control Moderation with Amazon Rekognition

Top-level category Second-level category Explicit Nudity Nudity Graphic Male Nudity Graphic Female Nudity Sexual Activity Instead of providing just a Illustrated Nudity Or Sexual Activity Adult Toys 'Safe' vs 'Unsafe' result, Suggestive Female Swimwear Or Underwear Male Swimwear Or Underwear Amazon Rekognition returns Partial Nudity Revealing Clothes a hierarchical list of labels Violence Graphic Violence Or Gore Physical Violence with confidence scores Weapon Violence Weapons Self Injury Visually Disturbing Emaciated Bodies Corpses Hanging Compliance and Quality Control

“At CBS, we place significant efforts to ensure we moderate inappropriate content within our programming as to not offend our global viewers or violate government regulations. This is supported by investments in manual methods to execute near real-time screening and editing of hundreds of hours of content every month. To scale our internal processes, we are looking to Amazon Rekognition to automate the moderation of our video content while leveraging the new feature of Custom Labels to further refine moderation models. This will enable us to automate the tagging of sensitive content such as nudity, obscene gestures, and violence, and speed up processing from hours to minutes.”

– Jamie Duemo, Senior Vice President, Multi-Platform Distribution, CBS Operations and Engineering Discovery and Monetization

Manually search all videos for clips with specific people, objects, scenes, activities, animated characters, or spoken words Discovery and Monetization

Amazon Rekognition Video Custom Labels Amazon Transcribe

Easily find specific videos and clips within your library using rich metadata from AI services

“All “Allvideos“All “Allvideos videoclips videos clips withclips clips with ” ” Character>” phrase>” Discovery and Monetization Celebrity Recognition and Face Search

Identify well known people in your video and image libraries to catalog footage and photos

Identify a person in a photo or video using a custom repository of face images Discovery and Monetization Object, Scene and Activity detection

Identify thousands of objects and scenes in images and videos; activities for video Discovery and Monetization Custom Labels

Comic Disc

Arthur and Buster Rock and Roll Hall of Fame Gold Record Discovery and Monetization

“Together with GrayMeta, using Amazon Rekognition has enabled us to make great progress in the last year to analyze, identify and tag content, including facial and object recognition of our 3,000 programs, which highlight our five-decade, 18,000-hour archive – the Videofashion Library. Previously, we had to manually search and make notes of each frame, logging everything in file boxes and cards with handwritten notes. Amazon Rekognition has helped us automate the process and drive significant savings by enabling us to quickly search and find content in our collection while driving revenue by unlocking the value of our archive. What used to take over an hour to watch and review now takes us 10 minutes, an 88% time saving. GrayMeta provided a number of services to Videofashion, including the migration of tape content into AWS, and through the deployment of GrayMeta Curio, the ability to visualize and use Rekognition data.”

– Anne V. Adami, Videofashion’s Co-Founder/Owner and Managing Editor Discovery and Monetization

“In today’s media landscape, the volume of unstructured content that organizations manage is growing exponentially. Using traditional tools users can have difficulty in searching through the thousands of media assets in order to locate a specific element they are looking for. By using the new feature in Amazon Rekognition, Custom Labels, we are able to automatically generate metadata tags tailored to specific use cases for our business and provide searchable facets for our content creation teams. This significantly improves the speed in which we can search for content and more importantly it enables us to automatically tag elements that required manual efforts before. These tools allow our production teams to leverage this data directly and provides enhanced products to our customers across all of our media platforms.“

- Brad Boim, Senior Director, Post Production & Asset Management, NFL Media Marketing and Advertising

{Beach,Family Family}Ad Vacations Break Ad {Soccer}SoccerAd CleatsBreak Ad

Contextual Advertising - Match advertisements to video content Marketing and Advertising

{Puppy} Brand Logo {Close-up face, Smiling, Girl}

Content Performance - Find which visual components matter Marketing and Advertising Face Analysis

Get attributes such as gender, age range, emotions expressed, smile, eyes open, glasses, and more for each detected face Marketing and Advertising

“With the introduction of Amazon Rekognition Custom Labels, marketers will be equipped with advanced capabilities within our Agile Creative Studio, enabling them to build and train the specific products (custom labels) that they care about within their ads, at scale, within minutes. Using VidMob’s integration of Amazon Rekognition, customers have historically been able to identify common objects but now the new ability for custom labels will make our platform even more targeted for every business. With a lift of 150% in creative performance and 30% reduction in human analyst time, this will adaptively extend their ability to measure their creative performance using VidMob’s Agile Creative Studio”

{ "Image": { "Bytes": blob, "S3Object": { "Bucket": "string", "Name": "string", "Version": "string" } }, "MaxLabels": number, "MinConfidence": number { } "Labels": [ { "Confidence": number, "Name": "string”, "Instances": [], DetectLabels "Parents":[] . . } ], ”LabelModelVersion": ”number" } Video API - Request and Response

{ { “ClientRequestToken": "string", "JobStatus": string, "JobTag": "string", "StatusMessage": string, "MinConfidence": number, "VideoMetadata": { "NotificationChannel": { "Format": string, "RoleArn": "string", "Codec": string, "SNSTopicArn": "string” "DurationMillis": number, }, "FrameRate": float, "Video": { "FrameWidth": number, "S3Object": { "FrameHeight": number "Bucket": "string", }, "Name": "string", "NextToken": string, "Version": "string” "Labels": [ } { } "Timestamp": number, } "Label": { StartLabelDetection "Name": string, "Confidence": float, JobId ”Instances": [], ”Parents": [], } GetLabelDetection } ], ... AWS Media2Cloud Solution

A serverless framework to ingest video archives into AWS, augment its metadata using machine learning services like Amazon Rekognition and Amazon Transcribe, and export assets into a Media Asset Management (MAM) system AWS Media2Cloud Solution

1. Drain the Existing Archive 2. Generate Content Value 3. Import into a MAM

MAM system Ingest dashboard

File transfer Authentication Amazon Cognito Amazon API Gateway Workflow orchestration Notifications Amazon Step Functions Amazon SNS Message broker AWS IoT Core

Pack & ship Transfer Accelerator Event driven functions AWS Snowball Amazon S3 Standardized Assets + Metadata

Ingest Proxies Metadata Archived content High speed link Ingest bucket Amazon S3 Amazon Lambda Amazon Lambda Amazon Lambda AWS Direct Connect Original metadata On-prem agent On-prem agent AWS Storage AWS DataSync UUID / MD5 Gateway Lifecycle Proxies / thumbnail policy Proxies / Machine learning Asset registry thumbnails metadata Media metadata

Machine learning Rapid Migrate MP4 Celebrities, labels, metadata UUID tag AWS Elemental faces, collections MediaConvert

MD5 checksum Archive storage Transcription Amazon Glacier

Sentiment, key phrases, Metadata Technical metadata Amazon DynamoDB LTO Migration entities, locations Crowdsource Tagging Master asset + Service Pool filename + barcode Hot search Crowdsource Tagging Amazon Elasticsearch Resources

Video Documentation: https://docs.aws.amazon.com/rekognition/latest/dg/video.html

Custom Labels: https://aws.amazon.com/blogs/machine-learning/announcing-amazon- rekognition-custom-labels/

Media2Cloud Solution: https://aws.amazon.com/solutions/media2cloud/ Using ML to optimize media supply chains

Simon Eldridge Chief Product Officer SDVI Corporation

Applying supply chain methodologies to the media industry with cloud native SaaS solutions

Rich broadcast and media heritage

AWS Technology Partner since 2013

Cloud-native architecture, Amazon Rekognition beta launch partner Archiving

Localization Normalization

Content Modification Receipt Distribution

Logging What is a media supply chain? Optimizing your media supply chain

EFFICIENCY AGILITY GROWTH Increase output by Adapt to evolving Gain rapid access to new automating repetitive operational business opportunities tasks, assigning people requirements as market and accelerate time to where they truly add demands shift. revenue by value. Clear, reliable cost Freedom to choose the implementing a true on- analysis and prediction best tools at any time. demand, vendor- provide actionable Gain full visibility, agnostic, elastic supply intelligence. security, and financial chain platform. control. How can I use ML data to optimize?

Agility Quickly adapt to new requirements Expand supply chain for new use cases Efficiency Locate content rapidly Accelerate content processing Growth Create automated relationships between content Do more with limited resources The Rally Media Supply Chain Platform supports technical agility, operational efficiency, and business growth by providing a scalable, dynamic infrastructure access

Adobe Premiere Panel, providing access to all time-based metadata from automated QC and AI services

• Assisted QC & fix • Compliance versioning • Segmentation • Distribution versioning • Caption validation • Metadata tagging • Project & EDL export and render Using ML data for Quality Control and Compliance

1. Create 2. Analyze 3. Run 4. Normalize 5. Review & proxy video using automated incoming process Amazon QC data content Rekognition Rekognition-Rally integration What Rally sends to Rekognition

self.topicArn, self.queueUrl = self.createTempTopicAndQueue(jobUuid) rv = rekognition.start_label_detection( NotificationChannel={ 'SNSTopicArn': self.topicArn, 'RoleArn': roleArn }, Video={ 'S3Object': { 'Bucket': jobInputs[0].file.bucketName, 'Name': jobInputs[0].file.keyname } } }) self.rekognitionJobId = rv['JobId'] except ClientError as ce: self.cleanUpTempTopicAndQueue() { Results and normalization "Labels Detection": { "occurrences": [ { "attributes": { "_durationSeconds": 0.033366700033366704, "_end": "00:01:39;28", "_durationFrames": 1, "message": "Blossom", "_start": "00:01:39;28" "Labels": [ }, { "tags": ["Labels Detection"], "providerName": "Rekognition", "Timestamp": 1968, "source": { "Label": { "end": "00:01:39;28", "Instances": [], "start_fr": "2996", "end_fr": "2996", "Parents": [ "confidence": 96.46537017822266, { "code": "GetLabelDetection", "Name": "Plant" "attributes.Confidence": 96.46537017822266, "attributes.Parents.0.Name": "Plant", } "message": "Blossom", ], "location": "Video Track 1", "Name": "Blossom", "start_ts": "1968", "attributes.Name": "Blossom", "Confidence": 96.46537017822266 "start": "00:01:39;28", } "end_ts": "1968" }, }, "name": "Blossom", ] "message": "Blossom", "location": { "start": "00:01:39;28", "durationFrames": 1, "end": "00:01:39;28", "durationSeconds": 0.033366700033366704 }, "severity": "warning" } }

“We saw a 70% improvement in time to market, with 80% cost savings.” “We’ve increased revenues 23%, while reducing operating costs by 25%.” “We now have 80% lower human involvement in our media processing.” Want more information on how Rally leverages AWS ML services to optimize supply chains?

Blog – https://amzn.to/2qNGwWi Contact us - www.sdvi.com - @SDVICorp - [email protected] Contact me - [email protected] - @eldrix4

Simon Eldridge [email protected] @SDVICorp

Blog post available at https://amzn.to/2qNGwWi