ML for EV3 with Scratch3 Camp II - Check-In

• Pre-survey for students: https://forms.gle/YuYgVMpmWKpRMgsp7

Check the Zoom Chat for the link (Only for new students. Not for returning students) • Test your ML account login. Ask your coach if you do not have credentials • Scratch to EV3 communication is NOT required today • We need a mask for a ML project today

Aug 14, 2020

Recorded lecture video will be available after the camp.

CJ Chung, Ph.D. Professor of CS Founder & Director of Robofest

Lawrence Technological University

Winners from the 1st Camp on July 10th . 1st place: https://youtu.be/BZzX7x26NII (Smart Mask Detector) . 2nd place: https://youtu.be/CmnwfPzYXIE (Smart Shopper)

Winners from the 2nd Camp on July 24 . Co-1st place: https://youtu.be/UEgeDiKJ6rc (Recognizing handwriting to turn on/off a EV3 robotic fan) . Co-1st place: https://youtu.be/LH5pInLapMo (Mood Recognition) . 3rd place: https://youtu.be/N14C1yYEP2U (Recognizing text as Sanitize or Sanitized)

. Details about the mini project competition will be announced at the end of today’s meeting . Top teams will win prizes sponsored by

In ML, Machines are taught to learn from example data

Math Model on computer

After training

Trained Math Model unseen data on computer

Group label: 0 Group label: 1 Group label: 2 Group label: 3 Group label: 4

? (3 5 4) unseen data

AI: mimics the intelligence of behavioral pattern of humans or other living entity

. ML is a technique where computers can Math be trained by using sample data to Model output answers without being explicitly on computer programmed. . ML algorithms build a mathematical model based on "training data", in order to make predictions or decisions without After training being explicitly programmed to do so. Trained Math Model Unseen data on computer

If n < 4.5, then dark

.Project 1: Face detection - Red Nose (Using pretrained ML model) .Project 2: Mask NoMask •2a: Images from the web, your ML model, no coding •2b: Images from your webcam, your own ML model, and Scratch coding .Project 3: Pose detection – Wrist Up or Down (Using pretrained ML model & Your ML model)

. Click on first . No need to sign up. Your coach has a username and password for each team . It is a paid account by usage. If you do not use it for 30 days, it may be closed . Your account can have up to 4 projects . Multiple devices can use the account. But projects in the account must be shared . If you do not have an account, please ask by Zoom-chat NOW!

. https://youtu.be/EjbHXMzeX4c (22min)

.Text .Number

.Image “Turn on the fan” or .Sound “It is hot here” Pac-Man: to avoid Ghost. See next slide

“&…..@$^” sound in alien language to go left Rock, Paper, Scissors

Do not need to follow. Please Just watch

Copyright © Chung 2020 23 Review: Scratch Block Categories . Motion blocks deal with the movement of sprites. They relate mainly to the x and y position and direction of the sprite. . Looks blocks are related to the appearance of sprites and the stage. . Sound blocks control sound. . Events blocks are related to various triggers in a project, or when one part signals another to run. . Control blocks run the basic flow of a project in the desired fashion. . Sensing blocks associate with sprites and the stage detecting conditions. . Operators blocks deal with many mathematical functions. . Variables blocks are for storing and accessing data. . My blocks are blocks that hold custom procedures for a selected sprite. . Images blocks contain costume & backdrop image reporters

Hat blocks are the C blocks loop the blocks blocks that start every or check if a condition is script. true.

Reporter blocks can hold Stack blocks perform numbers and strings. commands.

Boolean blocks are the Cap blocks are the blocks conditions — they are that end scripts. either true or false

Code on backdrop

It will take a few seconds to start the code. If no x & y updates, restart the script

1. Send current image to pre-trained model 2. The model sends back the x coordinate of the nose on the image

. Use a pre-trained machine learning model to do face detection on a live webcam video, and code animated effects on your nose . No need to use ML account. Go to: https://machinelearningforkids.co.uk/pretrained

. Click on “Get Started” button. . Click on button at the bottom left to open the Extensions window. . Click on the Video Sensing extension. . Open the Extensions window again. . Click on the Face detection extension to use the pre-trained model. . Delete the cat sprite by clicking the trash can icon. . Click on the green “cat+” button to choose a Sprite.

. Click on “Food” button and click on Strawberry . Click on “Costumes” tab . Select the leaves of strawberry and . press Delete key to remove the leaves . Click on Flip Vertical, then you will get . Reduce the size from 100 to 50 . Click on “Code” tab

. Check the New blocks from the 2 Extensions

Set x coordinate of the current Sprite

The size of the red- nose is proportional to the size of my nose

Turn on your camera and say “ML”

Model

Trained Mask Model Unseen image

. Go to: https://machinelearningforkids.co.uk/ . Click on the blue “Get Started” button . Log in your ML account. Please Zoom-chat if your team does not have an account . Click on Go to your Projects . . Enter Project Name and Recognising: . Click on CREATE button . Click on the Project: . Click on “Train” button

. Click on ; Create “Mask” and “NoMask” buckets . Open another browser tab and Google search “Mask face”. Click on “Images” . Drag 10 jpg or png pictures from the other browser windows and drop them to “Mask” bucket . Now Google search “Face” and click on Images . Drag 10 pictures from the other browser windows and drop them to “NoMask” bucket . Back to project . Click on “Learn & Test” button . Click on . Quiz time! (Around 10 minutes)

. When training is done, enter a web image address and press “Test with www” button

Stop Recording

Model

Scratch Trained webcam Mask code Model

. Go to: https://machinelearningforkids.co.uk/ . Click on the blue “Get Started” button . Log in your ML account. Please chat if your team does not have an account . Click on Go to your Projects . . Enter Project Name and Recognising . Click on button . Click on the Project . Click on “Train” button

. Click on ; Create “Mask” and “NoMask” buckets . Click on . Add 10 new examples of each label . Click on < Back to project . Click on “Learn & Test” button . Click on . Quiz time! (Around 10 minutes) . After the training is completed, click on

. Click on “< Back to project” . Click on “Make” button . Click on “Scratch 3” button . “Open in Scratch 3” . Check Your Project is included . Add Video Sensing Extension . Remove Sprite (code on backdrop) . (optional) Add Extension EV3

// If you do not wear the mask, you will hear the pop sounds

“Code on backdrop”

How do we draw the yellow circle (sun)?

Your Scratch code gets x, y coordinates of your right wrist Wrist Up Wrist Up from the pre-trained model and moves the current sprite (yellow circle) to the point

Wrist Down Wrist Down

. How to train your ML model to recognize wrist up or down? • Your scratch code adds training data (x, y coordinates) of “wrist up” examples to the “Up” bucket • Scratch code adds training data (x, y coordinates) of “wrist down” examples to the “Down” bucket Wrist Up • Train the model . How to recognize whether Up or Down? • Get the x, y coordinates of the wrist from the pre- trained model Wrist • Send the x, y coordinates to your trained model and Down get the label, either Up or Down

Model

Numbers, x and y

(x, y) (x, y)

Scratch code Trained (x, y) Down Model webcam

Copyright © Chung 2020 48 Project #3: Pose detection – Wrist Up or Down (1/4) . Go to: https://machinelearningforkids.co.uk/ . Click on the blue “Get Started” button . Log in your ML account. Please chat if your team does not have an account . Click on “Go to your Projects” . . Enter Project Name as “Wrist Up Down” and Recognising: “numbers” . Enter x and y values:

. Click on button . Click on the Project . Click on “Train” button . Click on ; Create “Up” and “Down” buckets (labels)

Copyright © Chung 2020 49 Project #3: Pose detection – Wrist Up or Down (2/4) . Click on “< Back to project” . Click on “Make” button . Click on “Scratch 3” button . . Check Your Project, is included . Add Extensions: Video Sensing & Pose Detection . Delete the cat sprite by clicking the trash can icon . Click on the cat+ button to choose a Sprite . Search “Sun” and Click on

. Rename the Sprite from “Sun” to “Right Wrist”; reduce size to 40

Hard to change to y: see tip

Tip: change when you see the mouse pointer, not hand

1. Create Project with labels 2. Collect data examples by using sensors in Scratch 3. Check the data collected in the Project; Add additional examples manually if needed 4. Train by Scratch code 5. Test with your Scratch code 6. If every test case works correctly, end data collection process; Otherwise Go to 3.

. Create any new project using the ML techniques you have learned. . Just ML project without Lego EV3 is fine. . Send me a video link and Scratch_file.sb3 by Sunday Aug 16 11:59pm, EDT . Selected 2 teams will win certificates and prizes . To get some ideas for vision related projects, • go to: https://machinelearningforkids.co.uk/#!/worksheets . Face Finder . Emoji Mask . Laser Eyes . Semaphore • Counting the number of wrist up & downs

a) More examples b) Fewer examples

98 9

Copyright © Chung 2020 60 Q. Which of these is NOT a good use of machine learning? Select 4. a) Sentiment analysis – recognizing the mood, tone, or emotion in text b) Translating text from one language to another c) Face recognition – recognizing a face in a photo and identifying who it is d) Road/Lane following for self-driving cars e) Recommending a movie based on films you have watched before f) Calculating the cost of something in different countries based on exchange rates g) Recognizing if an email is spam h) Adding big numbers together i) Counting the number of times a button is pressed j) Converting Celsius to Fahrenheit Copyright © Chung 2020 61 Q. The script detects the location of a nose. Select a closest answer that the code might display in the blue boxes below.

x y

. https://forms.gle/gmEzZX6KSra3uRhi8

. Mission tasks will be totally unknown until the day of competition . Since robots are built and coded at the competition, there is not a lot needed to prepare ahead of time. Practicing building a simple base quickly and brushing up on programming (driving to an edge, line following, measuring distances, calculating and displaying values) would be good preparation.

. Aug 28 (Junior), 8am (EDT) . Aug 29 (Senior), 8am (EDT) . A few spaces are available . Registration fee: $10 . Previous year challenges were posted on the UMC page at: https://robofest.net/index.php/current-competitions/unknown-mission-challenge

You may use this document as long as you give the instructor a credit CJ Chung (2020). “Introduction to Machine Learning (ML) for EV3 with Scratch3”, Robofest Online Summer Camp, Aug. 14 2020, www.robofest.net