Cloud Derby Hackathon This project is designed to help you learn Google Cloud Platform (GCP) in a fun way. Your objective is to build a self driving robot that plays a ball game against other robots. We provide all the instructions to build hardware and software, but you are free to experiment, add, or change anything you like. You will be learning and using various Google Cloud services:
● IoT Core ● App Engine (GAE) ● Dialog Flow ● Pub/Sub ● Cloud Functions (GCF) ● Security (IAM) ● Cloud Storage (GCS) ● TensorFlow ● Cloud Networking ● Compute Engine (GCE) ● Machine Learning (ML) ● Cloud SDK
See source code, list of the parts and rules of the game here: https://www.cloudderby.io.
Here are the high-level steps that you will complete to install and run the game. Each of these is described in the next section. 1. Complete the initial Project Setup. 2. Take and Prepare Images for ML Training. 3. Deploy and start Object Detection Module on the cloud. 4. Deploy and start the Driving Controller Module on the cloud. 5. Provision and start GoPiGo Car software on the car. 6. Go to the Driving Controller Web App and start the game. 7. Build Dialogflow agent to control the car with your voice.
For those who are interested in hosting Cloud Derby event, the list of materials and preparation steps for the event can be found on this site. Overview
There are two subsystems in this game: (1) your robot car, and (2) Google Cloud Platform. Software running on the cloud controls the movement of your car. In order to do it, several interconnected modules work together and interact with the software running on the car via PubSub and IoT Core. We use Dexter GoPiGo cars for this game with wide angle camera and distance sensor. Please see the logical architecture below.
● Driving controller (Navigation Logic) This component is responsible for consuming sensor information from the car and making driving decisions based on that data. It uses ML and other logic to make driving decisions in manual or automatic mode and send commands back to the car.
● Object Detection (Machine Learning APIs) This component is responsible for processing images and detecting locations of colored balls using
cloudderby.io 1
TensorFlow Object Detection API customized with the transferred learning to be tailored to colored balls.
● GoPiGo (Car) Software on the car runs inside of the Raspberry Pi and uses Google Cloud IoT Core to communicate to the Cloud Derby modules running on GCP. The car takes pictures and laser distance measurements and sends those to the cloud for processing. Cloud components will process this input and make decisions on the car movements and send it back to the car. The car software accepts those messages and makes turns and drives based on those commands from the cloud side logic.
During the building and training sections of the event, you will also use a Development environment
● Sensor simulator Sensor Simulator Module generates a hard coded set of signals faking real sensor information and sends it via PubSub to the cloud. This is useful for development and debugging and is not used for the game itself.
● Driving simulator Driving Simulator Module pulls commands from the Cloud and prints to the console. This is useful for development and debugging and is not used for the game.
Conventions
This event can be run in two different flavors: as an 8 hours event and a 4 hours event. In this manual we are using a convention to highlight what steps to do for each of these two kinds of events. No designation for the step means that the step needs to be done for both 8 and 4 hours events. In cases when you see the following:
cloudderby.io 2
- perform this step only if you are doing the 8 hours version of the derby.
- perform this step only if you are doing the 4 hours version of the derby.
We recommend that you copy & paste the commands to execute them in command line. If you type a command or its options incorrectly you may get all kinds of errors, including “permission denied” etc. because the project ID or other resource does not exist. Commands that you need to type on the command line are shown like this:
cat code.py
At times you’ll see a box with a Google beanie. This box contains interesting information or deeper level geek stuff that isn’t necessary to do to complete the lab. Feel free to explore or ignore.
Let’s get started!
cloudderby.io 3
Project Creation and Setup (15 mins)
Each team member should follow all of the steps in this tutorial independently in their own project. However, be aware that only one car is provided for the team and you will select one of your team’s GCP projects to be used to control the car at the end of the day during the competition.
In order to deploy and run Cloud Derby game you need to set up a project to develop and deploy your code. The following steps will help you to set up the development environment.
Setting up your GCP environment Pick up your user account card from the instructor if you are in the Google organized event. If you are doing this project on your own, feel free to use your own GCP account.
In a Chrome window, login to GCP Console http://console.cloud.google.com with the credentials provided to you by the instructor (for example: [email protected]).
If you are an instructor hosting Cloud Derby event, you can generate GCP user accounts using create-hackathon.sh script provided you are 1) Google Admin Superuser and 2) GCP Org Admin.
Create a new project
In the Google Cloud Console GUI, select the three bar menu at the right of the top menu bar and choose the sub-menu option IAM & admin > Manage Resources:
cloudderby.io 4
You may need to select proper organization cloudderby.io in the drop down menu:
Note that if you are not doing this project as part of the Google organized event and are working with your own organization (acme.com, etc.), you need to select your own organization and come up with your own folder structure. It does not have to match instructions below for names of folders.
Click CREATE PROJECT in the top sub-menu:
cloudderby.io 5
On the New Project page, enter a unique Project name that will be easy for the team to recognize and remember (for example: team15-dish-derby) .
Ensure that cloudderby.io is listed as the Organization.
Under Location click on the Browse button as you’ll need to select the folder that matches your team number under the folder
NOTE: Your user ID only has permission to create a project in your Team’s folder. All other folders will have warning signs indicating that you don’t have permission to create projects in them.
Folders allow companies to map their organization onto GCP and provides logical attach points for access management policies and Organization policies. Organizations can use folders to group projects under the organization node in a hierarchy. For example, your organization might contain multiple departments, each with its own set of GCP resources. Folders allow you to group these resources on a per-department basis.
To select the location, expand the cloudderby.io org node.
cloudderby.io 6
Expand the
Click on the team folder provided to you by the instructors (for example Team-1-resources) .
Take note of your “Project ID” to the right of the project name. This ID will be used for subsequent steps.
Then click Create to create a project:
Project IDs are globally unique identifiers for GCP projects and are the actual identifiers that GCP resources use. If your project name is unique enough, the ID will be the same as the name. In normal practice, we recommend that you name your project starting with your company name (cloudderby-myproject-dev) to help ensure uniqueness.
To go back to the home screen, click on the three-bar menu and select Home. Ensure that your project is selected in the pulldown menu.
Clone GitHub repository with Cloud Derby source code
We’ll be using Google Cloud Shell to execute our commands. Cloud Shell is an ephemeral Linux VM environment with persistent HOME directory for your user that automatically gets created. It includes GCP-specific tools such as the gcloud command line, as well as git and other open source tools.
cloudderby.io 7
Click the Activate Google Cloud Shell button at the top of the console window. A Cloud Shell session will open inside a new frame at the bottom of the console and displays a command-line prompt:
To make things easier to see, you can also expand the Cloud Shell window to full size by clicking on the Open in New Window button.
In this project we will be using the Cloud Derby source code on Github. Clone the source code into your Cloud Shell environment from GitHub onto your $HOME directory by executing this command in Cloud Shell.
cd $HOME
git clone https://github.com/GoogleCloudPlatform/cloud-derby.git
Cloud Shell provides an integrated code editor that we will be using for development purposes. To open the editor, click on the pencil icon:
We suggest that you use this editor for making changes to any file used later in this workshop. Here is what it looks like when it opens:
cloudderby.io 8
Optionally, you can use vi, vim, nano or your own development environment (we do not recommend it for today as our instructions assume the use of Cloud Shell, but overall flow should work with your favorite IDE on your laptop). If you do use your own development environment, we recommend one that runs Ubuntu 16+ as that is what has been validated for this course and is required for the scripts.
Configure your gcloud SDK environment Initialize your Cloud Shell with your individual account and project settings:
gcloud init
a. When prompted, select: [2] Create a new configuration b. Enter configuration name (make up your own name if you want): team15-may-18 c. Select your Cloud Derby account (most likely option 1) d. Select your project ID to use as default (remember you had created it earlier. If you forgot the name, look at the project name pull-down menu GCP console): your-project-id e. Select “n” if asked to configure a default Compute Region and Zone. The setup scripts will handle this later.
Verify that you are logged in with your own user id and in the project you just created by issuing the gcloud config list command:
[component_manager] disable_update_check = True [compute] gce_metadata_read_timeout_sec = 5 [core] account = [email protected] disable_usage_reporting = False project = team15-dish-derby [metrics] environment = devshell
In Cloud Shell, navigate to your home directory and run the project setup script. (Note that the $DEVSHELL_PROJECT_ID environment variable is automatically set to your current project):
cd $HOME/cloud-derby/setup && ./setup-project.sh $DEVSHELL_PROJECT_ID
When asked if you want to create service accounts and roles, answer y.
While the setup script is running, take a few minutes to review the script setup-project.sh you just ran. It contains valuable information and will help you understand how Identity and Access Management (IAM) can be managed in GCP.
cloudderby.io 9
IMPORTANT: This step creates a new script named $HOME/setenv-local.sh in your home directory. You will refer to and edit this file many times today, so c onfirm that you can see it now.
If instructed, update file $HOME/setenv-local.sh with the proper values for $ZONE and $REGION, otherwise leave them at their default values.
### These are Region and Zone where you want to run your car controller export REGION="us-central1" export ZONE="us-central1-f"
If you are outside of US: You may want to update the default region where your virtual machines will be created. Please review the list of available GCP regions with V100 GPU resources here and select one closest to where you are physically located at this time (examples: us-central1-f if you are in the US, europe-west4 if you are in London, etc.).
Click on the down arrow next to the project name to the right of the “Google Cloud Platform” text:
Select your project and click on it:
In order to open your project dashboard. Click on the three bars menu on the left top and click on the Home icon.
cloudderby.io 10
You should now see your project dashboard:
Congratulations! Your workspace is ready for Cloud Derby development and deployment.
cloudderby.io 11
ML - Image Preparation (90 min)
IMPORTANT: Only do this section if you are doing a 8 hour version of the event. We have provided you with a library of annotated images that you will use to train your model. For the 4 hours hackathon, please skip to the section entitled ML Model Training.
Let's get started with Machine Learning! Any data scientist will tell you that data preparation is the key to success, and it usually constitutes over 60% of the work in Machine Learning. As a starting point for this exercise we are providing you with about 2500 annotated images. The term “annotated” means that in addition to the JPG file with the image, we provide you with a bounding box describing the location of the object saved in an XML file.
Here are the high level steps in getting the Machine Learning to work for our robot: 1. Take images of balls and letters and annotate them with bounding boxes and labels (this section). 2. Train an object detector using the Tensorflow Object Detection API using transfer learning method. 3. Use the trained model to recognize color and location of balls in images taken by the car’s camera.
This section of the workshop is based on the Google Tensorflow Object Detection API on Github. To build the Cloud Derby module responsible for inference, we used as a starting point this tutorial: “Quick Start: Distributed Training on the Oxford-IIIT Pets Dataset on Google Cloud” as well as: “Creating an Object Detection Application Using TensorFlow”.
We will annotate images and prepare them to train an object detector. Among important factors of accurate image recognition are: 1. Number of images used for training. 2. Accurate annotation of images, including labels and bounding boxes (see example above). 3. Image augmentation - change of brightness, sharpness/focus, flip images, distortion of images, etc.
These factors will be the focus of this section. Other factors that impact image recognition include ML model hyperparameters – the number of training steps (too few steps and model is not trained well enough, too many
cloudderby.io 12
steps and you get overfitting and poor predictions), initial randomization of some model parameters, and different mathematical methods of the model.
As we mentioned earlier - the accuracy of the navigation of your robot greatly depends on the number of training images and quality of annotations. If you want to win the competition at the end of the day today, one of the things that can help you is a better job than other teams at collecting training images for your ML training. Hopefully every member of your team has a smartphone with a camera. Since you do not know what color of the ball and home base sign will be assigned to your team at the end of the day for the competition, each member of your team should take a few photos of color balls and symbols of the home base (to save time we recommend that you do not take more than 10 images total). While you take pictures, an important consideration to keep in mind is that the more diverse the dataset, the better.
● Images need to be taken in different light conditions (backlit, frontlit, shade, sun, etc.). ● Take images from different distances (between 1 inch up to 15 feet away from the object). ● Take images of balls and letters partially covered by other objects. ● During the training, the resolution of the image will be automatically reduced to about 1024 pixels - so you do not need to take pictures of resolution higher than this (you can if you want). ● You’re training a robot with a camera four inches off the floor. Get low. ● Here are some examples of pictures that you may want to take for annotations and training:
We suggest each member of your team take one or two images for each type of object. We have pre-installed Picasa on the Windows VM in case you want to edit images in any way before making annotations. The following objects need to be photographed: balls of Red, Green, Blue, Yellow color and letters A, B, C, D as shown above.
cloudderby.io 13
Note: Normally TensorFlow pre-trained models for Object Detection API work with images of low to medium resolution. In some cases high resolution images may hinder or even crash our training process because of configuration settings we selected. We have written a script to convert high resolution images to lower resolution. Feel free to annotate images of any pixel dimensions - our script will convert your jpg and your XML file into proper pixel size.
Build an Annotation VM GCP supports many different Operating Systems. We would like you to know that Windows is one of those systems and you can get Remote Desktop access to Windows, hence we decided to use this part of the tutorial to highlight Windows support.
Open GCP Console and login with credentials provided to you by instructor. We will use the project you have created earlier (See Project Creation and Setup).
You will be using two GCS buckets: one for the pictures you have taken just now, and another for the images with annotations that you will be creating later on.
Navigate to Google Cloud Storage and create a new GCS bucket by clicking on the CREATE BUCKET button.
Fill out the form with the following options and as shown below: ○ Name - the name of the bucket must be unique across all of GCP projects worldwide, not just yours. We recommend that you use the following naming convention for your bucket:
Click Create:
cloudderby.io 14
IMPORTANT: By uploading your images you agree that you are doing it under the Apache 2.0 license. Any image that you upload to your project during this workshop may be added to the training dataset for future workshops or open sourced to the public and visible on the internet. Therefore we recommend that you do not include pictures of people, your credit card numbers (duh), or other things that you do not want to be visible to the public.
Once the bucket has been created, click Upload files or Upload folder buttons to upload your images taken with your phone earlier. Later in this lab you will annotate those images and use them for training.
cloudderby.io 15
Next, you will create a new Windows VM on Google Compute Engine and use remote desktop to do image annotation. Login into GCP Console and navigate to Google Compute Engine. NOTE: you may not be able to use browser incognito mode for RDP - if that is the case, use normal browser mode.
Create a new VM by clicking the Create button:
In the section Boot disk, click on the Change button:
cloudderby.io 16
On the next screen, click on Custom images and select Source and Administration project in the drop down box.
NOTE: If you are not running this project under the cloudderby.io organization, you will not have the project listed above. If that is the case, please copy the image from the public GCS bucket gs://windows-vm-image-administration-203923/windows-image-labeling-image to your own project, rename it to windows-image-labeling-image.tar.gz and then use it to create your own VM image inside of your own project. For more details, please consult the product documentation on the steps to import custom VM images into your project and organization.
Next select windows-image-labeling-image as a starting point for your new VM, make sure you select boot disk type SSD and enter size of disk to be 60 GB as shown below, then click Select.
cloudderby.io 17
Now make sure you have your VM properties similar to what is shown in the image below, including the use of service account for your image identity. Select options as shown on the image below (note that
Click Create.
cloudderby.io 18
In the following steps we will use RDP to access WIndows VM. Check that the firewall rule for default-allow-rdp is present in the list.
cloudderby.io 19
Once the VM creation is complete, click on the arrow next to the RDP button and click Set Windows password.
In the dialog box type your username, click Set and copy and paste the password shown to you in the popup window. You may want to write down the password or take a picture for future reference (obviously don’t make a practice of doing this in real life).
cloudderby.io 20
Now you can click on the RDP button (you need Chrome with RDP extension to be able to do it). Chrome will open RDP window and ask you to login. Enter the password you created earlier and click OK:
After you log into Windows, feel free to close the Server Manager window (it opens automatically each time you login). In your Windows VM, click on the Google Cloud SDK Shell desktop shortcut.
In the step below we will clone the project source code from the repo to the local directory. Remember that we created this Windows VM we assigned the service account to be used as the identity of the VM. To make sure this is the case, please run the following command and note the account value in the bottom section entitled Current Properties.
gcloud info
The value of account should be the service account was created by you when running initial project setup (look into the “setup-project.sh” script if you are interested in details).
At this point we need to clone project source code into our Windows VM file system. In the Windows command line enter the following command - this will clone the Git repo into the local directory “cloud-derby”:.
cloudderby.io 21
cd c:\
git clone https://github.com/GoogleCloudPlatform/cloud-derby.git
Load your images Now is the time to download images that you took earlier with your cell phone into this Windows VM. Remember that you uploaded your images into a GCS bucket? Use that bucket name in the command below. Do not include the angle brackets <>.
gsutil -m cp -r gs://
Open Windows Explorer (folder icon at bottom of desktop) and expand “This PC” → “Local Disk (C:)” in left side navigation panel. Select a-robot-images folder.
Hint: Select ‘View’ Menu and Click ‘Extra Large Icons’ to see pictures without individually opening each one
Organize your photos into folders named after balls and letters of different colors. If you have a picture with balls of different color or several different letters in the same image, place a copy of that picture into each folder with corresponding color name. For example, the example image a few steps below can be put in every single folder because it has different color balls and all four letters. This means you will annotate it several times with each annotation being specific to the object depending on what folder holds the image.
● BlueHome folder is for pictures of the letter B ● GreenHome folder is for pictures of the letter C ● RedHome folder is for pictures of the letter D ● YellowHome folder is for pictures of the letter A
Your final folder structure should look like the following
cloudderby.io 22
Label your Images
We will now label your images and annotate them with bounding boxes. Start the free program called LabelImg software:
c:\LabelImageSoftware\labelImg.exe
You will repeat the same procedure annotating photos for each folder. Note that we have already placed several images in all of the folders and annotated some of them. Please check that every image in each folder is annotated. We will show it on the example of the BlueBalls folder. Click on the Open Dir icon:.
Navigate to the folder with one of the colors and select that folder:
cloudderby.io 23
In the right hand side check Use Default label and type ball (the label name does not matter as we will use the folder names to generate training records for TensorFlow, so you can name your labels anything you want).
On the left side click on Create RectBox.
cloudderby.io 24
Since we are using this example to annotate Blue Balls - please create annotations around the visible part of the blue ball (only the visible part!!!). In our training we are only going to annotate a single object even if there are multiple blue balls in the picture. Your box does not have to be perfectly touching the edges, but the closer it is the better the result. Here are two examples - one for annotating the blue ball and another for annotating B letter for home base.
Click Save button and make sure the name of the XML file is the same as the JPG file and is located in the same directory.
cloudderby.io 25
Do not worry about file naming scheme for images (we will be using folders names for generating training labels). The script that we provide later will automatically merge and resolve name conflicts between different team member files.
Now you can click Next Image and annotate all remaining images in the folder. You will end up with pairs of JPG and XML files. XML files have coordinates of the rectangle in Pascal VOC record format.
Repeat image annotation for one more folders. You do not need to annotate every single image if you feel that you understood the concept of annotation. Browse the folders and look at images and annotations. Each XML file provides bounding box for the ball object within an image and will be later used for transfer learning. Note that any images without annotations will be automatically discarded by our script.
Once you are done with the annotations, it is time to merge all files and annotations of all team members into a single directory. In order to do so, we provide you a script to upload results of your annotations and images into a team shared GCS bucket with unique zip file name for each team member.
Before you can run the script, please use GCP Console to create the bucket in your GCP project. Let's call this bucket annotated-images-${PROJECT}-version-${VERSION} - substitute ${PROJECT} with your proper project ID and ${VERSION} with the value of the VERSION in the file setenv-local.sh. The name of the bucket must be unique across all of GCP.
Edit the script we provided and update the values of two variables as follows (you have created this GCS bucket in steps earlier while the zip file will be created for you by the script. In windows explorer right click on c:\cloud-derby\cloud\ml\annotation\upload.bat and click Edit and make updates as follows and save the file:
cloudderby.io 26
set GCS_BUCKET=annotated-images-
Open the Google Cloud SDK Shell on your Windows VM and issue this command to upload images to GCS and make sure your command completes successfully:
c:\cloud-derby\cloud\ml\annotation\upload.bat
In Cloud Console browser window - check your storage bucket to confirm that your .zip file has been uploaded:
Since we will no longer be using Windows VM, we will stop it now for cost and environmental reasons (why waste energy when we don't have to?)
cloudderby.io 27
To stop the instance, select the checkbox to the left of Windows VM and click STOP:
Congratulations! You have annotated your own images and can train an object detection model!
cloudderby.io 28
ML - Model Training, Part I (25 min)
This module is an implementation of transferred learning with TensorFlow and is based on this Pets Tutorial as well as this CloudML Code lab. Additional information was used from the article Using your own dataset in Tensorflow Object Detection API. By using transfer learning we save hundreds of hours of compute by taking a pre-trained neural network with frozen weights, resetting the last layer and retraining it on our own images. Once the training is done, the frozen inference graph is saved into GCS.
Put simply, we’re going to take a trained model that is really close to the one we want, and tweak it to recognize the objects that our robots are looking for.
In this project we will be using a single Compute Engine Virtual Machine both for training our TensorFlow model and for running inference (i.e., making predictions or decisions) for our cars. The inference service will be used as the REST API endpoint later in the project by the Driving Controller application, so we need to open connectivity for the Driving Controller to talk to the Inference VM.
Create a Static IP for the Inference VM Since the Driving Controller will be running on Google App Engine (GAE), it needs to talk to a public IP address to communicate to our VM. Let’s first create this external public IP address that we’ll assign to the VM. In the Cloud Console open NETWORKING > VPC network > External IP addresses page:
Click Reserve Static Address.
cloudderby.io 29
Fill out the form as shown below: ● Name: ml-static-ip-$VERSION (the value of the $VERSION can be found in setenv-local.sh. Try running `grep VERSION $HOME/setenv-local.sh` in Cloud Shell) ● IP version: IPv4 ● Type: Regional ● Region: us-central1 <-This may not be the default, so please change it
Note: please use the same region as you have defined in the configuration file $HOME/setenv-local.sh in variable REGION. The default is us-central1.
cloudderby.io 30
Click on Reserve to create the static IP address.
Create a Training VM We’ll now create the VM. The creation script will use the address that you generated above. Run the following command in Cloud Shell to create a VM that we’ll use to train our TensorFlow model and to run Inference. Wait until the VM creation is complete (note that it will take some time to start the VM, so don’t worry about the “waiting” error messages while it starts)
cd $HOME/cloud-derby/cloud/ml/training && ./create-training-vm.sh
While the script is running, take a few minutes to review the script you just ran in the Cloud Shell graphical editor. Please note that the command to create the training and inference VM uses the name of the static IP
cloudderby.io 31
ML_IP_NAME (defined in setenv-global.sh) that we created earlier as you can see from the --address option in this code snippet. Also note that we created a VM with GPU acceleration as you can see from the --accelerator option. This will significantly speed up our model training process (see create-training-vm.sh):
… create_gpu_vm() { local VM_NAME=$1 local GPU_COUNT=$2 echo_my "Create VM instance '$VM_NAME' with '$GPU_COUNT' GPUs in a project '$PROJECT'..." # See docs: https://cloud.google.com/deep-learning-vm/docs/quickstart-cli # https://cloud.google.com/deep-learning-vm/docs/tensorflow_start_instance gcloud compute --project="$PROJECT" instances create $VM_NAME \ --zone $ZONE \ --image-family=tf-latest-gpu \ --image-project=deeplearning-platform-release \ --boot-disk-size=70GB \ --boot-disk-type=pd-ssd \ --machine-type n1-highmem-2 \ --accelerator="type=nvidia-tesla-v100,count=$GPU_COUNT" \ --service-account $ALLMIGHTY_SERVICE_ACCOUNT \ --maintenance-policy TERMINATE \ --restart-on-failure \ --subnet default \ --address $ML_IP_NAME \ --tags $HTTP_TAG,$SSH_TAG \ --metadata="install-nvidia-driver=True" \ --scopes=default,storage-rw,https://www.googleapis.com/auth/source.read_only
…
Once you see the success message below, your Training VM has been built.
************************************************************************* ML training VM Creation has completed. FINISHED: in 411.19 seconds (create-training-vm.sh). *************************************************************************
Configure the Training VM
Log into the newly created training VM (you can click on SSH button on the COMPUTE | Compute Engine dashboard next to your VM).
Note: Clicking on SSH will pop up a new browser window, so if you don’t see the VM window, make sure you enable popups in the browser for this site.
cloudderby.io 32
Clone the Git Repo into the ML GCE VM:
cd $HOME
git clone https://github.com/GoogleCloudPlatform/cloud-derby.git
From the training VM (aka in the SSH window, not in Cloud Shell), run project setup script (use your PROJECT_ID you defined earlier when creating your project):
cd $HOME/cloud-derby/setup && ./setup-project.sh
● When you are asked “Do you want to create new project? (y/n)” - answer “n”. ● When you are asked “Do you want to setup new roles, enable APIs and generate new service account now? (y/n)” - answer “n”.
If you performed the image annotation section earlier, all team members completed the upload of their annotated images into a shared Google Cloud Storage (GCS) bucket. It is now time to merge all images and annotations created by your team with images and annotations provided by your class instructors. Don’t worry if you didn’t provide any images - we have several hundred already in the training set from previous Cloud Derbys.
Prepare training images On your ML VM, edit the file $HOME/cloud-derby/cloud/ml/annotation/merge.sh and set GCS_BUCKET to the same value you set in Windows upload.bat file in steps above (annotated-images-
Now you can run the image merge script on your ML VM (it’s pretty sophisticated script that weeds out duplicate file names, removes files that have no annotations, and does all kinds of other cleanup on your images, including resizing, sorting, counting, etc.):
cloudderby.io 33
cd $HOME/cloud-derby/cloud/ml/annotation && ./merge.sh
You’ll see a lot of messages fly by as the VM copies training image files from Cloud Storage. Wait for the success confirmation message of the script before proceeding:
************************************************************************* Merge has completed successfully. FINISHED: in 128.19 seconds (merge.sh). *************************************************************************
Now that we’re set up, let’s train a Machine Learning model!
Train a TensorFlow model
IMPORTANT: This ML training step can take between 30 to 60 minutes. After you start the training in the steps below and while the training script is running, we’ll move on to the next two sections: 1) Driving Controller and 2) Car Assembly (optional) to get your car working in manual mode.
While still logged into the Training VM, in the file $HOME/setenv-local.sh make sure that the TRAINING_STEPS variable is set to a value between 6,000 and 8,000 (usually you would want to experiment to find the proper number of steps, but considering limited time today we will only run this training once). [Hint: grep TRAINING_STEPS $HOME/setenv-local.sh is a quick way to find the relevant line.]
The next step installs prerequisites needed for the app and downloads TensorFlow models repo from Github. Please note that we are using a Deep Learning VM image from GCE and it has NVIDIA CUDA GPU drivers and TensorFlow pre-installed. Once setup is complete, the TensorFlow model training will be started:
cd $HOME/cloud-derby/cloud/ml/training/ && ./run.sh
Take a few minutes to review the script you just ran.
T raining may take between a few minutes to several hours depending on the number of steps you configured. For our chosen Tensorflow model (faster_rcnn_resnet101_coco) with about 2,400 training images it takes ~40 minutes to complete 8,000 steps. (Note: we are experimenting with simpler and faster models to speed it up).
You can monitor your training VM in the console output and in GCE Console. You can safely ignore the warning lines about code deprecation – Tensorflow is constantly growing and evolving due to the efforts of the open source community, so some of our code may not be fully up to date.
To see details about the Training VM resource utilization, click on the name of the VM instance in the GCE console, then select the Monitoring tab.
cloudderby.io 34
G enerally, when you do this kind of project, for better accuracy of object detection, you need to experiment with the number of training steps and other settings on your model. We have enabled image augmentation in the model configuration file (see file $HOME/cloud-derby/cloud/ml/training/run.sh for details). More info about different augmentation options can be found in this StackOverflow discussion.
While we wait for the model training to complete, we’ll focus on setting up the cloud-based controller and your car.
cloudderby.io 35
Driving Controller (20 min)
The Driving Controller is a software module that is responsible for receiving sensor and other information from the car and making driving decisions based on that data. It uses ML and other logic to make driving decisions in manual or automatic mode and to send commands back to the car. Driving Controller can run in two different modes: (1) As a local NodeJS application in your own VM, or (2) as a serverless NodeJS application running in the cloud on Google App Engine. We’ll start by testing our code on a local VM, then we’ll deploy to App Engine.
Driving Controller uses SENSOR_SUBSCRIPTION subscription to listen for messages in the topic SENSOR_TOPIC, processes those messages and sends commands to the topic COMMAND_TOPIC. The car receives JSON formatted commands from the COMMAND_TOPIC and acts to execute those commands. Here are the steps to deploy Driving Controller logic:
Log into your GCP Cloud Shell and go to the Driving Controller folder:
cd $HOME/cloud-derby/cloud/controller
Before you can run self driving logic, you need to make sure your game settings in source code are set correctly. There may be variations in sizes of balls, cameras on the car, etc. Locate and review the following files and their content:
less $HOME/cloud-derby/cloud/controller/js/settings.js
Note the settings of the size of the ball and other variables. Use up and down arrows to explore and hit ‘q’ to return to the console. cloud/controller/js/settings.js
/************************************************************ Settings for the game ************************************************************/ module.exports = ({ // How many balls need to be collected to win the game BALLS_NEEDED: 3,
// Diameter of the ball (165.744 mm) BALL_SIZE_MM: 60.638,
// Width of the Home Base sign (letter size is 216mm) HOME_SIZE_MM: 200,
// Various labels returned by vision API, such as
// Various labels returned by vision API, such as
cloudderby.io 36
// Distance from the camera to the ball in a fully captured position MIN_DISTANCE_TO_CAMERA_MM: 21,
// Max car speed (wheel rotation degrees per second) MAX_SPEED: 1000,
You might be wondering why we need to know the focal lens of the camera, the size of the ball and other things? The way our logic has been implemented (and it is not the only way) we calculate distance to the ball by using two step approach: (1) Run Object Detection model to give us bounding box around the ball of known size, and (2) Calculating distance to the ball by using the following formula:
Object height on sensor / Focal length = Real Object height / Distance to Object
Calculating the turn angle is even simpler: (1) Using bounding box from the Object Detection step above we find the center of the object, and (2) calculate shift from the middle of the image knowing the angle of view of the lens. Once we know the angle and distance to the object, we give commands to the car to navigate to it - for details see file $HOME/cloud-derby/cloud/controller/js/navigation.js.
Deploy Driving Controller Locally
Open the $HOME/setenv-local.sh file and: ● Set variable DRIVING_CONTROLLER_LOCAL="true". With this variable set to true, the Driving Controller will be deployed locally to your Cloud Shell instance. ● Set variable USE_DEMO_INFERENCE="true". This variable forces Driving Controller to use inference VM that is provided to you for testing purposes. Since your own inference VM is still working to finish the training, we want to use demo VM to be able to test the controller setup.
You can really run the Driving Controller anywhere, including your own laptop or a VM on GCE. Deploying it locally allows for quick validation, debugging, and testing. We will deploy Driving Controller as Google App Engine service later once we validate local deployment. One of the benefits of Google App Engine is that you
cloudderby.io 37
can develop and debug applications locally and later deploy them into managed, autoscaled PaaS environment as we will do later.
To start the Driving Controller, simply run this script in Cloud Shell:
cd $HOME/cloud-derby/cloud/controller && ./run.sh
While the script is running, take a few minutes to review the script in the Cloud Shell file browser. Wait for the controller to start. Once it is running after a few minutes, you will see output similar to the following:
************************************************************************* STARTED: Start application 'driving-controller' (run.sh) ************************************************************************* [INFO] run.sh->File '/home/user6team15co/cloud-derby/cloud/controller/tmp/install.marker' was found = > no need to do the install since it already has been done. (0.02 seconds) [INFO] run.sh->Create Topics and Subscriptions for car to cloud communication... (0.04 seconds) name: projects/team15-dish-derby/topics/driving-commands-topic-2 [INFO] run.sh->Topic driving-commands-topic-2 found OK (1.40 seconds) name: projects/team15-dish-derby/topics/sensor-data-topic-2 [INFO] run.sh->Topic sensor-data-topic-2 found OK (2.77 seconds) name: projects/team15-dish-derby/subscriptions/sensor-data-subscription-2 [INFO] run.sh->Drop and create subscription for sensor data to avoid processing of old messages... (4.04 seconds) Deleted subscription [projects/team15-dish-derby/subscriptions/sensor-data-subscription-2]. [INFO] run.sh->Creating a subscription 'sensor-data-subscription-2'... (5.69 seconds) Created subscription [projects/team15-dish-derby/subscriptions/sensor-data-subscription-2]. [INFO] run.sh->INFERENCE_IP=104.198.76.21 (9.58 seconds) [INFO] run.sh->DEPLOY_LOCAL='true' (set it to false to deploy on GCP) - running on local machine (use this for test and dev only)... (9.60 seconds)
> [email protected] start /home/user6team15co/cloud-derby/cloud/controller/js > node app.js
***DRIVING-CONTROL-APP is starting up*** Listening on port 8080. Press Ctrl+C to quit. startListener(): started OK and ready ***DRIVING-CONTROL-APP.GET.main_page***
Note the line at the end stating that the Driving Controller is listening on port 8080. Now we will configure the Driving Controller to use full self driving mode. This is done using its web app that is served by the same NodeJs process we started in the step above. In order to open the Web UI for the driving controller that is started in your Cloud Shell on port 8080 (see image above).
Click on the Web Preview icon in your Cloud Shell window and select Preview on port 8080. This will open a new window in your browser with the Driving Controller interface:
cloudderby.io 38
Once you see Cloud Derby UI, click on Set driving mode to: Self driving:
In your browser window, you should see the following message. Click on Home to go back to the main page.
At this point the NodeJS Driving Controller app will listen for incoming messages from the car sensors, process them, invoke the remote Inference runtime using REST API, create control messages and send them back to the car via PubSub. The car software will subscribe to the topic where these control messages are published, interpret commands in those messages and issue appropriate signals to the car hardware.
Well, it won’t actually call the Inference runtime because we haven’t started it yet. For now we’re going to test
cloudderby.io 39
communication using simulators.
Simulate Messages Before we use Driving Controller to manage the real car, we will test its function by using software simulators. To do so, we will start another process that simulates messages from the car.
Open an additional Cloud Shell window to start simulation scripts by clicking on the + symbol.
Start the driving simulator (simulating the real car waiting for control messages) so it creates proper topic subscriptions.
cd $HOME/cloud-derby/car/simulator && ./driver.sh
Note: When we run it for the first time it will show no messages since we have not yet sent any “sensor” messages.
Let's send some fake sensor messages from the car simulator to the Driving Controller. Note that when you run it for the very first time, it will install npm and other prerequisites. This will take a few minutes and happens only once.
./sensor.sh
cloudderby.io 40
Verify that your first console window (which is running the Driving Controller) shows processed messages similar to the image below.
Now that Driving Controller has processed sensor message, we should be able to see some control messages being sent to the car. In order to see what those messages are, we will run the driving simulator once again in the second Cloud Shell window. This simulates your Car asking for any new messages from the Driving Controller.
./driver.sh
If this step completes successfully, you can close the second Cloud Shell window.
cloudderby.io 41
Deploy to Google App Engine We have validated that our car can send messages, the Driving Controller can receive and process them, and that the [simulated] Car can pull control instructions back from the Driving Controller. Now that you have validated the functionality locally, let’s deploy the controller to a production environment with App Engine. There are few steps to deploy the Driving Controller to App Engine:
● Go back to your first Cloud Shell window and stop your currently running Driving Controller by pressing Ctrl+C ● To deploy the same code set with the same functionality to App Engine, re-open the $HOME/setenv-local.sh file and set variable DRIVING_CONTROLLER_LOCAL="false". ● Now that we have verified that Driving Controller works with the demo inference VM, lets point it back to the one you are training. Open the $HOME/setenv-local.sh file and set variable USE_DEMO_INFERENCE="false". This variable forces Driving Controller to use inference VM in your own project. We will complete configuration of your inference VM when TensorFlow training completes. ● In Cloud Shell, start the Driving Controller by re-running the same script you used earlier (but this time it will deploy into the App Engine, instead of running it locally):
cd $HOME/cloud-derby/cloud/controller && ./run.sh
Wait for the controller to be deployed and started. It may take a few minutes to deploy and start the app. You will see a similar output as before which looks like the following, showing that we’re deploying the Driving Controller app into App Engine.
cloudderby.io 42
Take note of the URL provided in the green text at the end as you will need it for the next section. It will be in a format similar to https://driving-controller-dot-
[INFO] run.sh->Running on GCP URL=https://driving-controller-dot-team15-dish-derby.appspot.com/ (91.90 seconds)
NOTE: To find the URL of your App Engine app in the future, in the GCP console go to COMPUTE | App Engine. The link to your application will be in the upper right corner.
cloudderby.io 43
Congratulations! Your Driving Controller works and you are ready to set up your car in self driving mode.
cloudderby.io 44
GoPiGo Car setup (15 min)
Car assembly and configuration can start at any point in the workshop, but the remote control of the car with Machine Learning or Voice will only work after those other respective modules are complete. Additionally, the Machine Learning takes time to train the model (up to 1 hour), so it may be a good idea to do the car setup while ML training is running.
Software on the car runs on the Raspberry Pi board and uses Google Cloud IoT Core to communicate to the Cloud Derby application running on GCP. Here’s the basic flow: ● Your car takes pictures and uploads those pictures to a Cloud Storage bucket using the gcloud SDK. ● The car also makes distance measurements and sends other car data (battery,status, etc.) to the cloud using IoT Core with PubSub message service. ● Driving Controller running on App Engine will process this input and make decisions on the car movements and send it back to the car using a PubSub topic. ● Car software uses a PubSub subscription to get those control messages and takes actions based on those commands.
Notice that the car’s onboard LEDs light up in different colors after your car’s code startup: ● Green indicates that the car is ready to receive the command from the cloud. ● Red indicates that the car is processing a command. If your car stays lit Red for too long, it is an indication that there is a mechanical issue preventing a command from completing (you may want to lift the car and help it by turning the wheels gently). ● Yellow indicates the car is capturing and uploading an image and sensor data. If your car stays lit Yellow for too long, it is an indication that there is network latency issue between the car and the controller.
Configure the Car’s WiFi Note that a custom Raspberry Pi OS Image (or this image for cars with the Raspberry Pi 3B+) has already been downloaded and installed on the SD card for you (login username “pi” password “robots1234”). This is not the standard image that comes with the GoPiGo kit. We added Google Cloud SDK to the standard GoPiGo image so you don't have to spend time doing this. You will install the SD card as part of the car assembly as detailed in the instructions.
If you prefer to set WiFi configuration using the command line, here is a good article.
Plug in a Monitor to the HDMI port, keyboard and mouse into the USB ports on the Raspberry Pi. You only need these attached to the car for a few minutes to set up car WiFi and get the IP address of the car.
Power up the car using batteries. Please DO NOT use 12V DC power (unless instructed by the crew) and wait for the desktop to load.
Setup the Wifi on your car
cloudderby.io 45
● From the car’s desktop, enable wifi by clicking on the WiFi icon on the top right of the car’s screen and clicking on the “Turn On Wi-Fi” button
● Join the appropriate WiFi network as instructed by your derby crew.
● Open up the Terminal and type ip address list to get your IP Address on interface wlan0.
● From your laptop, ping this IP Address and try to SSH to your car via “SSH pi@
● When you configure the WiFi, please make sure you save the configuration of WiFi on your car and reboot the car while it is still connected to the monitor to make sure that IP address is still the same after reboot.
Make sure you’ve removed the lens cap from the car’s camera before proceeding. Cameras work best when they can see things.
While still attached to the monitor and keyboard, test the camera on your car by issuing the following command in the terminal window and looking at the resulting image that gets created in the directory where this command was issued.
raspistill -v -o test.jpg
Note: Camera documentation can be found here: https://www.waveshare.com/wiki/RPi_Camera_(G)
Don’t worry that the image is upside down. We’ll adjust for that in the code configuration. Also, since we are
cloudderby.io 46
going to be dealing with optics, please make sure your camera lens on the car is clean at all times or you may get poor results.
You are done working directly through the car’s interface, so disconnect the monitor, mouse and keyboard and take the car back to your work area. Keep the car powered up for the remainder of this section.
Install Cloud Derby source code on your Car
From your laptop SSH to your Raspberry Pi. The user name is pi, password is robots1234.
ssh pi@
Before you configure the car for your own project, you need to clean up any remains of any previous Cloud Derbies by removing a few files and directories while logged into the car. We’ll also create the $HOME/.secrets/ directory to place our Service Account key in later steps.
cd $HOME && rm -rf project-id.sh setenv-*.sh cloud-derby .secrets
mkdir $HOME/.secrets/
Clone the Cloud Derby code from Github.
git clone https://github.com/GoogleCloudPlatform/cloud-derby.git
Copy the GCP Service Account private key from your Cloud Shell environment over to the car. Back in your Cloud Shell in GCP, look at your Service Account key (this was created when you ran project setup script earlier):
cat $HOME/.secrets/service-account-secret.json
Service Account key looks like this:
{ "type": "service_account", "project_id": "team5-dish-derby", "private_key_id": "90f554d66c56ccb92ee50bc362c576eb03a2a105", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCKAcw9gQtMdS/w\nUik/surbHNO2B9HLOVLUu2x8T5gyDdRVJ JXdfTq1f/S2cL+ohZiNf/MJOS5raoaK\ngzHdK8T/Gge5xwVmVtWVz6JNsj3r4Rt0wG536UPTJo/gatCkyKOc6wRG4iFi6CtS\nZACKuqdRUg 8Cd4RfYr3olGNaaBY0PbMmSy302Zbt+u8PMU00HiPCWJf029YK4nWR\nE+xd2HuGlt3iZXjEEROd76rzHaeUiZkgPWyf/SzO0L0yMF2oIqnlF XzSwmyvZ/A+\nYxvMfuuDRppnKQA6Fbw8E8ET00rqI+Yn9YnAlvBb4kFa1dAiPxZpQsF7vfbmXpnm\ngwiKvqAFAgMBAAECggEACkibAwRAJs jesx4KBgA46\nnGph1aTp0BOTJH6K1C+C96xqpBaoEThou1qBBv1YDYYt4sfxYfuJB783DSPgOB57\n096eCoeGWjLVAqCHDEIahVxeJCQ2JU PiAX2wu1/dYpPWDh9pI7E/BvaNTmeVzPeZ\nq4IEf4jQNw0GeBFxluJ2qRwg5dcW8fe+Vj7AXIKBAoGAUqueo6n/L4coJ26mWn5T\nDqx8I5v 9csJ/TkTvfMQs6TZxl5jh1j22mv+s1eXhRQxOJfiZ1+JJtaEpDvGQPEBq\n8UK5M7taA4yg3paHiaJvX6xka49vKdfgIofTvGwRyrkliQNuRw q01lL0ewWqcZVp\nq7HOM0eMlt26COFX0oXYqEY=\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "115640835296926543173",
cloudderby.io 47
"auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cloud-derby-dev%4team5-derby.iam.gserviceaccount.com" }
Select all of the lines including the curly braces {} and copy them. In your car SSH session, go into $HOME/.secrets (note the . in the directory name) and create a file called service-account-secret.json. Paste the contents of the key into this file and save it. NOTE: all remaining steps in this section take place in the car SSH session.
cd $HOME/.secrets
nano service-account-secret.json
Activate your service account to be able to connect to the cloud (this is one command).
gcloud auth activate-service-account \ --key-file=$HOME/.secrets/service-account-secret.json
From the car (aka in the SSH window), run project setup script (use your PROJECT_ID you defined earlier when creating your project).
When you are asked “Do you want to create new project? (y/n)” - answer “n”.
When you are asked “Do you want to setup new roles, enable APIs and generate new service account now? (y/n)” - answer “n”.
cd $HOME/cloud-derby/setup && ./setup-project.sh
Check your gcloud config to ensure your project and service account are correct. Your service account will be in the format “cloud-derby-dev@
gcloud config list
[compute] region = us-central1 zone = us-central1-f [core] account = [email protected] disable_usage_reporting = False project = team15-derby-dish
cloudderby.io 48
Register your car in Google IoT Core In order to start using Google Cloud IoT Core we need to create a device registry and add our devices to that registry. You could do it via the console in the browser, or via an API. In this project we are providing you a script that uses the gcloud SDK to call GCP APIs to create the registry and add devices to it. You can find the complete script in $HOME/cloud-derby/car/driver/run.sh.
If your team is adding multiple cars to the same GCP project, please review the Car IDs that are already used in your project. If there are no registries that means that no one has registered an IOT device to the registry yet. We will be creating an IoT registry and registering your new car in an upcoming step. So let’s first open the IoT Registry (if you do not have a registry yet, it will be created later by our script).
Edit the file $HOME/setenv-local.sh to ensure you have a CAR_ID that is different from your teammates (the example above has two device IDs listed as iot-car-1 and iot-car-2).
On the car, navigate to the $HOME/cloud-derby/car/driver directory in the source tree and start the car software. This step will create IoT registry and add your car as authenticated IoT device to that registry:
cd $HOME/cloud-derby/car/driver && ./run.sh
If the command above results in an error “google.cloud not found”, then the Google Cloud SDK packages need to be installed or updated. Try manually installing the packages on your car:
pip install --upgrade google-cloud-pubsub google-cloud-storage pip install paho-mqtt
While the script runs, take a few minutes to review the script in the Cloud Shell file browser. After the command above completes, please go back to the GCP Console and review the new IoT Registry and the new device registered with your new CarID. You can find the console under BIG DATA | IoT Core.
cloudderby.io 49
Click on the registry name (car-iot-registry in the example above) and select Devices in the left hand menu. You should see your car registered. Note that in the example below where Device ID is iot-car-2, the value of CAR_ID was set to 2.
This step will only work when you complete setup and configuration of the Car Driving Controller. For the first time, the Driving Controller must be started before the car is started because it creates PubSub resources for communication. After the first time you can start and stop them in any order.
Now that the car software is up and running, we can test that the connection between the car and Driving Controller is established. By default, Driving Controller runs in fully automatic mode, but it can be switched into a manual mode by using the simple web app. To switch to manual mode and drive the car from the web page, navigate to the URL generated in the final step of the last section.
Remember that it has the form https://driving-controller-dot-
cloudderby.io 50
Click on Set driving mode to: Manual to control the car remotely from the browser.
On the manual control screen enter distance, turn angle or other options and click Send control message to the car. You should see your car execute the commands that you just sent to it.
Congratulations! You can now control your car through Google IoT Core through the cloud.
Troubleshooting your car In order to help troubleshoot the car, we have setup the three LEDs on the top of Dexter’s board to provide the status of three components: (1) ball color selected, (2) status of commands being processed onboard, and (3) driving mode. The below image outlines where these LEDs are located on the board and what the color indicates:
cloudderby.io 51
ML - Model Training, Part II (15 min)
The training script tails a file called nohup.out (this is what you see on the screen above). To exit back into command line, press Ctrl-C at any time after you see the Finished training message and continue with instructions (The tail command will never exit unless you terminate it).
Let’s check back in on our model training in the VM window that we opened earlier. Once the training is done you will see the following message (please note that the number of steps shown is defined by the environment variable TRAINING_STEPS that you checked earlier):
Check the model
After the training of the model has been completed, go to the console Cloud Storage page at STORAGE | Storage.
Verify that you see model checkpoints in the GCS bucket with a name similar to ‘
cloudderby.io 52
Review your model training performance Let’s see how well the model trained to recognize our images. You can review the results of the training using the TensorBoard dashboard by starting it in your training VM.
Go back to your training VM SSH window if you haven’t closed it (or start a new session by clicking SSH next to the name of the VM in the Compute Engine panel):
cd $HOME/cloud-derby/cloud/ml/training && ./tensorboard.sh
Look at the GCE Console and copy the External IP address of the ml-training-inference-## VM.
cloudderby.io 53
Open a new browser tab and open the TensorBoard at http://
After you look at the TensorBoard, please go back to the console in your training VM and kill the TensorBoard process:
ps -ef | grep tensorboard
Take note of the tensorboard process id and kill the process
kill -9
user6team15co@ml-training-inference-50:$ ps -ef | grep tensorboard user6te+ 10702 1 4 00:58 pts/0 00:00:09 /usr/bin/python3 /usr/local/bin/tensorboard user6te+ 10755 6920 0 01:02 pts/0 00:00:00 grep tensorboard
cloudderby.io 54
user6team15co@ml-training-inference-50:$ kill -9 10702
Export the frozen inference graph (the ML model) to our Cloud Storage bucket by running the script:
cd $HOME/cloud-derby/cloud/ml/training && ./export-checkpoint.sh
Wait until the export completes. You should see something similar to the following.
You can see the exported ML model in Cloud Storage in the folder named
Allow Traffic to the app by opening a port in a firewall Before we can run our Python program that responds to REST HTTP requests, we need to open up the HTTP port in the GCP firewall to allow external requests to reach our VM. In a previous section we ran the script to allow TensorBoard to accept requests on port 8081, but here we will have you create a firewall rule manually so that you can see how it’s done.
In Cloud Console UI open NETWORKING | VPC network > Firewall rules menu:
Click Create Firewall Rule:
cloudderby.io 55
Fill out the form using the following values and click Create (if you use different settings, be sure to update setenv-global.sh file with the variables HTTP_PORT, HTTP_TAG): ● Name: allow-http-8082 ● Network: default ● Targets: Specified target tags ● Target tags: http-from-all ● Source filter: IP ranges ● Source IP ranges: 0.0.0.0/0 ● Specified protocols and ports: tcp:8082
cloudderby.io 56
You should see the new firewall rule created with a tag named http-from-all. This tag has been applied to our inference VM.
cloudderby.io 57
By using tags instead of source IP addresses in our rules, we can easily apply firewall rules to new VMs by simply applying the tag to any instance rather than updating the rule with VM IP addresses or subnet ranges. To confirm that the tag is applied to the VM, you can select the instance in the console to view its details. Down near the bottom of the page you’ll see a section titled “Network tags” that displays all tags associated to that instance. You can dynamically add and remove tags to instances in GCP and they take effect immediately.
Create Inference web application and REST endpoint ML inference in this project is built as a Python application. It is responsible for detecting colored balls and other objects in images and returning labels and bounding boxes around detected objects back to the Cloud Controller. It is implemented as TensorFlow running on a VM with a single V100 NVIDIA GPU. This module can be called in two ways: (1) via REST call for production services, and (2) Web page for manual testing.
The ML inference logic uses a frozen inference graph from the ML Training step above. Script cloud/ml/setenv-ml.sh defines several important variables, including the inference graph location in Cloud Storage ($FROZEN_INFERENCE_GRAPH_GCS) and label map ($LABEL_MAP_GCS) environment variables. You can manipulate these variables to use results from different training runs if you desire.
We will be using the same VM we created earlier in the ML Training section. Log into the ml-training-inference-xx VM or use the same shell you have used earlier (you can click on the SSH button in the GCE dashboard).
In the Inference VM, run the script to install some additional prereqs into the VM and to download the frozen inference graph from GCS that we created during the previous step. Once that is done the Python app for object detection will be started:
cd $HOME/cloud-derby/cloud/ml/inference && ./run.sh
cloudderby.io 58
Note: the console output from the app is sent to the file python/nohup.out, which can be useful for troubleshooting. You can run ‘tail -f python/nohup.out’ for the remainder of the day to watch messages on the inference VM.
Manually Test Object Detection Find the public IP address for your service VM from the GCP Console:
Launch a browser to: http://
Test object detection with an image of your choice (here are a few sample images you can download and use). When you see the photo, click on the links BlueBall, YellowBall, etc, to see if it was detected.
cloudderby.io 59
Congratulations! Your Object Detection app works and you can use it for self driving.
cloudderby.io 60
Chatbot Control with Dialogflow (30 min - optional)
You should only do this section if you have completed all of the section above and have your car running in full self driving mode and you are ready for the competition at the end of the day.
Browse to https://console.dialogflow.com. Click the down-arrow
Click Create new agent:
Name the agent anything you desire.
Select the appropriate Google Cloud project from the drop down menu.
Click Create.
Download the Cloud Derby pre-built Dialogflow agent from this URL: https://storage.googleapis.com/cloud-derby-dialogflow/cloud-derby-df-agent.zip
Import the agent: 1. Click the gear icon next to your new agent 2. Click Export and Import 3. Click Restore from Zip 4. Click Select File 5. Type RESTORE in the bottom box 6. Click the RESTORE b utton (this will be greyed out until you type ‘RESTORE’ above it) 7. Click Done
cloudderby.io 61
Click Fulfillment. The next several steps setup a webhook using Firebase Functions that will accept parameters such as distance and direction from your voice agent and determine what command to send to the robot.
1. Toggle Webhook to DISABLED 2. Click Save 3. Toggle Inline Editor to ENABLED (you may need to wait for a few minutes for resources to be provisioned) 4. Copy this index.js code to inline editor index.js tab, replacing all existing code (DO NOT CLICK DEPLOY YET) 5. Copy this package.json code to inline editor package.json tab, replacing all existing code 6. Change COMMAND_TOPIC in index.js to your driving command PubSub Topic (line 16 in index.js). The topic will look like projects/team15-derby/topics/driving-commands-topic-2
cloudderby.io 62
7. Click Deploy. This will deploy a Firebase Function (also known as a Google Cloud Function) that you can view in the Cloud Console. 8. In Dialogflow, click Try it now and enter a phrase such as “drive forward 1 foot”.
Troubleshooting
In Dialogflow: Fulfillment -> Inline Editor -> View execution logs in Firebase console
Look for messages indicating the function ran and the PubSub message was sent. Also, verify the correct PubSub command topic is present.
Game preparation and play (15 min)
Now that we have all components setup we can configure the game and start playing! Here are the steps to start the competition:
Do NOT power down your car by pulling the cable out of the card - it will render your car permanently useless due to a hardware issue. If you need to power down the car, press the power button or use ssh to initiate Linux shutdown.
cloudderby.io 63
Install a fresh set of batteries into the car.
Deploy Car Driving Controller as a Google App Engine app (we did this earlier Car Controller module).
Login into your car and start the driving client:
cd $HOME/cloud-derby/car/driver && ./run.sh
Open the Car Controller UI and set the proper ball color assigned to you for the competition:
Click Set driving mode to: Self Driving to start the car for the competitive game.
Congratulations on building a self-driving, IoT and ML robot car powered by Google Cloud! For updates, please visit:
https://www.cloudderby.io
FAQ
1. My car is stuck and is not moving with the red light ON.
cloudderby.io 64
a. The reason the red light is ON and the car is not moving is most likely one of the wheel motors has not been able to complete the number of wheel rotations. To solve the problem lift the car off the ground and see if the motor completes rotations. If not, VERY GENTLY rotate one wheel few degrees. If this does not help, rotate the other wheel.
2.
cloudderby.io 65