Controlling the NAO: the Humanoid Robot MET 435W By: Mister Higgins [email protected] Chris Zachary [email protected] Adviser

MET 435W

By: Mister Higgins [email protected] Chris Zachary [email protected]

Adviser: Dr. Vukica Jovanovic Submitted: April 22, 2020 Letter of Transmittal April 22, 2020 Dr. Vukica Jovanovic Associate Professor Engineering Technology Department 214 Kaufman Hall, Old Dominion University Norfolk, VA 23529

Dear Dr. Jovanovic:

In compliance with the Senior Design Project requirements of MET 435, the technical report is attached for your review and approval. Mister Higgins and Chris Zachary are formally presenting a project named: “Controlling the NAO: The Humanoid Robot” as a final product for the Senior Design Project. Please consider the project for submission to the Department of Engineering Technology at Old Dominion University, Mechanical Engineering Technology Program. The main purpose of the document is to present an “Controlling the NAO: The Humanoid Robot” to learn how to operate this autonomous robot. The project team plans on doing a presentation that can be future used by middle schoolers to learn what is NAO robot and what it can do.

Sincerely,

Mister Higgins ______

Chris Zachary ______

APPROVAL: DATE: 04/22/2020

Abstract

NAO is a humanoid robot that is designed to work with humans in a variety of different ways and environments. NAO can be used to: -assess emotional changes in children, process those changes using facial recognition software, and give proper responses through various commands. The main purpose of this project is to learn how to operate this autonomous robot. It costs almost $10,000 (USD). The project team plans on doing a presentation that can be future used by middle schoolers to learn what is NAO robot and what it can do.

Table of Contents Abstract ...... 2 Equipment List ...... 4 Project Objective ...... 6 Introduction ...... 9 Discussion ...... 10 Why use the NAO Robot ...... 10 Choregraphe ...... 10 Challenges ...... 14 Conclusion ...... 16 Presentation ...... 16 Applications ...... 17 References ...... 18 Appendix ...... 19

Equipment List

Figure 1 NAO Robot (Softbank Robotics, 2020) This is NAO. Humanoid robots borrow inspiration from bipedal locomotion found in humans. Various developments of humanoid robots transpired in the previous century. These artificial helpers would be based on the electro-mechanical (mechatronics system), embedded with artificial intelligence that would work reliably and repetitively without injuring human beings. Having robots be “human-like” would make them more socially adaptable (Kanda, 2004). Various biped robots have been created in previous decades. Some of them were designed to showcase advanced capabilities of research departments from companies like Honda or Sony.

Figure 2: Different humanoid robots in history: a) Maschinenmensch "Robot; b) ASIMO at a Honda factory; c) Sony Qrio Robot; d) NAO Evolution

The NAO robot was developed by Aldebaran Robotics in 2006, later rebranded as SoftBank Robotics, and became one of the first humanoid robots placed on the mass market (Shamsuddin, 2011 SoftBank, 2016). It has seven senses integrated for natural interaction: moving, feeling, hearing and speaking, seeing, connecting, and thinking (SoftBank, 2016), and is a dynamically stable, bipedal robot that has controls based on the hybrid zero dynamics (Ames, 2012; Kofinas, 2012; Vukobatrovic, 2004; Vukobratvic, 2007). It is an autonomous robot with twenty-five degrees of freedom that can interact with the environment and adapt to events, such as falling to the ground, responding to someone talking to it, and alike. It has four directional microphones and loudspeakers, various sensors placed at its head, hands, and feet. It also has wireless connectivity with mobile devices. Its structure can be seen in Fig.2.

Figure 3. NAO Robot schematics

Project Objective

The main purpose of this project is to learn how to operate this autonomous robot. It costs almost $10,000 (USD). The project team plans on doing a presentation to middle schoolers by programming NAO to perform various functions like facial recognition, mathematical problems, and various dances.

Figure 4: Choregraphe will be used to program NAO.

Figure 5: NAO Charger (Softbank Robotics, 2019)

The charger for NAO is from the company SoftBank Robotics. The charger has an input of 100- 240 volts and has a current of 1.0 amperes with a frequency of 50-60 Hertz. The charger has an output of 24.8 volts and current of 2.0 amperes.

Figure 6: Mister’s MacBook Air

This is Mister Higgins’ laptop, a 2015 edition of the MacBook Air used primarily for schoolwork and to program NAO using Choregraphe software.

Figure 7: This is Chris Zachary’s laptop, a Lenovo IdeaPad S340.

This laptop uses a Windows 10 processor and has Choregraphe software downloaded to work with the NAO robot.

Figure 8: Ethernet port and ethernet cable

This approximately 15 feet long ethernet cable will connect with the pictured ethernet port, so that NAO will have a more stable connection and IP address. The purple port is used exclusively for this project and to connect to NAO. Thus, this cable will be connected to the internet so that programming can happen. When The pink port was used, they ran into problems because NAO was not connecting effectively to their laptops.

Introduction NAO is an autonomous, programmable humanoid robot developed by a French company called Aldebaran Robotics. This robot design is based on a human body with two arms and legs, a head and torso: hence, the name humanoid robot. Some features a humanoid robot may also have are a face with eyes and a mouth. Since NAO is autonomous, it can have cognitive and social experiences of a human as well. An example of NAO’s cognitive skill-set is the ability to identify its current environment, learn from the surroundings, then process whatever changes are needed.

Figure 9: NAO Robot kinematics

Our NAO is primarily white with grey joints and red trims. He also has 25 degrees of freedom, 2 cameras, 4 microphones, LED lights, 2 IR emitters and receivers, 2 speakers, 1 interior board, 9 tactile sensors, 8 pressure sensors , 1 GB RAM, 2 GB flash memory, a voice synthesizer and sonar rangefinder. NAO got has Ethernet and/or wireless networking capabilities that can be used for programming. There are two ways to program NAO: 1) Choregraphe and 2) NAOqi SDK, which allows usage of C++, C#, Java, Python, and MATLAB

The decision was made to learn how to use Choregraphe since no one in the group has extensive programming knowledge. Choregraphe is a visual-based programming tool that has options and commands drag and drop actions from the library (pre-programmed and customized boxes) and using wires to mutually connect commands.

Discussion

Why use the NAO Robot The NAO robot will be used to help introduce children to the STEM field in a fun and stimulating way. Many young, minority children do not get the affirming and pragmatic chance to positively experience the excitement of the STEM field at an early age, so they do not shy away from majors such as engineering or computer science if they decide to go to college. By introducing children to the NAO robot, it will give them insight into the robotics field and also show them a fulfilling time by having them interact with NAO. With these constructive interactions some of the children may take a liking to NAO and will hopefully expand their interest in the STEM field.

Choregraphe Choregraphe is a programming software used to create and control the behaviors of the NAO robot. Choregraphe has a vast array of actions that can be used and combined to recreate dances and totally unique movements. For example, Choregraphe allows NAO to move, walk, talk, ask questions, and respond to questions. It is also capable of playing uploaded music or sounds through its speakers.

Figure 10: This is what Choregraphe looks like when it is first opened.

At the top left of the screen is where files can be opened and saved. The green wave symbol at the top is used to connect a laptop or computer to the NAO robot. The red wave symbol next to the green one is used to disconnect the laptop or computer from NAO. On the bottom left side of the screen, is a box of libraries filled with different actions for NAO to perform. To the right of the screen, there are three basic standing positions for NAO to take up. Lastly, at the bottom right is the virtual robot that can be used to try out the programs when NAO is not able to be connected.

Figure 11: Simple program

The photo above shows what a complete program on Choregraphe looks like.This program tells NAO to stand up, walk a certain distance, sit down, and say “don’t be dirty, wash your hands”. To initiate the program-building process, the desired actions must first be dragged onto the screen. After completion, the black triangle is used to connect each of the actions together.

Figure 12: Clicking on the “say” button twice will pull up this screen asking the user to input words for NAO to say.

First, the language box must be changed from French to English. Then, the words are put into the dialogue box. After the dialogue is entered, the wrench under the “say” text box can be used to edit the voice tone and the speed at which NAO speaks. After all this is done, NAO will be ready to perform the actions it was programmed to do.

Figure 13: This is a more complex program on Choregraphe.

This program was designed to make NAO shake its arms up and down rhythmically. The “wait” command is used to delay the impending actions, so that the movements can flow the way the creator intended.

Figure 14: Choregraphe timeline

The most important function in this program is the “timeline” function. It is used to set specific actions that are not stored in NAO. The timeline controls are located near the top of the screen. It works by setting a start frame and an end frame with desired positions set certain frames apart. NAO’s position can be changed manually by moving the appendages by hand, or virtually using the motion controls.

Figure 15: The motion controls work by moving NAO’s joints from the computer.

The joints can be represented by two intersecting circles, or by arrows twisting around the arm. Both work relatively the same way by turning the circle or arrow the way you desire it to move, they go up to store position in frame, and then the position is set. It is better to use the arrows than the circles because the movements are easier to control. From this project, it can be said that using the timeline can be a very tedious task.

Challenges One of the problems that Chris and Mister faced was the lack of experience in programming. Since neither of them had ever seen or heard of Choregraphe until Professor Jovanovic sat them down and showed them how to use it. Once she assisted, they had to learn independently and practice with the program until they were able to do a presentation for middle schoolers.

Figure 16: Robo Phil NAO programing video

Chris and Mister used a few resources to learn how to program. One was popular “Youtuber”, Robo Phil, who had a YouTube series on how to program NAO to do various complex tasks like speech recognition, simple math problems, and facial recognition. Another resource used was an internet search for some programs online of NAO doing push-ups as well as Michael Jackson’s “Thriller” dance. Finally, Jovanovic helped keep them on track, and effectively simplify self-made complexities.

Another challenge encountered was connecting to NAO. It took a while to get NAO to connect to their laptops via Wi-Fi. NAO was not connecting because of the pink (wrong) port. The pink one was not working for whatever reason, but the purple port was the correct one to use. Chris’s laptop had an ethernet cable port, so he did not have difficulty connecting effectively; however, Mister’s laptop did not have connection capability.

Figure 17: Project schedule

The next hurdle they faced was time management. This is a semester-long project, so difficulties staying focused on the work that needs to be done as well as prioritizing classes, work, COVID- 19 pandemic evacuations, and family/social responsibilities. Preventive planning, scheduling frequent project meetings (2-3 per week) for three to four hours, and proactively starting on the report helped avoid panic and expedited project completion.

Figure 18: Final problem was the effects of COVID-19.

The university evacuation was a devastating blow since access was denied to lab Kaufman Hall 116 (NAO’s humble home). With social distancing, meetings were cancelled to practice new programs on NAO. Unfortunately, the outreach program showing STEM to minority students came to an end. Advisor Jovanovic suggested finishing up the report and having a zoom meeting for Oral presentation.

Conclusion

Presentation The second objective of this project was to introduce STEM to minority children through the usage of NAO. First, it involves one of two groups of children: 1) the NSBE Jr Chapter Initiative or 2) Mister’s Mentor Program Involvement. Since this is a hypothetical project, the engineers will work with participants selected by the reader of this report.

In order to work effectively with the students, programmers will need to secure additional robots, classroom space, presentation posters and items identified on the previously mentioned equipment list. Next, the kids would be asked to define STEM and identify engineering duties and concepts. From their responses, the engineers would assess the students’ prior knowledge of STEM and select appropriate and enjoyable activities for them.

Afterwards, NAO and its capabilities would be revealed to the kids. For example, first order of business, NAO would ask the children to wash their hands to prevent the spread of germs. Secondly, NAO would use facial recognition to apply custom expressions appropriate for each child. In addition, NAO would perform Michael Jackson’s “Thriller Dance”, and the Tai Chi routines, and other actions to show the humanoid’s complex movements. Hopefully, the kids become inspired to follow along with NAO as he performs push-ups and other activities, thus, promoting a challenging and exciting experience. Math anyone? NAO can show off his “math skills” to the future STEM scholars by doing simple multiplication and division problems. Finally, NAO would end the presentation by singing a song and saying “goodbye” to the young enthusiasts. Afterwards, the STEM engineers and facilitators would make closing remarks and ask final questions such as: • What did you enjoy about NAO? • What did you learn about engineering and robotics? • What about this experience created the most fun for you?

At the end of the day, assumptions are all we have, since we were unable to give a presentation to a target group due to COVID-19. We assumed the children were well-behaved, gained knowledge, and enjoyed their experience with NAO.

Applications There are many applications for the autonomous humanoid robot called NAO. Some of those applications include teaching, being a child’s companion, being a storyteller and a music speaker. The most useful of these applications would be teach. During this time of quarantine and isolation in 2020, NAO could fill in as a home teacher. Many schools across the country and the world have started switching to online classes. Unfortunately, Online classes just don't work for everyone. Should some effectively design a program that could teach Math, English, History, and Science, which would allow individuals to learn things at their own pace. NAO already has the capabilities to be programmed to write and do math, so it is only right to take advantage of them for the use of education. These programs could be downloaded and stored online, and the users would download them and use them at their own pace as needed. With NAO it can be shown that robotics can be used to help society in ways that were never imagined before.

References

Ames, A. D., Cousineau, E. A., & Powell, M. J. (2012). Dynamically stable bipedal robotic walking with NAO via human-inspired hybrid zero dynamics. Paper presented at the Proceedings of the 15th ACM international conference on Hybrid Systems: Computation and Control. https://www.researchgate.net/figure/Schematics-of-the-NAO-NIMA-Robot23_fig2_25649688 https://www.sciencedaily.com/terms/humanoid_robot.htm https://www.youtube.com/watch?v=SSD8DsVSrj4&list=PLGORloyIXX3EgyLIaLsDdzd- tMFbJoq9R Kanda, T., Ishiguro, H., Imai, M., & Ono, T. (2004). Development and evaluation of interactive humanoid robots. Proceedings of the IEEE, 92(11), 1839-1850. Kofinas, N. (2012). Forward and inverse kinematics for the NAO humanoid robot. Technical University of Crete, Greece. Kofinas, N., Orfanoudakis, E., & Lagoudakis, M.G. (2015). Complete Analytical Forward and Inverse Kinematics for the NAO Humanoid Robot. Journal of Intelligent & Robotic Systems, 77, 251-264. Shamsuddin, S., Ismail, L. I., Yussof, H., Zahari, N. I., Bahari, S., Hashim, H., & Jaffar, A. (2011). Humanoid robot NAO: Review of control and motion exploration. Paper presented at the Control System, Computing and Engineering (ICCSCE), 2011 IEEE International Conference on. Softbank Robotics (2019). Nao Robot Charger, Retrieved on April 14, 2020, from: https://www.robotlab.com/store/nao-charger Softbank Robotics (2020). Nao Robot, Retrieved on April 14, 2020, from: https://www.softbankrobotics.com/emea/en/nao SoftBank. (2016). Who is NAO? Retrieved October, 14, 2016, from https://www.ald.softbankrobotics.com/en/cool-robots/nao SoftbankRobotics. (2017). NAO Evolution. In N. E. .jpg (Ed.), Wikimedia Commons, the free media repository. Vukobratović, M., & Borovac, B. (2004). Zero-moment point—thirty five years of its life. International Journal of Humanoid Robotics, 1(01), 157-173. Vukobratovic, M., & Rodic, A. (2007). Contribution to the integrated control of biped locomotion mechanisms. International Journal of Humanoid Robotics, 4(01), 49-96.

Appendix

Figure 1: NAO Robot (Softbank Robotics, 2020)...... 4 Figure 2: Humanoid robotics in history……………...…..………………………………………..4 Figure 3. NAO Robot schematics…………………………..……………………………………..5 Figure 4: Choregraphe will be used to program NAO………………………………………….....6 Figure 5: NAO Charger (Softbank Robotics, 2019)...... 6 Figure 6: Mister’s MacBook Air…………………………………………………………………..7 Figure 7: This is Chris Zachary’s laptop…………………………………………………………..7 Figure 8: Ethernet port and cable………………………………………………………………….8 Figure 9: NAO Robot kinematic..……..…………………..………………………………………9 Figure 10: Choregraphe opening screen…………………………………………………………10 Figure 11: Choregraphe program simple………………..…………………………………….....11 Figure 12: Choregraphe say tab……………………………..…………………………………...11 Figure 13: Choregraphe advanced program……..……………………………………………….12 Figure 14: Choregraphe timeline ……………..…………………………………………………12 Figure 15: Nao motion controls………………………………………………………………….13 Figure 16: Robo Phil video..……………………………………………………………………..14 Figure 17: Schedule……………………………………………………………………………...15 Figure 18: COVID-19……………………………………………………………………………15