a g e | 1

Microsoft .NET and XNA Framework for the Robot

Michael Edoror B.S Computer Science „12 University of Maryland, Baltimore County

Mentor: Chad Jenkins, Ph.D Robotics, Learning, and Autonomy Lab Department of Computer Science Brown University

P a g e | 2

Table of Contents:

1. Introduction

2. ROS & Rosbridge

3. JSON

4. .NET & XNA Integration

5. Conclusions & Future Work

6. References

P a g e | 3

1. INTRODUCTION

The Robot Operating System (ROS) is an open source system which provides libraries and tools to help developers create robot applications. The most important feature with ROS is its abstraction of hardware and device specific engineering. Conceptually, ROS sits on top of hardware and write software to control the hardware.1

My research dealt with creating an interface which can communicate with the ROS runtime environment using .NET and XNA. This framework can publish and subscribe to information from ROS and its connected devices over a TCP/IP socket connection.

2. ROS & ROSBRIDGE

ROS executes its programs as nodes which can run simultaneously. In order for ROS entities to communicate with each other, ROS nodes publish data to listeners called topics. Conversely, in order for entities to receive information they subscribe to topics.

The Robotics Lab at Brown University have been developing software for ROS, creating many programs including special software that will allow for connection to ROS and ROS objects via the web. The interface is a ROS program called Rosbridge and that functionality allows to theoretically control ROS object from anywhere and on any type of computer. ROS has been developed and maintained primarily on Ubuntu but with Rosbridge and my research, I have created a framework for .NET and XNA.

The overall goal of Rosbridge is to be able to send and receive messages made by ROS objects on the ROS server. Messages in the correct format can be sent over to the ROS object through the topic. In order to receive messages, Rosbridge will subscribe to a certain topic and basically listen to what the ROS object publishes to that topic. P a g e | 4

For this project I used two ROS enabled robots. The first was an iRobot_Create. The robot is connected through a serial connection with a netbook that has ROS installed. I would optionally connect a Playstation Eye to the netbook in order to get a video stream from the robot. The second robot I used was an AR_Drone. I made no modifications to the drone and it connects wirelessly to a netbook which has ROS installed.

2. JSON

Rosbridge requires a certain format in order to send and receive data across the socket connection. This format is called JavaScript Object Notation (JSON). It is an open standard many programmers use to interchange data, usually for JavaScript. The exact form varies but for

Rosbridge, message data is arranged in the following manner; JSON messages are just sequences of “name : value” pairs wrapped in either quotes for strings, square brackets for arrays, or curly brackets for objects. For an example this is the JSON formatted string I used to subscribe to a topic in ROS. service = "{\"receiver\":\"/rosjs/subscribe\",\"msg\":[\"/sensorPacket\",0]}";

Figure 1. JSON Formatted String

3. MICROSOFT .NET & XNA

The .Net Framework is just like any other programming environment with the standard tools to help developers create software. The Framework supports programming in the typical paradigms: functional (F#), procedural (), and object oriented (C#). The most popular Integrated Development Environment (IDE) is Microsoft P a g e | 5

Visual Studio. In order to create my framework for the ROS connection I specifically used

Microsoft Visual Studio 2008 with Microsoft Framework version 3.5 and C#.

My first task was to create the client interface that communicates between ROS and the host

Windows Machine. The way Rosbridge was created makes this process pretty straight forward.

In fact, a JavaScript client interface was written in 73 lines of code. The process of creating the client is as follows: create a socket connection, connect the socket to the correct ROS server, and send the appropriate handshake. Afterwards, I would be able to send and receive messages from the ROS server. The big issue that I ran into was that after establishing the connection and sending the handshake, the messages I would summarily send over would not be seen by the

ROS server. In order to alleviate this issue I had to send multiple messages at the same time in order for ROS to register it as seen. It turns out that Rosbridge arbitrarily drops messages. When it was all said in done, I had a fully functional client interface now called WinRos which can send and receive messages from a ROS server all in around 85 lines of code. With all of that in place, I wrote a small teleoperator application that would be able to control an iRobot_Create and an AR_Drone. This application runs through the keyboard and can be found here.

Once the foundation of communication was reached, I had hopes on expanding on Windows development and making something interesting. I decided to integrate my existing code with

Microsoft XNA to create a .

The Microsoft XNA Framework is a collection of classes and libraries used to enhance development of games within the .NET framework. In addition to using XNA, I also used a , FlatRedBall (FRB). FRB is an engine written over top of XNA which encapsulates all of the coding intensive tasks of XNA like loading art assets or creating screens. FRB also has P a g e | 6 a tool called Glue which provides a visual-interactive interface for game development on XNA.

For the game I used FRB June Release and Microsoft XNA Game studio version 3.1. This game is made for XNA PC.

The process for creating the game was as follows: Use the existing WinRos interface to get a connection to Rosbridge through XNA, subscribe to topics that ROS objects on the server transmit like movement and video data, read/process the data from the socket, and display the data on the XNA side by drawing it to the screen. It seems straightforward but I still ran into a couple of issues.

Getting a video stream was probably the biggest issue. The XNA framework supports video but only static videos. Streaming video is not supported for the most part. Another issue is that getting video from the Rosbridge connection wasn‟t traditional in that the data I was reading was an encoded binary string. The normal forms to transmit video are of a compressed video format.

To deal with all of this, I had to do a little conversion gymnastics to decode the binary string in base64 format. Now in order to display the video I had to attach it to a Sprite instance. A sprite is the container of the game object or character in XNA. Every sprite has a texture, which is the actual image. In this case the texture is the video data, so I had to attach the video information to the texture. However texture information has to be in a certain form (in this case an array of bytes) so I further converted the binary string into an array of bytes and read it into the texture using a memory stream. The following code snippet shows the conversion and the next snippet is the process for attaching the video to the texture.

P a g e | 7

public MemoryStream decodeFrom64(string encodedData) { //decodes the binary string byte[] encodedDataAsBytes= System.Convert.FromBase64String(encodedData);

//creates an instance of memory which is just a buffer for the image data MemoryStream ms = new MemoryStream(encodedDataAsBytes, 0, encodedDataAsBytes.Length);

//writes the image data to the buffer ms.Write(encodedDataAsBytes, 0, encodedDataAsBytes.Length);

return ms; }

Figure 2. Binary String Conversion

//converts raw data to string image = Convert.ToString(((Hashtable)serviceReturn["msg"])["uri"]);

//removes the json part of the string image = image.Remove(0, 23);

//creates the file stream MemoryStream stream = decodeFrom64(image);

//these next lines return the buffer to the original position stream.Flush();

//zero base64 stream.Position = 0L;

//attaches the stream to the texture video.Texture = Texture2D.FromFile(FlatRedBallServices.GraphicsDevice, stream); video.ScaleX = 10.5f; video.ScaleY = 10.5f; video.Position = new Vector3(5, 0, 0);

Figure 3. Code Snippet of the complete process

After all of this, I was getting a very choppy image display. The problem was that the frame rate was moving faster than the display of the image. Slowing the frame rate from 60 frames per second to 5 really showed streaming in the video. There was still a 2 second delay between actual movement and the display, but display is a lot more functional than before. P a g e | 8

4 .CONCLUSIONS & FUTURE WORK

I feel very happy about what I accomplished this summer. I learned that I have the capability to develop software at a level that I thought wasn‟t reachable yet. Prior to this internship, I had no experience with the Robot Operating System. I had very little exposure to programming on the

Windows environment but now I feel very comfortable. I also did not have experience with developing applications, but now I do thanks to this internship and the constant pushing from my mentor.

I hope to continue the work I have done this summer. There is a lot I can do to improve and expand upon. For example, I would like to add complete drone functionality to the

WinRos_XNA interface. Being able to control a drone through XNA would be very exciting. I would also like to use a robot called the Neato vacuum robot which uses its laser range-finder to create mappings of its surroundings. With that I could possibly display this data on the XNA side to navigate robots by video and map. I would also like to make the application into an actual 360 . As of now it can only run on an XNA-enabled PC. In order to do this, I would need to upgrade my XNA game studio version and probably rewrite some code.

I would also like to add better art assets to the XNA interface. A large problem I had was the time it took to create the art for the game being that I was very indecisive and not capable of producing good art. Finally, I would also need to upgrade the code itself to make sure every part is as efficient as possible.

P a g e | 9

6. REFERENCES

1 Conley, Ken. "Ros/Introduction." R O S . o r g . Creative Commons Attribution 3.0, 18Jul2011. Web. 15 Aug 2011. .