Microsoft .NET and XNA Framework for the Robot Operating System
Total Page：16
File Type：pdf, Size：1020Kb
P a g e | 1 Microsoft .NET and XNA Framework for the Robot Operating System Michael Edoror B.S Computer Science „12 University of Maryland, Baltimore County Mentor: Chad Jenkins, Ph.D Robotics, Learning, and Autonomy Lab Department of Computer Science Brown University P a g e | 2 Table of Contents: 1. Introduction 2. ROS & Rosbridge 3. JSON 4. .NET & XNA Integration 5. Conclusions & Future Work 6. References P a g e | 3 1. INTRODUCTION The Robot Operating System (ROS) is an open source system which provides libraries and tools to help software developers create robot applications. The most important feature with ROS is its abstraction of hardware and device specific engineering. Conceptually, ROS sits on top of hardware and programmers write software to control the hardware.1 My research dealt with creating an interface which can communicate with the ROS runtime environment using .NET and Microsoft XNA. This framework can publish and subscribe to information from ROS and its connected devices over a TCP/IP socket connection. 2. ROS & ROSBRIDGE ROS executes its programs as nodes which can run simultaneously. In order for ROS entities to communicate with each other, ROS nodes publish data to listeners called topics. Conversely, in order for entities to receive information they subscribe to topics. The Robotics Lab at Brown University have been developing software for ROS, creating many programs including special software that will allow for connection to ROS and ROS objects via the web. The interface is a ROS program called Rosbridge and that functionality allows people to theoretically control ROS object from anywhere and on any type of computer. ROS has been developed and maintained primarily on Ubuntu Linux but with Rosbridge and my research, I have created a framework for .NET and XNA. The overall goal of Rosbridge is to be able to send and receive messages made by ROS objects on the ROS server. Messages in the correct format can be sent over to the ROS object through the topic. In order to receive messages, Rosbridge will subscribe to a certain topic and basically listen to what the ROS object publishes to that topic. P a g e | 4 For this project I used two ROS enabled robots. The first was an iRobot_Create. The robot is connected through a serial connection with a netbook that has ROS installed. I would optionally connect a Playstation Eye to the netbook in order to get a video stream from the robot. The second robot I used was an AR_Drone. I made no modifications to the drone and it connects wirelessly to a netbook which has ROS installed. 2. JSON Rosbridge requires a certain format in order to send and receive data across the socket connection. This format is called JavaScript Object Notation (JSON). It is an open standard many programmers use to interchange data, usually for JavaScript. The exact form varies but for Rosbridge, message data is arranged in the following manner; JSON messages are just sequences of “name : value” pairs wrapped in either quotes for strings, square brackets for arrays, or curly brackets for objects. For an example this is the JSON formatted string I used to subscribe to a topic in ROS. service = "{\"receiver\":\"/rosjs/subscribe\",\"msg\":[\"/sensorPacket\",0]}"; Figure 1. JSON Formatted String 3. MICROSOFT .NET & XNA The .Net Framework is just like any other programming environment with the standard tools to help developers create software. The Framework supports programming in the typical programming language paradigms: functional (F#), procedural (Visual Basic), and object oriented (C#). The most popular Integrated Development Environment (IDE) is Microsoft P a g e | 5 Visual Studio. In order to create my framework for the ROS connection I specifically used Microsoft Visual Studio 2008 with Microsoft Framework version 3.5 and C#. My first task was to create the client interface that communicates between ROS and the host Windows Machine. The way Rosbridge was created makes this process pretty straight forward. In fact, a JavaScript client interface was written in 73 lines of code. The process of creating the client is as follows: create a socket connection, connect the socket to the correct ROS server, and send the appropriate handshake. Afterwards, I would be able to send and receive messages from the ROS server. The big issue that I ran into was that after establishing the connection and sending the handshake, the messages I would summarily send over would not be seen by the ROS server. In order to alleviate this issue I had to send multiple messages at the same time in order for ROS to register it as seen. It turns out that Rosbridge arbitrarily drops messages. When it was all said in done, I had a fully functional client interface now called WinRos which can send and receive messages from a ROS server all in around 85 lines of code. With all of that in place, I wrote a small teleoperator application that would be able to control an iRobot_Create and an AR_Drone. This application runs through the keyboard and can be found here. Once the foundation of communication was reached, I had hopes on expanding on Windows development and making something interesting. I decided to integrate my existing code with Microsoft XNA to create a video game. The Microsoft XNA Framework is a collection of classes and libraries used to enhance development of games within the .NET framework. In addition to using XNA, I also used a game engine, FlatRedBall (FRB). FRB is an engine written over top of XNA which encapsulates all of the coding intensive tasks of XNA like loading art assets or creating screens. FRB also has P a g e | 6 a tool called Glue which provides a visual-interactive interface for game development on XNA. For the game I used FRB June Release and Microsoft XNA Game studio version 3.1. This game is made for XNA PC. The process for creating the game was as follows: Use the existing WinRos interface to get a connection to Rosbridge through XNA, subscribe to topics that ROS objects on the server transmit like movement and video data, read/process the data from the socket, and display the data on the XNA side by drawing it to the screen. It seems straightforward but I still ran into a couple of issues. Getting a video stream was probably the biggest issue. The XNA framework supports video but only static videos. Streaming video is not supported for the most part. Another issue is that getting video from the Rosbridge connection wasn‟t traditional in that the data I was reading was an encoded binary string. The normal forms to transmit video are of a compressed video format. To deal with all of this, I had to do a little conversion gymnastics to decode the binary string in base64 format. Now in order to display the video I had to attach it to a Sprite instance. A sprite is the container of the game object or character in XNA. Every sprite has a texture, which is the actual image. In this case the texture is the video data, so I had to attach the video information to the texture. However texture information has to be in a certain form (in this case an array of bytes) so I further converted the binary string into an array of bytes and read it into the texture using a memory stream. The following code snippet shows the conversion and the next snippet is the process for attaching the video to the texture. P a g e | 7 public MemoryStream decodeFrom64(string encodedData) { //decodes the binary string byte[] encodedDataAsBytes= System.Convert.FromBase64String(encodedData); //creates an instance of memory which is just a buffer for the image data MemoryStream ms = new MemoryStream(encodedDataAsBytes, 0, encodedDataAsBytes.Length); //writes the image data to the buffer ms.Write(encodedDataAsBytes, 0, encodedDataAsBytes.Length); return ms; } Figure 2. Binary String Conversion //converts raw data to string image = Convert.ToString(((Hashtable)serviceReturn["msg"])["uri"]); //removes the json part of the string image = image.Remove(0, 23); //creates the file stream MemoryStream stream = decodeFrom64(image); //these next lines return the buffer to the original position stream.Flush(); //zero base64 stream.Position = 0L; //attaches the stream to the texture video.Texture = Texture2D.FromFile(FlatRedBallServices.GraphicsDevice, stream); video.ScaleX = 10.5f; video.ScaleY = 10.5f; video.Position = new Vector3(5, 0, 0); Figure 3. Code Snippet of the complete process After all of this, I was getting a very choppy image display. The problem was that the frame rate was moving faster than the display of the image. Slowing the frame rate from 60 frames per second to 5 really showed streaming in the video. There was still a 2 second delay between actual movement and the display, but display is a lot more functional than before. P a g e | 8 4 .CONCLUSIONS & FUTURE WORK I feel very happy about what I accomplished this summer. I learned that I have the capability to develop software at a level that I thought wasn‟t reachable yet. Prior to this internship, I had no experience with the Robot Operating System. I had very little exposure to programming on the Windows environment but now I feel very comfortable. I also did not have experience with developing applications, but now I do thanks to this internship and the constant pushing from my mentor. I hope to continue the work I have done this summer. There is a lot I can do to improve and expand upon. For example, I would like to add complete drone functionality to the WinRos_XNA interface.