A Step by Step Runthrough of FRC Vision Processing

A Step by Step Runthrough of FRC Vision Processing David Gerard and Paul Rensing FRC Team 2877 LigerBots Contents 1 Abstract 2 2 Goals of Vision Processing 2 3 Vision System Configurations 2 3.1 Vision Processing on the roboRIO ........................ 4 3.2 Vision Processing on a Coprocessor ....................... 4 3.3 Vision Processing on Cell Phones ........................ 5 3.4 Vision Processing on the Driver's Station .................... 6 3.5 LigerBots 2018 Choice .............................. 6 4 Comparisons of Coprocessors 6 4.1 Coprocessor Operator Interface ......................... 8 5 Coprocessor Communication 9 6 Cameras 9 6.1 Choice of Camera ................................. 9 6.2 Camera Placement ................................ 11 6.3 Camera Mounting ................................. 11 6.4 Camera Cabling Challenges ........................... 12 7 Programming Environment 12 7.1 Language Choice: Python vs. C++ vs. Java . 12 7.2 Vision Processing Framework .......................... 13 8 Code Breakdown 13 8.1 Camera Calibration ................................ 14 9 Coordinate Systems and Geometry 14 9.1 Coordinate Systems ................................ 15 9.2 Coordinate Transformations ........................... 16 9.3 OpenCV Python Code .............................. 17 10 Additional Problems We Encountered 19 11 Appendix 20 11.1 Compiling OpenCV for the ODROID ...................... 20 11.2 Camera Tilt .................................... 21 1 1 Abstract This paper provides a detailed analysis of the implementation of vision processing for both rookie and veteran FRC robotics teams. After first hand experiences of vision processing throughout the 2018 Power Up season and years prior, we have recognized both our successes and failures. After compiling these key findings together, we created this guide to lead teams through current and future challenges, in hope to aid them in putting together a complete and accurate vision system in sync with their robot. 2 Goals of Vision Processing When first discussing ways to implement vision processing into a robot, it's important to consider the impact the work will have on your team's success. Often, depending on certain circumstances and requirements, vision may not be useful enough to consider adding to a repertoire of abilities. Many teams in the FRC 2018 game, FIRST Power Up, chose not to include vision processing in their robot due to the limited uses of cube and switch target detection throughout the game. Although, if chosen to be incorporated into a system correctly and efficiently, vision processing can in fact prove to be incredibly helpful in: • Creating autonomous routines • Constructing predefined actions for teleop • Enhancing the driver's abilities (streaming a camera feed to the driver station) • Programming the robot With vision processing, rather than using brute force reckoning during the autonomous and teleop periods, a team can instead ensure certain aspects of their gameplay and com- pletely change their strategies for the better. In addition to increasing the accuracy of the robot code, having a solid system for tracking targets helps drivers during a match with tough visibility or under the pressure of time. Just helping a robot reduce even a second or two from its cycle time by simply aligning to a small target or providing the driver with helpful information can make a significant difference, allowing a team to help out their alliance and possibly win close matches. Although, after deciding to incorporate vision processing into a design, many other aspects and trade-offs of the system must first be considered before continuing. They can notably affect the efficiency and effectiveness of the system. 3 Vision System Configurations A team must consider system configurations carefully and thoroughly examine what fits their requirements. Here are a few configurations the LigerBots considered throughout the years. Some have worked better for us than others, as explained later on, but it's up to a team themselves to figure out what works best for them. 2 • Processing directly on the roboRIO (USB or Ethernet camera) • Processing on a coprocessor (USB or Ethernet camera, results sent to roboRIO) • Processing on the driver's station (USB or Ethernet camera, results sent to roboRIO) • Processing on cell phone(s) (results sent to roboRIO) As seen in Figure 1, there is increasingly complex depth in assembling a complete set-up. We recommend going through at least the top level of the workflow below to understand each individual aspect. Figure 1: Various choices for the complete vision processing system 3 3.1 Vision Processing on the roboRIO Figure 2: Schematic of vision processing on the roboRIO One crucial item to consider while creating the basis for an efficient system is the platform on which to run processing code. This can be done either on the main processor itself or on a separate coprocessor. It is recommended to have a coprocessor run the vision code rather than running it directly on a roboRIO (unless the camera is simply being streamed directly to the driver's station without additional processing) because the vision processing may delay the robot code's cycle time on the roboRIO. This will create jolting movements due to a long loop period and numerous other possible problems. • OK for simple streaming to DS • Potential lag in robot control 3.2 Vision Processing on a Coprocessor Having a coprocessor will ensure that these types of problems will not happen and will increase the speed of the processing. Figure 3: Schematic for vision processing on a coprocessor 4 As shown in Figure 3, the radio connected to the roboRIO is also connected to the coprocessor. The coprocessor is constantly processing the images it receives from the attached USB camera. Then, through a system such as NetworkTables, the coprocessor sends the processed information to the roboRIO for the robot code to utilise during operation. • Pros: { Can run vision processing at a frame rate independent of the rate of images sent to DS { No risk of overloading the roboRIO • Cons: { Extra hardware: power supply, Ethernet { Communication delays (NetworkTables) 3.3 Vision Processing on Cell Phones For the 2017 season, the LigerBots did vision processing on cell phone cameras. We used 2 phones on the robot, one aimed forward for the gear placement mechanisms, and one angled upward to aid in targeting the Boiler. The phones were connected to the roboRIO using IP networking over USB cables. This has a number of perceived advantages: • A good camera is built into the phone • Vision processing can happen directly on the phone. Cell phone processors can be quite powerful, depending on the model. • The price can be reasonable, especially if you purchase past generation used phones. The LigerBots used two Nexus 5 phones. However, while the cameras did work reasonably well, the experience gained during the season convinced us that it is not our preferred solution: • The phones are large, heavy, and hard to mount. The mounting required a significant amount of space, particularly since we needed to remove the phone frequently (for charging, etc.). • The weight of the phone plus 3d printed mount meant that the cameras were signif- icantly affected by vibration when the robot was moving. This made the hoped-for driver assist very tricky. • An unexpected twist was that using 2 phones with USB networking required modifying, compiling, and installing a custom Android kernel on one of the phones. The issue was that the IP address assigned to the phone for USB networking is hard-coded in the kernel, and therefore the two phones had the same IP address, which would not work. A student installed a new kernel, but this is very specialized knowledge. 5 • The phones would get very hot during a match (we used them throughout for driver assist), and needed to be removed frequently to cool and be charged. • The vision processing needs to be programmed in Java (only), and it needs to be pack- aged as an Android app which auto-starts. Again, this is more specialized knowledge. Given all the factors, we decided we did not want to continue using the phones for vision processing. 3.4 Vision Processing on the Driver's Station While the LigerBots do not have any experience with this setup, many teams have success- fully used vision processing on the Driver's Station (DS) laptop. In this setup, a streaming camera is set up on the robot (Ethernet camera, or USB camera connected to roboRIO). A program runs on the DS which grabs a frame, processes it, and sends the necessary results back to the roboRIO via NetworkTables. • Pros: { Plenty of processing power on DS { Easy programming environment (Windows laptop with system of your choosing) • Cons: { Extra lag between image capture and results getting to roboRIO. Image frames and results are both transmitted over WiFi FMS. { Captured images should be shared between processing and Driver's Station (for display to driver), otherwise you will use double the bandwidth. 3.5 LigerBots 2018 Choice During the 2018 season, the LigerBots chose to incorporate a coprocessor into the robot design due to our experiences over the years. The coprocessor allowed our team to run a processing routine without interrupting the roboRIO. Additionally, having a coprocessor enabled additional further freedoms not only in the code, but also in the methods of communication between the two processors and other systems. 4 Comparisons of Coprocessors We wanted to investigate numerous coprocessors to find which

Load more