Seeing GIS Data Using AR Richie Carmichael of the Esri Applications Prototype Lab has been experimenting with using AR to visualize GIS content. He thinks of AR as simply adding to reality GIS in a way that enhances or improves upon & what you see displayed in a web or desktop Making GIS content available application. “GIS data is already a source of spatially en- in interesting and useful ways abled information that can be fed directly to your mobile device,” explained Carmichael. By Keith Mann, Esri Writer “Your phone knows where it is, the direction the camera is facing, and the inclination of the device in your hand. This information, combined with GIS data, not only adds to the end user’s experience but also makes GIS ac- cessible in a new and interesting way.” As AR becomes more prevalent in phone, tablet, and computer applications, more de- velopers will begin integrating GIS services If you own a smartphone, such as an iPhone Familiar Applications of AR and content to serve very real and practical or Android, you probably have an app that While the concepts of AR have been around purposes. Mansour Raad, a senior software uses augmented reality (AR). This technol- for decades, the technology has been applied architect at Esri, envisions AR apps that ogy superimposes digital information on to so many different types of media that it will channel GIS content to professional whatever you’re looking at through your is difficult to point to any one application as end users using mobile devices to help them phone’s camera. the embodiment of augmented reality. For make decisions on-site using the most up- These types of AR apps usually combine example, American football fans are used to-date and reliable information available. an assortment of miniaturized technology to seeing the virtual first down line that ap- “Imagine that you could point the camera devices—optical sensors, accelerometers, pears on the field during televised games— on your phone at the ground and see the GPS, gyroscopes, solid state compasses— they expect to see it. Movie directors have location and orientation of water pipes and along with context-sensitive information, portrayed augmented reality in movies such electric cables buried under the ground be- cause your AR app is cross-referencing the that is delivered as a web service. The service as Minority Report (2002) or Avatar (2009) content may be displayed as location-based to show how immersive technology might be GIS system with your location and giving labels (or billboards) that hover over or in used. Some of the most visually exciting ex- front of objects you see through the camera amples of AR entertainment are augmented display. The labels usually present additional projections or projection mapping in which information, including photos, but often computer imagery is projected onto physical provide reviews and contact information objects, such as a building facade, to create for things like restaurants, Wi-Fi hot spots, an augmented but realistic-looking new or houses for sale. You might use AR apps object that morphs continuously before the to discover and learn more about objects viewer’s eyes. around you. The military uses a type of AR in the Some AR apps allow you to use your phone head-up display (HUD) that appears on the camera to perform sophisticated measure- cockpit window (or view port) of a jet fighter. ments of distances and angles or even track These displays make it easier for pilots to cosmological events such as the azimuth of access critical information such as air speed  Richie Carmichael of the Esri Prototype Lab built this AR/GIS app using ArcGIS the sun. Still other AR apps amazingly rec- or angle of attack without having to break API for Silverlight and the open source ognize patterns in photos, diagrams, and their focus from the reality in front of them. Silverlight Augmented Reality Toolkit. It bar codes that then link you to additional Similarly, automobile manufacturers are orders and offsets overlapping imagery. Imagery is tagged with sensor information content or services, such as product reviews, also experimenting with HUD windshield such as capture date, percentage cloud price comparisons, or videos. displays to improve driver safety. cover, and satellite position.

28 au Spring 2011 esri.com Developer’s Corner you a kind of x-ray vision so that you can vis- viewfinder. Here the device’s location and ori- measurements are only approximations,” he ualize the infrastructure that is underneath entation play an important role. In a different said. “This may frustrate end users with GIS you,” explained Raad. case, the information seen by the camera is and CAD experience because they expect a “Not only would you be able to see hidden sent to a server and processed, then sent back high degree of accuracy and precision.” Both objects, you’d have access to their attrib- to you. Here, pattern recognition technology Emge and Carmichael recommend that AR utes, you’d be able to pull up engineering is being employed, and the response can be apps be used to query trusted GIS services diagrams, and even tap into real-time sensor as simple as playing a video or as complex as for precise measurements and reliable at- networks to view water pressure or amper- showing the results of spatial analysis.” tribute information instead of relying on a age.” Raad suggested that you might use Raad sees great potential for using the phone’s capabilities. your phone’s camera to take a photo of the pattern recognition capability of AR. “The Regardless of the popularity of AR or the augmented display and e-mail it to your image can be anything—a photo, a map, a current limitations of some mobile devices, project team or sync it with the GIS-enabled barcode, or even the stars in the sky. Using AR and GIS would seem to be a perfect work order system so that it can be accessed the camera, the AR app recognizes the pat- match, not only for consumers but for pro- by the project manager and the work crew. tern and does something that augments or fessionals who rely on geographic informa- “What makes the combination of AR and GIS mixes additional content with what you’re tion to make real-time decisions. really exciting is not just the visualization of seeing.” GIS content, which is very cool, but the inte- For example, Raad has created a sample gration of that content with the rest of the app that uses pattern recognition to let users enterprise. That’s what GIS is all about.” point an iPhone camera at a photo of a map, which triggers the loading of a video that Getting Started with AR shows a rotating globe displaying the same Development information. Raad used ArcGIS to compose AR app development is surprisingly straight- the 2D map data and drape it on a globe. forward. AR service providers, such as juniao Then he animated the globe and created the or Layar, provide both an AR browser and video, which is hosted on a server. Next, he the development resources needed to sim- registered a channel with junaio that links plify deployment for iPhone and Android. his video with the junaio AR application. To use the junaio app, you start by regis- Anyone with an iPhone can download the tering a “channel” with junaio. The channel junaio app, search for the channel, point the provides a callback to your server, where the iPhone camera at a photo of the map, and processing takes place. For example, if you view the rotating globe. use the app to find geographic information Carmichael also used pattern recogni- near your current position, junaio brokers tion. He linked a photo on a piece of paper the calls to your server, which processes to a related image of a map layer stack in his the request, then passes that information desktop application. However, in this case, back to your device. For a GIS developer, the layer stack appears to dynamically hover this means accessing the REST endpoints of above the photo—even if the photo is moved  Thomas Emge and Sathya Prasad, of the ArcGIS Server services as the link to dynam- around. Esri Applications Prototype Lab, created ic GIS content and tools or preprocessed Any developers using an AR toolkit and the AR app with ArcPhoto tools and Layar’s iPhone app. It georeferences images geographic information such as 3D terrains, one of the ArcGIS APIs for Silverlight, Flex, from a geodatabase and displays them as videos, or map layer stacks. or JavaScript can start building their own billboarded entities. “Augmented reality can be difficult to AR-GIS apps. The original ARToolkit was de- explain to people,” said Thomas Emge, also veloped by Hirokazu Kato of Nara Institute Learn More from the Esri Prototype Lab. “It’s often easier of Science and Technology in 1999 and was Raad maintains a very active personal blog at to show someone an AR application first, let released by the University of Washington thunderheadxpler.blogspot.com. Carmichael them experience it, and then start the con- Human Interface Technology Lab (HITLab). and Emge are regular contributors to the versation about what you want to do with it. Since 1999, numerous ARToolkit spinoffs Applications Prototype Lab blog at blog.esri. At first, there’s the cool factor that’s fun to have been created. In 2010, René Schulte com/Dev/blogs/apl/default.aspx. Both blogs do and easily impresses most people. Next, ported the ARToolkit to Microsoft’s contain posts on AR topics. you have to start thinking about design and Silverlight platform, which was released as usability.” the SLARToolkit. Sources for AR information Emge suggests that developers begin by Emge cautions developers to recognize the junaio: www.junaio.com considering the end user’s experience. “There limitations of mobile devices. “You may find Layar: www.layar.com are different ways to apply AR to your app,” that the limited accuracy of the GPS won’t HITLab: www.hitl.washington.edu/artoolkit said Emge. “In one case, you are overlay- allow you to place labels and billboards as Wikipedia: en.wikipedia.org/wiki/ ing additional information in your camera’s precisely as you’d like or that AR calculated Augmented_reality

esri.com Spring 2011 au 29