3D Augmented Reality Mobile Application Prototype for Visual Planning Support
Total Page:16
File Type:pdf, Size:1020Kb
3D Augmented Reality Mobile Application Prototype for Visual Planning Support Arnau Fombuena Valero Master’s of Science Thesis in Geoinformatics TRITA-GIT EX 11-010 School of Architecture and the Built Environment Royal Institute of Technology (KTH) Stockholm, Sweden November 2011 Abstract The aim of this thesis is to implement a prototype of a 3D Augmented Reality mobile application. Using 3D is becoming more and more common for professionals as well as users. A good example of that is Google Earth and its 3D view. Implementing a mobile application that takes advantage of 3D and Augmented Reality may be very useful for planning new constructions in both urban and non-urban areas allowing to visualize how the construction will be in the future and how it will interact with its surrounding environment. There is a great potential for such kind of applications. An example could be the modification of a certain area of a city; allowing the inhabitants of that city to preview the project and, hopefully, avoiding unnecessary conflicts related to that project. In non- urban areas this application is also very useful for helping decision making by visualizing, on site, how the project will be and its impact on the environment. In order to preview a future construction there is the need to have a 3D model. Therefore, a 3D format for that model is necessary. Since COLLADA is a 3D standard interexchange format it is used in this thesis. Together with COLLADA, the computer graphics imagery and gaming technology called OpenGL ES 2.0 is used. Using COLLADA and OpenGL ES 2.0 combined with the properties of the views’ layers, the camera input, the sensors in the mobile device and the positioning technologies permit obtaining successful results displaying a 3D object in an Augmented Reality mobile prototype application. Interface elements are implemented as well in order to bring basic usability tools. The results show the advantages of combining technologies in a mobile device and the problems derived from the low accuracy in positioning in such kind of devices. Thus, ideas for improving the positioning accuracy are discussed as well. Keywords: Augmented Reality, 3D, OpenGL ES, iOS, Planning. 1 Acknowledgments It is a pleasure to thank all the people who made this thesis possible. First, I would like to thank my supervisor Jesús Manuel Palomar Vázquez at Universidad Politécnica de Valencia. He has been very supportive and helpful. I am very grateful for the orientation and support I have received. I also would like to thank my supervisor Gyözö Gidófalvi at KTH who helped me outline the thesis and improve it with his comments. Thank you. Finally, I would like to thank my family for their unconditional love and support. 2 Table of Contents List of figures 1. Introduction 5 2. Related Work and Technology 7 2.1. Mobile Platforms 7 2.2. Augmented Reality 10 2.3. Core Animation 13 2.4. 3D Modeling 14 2.5. COLLADA 16 2.6. COLLADA Basic Structure 18 2.7. 3D Rendering Technologies 21 2.7.1. Possibilities outside of iOS 21 2.7.2. OpenGL ES 21 2.7.3. OpenGL ES 1.1 and OpenGL ES 2.0 25 2.7.4. Shaders in OpenGL ES 2.0 26 2.7.5. Engines based on OpenGL ES 33 2.8 Related Work 34 3. Methodology 35 3.1. Methodology Roadmap 35 3.2. Selected Technologies 37 3.2. Application’s Architecture 41 3.4. Implementation Process 44 3.4.1. Loading the COLLADA document 45 3.4.2. Use of OpenGL ES 2.0 46 3.4.3. Using the camera 50 3.4.4. Positioning and device orientation 51 3.4.5. Positioning implementation 54 3.4.6. 3D rotation 57 3.4.7. Interface 61 3.5. Methodology Summary 62 4. Results and Discussion 64 4.1. Results 64 4.2. Discussion 67 5. Conclusions and Discussion 68 5.1. Conclusion 68 5.2. Future Research: Improvements for Positioning 68 6. References 70 7. List of Software Used 75 3 List of Figures 2.1. Mobile Ecosystem Life Spans 7 2.2. Android TweetDeck Beta Users by OS Version. 8 2.3. Eniro 2D Augmented Reality. 10 2.4. AR Theodolite. 11 2.5. Colorblind AR. 11 2.6. AR Game. 12 2.7. Metro Paris. 13 2.8. COLLADA File Tags. 18 2.9. Library Geometries of a COLLADA File. 19 2.10. Example of a Mesh Using Polylist. 20 2.11. OpenGL ES Architecture. 23 2.12. OpenGL ES Data Types. 23 2.13. Normalized Cube. 24 2.14. OpenGL ES 2.0 Graphics Pipeline. 24 2.15. OpenGL ES 2.0 Vertex Shader. 27 2.16. OpenGL ES 2.0 Fragment Shader. 28 2.17. OpenGL ES View From the Program Point of View. 30 2.18. Triangle Strip and Triangle Fan. 32 3.1. Methodology Roadmap. 35 3.2. Operations to be Performed at Initialization. 41 3.3. Application Architecture. Sensors. 42 3.4. Application Architecture. Hierarchy From Bottom to Top. 42 3.5. OpenGL ES 2.0 Layers on iOS 46 3.6. Load a Texture in OpenGL ES 2.0. 47 3.7. Complex 3D Model Rendered in iOS Using OpenGL ES 2.0 48 3.8. View Structure. 49 3.9. Rotation Axis on iOS. 52 3.10. Definition of Custom Coordinate Class. 53 3.11. Setting the Notification Center for the Device’s Orientation. 54 3.12. Setting the Use of the Accelerometer. 54 3.13. Initializing the Location Manager. 55 3.14. Draft for the Computation of Rotations. 58 3.15. Sketch for Computing Planimetry Rotation. 59 3.16. Sketch for Computing the Altimetry Rotation. 59 3.17. Initialization of a Gesture Recognizer. 61 4.1. Capture of the 3D AR Application. 64 4.2. Capture of the 3D AR Application. Edit Mode. 64 4.3. Capture of the 3D AR Application. Landscape View. 65 4.4. Capture of the 3D AR Application. Edit Mode. 65 4.5. Capture of the Settings View. 66 4 1. INTRODUCTION The aim of this thesis is to implement a prototype of a 3D Augmented Reality mobile application. Augmented Reality (AR) is a growing research field covering from very expensive simulators to smaller applications in mobile devices. The advantages of the latter are the rather low cost and the bigger size of its market. Due to mainly those two reasons, the number of applications using Augmented Reality is increasing fast. However, those applications have something in common independently of their specific purpose: They are in 2D or, at best, using 3D-like views implemented in 2.5D. Details of one of the technologies allowing the easily implementation of 3D-like applications is explained in this thesis. Most of the low cost Augmented Reality applications are running on smartphones. Nowadays, the smartphones are also referred to as hand-held devices even though this term includes a wider range of devices. Smartphones are distributed with one of the multiple mobile operating systems available in the market. From the myriad of these operating systems there are two that are more used: Android and iOS. Both operating systems are very similar and both have advantages and disadvantages. In order to implement an Augmented Reality application using 3D a 3D format and the technology to render a 3D model is required. To solve this issue the COLLADA format has proved to be a very good solution together with OpenGL ES 2.0 as a rendering technology. The fact that the Khronos Group is authoring both COLLADA and OpenGL ES 2.0 guarantees the good synergy between the 3D format and the 3D rendering technology. OpenGL ES 2.0 has proved its success as a 3D rendering technology by being the standard in mobile devices. Furthermore, OpenGL ES 2.0 is based in OpenGL, which is widely used in 3D gaming development and the computer graphics imagery used in animation movies and special effects for both movies and series. However, the complexity of OpenGL ES 2.0 has caused a considerable learning effort and a large amount of time before being possible to understand it and use it correctly. The interface of the prototype application includes a number of features for improving its functionality. More specifically, 3D transformations and gestures have been implemented as an extra in order to give model handling functionalities and visualization tools to the application that, being a prototype, also includes the basic tools that a market-ready application should include such as the application settings. 5 The remainder of this thesis is organized as follows. After this introductory Section 1, Section 2 presents the related work and technologies used in this thesis including mobile platforms, augmented reality, 2.5D graphics in iOS, the description of existing 3D modeling software, the COLLADA 3D format and 3D rendering technologies for mobile devices focusing in OpenGL ES 2.0. Section 3 explains why the technologies used in this thesis are chosen. The application’s architecture and its implementation process are also described in this section. Section 4 shows the results of the application using and describing screen captures obtained from the prototype application where a geo- located 3D model is rendered on screen at a certain location. Finally, Section 5 presents the conclusions and discussion of this thesis as well as the advantages and disadvantages of the methodology used in this thesis. The discussion includes possible improvements for positioning considering the type of device used in this thesis. 6 2. RELATED WORK AND TECHNOLOGY 2.1. Mobile Platforms Nowadays, there are several operating systems supporting mobile devices as it is shown in Figure 2.1.