Interactive Visualization Methods for Mobile Device Applications Von der Fakultat¨ Informatik, Elektrotechnik und Informationstechnik der Universitat¨ Stuttgart zur Erlangung der Wurde¨ eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigte Abhandlung Vorgelegt von Joachim Diepstraten aus Leonberg Hauptberichter: Prof. Dr. T. Ertl Mitberichter: Prof. Dr. T. Strothotte Tag der mundlichen¨ Prufung:¨ 24.1.2006 Institut fur¨ Visualisierung und Interaktive Systeme der Universitat¨ Stuttgart 2005 ii iii To the two special persons I have met in my life iv Contents Lists ix List of Tables . ix List of Figures . xi List of Color Figures . xv Abstract xvii Zusammenfassung xviii 1 Introduction 1 2 3D Graphic APIs for Mobile Devices 5 2.1 Graphics Hardware . 5 2.1.1 Geometry Transfer and Primitives . 6 2.1.2 Vertex Processing . 7 2.1.3 Fragment Processing . 7 2.1.4 Frame buffer Operations . 7 2.2 The OpenGL ES API . 8 j 2.3 The Direct3D Mobile API . 9 2.4 Performance Analysis . 10 2.4.1 Software Rendering Library: Gerbera . 11 2.4.2 Software Rendering Library: Vincent . 13 2.4.3 Software Rendering Library: Klimt . 14 2.4.4 Conclusion . 15 vi CONTENTS 3 Remote Rendering Methods 17 3.1 Classification . 17 3.1.1 Render Remote . 17 3.1.2 Render Local . 18 3.1.3 Shared Visualization or Immediate-mode Rendering . 18 3.2 Different Render Remote Solutions . 18 3.3 An Image-based Remote Rendering Solution for Mobile Devices . 21 3.3.1 Dynamic Linking . 22 3.3.2 Library Architecture . 22 3.3.3 Evaluation of the previous Solution . 23 3.4 Improvements and Accelerations . 24 3.4.1 Data Rate Reduction . 25 3.4.2 Software Compression . 25 3.4.3 Using Different Rendering Styles to Increase Compression Ratios . 28 3.4.4 Motion Handling . 30 3.4.5 Latency Reduction . 30 3.4.6 Hardware Compression/Decompression . 31 3.5 An Alternative Remote Rendering Method . 35 3.5.1 Basic Concept . 36 3.5.2 Implementation . 38 3.5.3 Results . 41 3.5.4 Open Issues . 44 4 Rendering Transparent Surfaces 45 4.1 Transparency in Computer Graphics: General . 45 4.2 Transparency in Technical Illustrations . 46 4.2.1 Direct Transparency Methods for Line Drawings . 48 4.2.2 Direct Transparency Methods for Color Illustrations . 50 4.2.3 Indirect Method: Explosion Illustrations . 61 4.2.4 Indirect method: Cutaway illustrations . 62 4.2.5 Hybrid Method: Ghosting illustrations . 76 4.3 An Extension to Graphics Hardware . 78 CONTENTS vii 4.3.1 Architecture . 78 4.3.2 A Simulator for Vragments . 80 4.3.3 Benefits of Vragments . 81 4.3.4 Open Issues of Vragments . 85 4.3.5 Conclusion . 86 4.4 Transparency and Reflections in Cel Animation . 86 4.4.1 Analysis of Reflection Types . 88 4.4.2 Analysis of Transparency Types . 90 4.4.3 Implementation . 91 4.4.4 Results . 97 4.5 Improving the Rendering of Transparent Surfaces in Computer Graphics . 98 4.5.1 Transparency Models . 99 4.5.2 Proposal of a New Model . 103 4.5.3 Comparison of Models . 104 4.5.4 Implementation . 106 5 Three-dimensional Graphical User Interface for Mobile Devices 109 5.1 Why Three-dimensional Graphical User Interfaces? . 109 5.1.1 2D versus 3D on Desktop Computers . 110 5.1.2 2D versus 3D on Mobile Devices . 115 5.2 Design of a Platform Independent 3D GUI toolkit . 116 5.2.1 General . 116 5.2.2 Camera Control . 117 5.2.3 Occlusion of Windows . 117 5.2.4 Lighting . 119 5.2.5 Antialiasing . 119 5.2.6 Text Output . 120 5.2.7 Text Input . 120 5.2.8 Event Handling . 121 5.2.9 Geometry Management . 122 5.3 Results . 124 5.4 Open Issues . 127 viii Table of contents 6 Summary 129 6.1 Conclusion . 129 6.2 Summary . 131 6.3 Outlook . 133 A Color plates 135 B Bibliography 145 Lists List of Tables 2.1 Functions and features removed from OpenGL ES in the different rendering j pipeline stages compared to OpenGL. 11 2.2 Rendering performance of Gerbera for the different scenarios. 12 2.3 Rendering performance of Vincent for the different scenarios. 14 2.4 Rendering performance of Klimt for the different scenarios. 14 3.1 Frame rates for different scenarios. Left: 802.11b WLAN, right: Fast Ethernet. 27 3.2 Comparison of different image data size when using different rendering styles and lossless compression. 30 3.3 Comparison of the efficiency between Method I and Method II for getting the 2D line data. 41 3.4 Frame rates achieved on different mobile clients with different screen resolutions. 42 3.5 Amount of time spent on the server for each task of the two different test scenarios. 43 4.1 Performance measurements of the OpenGL implementation . 60 4.2 Performance measurements of the new Direct3D9 pixel shader 2 implementation of Method II. 61 4.3 Performance measurements for the Lancia engine model in frames per second. The tests were carried out with the OpenGL Geforce3 specific implementation. 75 4.4 Performance measurements in frames per second. The tests were carried out with the Direct3D9 pixel shader 2.0 implementation. 75 4.5 Performance measurements for two different scenarios. 98 5.1 Policy and dependency of different size parameters between parent, current, and child widget. 124 x List of Tables List of Figures 2.1 Current GPU programming model. 6 2.2 A schematic overview of the OpenGL ES pipeline. 9 j 2.3 Overview of the different OpenGL ES profile concerning footprint, performance, j and power consumption. 10 2.4 Overview of Vincent's runtime code generation method. 13 3.1 Remote visualization using the standard X display forwarding mechanism. 19 3.2 Overview of the remote rendering architecture as described by Engel et al. [ESE00]. 20 3.3 Overview of the SGI Vizserver remote rendering architecture. 20 3.4 Overview of the GLX architecture. 21 3.5 System architecture as described by Stegmaier et al. [SME02]. (1) The applica- tion issues a GLX request which is sent to the render server. (2) The application issues OpenGL calls, which are handled by the render device. (3) The library reads the contents of the frame buffer and (4) sends it to the interaction server using XPutImage request. (5) XEvents are sent from the interaction server to the application. 23 3.6 System architecture from Stegmaier et al. [SME02] used together with VNC. 23 3.7 Revised two-component architecture of the generic remote visualization system with custom image compression. 25 3.8 Visual quality comparison between a jpeg compressed rendered image (right) and the same image uncompressed (left). 26 3.9 Visual quality comparison between different compression techniques. (a) loss- less lzo, (b) block-based CCC, (c) transformation-based BTPC. 27 3.10 Example scenario taken to compare different data sizes. (a) per-pixel Blinn & Phong rendering (b) cool & warm shading (c) cel shading. 29 3.11 Visual explanation of the CCC algorithm. Upper part shows how the bit mask is generated. Lower part shows the encoding of the two colors for each 4x4 pixel block. ..