THINC: a Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices
Total Page:16
File Type:pdf, Size:1020Kb
THINC: A Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices Ricardo A. Baratto Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2011 c 2011 Ricardo A. Baratto This work may be used in accordance with Creative Commons, Attribution-NonCommercial-NoDerivs License. For more information about that license, see http://creativecommons.org/licenses/by-nc-nd/3.0/. For other uses, please contact the author. ABSTRACT THINC: A Virtual and Remote Display Architecture for Desktop Computing and Mobile Devices Ricardo A. Baratto THINC is a new virtual and remote display architecture for desktop computing. It has been designed to address the limitations and performance shortcomings of existing remote display technology, and to provide a building block around which novel desktop architectures can be built. THINC is architected around the notion of a virtual display device driver, a software-only component that behaves like a traditional device driver, but instead of managing specific hardware, enables desktop input and output to be intercepted, manipulated, and redirected at will. On top of this architecture, THINC introduces a simple, low-level, device-independent representation of display changes, and a number of novel optimizations and techniques to perform efficient interception and redirection of display output. This dissertation presents the design and implementation of THINC. It also intro- duces a number of novel systems which build upon THINC's architecture to provide new and improved desktop computing services. The contributions of this dissertation are as follows: • A high performance remote display system for LAN and WAN environments. This system differs from existing remote display technologies in that it focuses on the architecture of the system as a mechanism to improve performance, and not just on the remote display protocol and compression techniques. • A novel mechanism to natively support multimedia content in a remote display system in a way that is both transparent to applications and format indepen- dent. • pTHINC, a system to deliver improved remote display support for mobile de- vices, both in terms of performance and usability, and provide a competitive, and in some cases superior, alternative to native mobile applications. • MobiDesk, a desktop utility computing infrastructure that enables service provi- ders to host desktop sessions in fully virtualized environments. Hosted sessions can be remotely accessed using THINC, they can be migrated across computers to provide high-availability, and can be effectively and efficiently protected from denial of service attacks. • Moving beyond remote display, we show how THINC's architecture can be used to provide continuous, low overhead recording of a desktop. Alongside, we introduce a novel way to leverage desktop accessibility services to allow users to search their recording based on captured text content. We have implemented prototypes for these systems, and evaluated their perfor- mance in a number of scenarios, and compared it to representative alternatives when- ever possible. Our results demonstrate that THINC can provide superior remote display performance, and can be successfully used as a fundamental building block for new and improved desktop applications and services. Contents Contentsi List of Figures vi List of Tablesx List of Algorithms xi Acknowledgments xii 1 Introduction1 1.1 Contributions . 11 1.2 Dissertation Roadmap . 12 2 THINC Architecture 13 2.1 Remote Display Design . 13 2.2 Display Virtualization . 18 2.3 Remote Display Protocol . 25 2.4 Display Update Translation . 29 2.4.1 Offscreen Drawing . 33 2.5 Display Update Delivery . 35 i 2.6 Implementation . 39 2.6.1 Back End . 40 2.6.1.1 Creating Commands . 40 2.6.1.2 Adding and Manipulating Commands . 41 2.6.1.3 Abstracting Command Destinations . 47 2.6.1.4 Delivering Commands . 49 2.6.2 Front End . 51 2.6.3 Remote Display Implementation . 54 2.7 Experimental Results . 56 2.7.1 Web Browsing Benchmark . 60 2.7.2 Results . 61 2.8 Summary . 66 3 Multimedia 68 3.1 Video Support . 70 3.2 Audio Support . 73 3.3 Media Synchronization . 77 3.4 Implementation Details . 81 3.5 Experimental Results . 82 3.5.1 Experimental Setup and Benchmarks . 82 3.5.2 Results . 85 3.6 Summary . 95 4 Mobile Devices 96 4.1 pTHINC Usage Model . 99 ii 4.2 pTHINC System Architecture . 104 4.2.1 Display Management . 106 4.2.2 Video Playback . 107 4.3 Experimental Results . 109 4.3.1 Experimental Testbed . 110 4.3.2 Application Benchmarks . 112 4.3.3 Qualitative Results . 114 4.3.4 Quantitative Results . 117 4.4 Summary . 126 5 Desktop Virtualization 128 5.1 MobiDesk: Mobile Virtual Desktop Computing . 130 5.1.1 Display Virtualization . 135 5.1.2 Operating System Virtualization . 138 5.1.3 Network Virtualization . 142 5.2 A2M: Access-Assured Mobile Desktop Computing . 146 5.2.1 System Operation . 150 5.3 Experimental Results . 151 5.3.1 MobiDesk Virtualization Overhead . 153 5.3.2 MobiDesk Application Performance . 159 5.3.3 A2M Performance Evaluation . 164 5.3.3.1 Overall Performance . 166 5.3.3.2 Interactive Applications . 172 5.3.3.3 Wireless . 174 iii 5.4 Summary . 175 6 Display Recording and Text Capture 178 6.1 Display Recording . 179 6.2 Text Capture . 182 6.3 Playback . 185 6.4 Search . 186 6.5 DejaView . 188 6.6 Experimental Results . 190 6.7 Summary . 198 7 Related Work 199 7.1 Remote Display and Thin-Client Computing . 199 7.2 Multimedia Support . 206 7.3 Support for Mobile Devices . 209 7.4 Display Recording and Text Capture . 211 8 Conclusions and Future Work 213 8.1 Future Work . 215 Bibliography 220 A THINC Protocol specification 237 A.1 Packet Format . 237 A.1.1 Message Type and Flags . 238 A.1.2 Unused Field (mbz) . 238 A.2 Handshake Protocol . 239 iv A.2.1 Version verification . 239 A.2.2 Security Handshake . 239 A.2.3 Parameter Negotiation . 241 A.2.3.1 Packet Format of Server Replies . 242 A.2.4 Summary of Handshake Messages . 244 A.3 Remote Display Protocol . 245 A.3.1 Server Messages . 246 A.3.2 Client Messages . 254 v List of Figures 1.1 Remote Display Architecture . .2 1.2 Audio/Video playback performance of popular remote display systems6 2.1 Standard display architecture . 16 2.2 Standard display pipeline . 19 2.3 THINC virtual display architecture . 22 2.4 THINC architecture components . 23 2.5 Experimental Testbed . 57 2.6 Web Benchmark: Average Page Latency . 62 2.7 Web Benchmark: Average Data Transferred per Web Page . 62 2.8 Web Benchmark: THINC Average Page Latency Using Remote Sites 64 3.1 Audio Playback . 75 3.2 Audio Capture . 76 3.3 Experimental Testbed for Audio Capture/Playback Benchmark . 84 3.4 A/V Benchmark: A/V Quality . 86 3.5 A/V Benchmark: Total Data Transferred . 87 3.6 A/V Benchmark: THINC A/V Quality Using Remote Sites . 88 vi 3.7 Timestamp Deltas: MPEG-1 352x240 . 89 3.8 Timestamp Deltas: MPEG-1 480x260 . 90 3.9 Timestamp Deltas: QuickTime 480x360 . 90 3.10 Distribution of Timestamp Deltas: MPEG-1 352x240 . 91 3.11 Distribution of Timestamp Deltas : MPEG-1 480x260 . 91 3.12 Distribution of Timestamp Deltas: QuickTime 480x360 . 92 3.13 Mouth-to-ear latency overhead for VoIP applications . 93 4.1 pTHINC shortcut keys . 102 4.2 PDA Experimental Testbed . 110 4.3 pTHINC Web Screenshot: BBC News . 115 4.4 Native IE Screenshot: BBC News . 115 4.5 pTHINC Application Screenshot: Quicken . 115 4.6 Native Application Screenshot: Pocket Quicken . 115 4.7 PDA Browsing Benchmark: Average Page Latency . 118 4.8 PDA Browsing Benchmark: Average Page Data Transferred . 119 4.9 PDA Browser Screenshot: RDP 640x480 . 121 4.10 PDA Browser Screenshot: VNC 1024x768 . 121 4.11 PDA Browser Screenshot: ICA Resized 1024x768 . 122 4.12 PDA Browser Screenshot: pTHINC Resized 1024x768 . 122 4.13 PDA Video Benchmark: Fullscreen Video Quality . 123 4.14 PDA Video Benchmark: Fullscreen Video Data . 124 4.15 PDA Video Screenshot: RDP 640x480 . 126 4.16 PDA Video Screenshot: VNC 1024x768 . ..