Computer Engineering 2013 Mekelweg 4, 2628 CD Delft The Netherlands http://ce.et.tudelft.nl/ MSc THESIS Measurement of Video Conferencing Libraries using a Test Framework based on Profiling Maulik Prabhudesai Abstract Video Conferencing systems have become highly popular with the explosion of bandwidth and computing power. This has made high quality video conference possible on embedded hand held devices. This has created an ecosystem of companies developing applications which are powered by video conferencing libraries and a parallel ecosystem of companies who make those video conferencing libraries. An application developer has to design his system in a way that the video conferencing library (which is the backbone of the applica- tion) is efficiently integrated and is performing optimally. In order to do that, the developer must be acutely aware of features such CE-MS-2013-18 as platform compatibility, resource usage, performance boundaries and bottlenecks of the library in order to make an informed deci- sion. Hence the developer needs a test-bench which can evaluate the above mentioned features. This thesis mainly discusses the need to arrive at one such test framework followed by the framework itself. The framework designed in this thesis consists of a network emulator which is combined with a popular network protocol analyser. It pro- files the library and monitors variables essential to Quality of Service (QoS) like bandwidth usage, frame rates, frame/packet sizes and au- dio/video delay under varying network conditions. By observing the change in these variables, the QoS offered by the library can be determined. This information can also be extrapolated to understand the challenges in guaranteeing a minimum quality of experience (QoE). This thesis applies the framework on one chosen library (Library 1) and Google Hangouts and discusses the results. The performance of the library under varying network conditions (bandwidth, packet loss and latency) is observed, enabling Presence Displays BV to extract information about how the library works. Presence Displays BV is building a video conferencing enabled product and uses the framework discussed in this thesis to analyse video conferencing libraries in order to build the application around it. It has been observed that Library 1 favours video quality above other parameters and only compromises video quality to accommodate more users participating in a call. Google Hangouts favour audio, motion content in the video, interactivity and robustness over video quality. This information can be used to configure Library 1 in a way such that it delivers maximum QoE in the available resources. Faculty of Electrical Engineering, Mathematics and Computer Science Measurement of Video Conferencing Libraries using a Test Framework based on Profiling Requirement Specific Selection and Evaluation of Video Conferencing Libraries THESIS submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE in EMBEDDED SYSTEMS by Maulik Prabhudesai born in Mumbai, India Computer Engineering Department of Electrical Engineering Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology Measurement of Video Conferencing Libraries using a Test Framework based on Profiling by Maulik Prabhudesai Abstract ideo Conferencing systems have become highly popular with the explosion of bandwidth and computing power. This has made high quality video conference possible on embedded V hand held devices. This has created an ecosystem of companies developing applications which are powered by video conferencing libraries and a parallel ecosystem of companies who make those video conferencing libraries. An application developer has to design his system in a way that the video conferencing library (which is the backbone of the application) is efficiently integrated and is performing optimally. In order to do that, the developer must be acutely aware of features such as platform compatibility, resource usage, performance boundaries and bottlenecks of the library in order to make an informed decision. Hence the developer needs a test-bench which can evaluate the above mentioned features. This thesis mainly discusses the need to arrive at one such test framework followed by the framework itself. The framework designed in this thesis consists of a network emulator which is combined with a popular network protocol analyser. It profiles the library and monitors variables essential to Quality of Service (QoS) like bandwidth usage, frame rates, frame/packet sizes and audio/video delay under varying network conditions. By observing the change in these variables, the QoS offered by the library can be determined. This information can also be extrapolated to understand the challenges in guaranteeing a minimum quality of experience (QoE). This thesis applies the framework on one chosen library (Library 1) and Google Hangouts and discusses the results. The performance of the library under varying network conditions (bandwidth, packet loss and latency) is observed, enabling Presence Displays BV to extract in- formation about how the library works. Presence Displays BV is building a video conferencing enabled product and uses the framework discussed in this thesis to analyse video conferencing libraries in order to build the application around it. It has been observed that Library 1 favours video quality above other parameters and only compromises video quality to accommodate more users participating in a call. Google Hangouts favour audio, motion content in the video, inter- activity and robustness over video quality. This information can be used to configure Library 1 in a way such that it delivers maximum QoE in the available resources. Laboratory : Computer Engineering Codenumber : CE-MS-2013-18 Committee Members : Advisor: Koen Bertels, CE, TU Delft Advisor: Robbert Smit, Presence Displays BV Chairperson: Koen Bertels, CE, TU Delft i Member: Arjan van Genderen, Assistant Professor CE, TU Delft Member: Johan Pouwelse, Assistant Professor PDS, TU Delft ii This thesis is dedicated to the engineering and scientific community with the hope that after reading this, they will recognise me as one of their own. iii iv Contents List of Figures x List of Tables xi Acknowledgements xiii 1 Introduction 1 1.1 Background . .1 1.1.1 Problem Statement . .1 1.2 Thesis goals . .2 1.3 Thesis Contributions . .2 1.4 Thesis organisation . .3 2 Video Conferencing Technology and Considerations 5 2.1 Introduction . .5 2.2 Video Conferencing Technology . .6 2.3 Video Conferencing Stack . .8 2.3.1 User Interface Layer . .8 2.3.2 Media . .8 2.3.3 Network Stack . 11 2.3.4 Session Management in Video Conferencing . 17 2.3.5 A case study: WebRTC . 20 2.4 Topology . 21 2.4.1 Client-Server . 22 2.4.2 Peer-to-Peer . 23 2.4.3 Peer-to-Peer with Peer Contribution towards video distribution . 24 2.5 Quality of Service . 27 2.5.1 Challenges to the QoS . 29 3 Problem Description, System Requirements and System Design 31 3.1 Concept . 31 3.2 User Requirements . 31 3.2.1 Login Scenario . 32 3.2.2 Group formation scenarios . 32 3.2.3 User shuts down/disconnects without warning Scenario . 33 3.3 System Requirements . 33 3.3.1 Software System Requirements . 33 3.3.2 Video Conferencing Library Requirements . 34 3.3.3 Hardware Requirements . 35 3.4 System Design . 36 v 3.4.1 Topologies . 36 3.4.2 System Architecture . 38 3.4.3 Server's function in QoE management . 42 3.4.4 Hardware Device . 43 3.5 Conclusion . 44 4 Overview of Video Conferencing Libraries 45 4.1 Introduction . 45 4.2 Criteria for Evaluation . 45 4.2.1 Quality of Experience . 46 4.2.2 Maturity . 46 4.2.3 Robustness . 46 4.2.4 Technical Support . 46 4.2.5 Android Compatibility . 46 4.2.6 Multi-User Video Conferencing . 47 4.2.7 Availability of Source Code/API . 47 4.2.8 Price . 47 4.2.9 Intellectual Property . 47 4.2.10 Support for other functionalities . 48 4.2.11 Summary . 48 4.3 Overview of Video Conferencing Libraries/Applications . 49 4.4 Chosen Video Libraries for Implementation . 52 4.5 Criteria for Quality of Experience . 53 4.5.1 Metrics affecting QoE . 55 5 Framework 59 5.1 Introduction . 59 5.2 Controlling Network Conditions . 59 5.2.1 Network Emulation . 62 5.3 Framework Structure and Capabilities . 62 5.3.1 Audio Video Delay . 62 5.3.2 Audio Video Synchronisation . 64 5.3.3 Identification of the Server and subsequent analysis . 64 5.3.4 Packet Size, Frame Size and Frame Rate . 65 5.3.5 Video Quality . 66 5.3.6 Bandwidth Usage of a Call . 67 5.4 Summary . 68 6 Experiments and Analysis 69 6.1 Pre-experimentation modification . 69 6.2 Configuration Experiment . 70 6.2.1 Details of each configuration . 70 6.2.2 Results . 71 6.3 Effect of Packet Loss and Packet Corruption . 78 6.3.1 Effect of Packet Loss on Video with Motion . 83 vi 6.3.2 Effect of Packet Corruption . 84 6.4 Effect of Network Latency . 86 6.5 Effect of Bandwidth Variation . 88 6.5.1 Full Bandwidth . 88 6.5.2 Performance in Low Bandwidth Conditions . 92 6.5.3 Performance in Medium Bandwidth Conditions . 93 6.6 Pre-Connection Results . 97 7 Conclusions and Evaluations 101 7.1 Congestion - The main challenge to QoS . 101 7.1.1 How does each Library react to Packet Loss/Corruption ? . 102 7.1.2 How does each Library react to Network Latency ? . 102 7.1.3 How does each Library react to Bandwidth Constraints ? . 103 7.2 Effect of Heterogeneity in Hardware . 104 7.3 Effect of Varying Call Sizes . 104 7.4 Summary . 104 7.5 Evaluations and Future Recommendations . 105 Bibliography 111 vii viii List of Figures 2.1 A typical video conferencing system architecture based on WebRTC [10]7 2.2 Real-Time Transport Protocol Stack .