An Evaluation of Zoom and Microsoft Teams Video Conferencing Software with Network Packet Loss and Latency An Interactive Qualifying Project Report Submitted to the Faculty of WORCESTER POLYTECHNIC INSTITUTE In partial fulfillment of the requirements for the Degree of Bachelor of Science Eric Baccei, Alexander Clopper, Taylan Sel Submitted to: Professor Mark Claypool Worcester Polytechnic Institute 1 Abstract Video conferencing is an increasingly important form of communication, but there are few studies on how network quality affects new conferencing services. We studied Zoom and Microsoft Teams, video conferencing tools targeted towards businesses. Our user study had users rate their experiences during simulated normal conversation with network latencies and packet drop. Our study found that the network conditions did not significantly change user ratings, but Zoom had higher ratings than Microsoft Teams overall. Analysis of our benchmarking results showed that although network traffic for both services varied with network conditions, Zoom had a more constant bitrate than Teams which aligned with the responses received in the user study. 2 Table of Contents Abstract 1 Table of Figures 4 1- Introduction 6 2 - Background and Related Work 9 2.1 - Video Compression Technology 9 2.1.1 - MPEG 9 2.1.2 - H.261 / H.263 10 2.1.3 - Comparison 12 2.2 - Digital Communication Architectures 13 2.2.1 - Peer to Peer Connection 13 2.2.1 - Client-Server Connection 14 2.3 - Network Protocols 15 2.3.1 - TCP 15 2.3.2 - UDP 16 2.3.3 - MQTT 17 2.4 - Tools 18 2.4.1 - Wireshark 18 2.4.2 - Clumsy 19 3 - Methodology 20 3.1 - Video Conferencing Services 20 3.1.1 - Zoom 20 3.1.2 - Microsoft Teams 21 3.2 - Digital Tools 21 3.3 - Parameters 21 3.3.1 - Latency 22 3.3.2 - Packets 22 3.4 - Laptop 23 3.5 - Experimental Design 23 3.5.1 - User Study 24 3.5.2 - Benchmarking 25 4 - Analysis 27 4.1 - Benchmark Results 27 4.1.1 - Zoom 27 4.1.2 - Microsoft Teams 29 3 4.1.3 - Zoom/Teams Comparison 31 4.2 - User Study Results 34 4.2.1 - Demographics 34 4.2.2 - Zoom 35 4.2.3 - Microsoft Teams 37 4.2.4 - Zoom Teams Comparison 39 4.3 - Overall Results 41 5 - Conclusion 43 6 - Future Work 45 References 46 4 Table of Figures Figure 1: MPEG Frame display order and prediction patterns. --------------------------- Page 10 Figure 2a: Visualization of macroblocks within a GOB for H.261. ------------------------ Page 11 Figure 2b: Visualization of blocks within a macroblock for H.261. ------------------------ Page 11 Figure 3: Visual Description of PB frames in the H.263 compression technology. ----- Page 12 Figure 4: Comparison of MPEG and H.26X series ------------------------------------------ Page 13 Figure 5: Example of peer to peer based communication systems ------------------------ Page 14 Figure 6: Example of server based communication system. -------------------------------- Page 15 Figure 7: Shows the difference between how TCP and UDP sends packets and responds to packet loss. --------------------------------------------------------------- Page 16 Figure 8: Shows how messages are sent and received using MQTT. ---------------------- Page 18 Figure 9: Packet disassembling and reassembling.-------------------------------------------- Page 22 Figure 10: Laptop Specifications ----------------------------------------------------------------- Page 23 Figure 11: The questions participants filled out after each “round.”------------------------- Page 25 Figure 12: List of latency and packet drop levels. ---------------------------------------------- Page 25 Figure 13: Graph of bytes per second based on packet drop chance for Zoom. ------------ Page 28 Figure 14: Graph of bytes per second based on latency levels for Zoom. ------------------- Page 29 Figure 15: Graph of bytes per second based on packet drop chance for Microsoft Teams. ----------------------------------------------------------------------------------- Page 30 Figure 16: Graph of bytes per second based on latency levels for Microsoft Teams. ----- Page 31 Figure 17: Comparison between Zoom and Teams bytes per second with no latency. ---- Page 32 Figure 18: Comparison between Zoom and Teams bytes per second with 200 ms latency. ---------------------------------------------------------------------------------- Page 32 Figure 19: Comparison between Zoom and Teams bytes per second with 0% packet drop chance. ---------------------------------------------------------------------------- Page 33 Figure 20: Comparison between Zoom and Teams bytes per second with 2% packet drop chance. ---------------------------------------------------------------------------- Page 33 5 Figure 21: Table of reported demographic information for participants. -------------------- Page 34 Figure 22: Table of participant responses to background questionnaire. 5 represents more experience/better experience, where 1 represents less experience/worse experience. ------------------------------------------------------------------------------ Page 34 Figure 23: Table of majors for the participants. Page 35 ------------------------------------------------- Figure 24: Graph of average user ratings based on latency for Zoom. ---------------------- Page 36 Figure 25: Graph of average user rating based on packet drop chance for Zoom. --------- Page 37 Figure 26: Graph of average user rating based on latency for Microsoft Teams. ---------- Page 38 Figure 27: Graph of average user rating based on packet drop chance for Microsoft Teams. ----------------------------------------------------------------------------------- Page 39 Figure 28: Graph of comparison between Zoom and Teams user ratings with no latency. Page 40 Figure 29: Graph of comparison between Zoom and Teams user ratings with 200 ms latency. ---------------------------------------------------------------------------------- Page 40 Figure 30: Graph of comparison between Zoom and Teams user ratings with 0% packet drop chance. ---------------------------------------------------------------------------- Page 41 Figure 31: Graph of comparison between Zoom and Teams user ratings with 2% packet drop chance. ---------------------------------------------------------------------------- Page 41 6 1 - Introduction With the first experiments with video conferencing in the 1920s, it was considered a [1] futuristic idea. ​ Video conferencing was in many science fiction forms of media, but seemed far ​ from reality for the consumer. This all changed in the computer revolution of the 1980s with the increases in data communication infrastructure and the advent of video codecs. Additionally, the growth of mobile phones and webcams began to spur the growth of the video call and conference. One of the first widely available cameras, QuickCam, was even named as one of the most important technological advancements of the early 2000s. Although multiple versions of video calling services emerged over the years, Skype seemed to gain the most ground due to its free, cross platform service. Today, there are dozens of services for video calling, voice over IP communication, and video conferencing. As a business, video conferencing has quickly grown. In 2018, it was estimated that revenue related to video conferencing was over 7.8 billion dollars (US), with some 32.8 million [2] video conferencing devices. ​ Video conferencing is expected to grow up to 13.82 billion dollars ​ by 2023. There are now multiple competitors for all markets. Both Microsoft Teams and Zoom are services for the business sector. These have the capacity to host a large number of users at once in the same room, the ability to share screens, and create leaders; all useful tools for [3] running digital meetings. ​ However, the technology is being used for more than just meetings. ​ Hospitals can use video calling and conferencing to interact with patients, even to help facilitate [4] new ways to treat mental health. ​ Educators are also making full use of video calling. Even ​ 7 governments use these services now, with customers such as the Department of Homeland Security.[5] ​ Zoom is a video conferencing service designed for a range of businesses, with the ability to conference with up to 1000 people at once, and up to 49 people can display their screens at [3] once. ​ Zoom also allows for versatility via connectivity options, including a mobile app, ​ downloadable client and web client. Zoom has also expanded its features to include administrative options and a chat feature which make it ideal for business functions. Zoom has expanded its base to nearly 13 million active users as of February, 2020.[6] ​ Microsoft Teams is also designed for businesses, although more as an all in one tool rather than just for video conferencing. Teams offers a variety of tools, not all related to calling. The Teams platform can do file storage, file collaboration, and has a calendar application with reminders. Microsoft Teams also has many educational uses, such as being used to assign [7] assignments and quizzes, and allows for text, video and audio from student to teacher. ​ Teams’ ​ audience has grown to 20 million active users as of November, 2019.[8] ​ Most studies on video conferencing are older studies with older technologies. For example, Skype has had extensive research done on it, both as voice over IP service and as a [9][10] [9] video calling service. ​ One such study was conducted by Xinggong Zhang and Yang Xu, ​ ​ studied the effects of packets and bit rates on Skype, with user input for the quality of video. Batu Sat and Benjamin Wah compared Google Talk