Architectures and Algorithms for Real-Time Web-Based Collaboration Cristian Gadea A thesis submitted in partial fulfillment of the requirements for the Doctorate in Philosophy degree in Electrical and Computer Engineering School of Electrical Engineering and Computer Science Faculty of Engineering University of Ottawa © Cristian Gadea, Ottawa, Canada, 2021 \Great things are done by a series of small things brought together." Vincent Van Gogh ii Abstract Originating in the theory of distributed computing, the optimistic consistency control method known as Operational Transformation (OT) has been studied by researchers since the late 1980s. Algorithms were devised for managing the concur- rent nature of user actions and for maintaining the consistency of replicated data as changes are introduced by multiple geographically-distributed users in real-time. Web-Based Collaborative Platforms are now essential components of modern orga- nizations, with real-time protocols and standards such as WebSocket enabling the development of online collaboration tools to facilitate information sharing, content creation, document management, audio and video streaming, and communication among team members. Products such as Google Docs have shown that centralized web-based co-editing is now possible in a reliable way, with benefits in user produc- tivity and efficiency. However, as the demand for effective real-time collaboration between team members continues to increase, web applications require new syn- chronization algorithms and architectures to resolve the editing conflicts that may appear when multiple individuals are modifying the same data at the same time. In addition, collaborative applications need to be supported by scalable distributed backend services, as can be achieved with \serverless" technologies. While much existing research has addressed problems of optimistic consistency maintenance, previous approaches have not focused on capturing the dynamic client-server inter- actions of OT systems by modeling them as real-time systems using Finite State Machine (FSM) theory. This thesis includes an exploration of how the principles of control theory and hierarchical FSMs can be applied to model the distributed system behavior when processing and transforming HTML DOM changes initiated by multiple concurrent users. The FSM-based OT implementation is simulated, including with random inputs, and the approach is shown to be invaluable for or- ganizing the algorithms required for synchronizing complex data structures. The real-time feedback control mechanism is used to develop a Web-Based Collabo- rative Platform based on a new OT integration algorithm and architecture that brings \Virtual DOM" concepts together with state-of-the-art OT principles to enable the next generation of collaborative web-based experiences, as shown with implementations of a rich-text editor and a 3D virtual environment. iii Acknowledgements This thesis was prepared at the Network Communications and Control Technolo- gies (NCCT) Laboratory of the University of Ottawa. I am grateful for the open- ness and guidance of my supervisor, Dr. Dan Ionescu, who always encouraged me to tackle new challenges and never hesitated to offer help. His input and experience was invaluable throughout the realization of this work. This thesis would not have been possible without the generous assistance of other members of the NCCT Lab. Their depth of knowledge and devotion to techni- cal research created a positive working atmosphere that motivated me to learn about many new technologies. Bogdan Ionescu, Bogdan Solomon, Mircea Trifan, Laurentiu Checiu, Juliano Garcia and Daniel Hong all contributed to the projects mentioned in this thesis. Finally, I would like to thank my parents and my brother for their continuous love, support and encouragement throughout this entire journey. iv Publications Content from this thesis has previously appeared in the following publications: Refereed Journal and Conference Papers 1. C. Gadea, B. Ionescu, and D. Ionescu, \A Finite State Machine Approach to Collaborative Computing," in Transactions on Parallel and Distributed Systems. IEEE Computer Society, 2021, submitted. 2. , \A Control Loop-based Algorithm for Operational Transformation," in SACI 2020: Proc. of 14th Int. Symp. on Applied Computational Intelli- gence and Informatics. IEEE Computer Society, 2020, pp. 247-254. 3. , \New Algorithms and Methods for Collaborative Co-Editing Using HTML DOM Synchronization," in CIC 2018: Proc. of 4th IEEE Int. Conf. on Collaboration and Internet Computing. IEEE Computer Society, 2018, pp. 217-226. 4. , \A Hybrid FSM Rule-Based Approach for the Real-Time Control of Web-Based Collaborative Platforms," in INES 2018: Proc. of 22nd Int. Conf. on Intelligent Engineering Systems. IEEE Computer Society, 2018, pp. 27-32. 5. , \Modeling and Simulation of an Operational Transformation Al- gorithm using Finite State Machines," in SACI 2018: Proc. of 12th Int. Symp. on Applied Computational Intelligence and Informatics. IEEE Com- puter Society, 2018, pp. 119-124. 6. C. Gadea, D. Hong, D. Ionescu, and B. Ionescu, \An Architecture for Web- based Collaborative 3D Virtual Spaces using DOM Synchronization," in v vi CIVEMSA 2016: Proc. of Int. Conf. on Computational Intelligence and Virtual Environments for Measurement Systems and Applications. IEEE Computer Society, 2016, pp. 16. 7. C. Gadea, M. Trifan, D. Ionescu, M. Cordea, and B. Ionescu, \A Microser- vices Architecture for Collaborative Document Editing Enhanced with Face Recognition," in SACI 2016: Proc. of 11th Int. Symp. on Applied Com- putational Intelligence and Informatics. IEEE Computer Society, 2016, pp. 441-446. 8. C. Gadea, M. Trifan, D. Ionescu, and B. Ionescu, \A Reference Architec- ture for Real-Time Microservice API Consumption," in Proc. of the 3rd Workshop on CrossCloud Infrastructures & Platforms. ACM, 2016, p. 2. 9. B. Ionescu, D. Ionescu, C. Gadea, B. Solomon, and M. Trifan, \An Architec- ture and Methods for Big Data Analysis," in SOFA 2014: Proc. of the 6th Int. Workshop Soft Computing Applications, Advances in Intelligent Sys- tems and Computing, vol 356. Springer International Publishing, 2016, pp. 491-514. 10. B. Ionescu, C. Gadea, B. Solomon, M. Trifan, D. Ionescu, and V. Stoicu- Tivadar, \A Chat-Centric Collaborative Environment for Web-Based Real- Time Collaboration," in SACI 2015: IEEE 10th Jubilee Int. Symp. on Ap- plied Computational Intelligence and Informatics. IEEE Computer Society, 2015, pp. 105-110. 11. B. Ionescu, C. Gadea, B. Solomon, D. Ionescu, V. Stoicu-Tivadar, and M. Trifan, \A Cloud Based Real-Time Collaborative Platform for eHealth," Studies in Health Technology and Informatics, vol. 210, p. 919-923, 2015. Book Chapters 1. B. Solomon, D. Ionescu, C. Gadea, and M. Litoiu, Migrating Legacy Appli- cations: Challenges in Service Oriented Architecture and Cloud Computing Environments. IGI Global, November 2012, ch. Geographically Distributed Cloud-Based Collaborative Application. Contents Abstract iii Acknowledgements iv Publicationsv List of Figures xi List of Tables xv List of Algorithms xvi List of Listings xvii Abbreviations xx 1 Introduction1 1.1 Motivation & Research Objectives...................1 1.2 Thesis Contributions..........................6 1.3 Thesis Methodology & Organization.................8 2 Background & Related Work 11 2.1 Groupware................................ 11 2.2 Optimistic Consistency Control.................... 13 2.2.1 Operational Transformations.................. 13 2.2.1.1 \Causality Preservation" Requirement....... 14 2.2.1.2 \Convergence" Requirement............. 17 2.2.1.3 \Operation Intention Preservation" Requirement. 26 2.2.1.4 \User Intention Preservation" Requirement.... 27 2.2.1.5 Jupiter Algorithm.................. 30 2.2.1.6 SOCT4 Algorithm.................. 35 2.2.2 Commutative Replicated Data Types............. 37 2.3 Collaboration Platforms........................ 38 vii Contents viii 2.3.1 NCCT Lab Projects...................... 38 2.3.2 Wave & Google Docs...................... 39 2.3.3 ShareDB............................. 42 2.3.4 Generic Collaboration Infrastructure............. 47 2.3.5 The Pulsar System....................... 47 2.3.6 Firebase............................. 48 2.3.7 Other Relevant Work...................... 51 3 High-Level FSM Model Design for Collaborative Platforms 54 3.1 Motivations for a New Architecture.................. 55 3.2 FSMs & DOM.............................. 57 3.3 Architecture Requirements....................... 59 3.4 Architecture and Component Design................. 61 3.5 High-Level FSMs for Operational Transformations.......... 65 4 Control Loop-based OT Algorithm 70 4.1 Client FSM............................... 72 4.2 Client FSM with Buffer......................... 76 4.3 Controller FSM............................. 79 4.4 CLOT Algorithm Analysis....................... 82 5 DOM-Based FSM Model Design 85 5.1 Modeling the Required DOM Operations and Algorithms...... 85 5.1.1 Defining New DOM Operations................ 86 5.1.2 Operation Maps & Algorithms................. 89 5.1.2.1 Initialization & SET DOM............. 90 5.1.2.2 Feedback DOM Observer.............. 92 5.1.2.3 Comparison Logic.................. 93 5.1.2.4 Decision Maker.................... 94 5.2 DOM-Based Operations and Examples................ 98 5.2.1 insertNode............................ 99 5.2.2 deleteNode........................... 103 5.2.3 moveNode............................ 104 5.2.4 insertData............................ 106