Architecture Practices for Complex Contexts John R. Klein 2017 SIKS Dissertation Series No. 2017-28 The research reported in this thesis has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems. Promotiecommissie: prof. dr. ir. Paris Avgeriou prof. dr. Jan Bosch prof. dr. Sjaak Brinkkemper dr. Remco de Boer prof. dr. Patricia Lago This publication incorporates portions of the following, (c) 2013-2017 Carnegie Mellon University, with special permission from its Software Engineering Institute: “A systematic review of system-of-systems architecture research” by J. Klein and H. van Vliet. “Application-Specific Evaluation of NoSQL Databases” by J. Klein, I. Gorton, N. Ernst, P. Donohoe, K. Pham, and C. Matser. “Architecture Knowledge for Evaluating Scalable Databases” by I. Gorton, J. Klein, and A. Nurgaliev. “Common Platforms in System-of-Systems Architectures: Results from an Exploratory Study” by J. Klein, S. Cohen, and R. Kazman. “Design Assistant for NoSQL Technology Selection” by J. Klein and I. Gorton. “Distribution, Data, Deployment: Software Architecture Convergence in Big Data Systems” by I. Gorton and J. Klein. “Model-Driven Observability for Big Data Storage” by J. Klein, I. Gorton, L. Alhmoud, Laila, J. Gao, C. Gemici, R. Kapoor, P. Nair, and V. Saravagi. “Performance Evaluation of NoSQL Databases: A Case Study” by J. Klein, I. Gorton, N. Ernst, P. Donohoe, K. Pham, and C. Matser. “Runtime Performance Challenges in Big Data Systems” by J. Klein and I. Gorton. “System-of-Systems Viewpoint for System Architecture Documentation” by J. Klein and H. van Vliet. ANY MATERIAL OF CARNEGIE MELLON UNIVERSITY AND/OR ITS SOFTWARE ENGINEERING INSTI- TUTE CONTAINED HEREIN IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER IN- CLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVER- SITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This publication has not been reviewed nor is it endorsed by Carnegie Mellon University or its Software Engi- neering Institute. ISBN 978-94-6295-682-7 NUR 982 Copyright © 2017 John Robert Klein All rights reserved unless otherwise stated Cover image “Systems of Systems of Systems” by John Klein Published by ProefschriftMaken || www.proefschriftmaken.nl Typeset in LATEX by the author VRIJE UNIVERSITEIT Architecture Practices for Complex Contexts ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad van Doctor aan de Vrije Universiteit Amsterdam, op gezag van de rector magnificus prof.dr. V. Subramaniam, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Exacte Wetenschappen op dinsdag 26 september 2017 om 9.45 uur in de aula van de universiteit, De Boelelaan 1105 door John Robert Klein geboren te Elizabeth, New Jersey, Verenigde Staten promotor: prof.dr. J.C. van Vliet Contents 1 Introduction 1 1.1 Background . 2 1.1.1 Systems of Systems . 2 1.1.2 Data-Intensive Systems . 4 1.1.3 Software Architecture Practices . 6 1.2 Objectives and Research Questions . 13 1.3 Research Methods . 16 1.4 Thesis at a Glance . 17 1.5 Outline and Origin of Chapters . 17 2 Systematic Review of Systems-of-Systems Architecture Research 21 2.1 Introduction . 22 2.2 Research Method . 23 2.2.1 Search Strategy and Data Sources . 23 2.2.2 Search Results . 26 2.2.3 Data Extraction and Synthesis . 29 2.3 Results and Discussion . 30 2.3.1 Demographic Data . 30 2.3.2 Type of Research Result Reported . 32 2.3.3 Architecture Task Focus . 34 2.3.4 Application Domain . 35 2.3.5 Quality Attribute Focus . 36 2.3.6 Technology Maturity . 36 2.3.7 Impacts on research and practice . 38 2.3.8 Study Limitations and Threats to Validity . 41 2.4 Conclusions . 42 3 Common Software Platforms in System-of-Systems Architectures: The State of the Practice 45 3.1 Introduction . 46 3.1.1 System-of-Systems Context . 46 3.1.2 Platforms, Product Platforms, & System-of-Systems Platforms . 46 3.1.3 Goals of this Study . 48 3.2 Research Method . 48 3.3 Interview Questions . 49 v CONTENTS 3.4 Results and Discussion . 51 3.4.1 Architecture Framing and Processes . 51 3.4.2 Challenges and Patterns of Success . 52 3.4.3 Solution constraints . 55 3.5 Conclusions . 56 4 Design Assistant for NoSQL Technology Selection 59 4.1 Introduction . 60 4.2 Related Work . 62 4.3 Feature Taxonomy . 63 4.3.1 Data Model . 64 4.3.2 Query Languages . 65 4.3.3 Consistency . 66 4.3.4 Scalability . 67 4.3.5 Data Distribution . 69 4.3.6 Data Replication . 71 4.3.7 Security . 71 4.4 Knowledge Base Overview . 74 4.4.1 Semantic Knowledge Model . 75 4.4.2 QuABaseBD Implementation of Feature Taxonomy . 79 4.5 QuABaseBD Use Cases . 83 4.6 Demonstrating Feature Taxonomy Efficacy . 85 4.7 User Trials . 87 4.8 Further Work and Conclusions . 89 5 Application-Specific Evaluation of NoSQL Databases 91 5.1 Introduction . 92 5.2 Electronic Health Record Case Study . 93 5.2.1 Project Context . 93 5.2.2 Evaluation Approach . 93 5.3 Prototype and Evaluation Setup . 96 5.3.1 Test Environment . 96 5.3.2 Mapping the data model . 97 5.3.3 Generate and Load Data . 98 5.3.4 Create Load Test Client . 98 5.3.5 Define and Execute Test Scripts . 99 5.4 Performance and Scalability Test Results . 100 5.4.1 Performance Evaluation—Strong Consistency . 101 vi CONTENTS 5.4.2 Performance Evaluation—Eventual Consistency . 104 5.5 Lessons Learned . 106 5.5.1 Essential Issues . 108 5.5.2 Accidental Issues . 110 5.6 Further Work and Conclusions . 111 6 System-of-Systems Viewpoint for System Architecture Documentation 113 6.1 Introduction . 114 6.2 Related Work . 116 6.3 Approach . 117 6.3.1 Problem Investigation—Identify Stakeholders and Concerns . 117 6.3.2 Treatment Design—Define the Architecture Viewpoint . 119 6.3.3 Treatment Evaluation—Active Design Review by Expert Panel . 125 6.4 Analysis and Results . 129 6.4.1 Expert Panel Demographics . 129 6.4.2 Active Review Question Responses . 129 6.4.3 Subjective Questions . 131 6.4.4 Interpretation and Viewpoint Rework . 133 6.4.5 Threats to Validity . 135 6.5 Conclusions and Future Work . 136 6.6 Appendix: Viewpoint Definition . 137 6.6.1 Viewpoint Name . 138 6.6.2 Viewpoint Overview . 138 6.6.3 Concerns Addressed by this Viewpoint . 138 6.6.4 Typical Stakeholders . 138 6.6.5 Model Kinds/Metamodels . 138 6.6.6 Correspondence rules . 145 6.6.7 Operations on views . 145 6.6.8 Examples and Notes . 145 7 Runtime Performance Challenges in Big Data Systems 149 7.1 Introduction . 150 7.2 Characteristics of Big Data Systems . 151 7.3 The Need for Observability . 153 7.4 Related Work . 155 vii CONTENTS 7.5 Our Approach . 156 7.5.1 Model-Driven Design Time Toolkit . 157 7.5.2 Monitoring and Analysis Runtime Framework . 160 7.6 Future Work . 162 7.7 Conclusion . 163 8 Model-Driven Observability for Big Data Storage 165 8.1 Introduction . 165 8.2 Architecture and Implementation . 167 8.2.1 Overview of the Observability Architecture . 167 8.2.2 Metamodel . 170 8.2.3 Model Editor Client . 170 8.2.4 Runtime Metric Collection . 172 8.2.5 Metric Aggregation and Visualization . 173 8.3 Performance Results . 173 8.4 Prior Work . 176 8.5 Conclusions and Future Work . 177 9 Conclusions 179 9.1 Answer to the Research Questions . 179 9.1.1 Practice Area—Architecture Design . 180 9.1.2 Practice Area—Architecture Documentation . 181 9.1.3 Practice Area—Architecture Evaluation . 182 9.2 Answering the Main Research Question . 183 9.3 Further Research . 184 9.3.1 Continuing this work . 184 9.3.2 Complementing this work . 185 10 Samenvatting 187 10.1 Nederlandse samenvatting . 187 10.2 English Summary . 188 viii Acknowledgements I am thankful.