(2008). Exploring the Impact of Systems Architecture

Exploring the Impact of Systems Architecture and Systems Requirements on Systems Integration Complexity

Dr. Rashmi Jaini, Anithashree Chandrasekaran, George Elias, Dr. Robert Cloutier Stevens Institute of Technology [email protected], [email protected], [email protected], [email protected]

Abstract development projects in the government The need to perform faster systems sector. integration of complex systems require the Keywords: System Architecture, System architect and design team understand how Integration, Integration Complexity, System the selected architecture and design Requirements components will impact the system Introduction integration processes complexity. System It is necessary to determine if integration process complexity is an integration of the defined physical elements outcome of the interaction between degree and interfaces in system architecture of feasibility and level of effort required to contributes to system integration process understand, describe, implement, manage complexity. Today’s systems are usually and document the system integration process expected to function independently and in for a given system development and cooperation with the existing systems operational environment. This paper (system of systems). The cooperating analyzes the cause and effect relationships systems undergo frequent changes and between the system requirements, interoperability becomes a key factor and a architecture and the system integration major contributor to system integration processes complexity. In order to address process complexity. Interoperability is based systems integration issues upfront in the on the existence of the conceptual view design phase it is necessary to determine if [Carney and Oberndoff, 2004]. The the architecture and design of components, conceptual view can be embodied in sub-systems, processes and interfaces requirements and architecture/design and the impacts (and to what extent) system system architecture determines the level of integration process complexity. This paper interoperability. The system integration also defines, and analyzes the impact of the process complexity includes interoperability. different system architecture and The major focus of the paper is on requirements factors on system integration the cause and effect relationship between process complexity. A research framework system requirements, system architecture, is developed to understand the cause and and system integration process complexity. effect relationships between the system The paper is based on the research that requirements, architecture and integration includes system engineering literature process. Finally, the paper proposes reviews, study of industry best practices and recommendations based on the causality a survey of the system engineering leads for results. These conclusions are based on eight different development projects in a research undertaken by the authors on 8 government organization. The research methodology used for this study is provided in Figure 1. This paper is organized in a were selected. The selection of these similar outline, first establishing the context activities and factors were based on the by defining and discussing system assumption that these activities contribute integration process, and its complexity towards better requirements and architecture classification. In order to understand, and would also reduce the system analyze and prioritize the causal factors a integration process complexity, and improve cause and effect relationship model was the quality of system verification and developed and relevant attributes of good validation. A survey was then developed to requirements and architecture were verify and validate this. Finally, this paper identified. These requirements and discusses a set of research questions and the architecture attributes define the activities findings to further provide a meaningful involved in these two critical phases and analysis of these relationships between enable a relatively simplified integration system requirements, architecture and process. Next, a set of activities/factors of integration process complexity. system requirements and system architecture

Research Methodology: Cause and Effect Relationship between SR, SA and SI Process Complexity

Our research knowledge base in SI Research on factors of system Research on system integration models, methodologies and requirements and architecture that process complexity

practices approaches impact SI process Review of Review Literature & Literature Industry best best Industry

Define SI process Identify the cause and effect relationships complexity & its between system requirements, architecture classification and integration process complexity on the reviews on Synthesis based based Synthesis

Identify significant factors of system Research Questions & requirements and system architecture that Survey analysis Survey Development result in improving SI process & quality of synthesis

analysis & & analysis V&V SI process SI process complexity survey survey complexity

Figure 1 Research Methodology System Integration Process Systems integration is the process Complexity that links the systems engineering life cycle System integration (SI) process is process from requirements to verification defined as a set of activities that transforms and validation and ultimately the stakeholder requirements into an implementation of the system as shown in operational system by unifying the process the Figure 2. This linking process is also components and product components while called traceability, which is an integral ensuring compliance to the specified levels component of an effective systems of component operations and integration process. interoperability/interdependence. The system architecture and design is determined by the system requirements

2 which are originated and derived from the allocates the system functionality into stakeholder requirements. Requirements components, sometimes referred to as specification involves translating system Hardware Configuration Items (HWCI) and requirements into a formal representation of Computer Software Configuration Items interrelated units. The output of this process (CSCI), and defines the interfaces between serves as a blueprint of the system which these configuration items. The complexity of guides and controls all the subsequent the systems integration process may be development activities [Dogru, et.al, 1992]. determined by the functional and physical The architecture and design decomposes and decomposition of the system architecture. REQUIREMENTS FUNCTIONAL/PHYSICAL ARCHITECTURE

S Input Reqts System Y S Output Reqts T Processing Reqts E M Non-Functional Reqts SubSystem 1

S U Input Reqts B S Output Reqts Y SubSystem 2 S T Processing Reqts E M Non-Funct. Reqts Systems Integration 1 S U Input Reqts B S Output Reqts Y S CSCI 1 – Application X T Processing Reqts E M Non-Funct. Reqts 2 CSCI 2 – Application Y

HWCI 1 – Processor CSCI 1 Reqts . CSCI 1 Reqts As you create your architecture (design), you . . HWCI 1 Reqts determine the requirements allocation and trace . . between the requirements and design . Figure 2 System Integrationii Figure 2 demonstrates the process by providing strategy of system decomposition, which the functional architecture enables the data storage, inter-process and user requirements to be traced and defined for communication each of the configuration items that are a The importance of the system integration part of the physical architecture. These process in the development of a system and requirements form the basis for hardware its successful implementation can be and software detail design and development. understood and appreciated by Requirements are usually allocated to understanding and familiarizing with these functions (within the functional SI process activities. Despite the large architecture), that are performed by physical investments that are made, many integration elements. In [Rossak and Prasad, 1991] efforts fail for a number of technical, Rossak and Prasad state that integration organizational and management, and architectures for system of systems serve as migration planning reasons. The technical a general pattern to define the basic layout issues include the following [Smith, et al. of an integrated system. The integration 2002]: architectures deal with the development of new system parts and the post-facto ¾ Legacy systems originated from integration of already existing solutions by

unplanned, stovepipe development or understand, describe, implement, manage were developed as batch, single tier. and document the system integration process ¾ Data is specific to the applications and activities are shown in Table 1. This list is was not designed for sharing. based on our system integration framework ¾ The legacy systems were not initially and system integration process model (SIPM) designed for new quality-attribute [Jain, et al. 2007a, 2007b]. requirements and are being affected by Table 1 Factors that impact degree of needs for interoperability, performance, feasibility and level of effort required for security, and usability. SI ¾ The scope for the integration effort is not adequately defined. Factors that impact ¾ Decisions are made without performing Degree of feasibility Level of effort required adequate analyses (sometimes referred to Availability of the Documentation of the as “management by magazine”) integration (including system requirements V&V) methodologies The exponential development and and tools growth of technology and increase in user Availability of the System Integration demands has resulted in increase of required external and Planning, Control and complexity in the system integration and internal interfaces Management development process. Complexity in this Scope of COTS Documentation of context is the degree to which a system or requirements (interface system integration component has a design or implementation and interoperability) specifications (including that is difficult to understand and verify V&V) [IEEE 610-12, 1990]. Complexity is also Scope of legacy Familiarity and defined as the degree of complication of a requirements (interface knowledge of the system or system component, determined by and interoperability) integration (including V&V) methodologies such factors as the number and intricacy of and tools interfaces, the number and intricacy of conditional branches, the degree of nesting, Adherence to the Familiarity and standards, regulations knowledge of the and the types of data structures [Evans and and guidelines required external and Marciniak, 1987]. Any measurement of internal interfaces process complexity requires measurement of Planned programmatic Knowledge of the system situation complexity, action complexity and resources for the system constraints, context, intention complexity. [Howard, et al. 2004] integration process environment and scope We define Level of performance Specification of metrics to be supported performance metrics System Integration process complexity as (What, How, When to an outcome of the interaction between measure? Who will degree of feasibility and level of effort measure?) required to understand, describe, Level of Specification of implement, manage and document the operational/service operational/service system integration process for a given metrics to be supported metrics (What, How, system development and operational When to measure? Who will measure?) environment. Some of the factors that impact the degree of feasibility and the level of effort required to

Factors that impact programmatic complexity are allocated Degree of feasibility Level of effort required budget, cost associated with system integration activities, schedule, Degree of automation of the SI process materials/equipments/tools, and skill sets. Configuration complexity: The complexity Number of dependencies of system integration process due to the for SI process activities (Degree of concurrency inconsistencies and lack of control in the of the SI process) development of process and product. The major factors of configuration complexity Most of the research discussion on are the control and management of changes, system integration complexity is in the volatility and clarity of documentation and context of the complexity associated with specifications, integration baselines, and the entire system. Currently, system integration personnel communication. integration complexity is mainly focused on Operational complexity: This complexity the degree of testing required and interface results from providing and supporting the complexity. But there is a need to look at required level of operational support and system integration complexity from both the system availability. The system integration process and product point of view in order to operational complexity can be observed in ascertain associated risks in a the required level of effort to integrate, comprehensive manner. . To address these verify and validate the availability and aspects we classify system integration operational support of the system and its process complexity as Technical, sub-systems. Programmatic, Configuration, Operational, Organizational complexity: The complexity and Organizational. This classification is of system integration process due to the based on how the system lifecycle and the organizational strategy, compliance, development activities impact the system processes and product line. The major integration process. These are described factors of organizational complexity are below. service level agreements of sub-systems and Technical complexity: The complexity of interfaces, standards, regulations and system integration process due to the guidelines for system integration, and legacy required system capabilities and functions. system integration. The major factors of technical complexity are the integration technologies required to Designing for integration achieve the system capabilities and Most of the factors that result in SI functions, feasibility and performance of the process complexity are external to the interfaces, and strategies, methodologies and context of system integration process. The tools for verification and validation of the system integration processes and activities technical effectiveness of the system and its receive most of its inputs, controls and sub-systems. triggers from system requirements and Programmatic complexity: Programmatic system architecture (Figure 3, Figure 4, complexity includes system integration Figure 5). This highlights the direct cause process complexity arising out of the and effect relationship between the system variance between the planned and available requirements, architecture and system resources needed to support the system integration. System requirements and integration process over its lifecycle. The architecture has a direct impact on system major variances resulting in such

integration. architecture on system integration. System System architecture also receives architecture can play a direct causer or a inputs, triggers and controls from system moderator or a mediator depending on the requirement activities. Hence there is a system context, and concept. mediating and moderating effect of system

Figure 3 Context Diagram for the Integration Process [SE Handbook, 2006]

Figure 4 Context Diagram for the Verification Process [SE Handbook, 2006]

Figure 5 Context Diagram for the Validation Process [SE Handbook, 2006] The goodness of system architecture An attribute is a “property or and requirements can be defined by some characteristic of an entity that can be attributes. The attributes of a system or distinguished quantitatively or qualitatively architecture are important because they are by human or automated means [ISO/IEC used to describe the properties of the system 15939, 2002].” The attributes of a system or or the system architecture in a unique an architecture are used to describe the distinguishing manner [Elias and Jain, properties of the system or the SA in a 2007]. System attributes are a key link unique distinguishing manner. Because between good system architecture and the attributes are measurable they are ideal for system’s ultimate development cost and describing and monitoring many systems schedule [McCabe and Pollen, 2004]. A engineering tasks [ISO/IEC 15939, 2002]. system attribute denotes a characteristic or The use of attributes to measure and category of requirements commonly evaluate systems and systems architecture is demanded of systems. Recent work suggests valuable in making decisions and tradeoffs that system attributes offer an effective set [Bass, 2003]. In this paper, we focus on of perspectives to evaluate architectural important attributes that contribute to decisions and tradeoffs [Bass, 2003]. The set System Integration Complexity (SIC). By of attributes shown in Figure 6 provides understanding which attributes of SA affect standard guidelines for system requirements SIC one could use this information and system architecture. By addressing some throughout the requirements, architecting, of these attributes of system architecture and and development phases of the system requirements some of the significant system lifecycle. integration issues can be contained and SIC can be affected by multiple controlled effectively resulting in a factors. We are primarily concerned with relatively simplified system integration system architecture aspects that impact SIC process. A comprehensive list of attributes such as commonality, modularity, standards- of architecture can be obtained from SEI based, and reliability, maintainability, and Quality Measure Taxonomy [SEI, 2006], testability (RMT) as shown in Figure 6. [Clements, 2001], and [Elias and Jain, 2007]. Integration Complexity

Commonality Modularity RMT Standards Based

1. Commonality in Hardware and 4. Physical Modularity 8. Required Level of 11. Interface Openness Software Subsystems 5. Functional Modularity Reliability 12. Open System Orientation 2. Percentage of Familiar 6. Orthogonality 9. Maintainability 13. Consistency Orientation Technology 7. Abstraction of System 10. Testability Factors 3. Operational Commonality Architecture

Requirements

14. Clear Identification of the System Requirements 15. Prioritization of System Requirements 16. Feasibility Analysis for System Requirements 17. Risk Analysis for System Requirements 18. Categorization of System Requirements Based on its Type Figure 6 Architectural attributes that impact SIC Commonality refers to the systems software packages implemented, number of design where a component or subsystem can languages, number of compilers, average be used in more than one place in the number of software instantiations. system. Commonality directly corresponds Modularity is the degree to which a to the degree to which components and system is structured as a configuration of subsystems are reused within the system. smaller, self-contained units with well- Commonality is highly related to reuse when defined interfaces and interactions (i.e., multiple systems have subsystems in independently testable), moderating design common. Reusability is the degree to which complexity and enhancing its clarity, and a module, component, system or other work enabling design and functional flexibility product can be used more than once [IEEE and variety for the system as a whole 610-12, 1990]. Operational commonality has [McCabe and Pollen, 2004], [Open Systems to do with how close-related systems or Joint Task Force, 2005]. Modularity is subsystems are operationally similar. This usually associated with open systems design, directly affects the attributes of usability and but in this case we are considering open maintainability. Another aspect of systems design to be a design standard, commonality is architecting with a view to while modularity is a good design practice use familiar technology. Familiarity of on its own. Nevertheless, both of these technology helps design more open and concepts are covered regardless of where flexible architectures. Factors that they are placed in this structure. Abstracting negatively affect commonality are: the the system architecture enables the architect number of unique Line Replacement Units to review several alternative solutions before (LRU), number of unique fasteners, number choosing the one that will be implemented. of unique cables, number of unique Differentiating the layers of functionality standards implemented, number of unique and the structure that is required to support through abstraction simplifies the choice in Testability is the degree to which a many cases. Abstraction also facilitates system or component facilitates the reusability and portability by preserving the establishment of test criteria and the boundaries of the different layers [Jorgensen performance of tests to determine whether and Philpott, 2002]. those criteria have been met [IEEE 610-12, Standards based systems are 1990]. Examples of issues that affect designed and architected based on open testability: number of LRUs covered by BIT standards. Open systems employ a system (Built In Test), logging/recording capability, design philosophy which provides for ability to create system state at time of interoperability and portability. In addition system failure, online testing, ability to be to using an open systems approach, the operational during external testing, ease of architecting or implementing organization access to external test points, automated may choose to document their own design input/stimulation insertion). standards. Organizational design standards Requirements are the life-blood of create consistency in the way systems are systems development. Every man-made architected and designed by an organization. system starts with requirements regardless The downside to designing to standards is of whether they are implicit, explicit, good that they may impose constraints that or bad. The first phase of the systems become problematic in relation to engineering process is driven by collecting, technology issues, or may not even be composing and analyzing systems possible in some cases. While designing to requirements. These requirements drive the standards may not always be possible or success or failure of the system throughout desirable, it is always important to consider the lifecycle. The decisions made through design standards. detailed design commit approximately 80% Reliability is often a very important of the cost of the system [Buede, 2000]. factor in the design of a system. Reliability Therefore, requirements are a very important has two classic definitions that apply to driver of the systems engineering process systems: 1) The duration or probability of and require substantial attention. failure-free performance under stated Requirements that are generated as a result conditions, and 2) The probability that an of stakeholder concerns, and written in the item can perform its intended function for a context of the key attributes about to be specified interval under stated conditions. listed, increase that likelihood of system (For non-redundant items, this is equivalent success. to definition (1). For redundant items, this is Based on our research, we identified equivalent to mission reliability [MILSTD- the following 18 activities that characterize 1388-1A, 1983]. the impact of system requirements and Unfortunately, when systems are not architecture related attributes on systems designed with maintainability in mind it is integration complexity. usually left out. Maintainability is the ease 1. Commonality in hardware and software with which a software system or component subsystems (such as number of unique can be modified to correct faults, improve Line Replacement Units (LRU), number performance, or other attributes, or adapt to of unique fasteners, number of unique a changed environment [IEEE 610-12, cables, number of unique standards 1990]. Methods to measure maintainability implemented, number of unique are mean time to repair (MTTR), maximum software packages implemented, number fault group size, etc. of languages, number of compilers,

average number of software 12. Open system orientation (hardware and instantiations) software standards) 2. Percentage of familiar technology used 13. Consistency orientation (common in the system of interest guidelines for implementing diagnostics 3. Operational commonality (such as and performance monitoring and fault percentage of operational functions localization) automated, number of unique Skill codes 14. Clear identification of the system required, estimated operational training requirements (uniquely identified (i.e., time, estimated maintenance training number, name tag, mnemonic, time) hypertext) and reflect linkages and 4. Physical modularity (such as ease of relationships) system element upgrade, ease of 15. Prioritization of system requirements operating system upgrade) (based on stakeholders’ input and 5. Functional modularity (such as ease of criticality analysis) adding new functionality, ease of 16. Feasibility analysis for system upgrade existing functionality) requirements 6. Orthogonality (examples are functional 17. Risk analysis of system requirements requirements fragmented across multiple 18. Categorization of system requirements processing elements and interfaces, based on its type (such as input, output, throughput requirements across reliability, availability, security, interfaces, common specifications environmental, performance, interface, identified) testing) 7. Abstraction of system architecture Some of the significant sources of 8. Required level of reliability (factors such the causality analysis include [Verma and as fault tolerance, percentage of system Johannesen, 2000], Architecture Tradeoffs loading (processor loading, memory Analysis Method (SEI ATAM), IBM loading, network loading, etc…)) Architecture Evaluation Methodology (IBM 9. Maintainability in terms of expected AEM), [Bachmann, et al. 2000], MTTR (mean time to repair), maximum [Bachmann, et al. 2002], [Kazman, et al. fault group size, accessibility and 2003], [Bachmann, et al. 2003], [IEEE 1233, required system operational during 1998], [IEEE 830, 1998], INCOSE SE maintenance Handbook v3 [SE Handbook, 2006]. These 10. Testability factors (Examples: number of system architecture and requirements LRUs covered by BIT (BIT Coverage), activities when implemented can result in reproducibility of errors, certain clearly observable patterns in system logging/recording capability, ability to architecture and requirements [Cloutier, create system state at time of system 2006], [Cloutier and Verma, 2007]. By failure, online testing, ability to be studying and mining similar systems, operational during external testing, ease common patterns are found that go well of access to external test points, beyond software patterns in common use automated input/stimulation insertion) today. These patterns can form the basis for 11. Interface openness (such as number of entire subsystems of a complex system. The Interface Standards/Interfaces, multiple Perform C2 (command and control) is an vendors, multiple business domains, example of such a pattern [Cloutier and standard maturity) Verma, 2006]. In this system architecture pattern, the smaller plan, detect, control,

10 and engage patterns are assembled into a using fasteners of the same specification pattern language called Perform C2. One of within and among the subsystems. the motivators documented for the use of Each of the identified system system patterns is the value of managing requirements and architecture activity and complexity. When a pattern such as the factor help address at least one category of Perform C2 is implemented and integrated system integration process complexity. The into a system, the integration complexities impact of these activities on the complexity are better understood, and can be leverage in category is a result of the dependencies subsequent implementation and integration between the complexity factors and the efforts if the same pattern is used again. The activities. A mapping was created to provide same can be said for patterns that evolve an overview of how each activity/factor when developing embedded system in impact the system integration process compatible languages or common platform, complexity categories. The mapping is shown in Table 2.

Table 2 System Architecture and Requirements Cause and Effect on System Integration Process Complexity

System Architecture and Technical Programmatic Configuration Operational Organizational Requirements Factors complexity complexity complexity complexity complexity Commonality in hardware and software subsystems X Percentage of familiar technology X X X Operational commonality X X X Physical modularity X Functional modularity X Orthogonality X X Abstraction of system architecture X X X Required level of reliability X Required level of maintainability X Testability X Interface openness X X Open system orientation X X X Consistency orientation X X Clear identification of the system requirements X X X X Prioritization of system requirements X X System requirements feasibility analysis X X X System requirements risk analysis X X X X Categorization of system requirements X X X X X

Formulating hypothetical SI architecture and requirements with system complexity causal relationships integration process complexity. Through the course of this work, research questions were constructed to assist 1. Null Hypothesis: There is a cause and in the understanding of how and to what effect relationship between system extent the identified system architecture and requirements, system architecture and requirements factors impact the system system integration process complexity. integration process. By analyzing these 2. Null Hypothesis: There is a cause and research questions we arrive at some effect relationship between system recommended practices to handle the cause requirements, system architecture and and effect relationship between system quality of V & V. architecture and requirements that would 3. What are the three most important simplify the system integration process and factors of system requirements and reduce the dependencies between these architecture that could reduce system development phases. Addressing some of integration complexity? the system integration issues during the 4. What are the three most important system requirements and architecture phase factors of system requirements and provides a strong foundation for the architecture that could improve quality integration phase and help manage the of system verification and validation? associated criticality of these issues. When 5. Does system architecture impact the the foundation for the system integration complexity of system integration? activities and processes is initiated early in Hypothesis: System architecture the lifecycle, feedback and iterations during improvements can reduce system development helps to refine and improve integration complexity. system integration strategies, planning and 6. What are the three most important quality. architectural factors for system The two factors (dependent integration complexity? variables) considered for this research study 7. Does system architecture impact the were the system integration complexity quality of system verification and (system integration process complexity) and validation? Hypothesis: System quality of system verification and validation architecture improvements can improve (V&V). The most important phase of system the quality of system verification and integration is the system verification and validation. validation. V&V are the phases of system 8. What are the three most important engineering where the required functionality architectural factors for quality of comes together with all the interfaces verification and validation? completed. Hence studying the quality of 9. Does the level system reliability and V&V along with the SI complexity is maintainability impact the complexity of important and provides a comprehensive system integration and the quality of view of system integration. The independent verification and validation? Hypothesis: variables are the system architecture and Higher levels of reliability and requirements factors discussed in the maintainability in system design reduce previous section. We formed the following the complexity of system integration and research questions to understand the cause increase the quality of verification and and effect relationships between system validation. 10. Does an improved requirement questions was piloted to the SE personnel at engineering process results in reduced a government organization to obtain their system integration complexity and better feedback on the survey clarity and quality of verification and validation? terminologies used. After the completion of 11. Does familiarity of technology lead to the pilot, data was collected on 8 different reduced system integration complexity development projects. The survey responses and better quality of verification and are shown in Table 3, and Table 4. The validation? impact of the causal factors of requirements A survey questionnaire addressing each of engineering on system integration these identified system architecture and complexity is shown in Figure 7, and Figure requirements factors was developed. The 8. The impact of the causal factors of system survey questions are designed to verify if architecture on system integration there is a cause and effect relationship complexity is shown in Figure 9, and Figure between each of the identified 18 factors on 10. The impact of the causal factors of system integration complexity and quality of requirements engineering on quality of system verification and validation and also verification and validation is shown in measure their impacts. The survey questions Figure 11, and Figure 12. The impact of the were also designed to capture the comments causal factors of system architecture on (thoughts and inputs) of the participants on quality of verification and validation is each of these 36 cause-and-effect shown in Figure 13, and Figure 14. relationships. The questionnaire with 36 Table 3 System Integration Process Complexity: Survey Results # SI is improved as a result of Not Improved (%) Improved (%) Partially Moderately Significantly Exceptionally

1 Categorization of system requirements 25 13 50 0 13 based on its type 2 Clear identification of the system 0 0 38 25 38 requirements 3 Commonality in hardware and software 0 0 63 25 13 subsystems 4 Decrease in abstraction of the system 67 17 17 0 0 architecture 5 Decrease in expected maintainability 71 14 14 0 0 6 Decrease in interface openness 29 29 14 29 0 7 Decrease in level of required reliability 43 29 14 14 0 8 Decrease in orthogonality 01429570 9 Decrease in testability factors 33 17 33 17 0 10 Increase in consistency orientation 0 29 43 29 0 11 Increase in functional modularity 0 0 43 43 14 12 Increase in open system orientation 25 25 13 38 0 13 Increase in operational commonality 13 25 50 13 0 14 Increase in percentage of familiar 13 13 25 50 0 technology 15 Increase in physical modularity 0 0 57 29 14 16 Prioritization of system requirements 0 13 38 25 25 17 System requirement feasibility analysis 25 13 38 13 13 18 System requirement risk analysis 13 13 38 25 13

Improvement in SI as a result of Requirements Engineering

100% Improved 80% Exceptionally 60% Improved 40% Significantly SI 20% Improved 0% Moderately Improved Partially

Not Improved of of of Clear analysis feasibility Requirement Requirement Prioritization risk analysis risk identification requirements requirements requirements requirements requirements Categorization

Requirements Engineering

Figure 7 SI process improvements based on specific Requirements Engineering activities

Improvement of SI Process SI complexity as (complexity) as a result of System Architecture a result of Requirements Engineering

22% Not Improved 24% Not Improved 33% Moderate 37% Moderate Improvements Improvements Sign ifican t Significant Improvements Improvements

45% 39%

Figure 8 SI process improvements based Figure 9 SI process improvements based on Requirements Engineering on System Architecture

Improvement in SI as a result of System Architecture 100%

80%

60%

40% SI 20%

0% factors level of required physical interface familiar testability openness openness functional Increase in Increase in Increase in modularity modularity Decrease in Decrease modularity modularity Decrease in Decrease Decrease in Decrease Increase in consistency orientation orientation software orientation orientation Increase in Increase in expected open system open Decrease in operational Decrease in Decrease architecture percentage of percentage Decrease in abstraction of abstraction commonality commonality orthogonality orthogonality Commonality in hardware and maintainability Architecture

Figure 10 SI process improvements based on specific System Architecture activities [Legend same as Figure 7] Table 4 Quality of Verification and Validation: Survey Results # Verification and Validation is Not Improved (%) improved as a result of Improved (%) Partially Moderately Significantly Exceptionally

1 Categorization of system requirements based on 25 13 38 13 13 its type 2 Clear identification of the system requirements 0 0 38 25 38 3 Commonality in hardware and software 0 0 75 25 0 subsystems 4 Decrease in abstraction of the system architecture 50 33 0 0 17

5 Decrease in expected maintainability 43 29 14 0 14 6 Decrease in interface openness 33 17 33 17 0 7 Decrease in level of required reliability 43 14 14 14 14 8 Decrease in orthogonality 17 0 17 50 17 9 Decrease in testability factors 33 0 50 0 17 10 Increase in consistency orientation 14 29 29 29 0 11 Increase in functional modularity 17 17 33 17 17 12 Increase in open system orientation 25 38 13 25 0 13 Increase in operational commonality 013502513 14 Increase in percentage of familiar technology 13 13 38 25 13 15 Increase in physical modularity 017331733 17 Prioritization of system requirements 13 25 25 13 25 18 System requirement feasibility analysis 25 25 25 25 0

Quality of V&V as a result of Requirements Engineering

100% V Improved 80% Exceptionally 60% Improved 40% Significantly 20% Improved 0% Moderately Quality of V& of Quality Improved Partially

Not Improved of of of Clear analysis feasibility Requirement Requirement Prioritization analysis risk identification requirements requirements requirements requirements Categorization

Requirements Engineering

Figure 11 V&V quality improvements based on specific Requirements Engineering activities

Quality of V&V as Quality of V&V as a result of Requirements a result of System Architecture Engineering

26% Not Improved 25% Not Improved 30% 36% Moderate Moderate Improvements Improvements Sign ifican t Sign ifican t Improvements Improvements

44% 39%

Figure 12 V&V quality improvements Figure 13 V&V quality improvements based on Requirements Engineering based on System Architecture

Quality of V&V as a result of System Architecture

100%

V 80%

60%

40%

20% Quality of V&

ss ty n ty ms e i o on y te al ctors ti i og ability ta at nality itecture n on n nt ol bsys h og y fa e ie e openn h modulari su c ori l or techn re a ort a m r erf in stabilit ncy l commo ia d maintai t quired reliability e e l twa e re se st mi t a nction syste a sof e u n f cr onsi f nd se in in e c n pe erationa of a rease in t o p e n expec D c in n o g re a n abstraction i of arc e i ta i e D se wa e Decre crease i a en Increase in physical modularity rd crease In re rc a c rease in pe ecreas In In c in h ecreas D In in ty D Decrease in level of li ase na cre In ommo C System Architecture

Figure 14 V&V quality improvements based on specific System Architecture activities [Legend same as Figure 11] Significant Cause and Effect consolidated view of this analysis and Relationships synthesis. The significant cause and effect The results of the survey were relationships between system requirements, analyzed based on the research questions. system architecture, integration process The following sections provide a complexity, and quality of V&V are shown in the fishbone charts shown in Figure 15 and Figure 16.

Figure 15 Major Factors Impacting SI process complexity

Figure 16 Major Factors Impacting Quality of System Verification & Validation complexity and in improving the quality of Impacts of Requirements Engineering on verification and validation are discussed SIC and Quality of V&V below. The results of the survey confirmed Clear identification of system requirements: the belief that an improved requirement There has been a good amount of engineering (RE) process results in reduced research and improvements in the field of system integration complexity and a higher requirements engineering. The major focus quality requirements verification and area of these research, methods and tools is validation. The improved RE process will the ambiguity and volatility associated with result in significant improvements by the system requirements. Subject + Verb + reducing system integration complexity and Modifier format is commonly used for clear improving the quality of V&V. The system requirements. Traceability and significant system requirements factors that testability of requirements have to be result in reducing the system integration

17 concentrated during requirements elicitation prioritization of system requirements. The and analysis. A number of tools on the effort required for integration, verification market today help system engineers trace and validation can be planned and executed requirements and specify the associated test effectively by performing this activity early requirements. But the traceability in most of in the lifecycle with stakeholder these tools ends during the requirements participation. By prioritizing and phase. Even though there is a good concentrating on core functionalities/ traceability between the originating and capabilities first and building upon them can derived requirements, the benefits are also help in configuration management. This limited when there is no traceability across activity helps reduce both the technical and all phases of development. Stakeholder configuration complexity of integration. involvement throughout the system development is a key to identify clear Impacts of Systems Architecture on SIC system requirements and transition them and Quality of V&V effectively across the system lifecycle. The results of the survey suggest that By performing this activity the there will be a significant improvement in system capabilities and functions can be the system integration complexity and the clearly identified. This activity impacts quality of V&V by adopting some of the every phase of system integration (derive suggested system architecture practices. integration requirements, develop Current systems engineering researchers integration architecture, plan integration, focus on architectures that are integration implement based on the architecture and friendly. The mode of system development plan, and verify and validate the system and has shifted away from being manufacturing- its interfaces) and significantly impacts the centric to being integration-centric due to technical complexity of integration. By the fact that use significant percentage of identifying clear requirements, the cost COTS, strategic outsourcing, value based overruns and schedule slippage can be capabilities development, and need for avoided due to the requirements ambiguity. supply chain excellence. Another Therefore, requirements traceability architecture trend is the migration toward also impacts programmatic complexity. By service oriented architectures (Service identifying clear operational support and Oriented Architectures focused around availability requirements and other supportability of operational scenarios) and constraining requirements the risks event driven architecture which are even associated feasibility and effort required can more integration friendly [Kumar, et. al., be analyzed up in the lifecycle and can be 2005], [Krishnamoorthy, et. al., 2005] The mitigated. The result is lower integration factors of system integration process operational complexity and organizational complexity are addressed earlier in the complexity. lifecycle and addressed as a part of the Prioritization of system requirements: architecture by adopting these integration During system development friendly architecture methodologies. Some concentrating on value added activities is of the system architecture factors that impact very important. In order to provide the value system integration process complexity added activities the system requirements evolve when these architecture need to be prioritized earlier in the lifecycle. methodologies are adopted. Future research One of the key success criteria for based on these research findings has been evolutionary or iterative development is proposed at Stevens Institute of

Technology, and should result in practices, reduced significantly by increasing methods, and tools that will aid in creating functional modularity in system architecture. system architectures which utilize patterns at The increase in functional modularity in the system and subsystem levels, and system architecture is a major contributor to displaying high degrees of modularity. rapidity in system development by adding Good architecture should exhibit ease of building upon or upgrading the characteristics such as Time Sensitivity, existing functionalities. This factor is a key Context Sensitivity, and Stakeholder in evolutionary and iterative system Sensitivity. Based on the survey results the developments. The risk associated with factors of system architecture that impacted system architecture during rapid system system integration process complexity and development would also be reduced by quality of system verification and validation addressing functional modularity. This the most and results in at least partial reduces the technical complexity associated improvement are with system integration process. Commonality in hardware and Increase in operational commonality: software subsystems: This system Quality of verification and validation can be architecture factor emerged to be an improved significantly by increasing the important factor impacting both system operational commonality. Commonality is integration process complexity and quality the extent to which the system is made up of of system verification and validation. common hardware and software Commonality in subsystems helps reduce components, utilizes familiar technologies, the effort required for integration, and is automated reducing the effort of verification and validation. The degree of training and maintenance. Operational unique interfaces, platforms, technology commonality in system architecture results used, and protocols are reduced resulting in operational requirements that can be familiarity of subsystems. This reduces the easily verified and validated. Automation technical complexity associated with system also reduces the effort involved in V&V. integration process. Decrease in orthogonality: This Increase in physical modularity: factor of system architecture results in Quality of verification and validation can improvements in both quality of verification have at least moderate improvements and and validation and system integration System integration process complexity can process complexity. Orthogonality of system be at least partially reduced by increasing architecture can be reduced by performing the physical modularity in system one-to-one functional mapping. This factor architecture. The physical and functional reduces both the technical and configuration modularity in architecture facilitates both complexities of system integration process. modernization and replacements of legacy Increase in percentage of familiar systems. This significantly reduces the technology: By having familiar technologies complexity associated with the legacy in system architecture quality of V&V and system integration. Modularity also results system integration process can be improved in higher levels of maintainability and moderately. This can be achieved by support. This reduces the technical adopting good product line architectures and complexity associated with system nested layer architecture. This factor reduces integration process. the programmatic and configuration Increase in functional modularity: complexities of system integration process. System integration complexity can be

Functional modularity as a activities. However, the results presented in characteristic of system architecture plays an this paper indicate these activities could important role in reducing system result in reduced total cost of ownership and integration complexity. However, it does not improved system operational effectiveness. seem to improve system verification and Summary validation. Also we can observe that The specific nature of application- operational commonality plays an important domain plays a critical role in the impact of role in improving system verification and the system architecture and system validation but not in reducing system requirements related factors on system integration complexity. These differences integration process complexity. The findings could be attributed to the nature of the tasks of this study were based on the survey of involved in system integration and system one government organization. They indicate verification and validation. Modularity is the that attention to key architecture and extent to which the system is made up of requirements attributes during the early well defined, functionally non-overlapping, development activities can have significant modular elements with well documented impact during systems integration, while interfaces allowing updates to or resulting in reduced systems integration replacements of a portion of the system complexity. Further research to extend the without affecting the remainder of the study across domains and industry sectors is system. Functional modularity in system required to generalize the findings. architecture results in standard functional References requirements fragmented across multiple 1. Bachmann, F., Bass, L., Chastek, G., processing elements and interfaces. It also Donohoe, P., Peruzzi, F., “The enables adding new functionality with Architecture Based Design Method”, minimum disruption. By doing so system Software Engineering Institute, Carnegie integration is simplified. Mellon University (SEI-CMU), 2000. The results also indicate that the 2. Bachmann, F., Bass, L., Klein, M., most of the factors that impact system “Deriving Architectural Tactics: A Step integration complexity also impact quality Toward Methodical Architectural Design”, of system verification and validation. By SEI-CMU, 2003. adopting these activities in system 3. Bachmann, F., Bass, L., Klein, M., requirements and architecture critical issues “Illuminating the Fundamental of system development are addressed early Contributors to Software Architecture in the lifecycle. This provides more Quality”, SEI-CMU, 2002. bandwidth for mitigating the risks associated 4. Bass, L., Clements, P., Kazman, R., with these issues. By doing so adverse “Software Architecture in Practice”, consequences of these risks are understood Addison-Wesley, 2003. and addressed resulting in a more likely 5. Buede, D., “The Engineering Design successful system development and of Systems”, Wiley Series in Systems operation. Normally there are no system Engineering, 2000. operational effectiveness and total cost of 6. Carney, D., Oberndoff, P., ownership tradeoffs being performed during “Integration and Interoperability Models these activities. Therefore, one might decide for System of Systems”, Carnegie Mellon there is no need to perform optimization of University, 2004. system operational effectiveness and total 7. Clements, P., Kazman, R., Klein, M., cost of ownership when adopting these “Evaluating Software Architectures:

Methods and Case Studies”, Addison- of Technology, 2007a. Wesley, 2001. 20. Jain R., Erol, O., Chandrasekaran, 8. Cloutier, R., “Applicability of A., “Proposing a System Integration Patterns to Architecting Complex Readiness Framework”, Working Paper, Systems”, Doctoral Dissertation, Stevens Stevens Institute of Technology, 2007b. Institute of Technology, 2006. 21. Jorgensen, R., Philpott, I., 9. Cloutier, R., Verma, D., “Applying “Architectural Abstractions”, INCOSE Pattern Concepts to Systems (Enterprise) Symposium, 2002. Architecture”, Journal of Enterprise 22. Kazman, R.., Nord, R.., Klein, M., Architecture, May 2006. “A Life-Cycle View of Architecture 10. Cloutier, R., Verma, D., “Applying Analysis and Design Methods”, SEI-CMU, the concept of patterns to systems 2003. architecture”, Systems Engineering 10(2): 23. Krishnamoorthy, V., Unni, N.K., 138-154, 2007. Niranjan, V., “Event-driven service- 11. Dogru A., Delcambr, S., Bayrak, C., oriented architecture for an agile and Christiansen, M., Tanik, M., “The scalable network management system”, development of an integrated system IEEE Conference on Next Generation Web design environment”, Proceedings of the Services Practices, August 2005. second international conference on systems 24. Kumar Harikumar, A., Lee, R., Chia- integration, ICSI, 1992, pp. 691-698. Chu C., Hae-Sool Y., “An event driven 12. Elias, G., Jain, R., “Exploring architecture for application integration Attributes for Systems Architecture using Web services”, IEEE Conference on Evaluation”, CSER, 2007. Information Reuse and Integration, 13. Evans, M. W., Marciniak, J., August, 2005. “Software Quality Assurance and 25. McCabe, R., Pollen, M., “Evaluating Management”, John Wiley & Sons, 1987. Architectures With System Attributes”, 14. Howard, N., Rolland, C., Qusaibaty, Software Productivity Consortium, 2004. A., “Process complexity: Towards a theory 26. Mil-Std-1388-1A, “Logistics of intent-oriented process design”, INFOS, Support Analysis”, DOD, Pars. 20, April 2004. 1983. 15. IEEE 1233, “IEEE guide for 27. Open Systems Joint Task Force, developing system requirements Open Systems Joint Task Force, specifications”, IEEE, 1998. http://www.acq.osd.mil/osjtf/, 2005. 16. IEEE 610-12, “IEEE Standard 28. Rossak, W., Prasad, S., “Integration Glossary of Software Engineering Architectures – a framework for systems Terminology”, IEEE Computer Society, integration decisions”, Proceedings of the 1990. IEEE international conference on systems, 17. IEEE 830, “IEEE recommended man, cybernetics, 1991, pp. 545-550. practice for software requirements 29. Smith, D., O’Brien, L., specifications”, IEEE, 1998. Kontogiannis, K., Barbacci, M., “The 18. ISO/IEC 15939, “Software Architect: Enterprise Integration”, SEI engineering – Software Measurement News, Vol. 5, No. 4, 2002. Process”, 2002. 30. Software Engineering Institute (SEI), 19. Jain, R., Chandrasekaran, A., Erol, Carnegie Mellon, “Quality Measures O., “A System Integration Process Model Taxonomy”, (SIPM)”, Working Paper, Stevens Institute http://www.sei.cmu.edu/str/taxonomies/

qm_tax_body.html, 2006. George M. Elias is a Systems 31. Systems Engineering Handbook v3, Engineering Doctoral Candidate at Stevens INCOSE, 2006. Institute of Technology. Mr. Elias is a 32. Verma, D., Johannesen, L.H., Systems Engineer at ITT Electronic Systems “Supportability assessment and evaluation in Clifton, New Jersey. Mr. Elias has an during system architecture development”, Undergraduate degree in Information INCOSE Symposium, Minneapolis, MN, Systems from Rutgers and New Jersey July 2000, pp. 699–706. Institute of Technology and a Masters in Biography Computer Science from Stevens Institute of Dr. Rashmi Jain is Associate Technology. Mr. Elias has research interests Professor of Systems Engineering at Stevens in Systems Architecture and Systems Institute of Technology. Dr. Jain has over 15 Engineering Attributes. years of experience of working on socio- Robert Cloutier (Rob) has over 20 years economic and information technology (IT) experience in systems engineering, software systems. Over the course of her career she engineering, and project management in has been involved in leading the both commercial and defense industries. His implementation of large and complex research interests include Systems systems engineering and integration Engineering Patterns and modeling complex projects. Dr. Jain is currently the Head of systems with UML/SysML, Reference Education and Research for International Architectures, and agent based technology Council of Systems Engineering (INCOSE). applied to systems engineering. Rob Her teaching and research interests include received his Ph.D. in Systems Engineering systems integration, systems architecture from Stevens Institute of Technology, and and design, and rapid systems engineering. also holds an M.B.A. from Eastern Dr. Jain is Head of Education and Research University, and a B.S. from the United of INCOSE. In this role she is leading the States Naval Academy. He is an Adjunct development of a reference Systems Professor for Eastern University and chairs Engineering curriculum. She holds Ph.D. the Rowan University Electrical and and M.S. degrees in Technology Computer Engineering Department Industry Management from Stevens Institute of Advisory Board. Rob belongs to the Technology. International Council on Systems Anithashree Chandrasekaran is a Engineering (INCOSE), is a member of the Doctoral Candidate in the School of Systems Technical Leadership Team. He is also an and Enterprise at Stevens Institute of active member of the Association of Technology. Her research interests includes Enterprise Architects, and IEEE. Finally, he Rapid Systems Development and its and his wife raise puppies for The Seeing processes, Development process Eye to serve as guide dogs to the blind. reengineering, Risk Management and i Modeling, System Integration, System Dr. Rashmi Jain is the primary author of this paper. All questions and comments should be addressed to Design and Architecture. She obtained her her B.E. in Electrical and Electronics ii Adapted from SYS 625 course notes, SEEM, Engineering from P.S.G. College of Stevens Institute of Technology, 2005 Technology, India. She obtained her M.S. in Systems Engineering from Stevens Institute of Technology. She is the president of Stevens INCOSE student chapter.