Spiral Acquisition of Software-Intensive Systems of Systems Barry Boehm, USC; Victor Basili, U. of Maryland; A. Winsor Brown, USC; and Richard Turner, George Washington U. and OSD/Software-Intensive Systems December 2003

1. Introduction Current trends toward transformation of warfare (and other large-scale competitive pursuits) into network-centric and knowledge-based systems of systems show great promise for competitive advantages over traditionally organized federations of largely independent components. However, these transformational systems of systems are critically dependent on the successful functioning of their computer software (Harned- Lundquist, 2003; Rechtin-Maier 1997]. This paper summarizes the benefits provided by software for such transformational systems and identifies a top-10 list of associated risks and challenges that need to be resolved in developing and evolving software-intensive systems. It then summarizes the WinWin Spiral Model discussed earlier in CrossTalk [Boehm-Hansen, 2001; Boehm-Port, 2001], and shows how its application can be used to mitigate the top-10 software-intensive system risks and challenges.

2. Software Benefits for Transformational Systems Managers of software-intensive system acquisitions and operations often wish that the software and its invisible risks and problems would just go away and leave them with simpler, tangible hardware configurations to manage. However, pure hardware- based solutions could not provide several key benefits that software can provide for complex transformational systems of systems. These include:  The need to accommodate many combinations of mission options. Trying to accommodate these in hardware leads to earlier freezing of option choices, more complex hardware production, inflexible human computer interfaces, and very expensive and time-consuming hardware field upgrades. Accommodating the options in software enables later option changes, simpler hardware, adaptable human-computer interfaces, and distributed electronic upgrades.  The need for rapid response to change. The pace of unforeseeable change from competitive threats, technology, stakeholder preferences, organizational realignments, and the environment continues to accelerate. The need to accommodate such change runs into much the same set of hardware difficulties and software opportunities as does the accommodation of many options. Also, many sources of software change such as new networking options, user interface devices, security devices, and tax or regulatory changes are often accommodated by commercial-off-the-shelf (COTS) software upgrades made by vendors needing to stay competitive.  The need for fielding of partial capabilities. Some options for doing this in hardware are available via line replaceable units and plug-compatible interfaces, but again with higher needs to pre-commit to interface choices. Current DOD policies emphasizing evolutionary acquisition [DOD, 2003] are much easier to accommodate via simpler hardware platforms and evolutionary software upgrades.

3. Associated Software-Intensive Systems of Systems (SISOS) Risks and Challenges. The benefits to transformational systems provided by software capabilities make such systems sufficiently software-intensive that software considerations need to be integrally addressed during system engineering. But along with the software benefits above come associated risks and challenges.

The ability to accommodate many combinations of options comes with the challenge that the more mission options there are to be accommodated, the more software there needs to be, and the longer it will take to deliver the software- intensive system of systems (SISOS). A large SISOS such as the national air traffic control system, the International Space Station, the Army Future Combat System, or a major integrated industrial manufacturing and supply chain management system will have over 10 million source lines of code (10 MSLOC, or 10,000 KSLOC) that need to be developed and integrated. SISOS managers are frequently surprised to see the first results of a software cost and schedule estimation model run, indicating that 10,000 KSLOC of software will take at least 9 years to develop using traditional methods. Most software cost-schedule models have calibrated relationships indicating that the calendar time required for software development scales roughly as the cube root of the size in KSLOC, or roughly

Development Time = 5 * cube root (KSLOC).

Table 1. Software Development Time vs. Size

Size (KSLOC) 300 1,000 3,000 10,000 Time (months) 33 50 72 108

Clearly, if the SISOS is needed quickly, replacements for traditional software methods are needed. These go from processes enabling more concurrent development to acquisition methods that are both less bureaucratic and more able to control massive concurrent development. The need for rapid response to change exacerbates these risks and challenges. Traditional requirements management and change control processes are no match for large volumes of change traffic across the multitudes of suppliers and operational stakeholders involved in a SISOS. Further challenges are that many of the sources of change (externally interoperating systems, COTS products) are outside the program’s span of control.

The criticality, software-intensiveness, and cross-cutting nature of many of these changes means that traditional project organizations with software element managers buried deep in the management structure will not meet the challenge of rapid and effective response to change. And traditional contracting mechanisms and incentive structures optimized around low-cost delivery to a fixed specification will have exactly the wrong effect on rapid cross-supplier adaptation to change. Technically, software architectures sacrificing ease of change for incremental performance gains will also make rapid change unachievable, and much more up- front work on architecture tradeoff analysis is needed to get the right balance among performance, dependability, ease of use, and adaptation to change.

The benefits of “free” software COTS adaptation to change also have their risks and challenges. COTS changes are determined by COTS vendors. Each time this happens, the SISOS integrators are presented with a difficult challenge over which they have limited control. [Meyers-Oberndorf, 2001] And on a SISOS, it will happen a lot. Four years of survey data from the annual Air Force/ Aerospace Corporation Ground Systems Architectures Workshops indicate that the average COTS product in the satellite ground systems domain undergoes a new release every 8-10 months. On a complex SISOS with dozens of suppliers making commitments to over 100 different COTS products, at least 10 new releases will impact the program every month, and the opportunities for proliferating incompatibilities are immense.

A major organizational-management implication of the need to coordinate supplier commitments to potentially-incompatible COTS and non-developmental item (NDI) software components is that the Total System Performance Responsibility (TSPR) acquisition structure is not viable for a SISOS. Leaving dozens of suppliers of component systems to make hundreds of commitments to incompatible COTS and NDI components will not work. On the other hand, centralized management of most supplier decisions will not work either. And there is a significant risk of supplier micromanagement if the SISOS system integrator is a contractor staffed by people more familiar with making detailed development decisions versus making acquisition leadership decisions. Thus, there is a need to balance the acquisition strategy to determine “how much commonality is enough” for each aspect of the SISOS. As we will see below, this type of question can be well addressed by risk analysis within the framework of the risk-driven spiral model.

The software benefits of enabling early fielding of partial capabilities are also accompanied by risks of overoptimizing on the early capabilities and overoptimistically assuming that any sort of software architecture and code can be easily modified later—for example, by rapidly developing insecure software and assuming that it can easily be made secure later. This assumption is invalid for most software, as is still shown by empirical data that the cost of software changes on large projects goes up by a factor of about 100 from requirements specification to post- deployment change [Boehm-Basili, 2001]. But this factor can be reduced significantly by thorough software architecting for change and risk management, as on the USAF/TRW CCPDS-R Project [Royce, 1998]. 4. WinWin Spiral Model Overview Current Department of Defense (DOD) acquisition policy in DOD Directive 5000.1 and DOD Instruction 5000.2 strongly emphasizes the use of evolutionary acquisition and spiral development [DOD, 2003]. Several versions of the spiral model have been used successfully on DOD and other systems; here we focus on the WinWin Spiral Model being used on probably the largest and most transformational system of systems under development today: the U.S. Army/ DARPA Future Combat Systems Program, with Boeing and SAIC serving as Lead System Integration contractors. The WinWin Spiral Model applies not only at the software level, but also at the systems level. Previous CrossTalk articles [Boehm-Hansen, 2001; Boehm-Port, 2001] have described the WinWin Spiral Model and applications of its use. Here we provide a short overview and update with respect to its application to SISOS. Detailed guidelines on its use are provided at the USC Center for Software Engineering’s MBASE web site at http://sunset.usc.edu/research/MBASE [USC-CSE, 2003].

Figure 1 summarizes the WinWin Spiral Model. Its major distinguishing feature is a cyclic process of concurrently engineering the increasingly detailed definition of a project’s products (requirements, architecture, design, deliverable system elements) and processes (plans, schedules, budgets, responsibilities), rather than sequentially engineering the products as in a waterfall process. Its radial dimension is the project’s cost or duration, whichever is more critical, and the major objective is to produce a satisfactory system at the lowest resulting cost or duration, primarily by avoiding unnecessary rework.

Its definition of a “satisfactory system” is that the system produces mutually satisfactory (win-win) results for all of its success-critical stakeholders (users, operators, acquirers, developers, suppliers, testers, interoperators, maintainers, higher managers, others as appropriate). A win-lose system outcome will generally turn into a lose-lose, as a losing success-critical stakeholder will see no reason to commit to the project’s success, or will try to become a winner at another stakeholder’s expense. Thus, the box at the left in Figure 1 indicates that the model strongly emphasizes identifying a system’s success-critical stakeholders; engaging them in identifying their win conditions (objectives, constraints, priorities, preferred solution elements); and negotiating, contributing to, reviewing, and committing to a win-win set of product and process elements. For a SISOS with many diverse stakeholders, the win- win process will be complex, but certainly preferable to a win-lose or lose-lose outcome.

The WinWin Spiral Model’s second main driver in Figure 1 is risk management. The spiral model does not produce a single one-size-fits-all process. It is rather a process model generator, in which the project’s particular risk patterns determine the order and level of detail for each activity and product. Thus, a waterfall process is a special case of the spiral for some risk patterns, and a series of exploratory prototypes ending with an operational working system is another. Thus, “unrolling” the spiral to produce an incremental waterfall or a Vee process model is a misapplication of the spiral model, particularly for the risk patterns of a SISOS (many diverse stakeholders, many options, emergent requirements, early fielding, rapid change). Risk management also helps determine “how much is enough?” of each activity or product, as will be discussed below in a SISOS context under Risk 3, Achievable Software Schedule.

The third key driver in Figure 1 is the set of anchor point milestones. These enable the concurrent engineering of many artifacts in a SISOS to be tracked and managed. Table 2 summarizes the content of the first two anchor point milestones, Life Cycle Objectives (LCO) and Life Cycle Architecture (LCA). The major new feature in the LCO and LCA anchor point milestones is the fourth key spiral model driver in Figure 1, the Feasibility Rationale. In preparing for an LCO or LCA milestone review, it is not sufficient for developers to prepare just Table 2. LCO and LCA Anchor Points -Risk-driven level of detail at each milestone Milestone Life Cycle Objectives (LCO) Life Cycle Architecture (LCA) Element Definition of  Top-level system objectives and scope  Elaboration of system objectives and scope Operational - System boundary by increment Concept - Environment parameters and assumptions  Elaboration of operational concept by - Evolution parameters increment  Operational concept System  Exercise key usage scenarios  Exercise range of usage scenarios Prototype(s)  Resolve critical risks  Resolve major outstanding risks Definition of  Top-level functions, interfaces, quality  Elaboration of functions, interfaces, quality System attribute levels, including: attributes by increment Requirements - Growth vectors - Identification of TBDs (to-be-determined - Priorities items)  Stakeholders’ concurrence on essentials  Stakeholders’ concurrence on their priority concerns Definition of  Top-level definition of at least one feasible  Choice of architecture and elaboration by System and architecture increment Software - Physical and logical elements and - Physical and logical components, relationships connectors, configurations, constraints Architecture - Choices of COTS and reusable software - COTS, reuse choices elements - Domain-architecture and architectural  Identification of infeasible architecture style choices options  Architecture evolution parameters Definition of  Identification of life-cycle stakeholders  Elaboration of WWWWWHH* for Initial Life-Cycle - Users, customers, developers, Operational Capability (IOC) Plan maintainers, interpreters, general public, - Partial elaboration, identification of key others TBDs for later increments  Identification of life-cycle process model - Top-level stages, increments  Top-level WWWWWHH* by stage Feasibility  Assurance of consistency among elements  Assurance of consistency among elements Rationale above above - Via analysis, measurement, prototyping,  All major risks resolved or covered by risk simulation, etc. management plan - Business case analysis for requirements, feasible architectures * WWWWWHH: Why, What, When, Who, Where, How, How Much paper requirements, architectures, or plans. They must also produce the evidence that these represent a feasible, worthwhile, and stakeholder-satisfactory combination of artifacts. This evidence may be produced via analyses, prototypes, models, simulations, benchmarks, or working production components. But if it is missing or inadequate, the review should not be considered as passed. Table 3 summarizes the pass-fail criteria for the LCO, LCA, and IOC (Initial Operational Capability) and LCA milestones (roughly equivalent to DOD Milestones A, B, and C). Note that Table 3 includes one way of passing the LCA milestone without a complete Feasibility Rationale, which is to cover any unresolved risks by a risk management plan. This is a good way to track progress toward passing an anchor point milestone: to track the ability to produce a credible Feasibility Rationale for each requirement or key performance parameter, and to identify its satisfaction as a risk to be managed if not.

Table 3. LCO, LCA, and IOC Pass/Fail Criteria -Documented by Feasibility Rationale

LCO LCA IOC For at least one architecture, a For a specific detailed An implemented architecture, an system built to that architecture architecture, a system built to operational system that has: will: that architecture will:  Realized the Operational  Support the core  Support the elaborated Concept Operational Concept Operational Concept  Implemented the initial  Satisfy the core  Satisfy the elaborated operational requirements Requirements Requirements  Prepared a system operation  Be faithful to the  Be faithful to the and support plan Prototype(s) Prototype(s)  Prepared the initial site(s) in  Be buildable within the  Be buildable within the which the system will be budgets and schedules in budgets and schedules in deployed for transition the plan the plan  Prepared the users,  Show a viable business  Have all major risks operators, and maintainers case resolved or covered by a to assume their operational  Have its key stakeholders risk management plan roles committed to support the  Have its key stakeholders Elaboration Phase (to LCA) committed to support the full life cycle Figure 2. The Cost of Unvalidated Requirements/Architecture/Plans Packages: 15-Month Architecture Rework Delay

Figure 2 shows an example of the need for a sound Feasibility Rationale. It shows what happened on a large TRW government contract during the early 1980’s, using a sequential waterfall acquisition approach. The initial mission analysis identified a need for a 1- second response time to user queries. This was passed along as a contract requirement and reviewed at a Software Requirements Review without any information on its achievability using the originally-proposed architecture, which was able to achieve 1- second response times for small workloads. In subsequently evaluating its performance on the large required workloads, the developers found that the original architecture could not scale up to meet even a 2-second response time.

The developers explored alternative architectures to meet the 1-second response time requirements. After about a year, they had developed and validated an architecture involving 27 super-midi computers rapidly caching data to keep it within 1 second of any user query. However, its cost was $100 million instead of the $30 million budgeted for the system. When the customers discovered this, they and the developers did some user interface prototyping and found that a 4-second response time was satisfactory for 90% of the queries, and that the other 10% were not critical anyway.

Once the response time requirement was changed to 4 seconds, the original architecture became feasible and affordable. But this was only resolved after a 15- month delay in architecture development and rework. Had the initial review required a Feasibility Rationale for the architecture and the 1-second requirement, this problem would have been either anticipated or much more quickly resolved. 5. Top-10 SISOS Risks and Spiral Mitigation Strategies Here is a prioritized top-10 list of SISOS risks, based on our SISOS experiences in the areas of C4ISR systems, space systems, the Army Future Combat Systems program, the U.S. National Air Traffic Control System, and commercial network-centric systems of system; along with the results of a number of DOD Tri- Service Assessment Initiative reviews [McGarry, 2003].

Risk 1. Acquisition Management and Staffing The biggest risk in acquiring a SISOS is the risk of committing to acquisition practices and strategies that may still work for some systems but are incompatible for a SISOS. This happens most often via legacy policies and cultures that assume that SISOS requirements can be predetermined and allocated to hardware, software, and humans before architecting the SISOS and contracting for its component systems. Examples are MIL-STD-1521, “Reviews and Audits,” which imposes a set of sequential reviews of the requirements, preliminary design, and detailed design on the SISOS process; and the Software Capability Maturity Model’s Requirements Management Key Process Area, which says “Analysis and allocation of the system requirements is not the responsibility of the software engineering group but is a prerequisite for their work.” [Paulk et al., 1994].

Actually, many of the requirements of a SISOS are emergent with development and use rather than prespecifiable. As seen in the discussion of Figure 2, feasible technical requirements emerge through development and prototyping experience. And feasible human-computer interface and decision support requirements emerge through software and system exercise and use.

The WinWin Spiral Model addresses this risk through its risk-driven concurrent engineering and evolutionary development of SISOS products and processes. Mature, highly precedented systems can be prespecified with low risk; immature systems or unprecedented combinations of systems may need several spiral cycles of risk resolution to get the right combination of requirements, architecture, system elements, and life cycle plans.

The second major risk element is the lack of rapid response to change that happens in traditional project organizations in which software expertise and decision authority is scattered at low management levels across the various project elements. As discussed in Section 3, for rapid and effective SISOS response to change, a project needs an organization with an integrating software and information processing leader reporting directly to the project manager, with strong software dotted-line management through counterpart software and information processing leaders reporting directly to each Integrated Product Team (IPT) leader and system-supplier project manager. The project also needs strong software networking within the SISOS IPT’s, which may include IPT’s for sensors, networks and communications, command and control, ground/sea/air/space vehicles, logistics, training, integration and test, modeling and simulation, and infrastructure. It further needs collaborative supplier integration and support of concurrent incremental development, as discussed below under Risk 4 (Supplier Integration) and Risk 5 (Adaptation to Rapid Change).

The third major risk is key staff shortages and burnout. Some sectors such as defense are challenged in competing with commercial industry for top software talent [Harned-Lundquist, 2003]. The key system and software personnel on a SISOS have little time to do so their assigned work after participating in all or most of the coordination meetings that a SISOS requires. These include meetings among IPT’s, supplier hierarchies, management hierarchies, customers, external interoperators, process groups, risk management groups, and numerous other specialty groups. And a SISOS evolutionary acquisition project can go on for years, leading to a high risk of key staff burnout.

Risk mitigation practices for this source of risk include reducing the risk impact via career path development and mentoring of junior staff to provide replacements for key personnel; and reducing the risk probability via such mechanisms as increment completion bonuses, flowdown of contract award fees to project performers, and recognition initiatives for valued contributions.

Risk 2. Requirements/Architecture Feasibility

The biggest risk here is committing to a set of requirements or to an architecture without validating its feasibility. Its nature and criticality were exemplified by the commitment to a 1-second response time requirement by the project illustrated in Figure 2. Often, the rework penalty is even worse if the project proceeds to implement an inappropriate architecture into hardware and software. In general, requirements/architecture infeasibilities with respect to quality factors such as response time, throughput, security, safety, interoperability, usability, or evolvability have the highest risk exposures. They are discussed further under Risk 6 (Software Quality Factor Achievability). The WinWin Spiral Model’s anchor point milestone pass/fail criteria and Feasibility Rationale explicitly address this risk. They prevent a project’s marrying its architecture in haste and having to repent at leisure – that is, if any leisure time is available.

Risk 3. Achievable Software Schedules Sometimes software cost is the most critical resource constraint, but for a SISOS, the large volume of software tends to put the software development schedule more on the project’s critical path than it is for simple systems. This discussion of Table 1 in Section 3 clearly shows the magnitude of the risk. The WinWin Spiral Model’s anchor point milestones and Feasibility Rationale again directly address this issue. Schedule Feasibility should be addressed both by software cost and schedule estimation models (using and comparing the results of two independent models is a good practice), and by explicit development and critical-path analysis of project activity networks (probabilistic activity networks are more conservative and realistic). Techniques for accelerating the project’s schedule below the times estimated in Table 1 include the major techniques for increasing overall software productivity (software reuse, COTS, reducing rework, top personnel, better tools), plus two techniques focused on improving schedule directly.

Architecting and organizing for massive concurrent development If the SISOS could be architected so well that supplier-developed components could instantly plug and play, Table 1 indicates that organizing the project into many 300-KSLOC components would get the job finished in 33 months vs. 108. Unfortunately, the effects of architectural imperfection and continuing change makes seamless integration an unrealistic objective, but the relative gains of reducing integration rework are worth trying to achieve. The other main problem is the fraction of development time it takes to produce a fully validated integration architecture, which increases with the amount of software needing to be integrated. Figure 3, from [Boehm-Turner, 2004], shows how this tradeoff between architecting time before finalizing SISOS supplier specifications and rework time can be analyzed by the well-calibrated COCOMO II Architecture and Risk Resolution Factor [Boehm, 2000]. It shows that for a 10,000-KSLOC SISOS, the “sweet spot” minimizing the sum of architecting time and rework time occurs at about 37% of the development time for architecting, with a relatively flat region between 30% and 50%. Below 30%, the penalty curve for premature issuance of supplier specifications is steep: a 17% investment in architecting yields a rework penalty of 48% for a total delay of 65% compared with a rework penalty of 20% and a total delay of 50% for the 30% architecting investment. This curve and its implications were convincing enough to help one recent SISOS to add 18 months to its schedule to improve the architectural specifications used by the suppliers. Figure 3 How Much Architecting is Enough?

2. The Schedule As Independent Variable (SAIV) Process The SAIV process, also described in detail in CrossTalk [Boehm et al, 2002], is a special case of the WinWin Spiral Model that applies when there is a strong need to produce an Initial Operational Capability (IOC), by a particular date, but the exact nature of the IOC is not well specifiable in advance. This situation fits a good many SISOS acquisitions. The SAIV process, which is compatible with the WinWin, incremental, and concurrent development processes discussed above, operates as follows:  Work with stakeholders in advance to achieve a stored product vision, realistic expectations, and prioritized requirements;

10 KSLOC  Estimate the maximum size of software buildable with high confidence within the available schedule;  Define a core-capability IOC content based on priorities, end-to-end usability, and need for early development of central high-risk software;  Architect the system for ease of dropping or adding borderline-priority features;  Monitor progress; add or drop features to accommodate high-priority changes or to meet schedule.

The SAIV process has been used successfully to date on one recent very large SISOS. For example, lower-priority requirements originally within the IOC set such as automatic real-time natural language translation were deferred to create an achievable core- capability IOC.

Risk 4. Supplier Integration As the SISOS system suppliers integrate their architectures and components and jointly respond to changes, they will need to share information and rapidly collaborate to negotiate changes in their products, interfaces, and schedules. As these suppliers are traditionally wary competitors used to performing to fixed specifications, there are significant risks of slow and inflexible adaptation to SISOS sources of change. The well- calibrated COCOMO II “Team Cohesion” factor yields an added 66% in effort and up to 30% in added schedule between seamless team cohesion and very low team cohesion. In mitigating these risks, the win-win aspects of the WinWin Spiral Model became paramount. Strategies for achieving win-win supplier participation include making them first-class stakeholders in negotiating their parts of SISOS objectives, constraints, priorities, and preferred alternatives in Figure 1; establishing early training and teambuilding activities for selected suppliers; pro-actively identifying needs for supplier collaboration and networking of their lead software and system architects; and establishing contract provisions and award fee criteria for effective collaboration in such areas as schedule preservation, continuous integration support, cost containment, technical performance, architecture and COTS compatibility, program management and risk management. An example award fee evaluation process and criteria are provided in [Reifer-Boehm, 2003].

Risk 5. Adaptation to Rapid Change As discussed in Sections 2 and 3, adaptation to change is a SISOS necessity, but continuous adaptation to change across dozens of suppliers, IPT’s, and external interoperators can be completely destabilizing. Within the WinWin Spiral Model, the best strategy for balancing change and stability is incremental development. As seen in Figure 1, the spiral cycles combine architecting and development early, with key parts of high-risk elements developed in a Build 1 and used as part of the Feasibility Rationale for the SISOS LCA milestone. The post-LCA builds have the suppliers concurrently developing increments of capability within the validated architecture established at the LCA milestone. In order to stabilize development, proposed changes are deferred as much as possible to later builds, and the SAIV process can be used to drop lower-priority features not needed by other suppliers in order to keep on a common schedule. This process is similar to the Microsoft synchronize-and-stabilize process discussed in [Cusumano- Selby, 1995], and works best if there is some slack built into the end of each build. Other strategies for adaptation to rapid change include pro-active technology-watch, COTS- watch, and interoperability-watch activities to anticipate, experiment with, and prepare to respond to sources of change; cross-supplier and cross-IPT networking for rapid team adaptation; change-anticipatory architectures; and agile change control and version control capabilities.

Risk 6. Software Quality Factor Achievability As discussed under Risk 2, software quality factors are the most difficult sources of SISOS requirements/architecture feasibility risk. These factors (response time, throughput, security, safety, interoperability, usability, evolvability, others) are strongly scenario-dependent and interact in complex ways that cut across supplier and IPT boundaries. A good example is a vehicle self-defense timeline, which imposes performance requirements and tradeoffs across sensor, networking, fusion, command- control, vehicle maneuvering, electronic warfare, defensive weapon employment, and software infrastructure elements of a SISOS, along with additional tradeoffs between performance, security, usability, safety, and fault tolerance. Another example is the 1- second response time requirement discussed with Figure 2. A key WinWin Spiral strategy for quality factor achievability is to establish a quality factor trade space by replacing single-value quality factor requirements by a range between acceptable and desired values (for example, 1-second response time is desired but 4-second response time is acceptable). This provides the system and software architects with enough degrees of freedom to converge to a mutually acceptable or win- win combination of achievable quality factor values. Another key strategy is the use of the SEI Architecture Tradeoff Analysis Method (ATAM) for stakeholder establishment, prioritization, and assessment of the quality factor values achievable with a given architecture; and identification of strategies to bring the quality factor values up to acceptable levels [Clements et al., 2002].

Risk 7. Product Integration and Electronic Upgrade SISOS software needs to be integrated across supplier hierarchies, IPT domains, computing platforms, vehicle platforms, critical scenarios, operational profiles, system modes, and friendly-to-adversarial environments. Having too much or too little concurrency across these dimensions of integration can cause significant delays and rework. And rework needs to be fed back to developers, who will already be busy developing the next build, causing even further SISOS delays. The benefits of electronic software upgrades discussed in Section 2 come with several types of version-mismatch risks. Examples are putting the wrong version’s upgrades onto a platform in the field, or having part of the fielded platforms running version 3.5 of the SISOS software while the others are running version 3.4. These mismatches can cause such problems as software crashes, communication outages, out- of-synchronization data, or mistaken decisions. Key WinWin Spiral strategies for addressing these risks include up-front involvement of software-oriented integration, test, supportability, and maintenance personnel in win-win negotiations affecting their ability to perform; early establishment, usage, and incremental growth of software and system integration laboratories for the overall SISOS and for its key IPT areas; and architecting the software to accommodate continuous operation and synchronized upgrades (for example, by enabling parallel operation of old and new versions while validating and synchronizing an upgrade).

Risk 8. Software COTS and Reuse Feasibility Sections 2 and 3 discussed the benefits of “free” software COTS adaptation to change, and some of the SISOS risks involved in synchronizing COTS upgrades across a wide variety of independently-evolving COTS products. Not only do COTS products typically undergo new releases every 8-10 months, but also their vendors typically continue support of only the most recent 3 releases. For a SISOS, with many suppliers developing ambitious capabilities within tight budgets and schedules lasting over 30 months, there is a temptation not to upgrade the COTS products and to deliver unsupported versions [Basili-Boehm, 2001]. In one case, we encountered a large system delivered to the customer and users with 55 of its 120 COTS products operating on unsupported releases. WinWin Spiral mitigation strategies for COTS related risks include contract provisions prohibiting the delivery of unsupported COTS components; including key COTS vendors as strategic partners and success-critical stakeholders; pro-active COTS- watch experimentation and participation in user groups; operation of a SISOS-wide COTS product and version tracking and compatibility analysis activity; and development and execution of a strategy for periodic synchronized COTS upgrades. Software reuse is a powerful strategy of reducing software cost and schedule, but frequently estimates of 80% software reuse on a suppliers’ system turn out to be more like 40% once the different natures of the SISOS and thelegacy software are recognized, presenting serious budget and schedule overrun risks. WinWin Spiral strategies for mitigating software reuse risks include validating the compatibility of supplier reuse components with SISOS product line architectures, constraints, and assumptions; continuous data analysis of actual vs. estimated reuse parameters and recalibration of reuse estimates; and performing root cause analyses of reuse successes and failures. Further reuse and product line best practices can be found in [Reifer, 1997] and [Clements-Northrop, 2002].

Risk 9. External Interoperability Large SISOS are likely to require interoperability with over 100 independently evolving external systems (and even more, if COTS components are included). As with COTS, there are major risks of some or all of the SISOS systems getting out of synchronization with these external systems. And as with COTS, major WinWin Spiral strategies for risk mitigation include establishing pro-active stakeholder win-win relationships with critical interoperability systems, including memoranda of agreement on interoperability preservation; participation in analyses and negotiations of the impact of new interoperability standards and services such as the Global Information Grid, Joint Technical Architecture, and Net-Centric Operations and Warfare Reference Model; operating an external systems interoperability tracking and compatibility analysis activity; and inclusion of external interoperability in modeling, simulation, integration, and test capabilities.

Risk 10. Technology Readiness The scale and mission scope of a SISOS may far exceed the capabilities of technologies that may have demonstrated considerable maturity in smaller systems or friendlier environments. Examples are adaptive mobile networks, autonomous agent coordination capabilities, sensor fusion capabilities, and software middleware services. Assuming that a technology’s readiness level on a smaller system will be valid for a SISOS runs a major risk of performance shortfalls and rework delays. Primary WinWin Spiral risk mitigation strategies focus on satisfaction of a Feasibility Rationale for the key advanced technologies, including the exercise of models, simulations, prototypes, benchmarks, and working SISOS applications representing SISOS normal, crisis, and adversarial scenarios. Risk management strategies include identification of fallback technology capabilities should key new technologies prove inadequate for SISOS usage.

6. Conclusions Competitive pressures for increased integration and high performance of commercial, industrial, and public services capabilities such as military defense or homeland security are leading many organizations toward acquisition of multi-domain and multi-supplier systems of systems. The increasing number of options and frequency of changes encountered by these systems of systems are making them increasingly software-intensive. The acquisition of such software-intensive systems of systems (SISOS) has many differences from acquisition of traditional systems. Besides the significantly larger number of options, changes, suppliers, and domains to accommodate, there are significantly larger numbers of external interfaces, COTS products, coordination networks and meetings, operational stakeholders, and emergent vs. prespecifiable requirements. These differences in scope, scale, and dynamism have made traditional acquisition practices increasingly inadequate. For example, the Total System Performance Responsibility acquisition approach is not viable for suppliers of component systems within a system of systems, nor is centralized micromanagement of these suppliers. Current initiatives toward evolutionary acquisition and spiral development are promising, but many new practices need to be worked out. Experiences on several SISOS have identified both a set of top-10 SISOS risks and corresponding risk mitigation strategies currently being applied on some SISOS. The top 10 SISOS risks are:

1. Acquisition management and staffing 2. Requirements/architecture feasibility 3. Achievable software schedules 4. Supplier integration 5. Adaptation to rapid change 6. Software quality factor achievability 7. Product integration and electronic upgrade 8. Software COTS and reuse feasibility 9. External interoperability 10. Technology readiness

Applying their corresponding risk mitigation strategies within a WinWin Spiral development and evolutionary acquisition process is meeting with some success, and appears to be a good starting point for identifying and coping with SISOS risks. But much more experience on SISOS acquisition and development will be needed to achieve mature SISOS acquisition capabilities

7. References

[Basili-Boehm, 2001] V. Basili and B. Boehm, “COTS-Based Systems Top-10 List,” Computer, May 2001, pp.91-93.

[Boehm et al., 2000] B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, 2000.

[Boehm-Basili, 2001] B. Boehm and V. Basili, “Software Defect Reduction Top- 10 List,” Computer, January 2001 pp. 135-137.

[Boehm-Hansen, 2001] B. Boehm and W. Hansen, “The Spiral Model as a Tool for Evolutionary Acquisition,” CrossTalk, May 2001, pp. 4-11.

[Boehm-Port, 2001] B. Boehm and D. Port, “Balancing Discipline and Flexibility with the Spiral Model and MBASE,” Cross Talk, December 2001, pp. 23-28.

[Boehm et al., 2002] B. Boehm, D. Port, L. Huang, and W. Brown, “Using the Spiral Model and MBASE to Generate New Acquisition Process Models: SAIV, CAIV, and SCQAIV,” CrossTalk, January 2002, pp. 20-25.

[Boehm-Turner, 2004] B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley, 2004.

[Clements et al., 2002] P. Clements, R. Kazman, and M. Klein, Evaluating Software Architectures: Methods and Case Studies, Addison Wesley, 2002.

[Clements-Northrop, 2002] P. Clements and L. Northrop, Software Product Lines: Practices and Patterns, Addison Wesley, 2002.

[Cusumano-Selby, 1995] M. Cusumano and R. Selby, Microsoft Secrets, The Free Press, 1995. [DoD, 2003] DOD Directive 5000.1, “The Defense Acquisition System” and DoD Instruction 5000.2, “Operation of the Defense Acquisition System,” U.S. Department of Defense, May 12, 2003.

[Harned-Lundquist, 2003] D. Harned and J. Lundquist, “What Transformation Means for the Defense Industry,” The McKinsey Quarterly, November 3, 2003, pp.57-63.

[Meyers-Oberndorf, 2001] B.C. Meyers and P. Oberndorf, Managing Software Acquisition: Open Systems and COTS Products, Addison Wesley, 2001.

[Paulk et al., 1994] M. Paulk, C. Weber, B. Curtis, and M. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison Wesley, 1994.

[Rechtin-Maier, 1997] E. Rechtin and M. Maier, The Art of Systems Architecting, CRC Press, 1997 (2nd ed: Maier-Rechtin, 2001).

[Reifer, 1997] D. Reifer, Practical Software Reuse, Wiley, 1997.

[Reifer-Boehm, 2003] D. Reifer and B. Boehm, “A Model Contract/Subcontract Award Fee Plan for Large, Change-Intensive Software Acquisitions,” USC-CSE Tech Report USC-CSE-03-511, April, 2003.

[Royce, 1998] W.E. Royce, Software Project Management, Addison Wesley, 1998.

[USC-CSE, 2003] USC Center for Software Engineering, “Guidelines for Model-Based (System) Architecting and Software Engineering,” 2003: http://sunset.usc.edu/research/MBASE