ISA 201 Information System Acquisition

11/15/2016

Lesson 12 Systems Design Considerations

1 11/15/2016

Learning Objectives Today we will learn to:

• Identify at least three reasons why interface management and documentation are important to the development of DoD Software Intensive Systems. • Identify best practices in order to reduce software assurance vulnerabilities. • Identify the underlying reasons for DoD policy regarding employing an Open Systems Approach (OSA) and its associated principles. • Recognize potential areas of software safety issues and risks when reviewing acquisition documents. • Explain the characteristics and importance of sound data management practices during the development process. • Describe why continuing the spectrum management process throughout the lifecycle of an IT program is critical. • Recognize the benefits of using Open Source Software in your systems design. • Discuss how designing for accessibility impacts software development for DBS and DoD Infrastructure systems

Systems Design 3

Lesson Overview Lesson Plan

• Software Assurance • Interface Management Software • Software Safety Assurance

• Open Systems Approach (OSA) Data Interface Management Management • Open Source Software • Data Management Accessibility

• Spectrum Management Software Spectrum • Accessibility Safety Management Open Systems and • COTS Open Source, COTS • Design Practices Exercise

DAG Chapter 4 Systems Engineering Design Considerations

Systems Design 4

2 11/15/2016

Design Practices

Systems Design 5

Lesson Overview Lesson Plan Status • Software Assurance • Interface Management • Software Safety • Open Systems Approach (OSA) • Open Source Software • Data Management • Accessibility • Spectrum Management • COTS • Design Practices Exercise

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 6

3 11/15/2016

Software Assurance Defined

• Software Assurance—the level of confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at any time during its life cycle, and that it functions in the intended manner [CNSSI No. 4009]. • Software Assurance—The level of confidence that software functions as intended and is free of vulnerabilities, either intentionally or unintentionally designed or inserted as part of the software throughout the lifecycle (NDAA 2013 Section 933). • Software assurance focuses on the management of risk and assurance of safety, security, and dependability within the context of system and software life cycles [ISO/IEC 15026].

Software Assurance in Acquisition: Mitigating Risks to the Enterprise, Building Security In, Oct 2008 Systems Design 7

Software Assurance & Public Law

• Public Law 111-383, National Defense Authorization Act (NDAA) for Fiscal Year 2011, section 932, Strategy on Computer Software Assurance: - Required report back to congress on SwA status • Public Law 112-239-January 2, 2013, NDAA for Fiscal Year 2013, Section 933, Improvements in Assurance of Computer Software Procured by the Department of Defense: - A research and development strategy to advance capabilities in software assurance and vulnerability detection - The state-of-the-art of software assurance analysis and test - How the Department might hold contractors liable for software defects or vulnerabilities • Public Law 113-66, NDAA for Fiscal Year 2014, Section 937, Joint Federated Centers for Trusted Defense Systems for the Department of Defense: - Establishes Joint Federated Assurance Center (JFAC) which provides vulnerability analysis and mitigation support to programs Systems Design 8

4 11/15/2016

Software Assurance & DoD Policy

• DoDI 5000.02 Operation of the Defense Acquisition System - Baselines SwA policy for covered systems - Requires automated SwA tool use and SwA practice across the DoD Acquisition life cycle • DoDI 5200.44 Trusted Systems and Networks mandates: Automated SwA tool use across the life cycle - Risk management of mission-critical functions and components - Testing and Evaluation, to detect HW/SW vulnerabilities - DoD-unique application-specific integrated circuits (ASICs) must be procured from trusted certified suppliers - Plans and mitigations documented in program protection and information assurance activities

Systems Design 9

Building Secure Software

• 75% of current hacks target applications* • SW must be able to:** - Resist Attack or withstand anticipated attacks - Recover rapidly and mitigate damage from attacks • Keys to secure software:** - Security focused development—processes that help developers root out and remove exploitable defects…or prevent entirely - Security enhanced acquisition—processes that address risks in the supply chain - Build Security In–https://buildsecurityin.us-cert.gov/

“Functional Correctness must be exhibited even when SW is subjected to hostile conditions; therefore, claims about system reliability, integrity and safety must include provisions for built-in security of enabling SW.”**

Sources: *Gartner; **DHS National Cyber Security Division Systems Design 10

5 11/15/2016

Primary Causes of Software Vulnerabilities

• Lack of Developer Motivation: Because consumers reward software vendors for being first to market and adding new features to their products, rather than for producing software that is better or more secure, vendors have no financial incentive to do the latter. • Lack of Developer Knowledge: Software is so complex; it exceeds the human ability to fully comprehend it, or to recognize and learn how to avoid all of its possible faults, vulnerabilities, and weaknesses. This factor, combined with lack of developer motivation, is often used as an excuse for not even teaching developers how to avoid those faults, vulnerabilities, and weaknesses that are within their ability to comprehend and avoid. • Lack of Technology: Current tools are inadequate to assist the developer in producing secure software or even to reliably determine whether the software the developer has produced is secure.

Andy Ozment, “Research Statement” [web] (Cambridge, UK: University of Cambridge,).

Systems Design 11

Building Security In

Vulnerable software may permit the following: • Unintentional errors leading to faulty operations that result in destruction of information or major disruption of operations • Intentional insertion of malicious code intent on loss of life, destruction of information, major disruption of operations, or even destruction of critical infrastructure • Theft of vital information that is sensitive or classified • Theft of personal information • Changed product, inserted agents, or corrupted information. Software Assurance in Acquisition: Mitigating Risks to the Enterprise, Building Security In, Oct 2008

“Ultimately, we believe that there is no alternative to making security a part of the software contracting. Currently, we believe that there are serious misunderstandings about the security of code being delivered under many software development contracts. This can only lead to expensive litigation and a decision made by individuals with little software experience or understanding.” The Open Web Application Security Project (OWASP) Foundation

Systems Design 12

6 11/15/2016

Lesson Overview Lesson Plan Status

• Software Assurance • Interface Management • Software Safety • Open Systems Approach (OSA) • Open Source Software • Data Management • Accessibility • Spectrum Management • COTS • Design Practices Exercise

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 13

Interface Management Defined

• Interface - A boundary across which two independent systems meet and act on or communicate with each other • Interface Management - The management of communication, coordination and responsibility across a common boundary between two or more organizations, phases, or physical entities which are interdependent. - All of the interfaces between co-functioning items need to be identified and documented so that their integrity may be maintained through a disciplined configuration control process.

Systems Design 14

7 11/15/2016

Interface Management Imperatives

• Design/define and verify interfaces early in the Product Life Cycle • Control interfaces to ensure they are within designated boundaries • Design interfaces into program hierarchy • Develop an Interface Management Plan (IMP) • Develop Interface Control Working Groups (ICWGs) • Document interfaces and make available to stakeholders through Interface Control Documents (ICDs)

Systems Design 15

Interface Management Plan (IMP)

An IMP is a part of a configuration management plan that: • Documents a system’s internal and external interfaces and their requirement specifications • Identifies preferred and discretionary interface standards and their profiles • Provides justification for selection and procedure for upgrading interface standards • Describes the certifications and tests applicable to each interface or standard

Systems Design 16

8 11/15/2016

ICDs and ICWGs

• Interface Control Documents (ICDs) - Documents interfaces - Formal means of establishing, defining, and controlling interfaces and for documenting detailed interface design definition - For each of the interface types, the ICD typically exists between configuration items to establish and specify interface definition and design • Interface Control Working Group (ICWG) - A specialized technical working group comprised of appropriate technical representatives from the interfacing activities and other interested participating organizations. - Serves as a forum to develop and provide interface requirements, as well as, focus on interface detail definition and issues - Detailed description and requirements are provided in the ICWG Charter.

Systems Design 17

Interface Management Process Flow

From MIL-HDBK-61A Systems Design 18

9 11/15/2016

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 19

Software Safety

• Both a subset of Software Development and Software System Safety • The process of identifying Safety-Critical Software and assessing the criticality • A series of requirements, design, and verification activities used to demonstrate that software will meet the customer’s safety requirements to an acceptable level of confidence.

Systems Design 20

10 11/15/2016

Software Safety Failure Example

The British destroyer H.M.S. Sheffield was sunk in the Falkland Islands war. According to one report, the ship's radar warning systems were programmed to identify the Exocet missile as "friendly" because the British arsenal includes the Exocet's homing device and allowed the missile to reach its target, namely the Sheffield.

Systems Design 21

Software Safety Definitions

• Software System Safety—The optimization of system safety in the design, development, use, and maintenance of software systems and their integration with safety-critical hardware systems in an operational environment. • Safety-Critical—A term applied to a function, condition, event, operation, process, component, or other element of a system whose proper recognition, control, performance or tolerance is essential to safe system operation or use (e.g., safety- critical function, safety-critical path, safety critical component). • Safety-Critical Software Function (SCSF)—A function, whose proper execution is necessary to reduce the risk of a hazard OR if performed incorrectly, performed out- of sequence, or not performed could contribute to a hazard, compromise a safety mitigation feature, or result in a diminished level of safety. • Mishap—An unplanned event or series of events resulting in death, injury, occupational illness, or damage to or loss of equipment or property, or damage to the environment. • Hazard—Any real or potential condition that can cause: 1. Injury, illness, or death to personnel; 2. Damage to or loss of a system, equipment or property; or 3. Damage to the environment. Systems Design 22

11 11/15/2016

Software Safety Ground Rules

• Software is a hazard cause - Software is a causal factor or contributes to a hazard - Two types of hazardous functions - Mitigating or “Must Work” functions (e.g. flight controls, …) - Contributing or “Must Not Work” functions (e.g., premature missile firing, …) • Software hazard contributors result from inadequate System and Software design and requirements - Caused by lack of design foresight, system development errors, inadequate requirements specifications, … - Hardware failures can modify intended software behavior • Traditional Independent Verification & Validation (IV&V) does not address Software System Safety - IV&V is focused on all mission critical software functions and not traditionally focused on safety-critical software functions - Software System Safety focuses on safety-critical software functions as hazard causes

Systems Design 23

Safety of Software

Software System Safety Software Safety System Safety and Risk Assess Activities Software Safety Development Assurance Activities • System Safety Analyses to include software & hazard reports • Participate in Hazard Analysis to identify specific S/W causes and faults • Perform hazard risk assessments • Assist in identification of safety-critical Identify System Safety requirements • software functions, SCRs, & criticality Identification of safety critical software functions, safety-critical software requirements • Flag SCRs & criticality in requirements (SCRs), and associated criticality traceability database • Identification of risk mitigation and perform risk • Trace requirements to design & code assessment • Perform development assurance analyses and • Produce Safety Assessment Report tests to ensure software meets (Level of Rigor) LOR & (Safety Integrity Level) SIL • Provide V&V evidence to System Safety for final risk assessment Systems Design 24

12 11/15/2016

Software Safety Lessons Learned

• Most Software Safety problems come from vague or incomplete requirements that are misunderstood or misinterpreted. • Legacy/Reused software must be included in Software System Safety Analyses • Perform Software System Safety Analyses early in program lifecycle • Safety-Critical Software requirements derived from Software System Safety Analyses must be included and identified as Safety-Critical in the Software Requirement Specification (SRS)

Systems Design 25

Software Safety Lessons Learned (Continued)

• Correct implementation of software exception handlers MUST BE VALIDATED (exceptions are runtime errors with undesired results or that disrupt normal program flow—handlers are built to assist the system to safely counteract the exception/error) • Safety concerns from Software System Safety Analysis must be addressed in Software Test Plans and Procedures (Boundary value testing, Full integration end-to-end testing) • Safety-Critical software must undergo full integration end-to-end testing, all software must be functioning and not inhibited

Systems Design 26

13 11/15/2016

Lesson Overview Lesson Plan Status

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 27

Defining “Open”

• Open Source Software, Open Standards, Open System Architecture • Open Source Software—Copyright license terminology. End user granted specific rights to access, review, modify, distribute software without seeking permission from the copyright holder or paying license fees. • Open Standards—A general, descriptive term referring to publicly available standard for design. • (Modular) Open Systems Architecture (OSA)—OSA is a DoD- centric business and technical strategy for developing or modernizing (technology/weapons) systems that relies on modular design of components and utilizes open standards and specifications.

Systems Design 28

14 11/15/2016

Open Systems Approach/Architecture (OSA)

Open Systems Architecture (OSA)*—OSA is a DoD-centric business and technical strategy for developing or modernizing (technology/weapons) systems that relies on modular design of components and utilizes open standards and specifications.

*Previously called MOSA: Modular Open Systems Approach

Systems Design 29

OSA Characteristics

• An integrated business and technical strategy that: - employs modular design, - uses widely supported and consensus-based standards for their key interfaces, and - has been subjected to successful validation and verification tests to ensure the openness of their key interfaces. • The DoD preferred approach for implementation of open systems is Open Systems Architecture (OSA).

DoDI 5000.02, 7 Jan 2015 states: “Program managers are responsible for applying open systems approaches in product designs …”

Systems Design 30

15 11/15/2016

OSA Benefits

• OSA is a strategy for developing a new system or modernizing an existing one. • OSA assists acquisition and engineering communities to design for affordable change, employ evolutionary acquisition and spiral development, and develop an integrated roadmap for system design and development. • Basing design strategies on widely supported open standards increases the chance that future changes to the system will be integrated in a cost-effective manner.

http://www.acq.osd.mil/se/initiatives/init_osa.html

Systems Design 31

Example Open Systems Approach (OSA)

http://acqnotes.com/acqnote/careerfields/modular-open-systems-approach Systems Design 32

16 11/15/2016

Open Systems Requirements

Open system features may be specified as: • design requirements (e.g., mandated open standards and protocols); • derived requirements (e.g., need for open interfaces to facilitate interoperability); • design constraints (e.g., need to adhere to open interface specifications as system components are designed); • architectural attributes (e.g., need for an adaptable, upgradable, and reconfigurable system architecture); • design considerations (e.g., taking into consideration modular and open systems design benefits and concerns); and • business strategies to gain access to competitive sources of supply and effectively manage technological obsolescence.

Systems Design 33

DoD OSA Guidebook

• The guidebook provides recommendations for writing an OSA-based statement of work, guidance on special interest requirements, recommended contract line items, and guidance on obtaining intellectual property and data rights to support full life-cycle competition and recommended CDRLs.

Systems Design 34

17 11/15/2016

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 35

Free and Open Source Software (FOSS)

• Open source doesn't just mean access to the source code: - Free redistribution - Source code - Derived works - Integrity of the author's source code - No discrimination against persons or groups - No discrimination against fields of endeavor - Distribution of license - License must not be specific to a product - License must not restrict other software - License must be technology-neutral Opensource-http://opensource.org/ Systems Design 36

18 11/15/2016

FOSS Examples

• Well known: - Mozilla (Firefox, Thunderbird, Sunbird, Seamonkey) - Java - Open Office - Linux - Apache - Perl - Python - Ruby

Systems Design 37

Who's Using FOSS

• Every branch of US government • Google • Yahoo • Ticketmaster • Hackers • IBM • Sun • Oracle • Regular people—you!

Systems Design 38

19 11/15/2016

FOSS Advantages

• Try Before You Buy • Security (also a disadvantage) • Quality - A software package created by a handful of developers, or a software package created by thousands of developers? • Cost • Support Options • Interoperability • Code Coverage Tools

Systems Design 39

FOSS Disadvantages

• Security (also an Advantage) • Probably not optimized for the functions you need performed. • May require sophisticated professional guidance on licensing issues • May have little documentation • “Hidden” costs - Hosting, training, etc. • Project longevity/support - Developers could lose interest and leave • No one is obligated to help you

Systems Design 40

20 11/15/2016

What's the difference?

• COTS • GOTS • FOSS • Freeware

Systems Design 41

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 42

21 11/15/2016

Data Management

• Definition - The process of applying policies, procedures, and tools for the identification and control of data requirements, for assuring the adequacy of data and for facilitating the timely, economical acquisition and availability of data, including digital delivery or access. In simple terms, DM is the process for the acquisition of data (access or delivery) through contractual vehicles, so that data is available for use by authorized users. The type of data to which this applies includes research and development, acquisition, and logistics information. • Critical to the process of system design and development. • Data management and standardization reduces the cost, complexity, and overall level of resources expended on the development of software and computer system data components

Systems Design 43

Software Development and Data Rights

• Data rights refer to intellectual property regarding the use of the data developed, accessed, and/or delivered under a Government contract. • Involve proprietary, restrictive, Government purpose, unlimited, limited, and may include patents, copyrights, and other data right provisions. • Necessary in the determination of release, duplication, and disclosure of technical data. • Data rights are generally determined by whose money is used in the development of the data. - If the data is developed with Government funding, then the Government has the right to access and receive the data with unlimited rights. - If data is developed with private sector funding, the Government, generally, will be allowed Government Purpose Rights. • The FAR 52.227 and DFARS 252.227 and other related sections set forth the policies, procedures, and implementing instructions relating to the requirements for the acquisition of technical data and computer software. • The Government should review the validity of any asserted restriction on technical data under a contract before acceptance of the data, but not later than three years after data delivery or final payment, whichever is later. Source: Acquisition Data Management: Body of Knowledge (acc.dau.mil)

Systems Design 44

22 11/15/2016

Data Management Policy

• Data is an essential enabler of network-centric warfare (NCW) and shall be made visible, accessible, and understandable … • Data assets shall be made visible by creating and associating metadata (“tagging”) … • Data assets shall be made accessible by making data available in shared spaces. • Data assets shall be made understandable by publishing associated semantic and structural metadata in a federated DoD metadata registry • To enable trust, data assets shall have associated information assurance and security metadata, and an authoritative source … shall be identified … • Data interoperability shall be supported by making data assets understandable and by enabling business and mission processes to be reused where possible • Semantic and structural agreements for data sharing shall be promoted through communities (e.g., communities of interest (COIs)), … • Data sharing concepts and practices shall be incorporated into education and awareness training and appropriate DoD processes.

Sharing Data, Information, and Information Technology (IT) Services in the Department of Defense August 5, 2013

Systems Design 45

Lesson Overview Lesson Plan Status

23 11/15/2016

Section 508 Amendment To The Rehabilitation Act Of 1973

What is the purpose of this law?

The purpose of Section 508, Title 29 U.S. Code paragraph 794.d, is to provide disabled employees and members of the public access to information that is comparable to access available to others (unless undue burden would be imposed on the agency).

Why is this important to acquisition professionals?

1) Addresses electronic and information technology (EIT) procurement, development, maintenance and use. 2) Section 508 standards apply to people with visual, auditory, physical, speech, cognitive and neurological impairments. 3) Section 508 lists the types of technologies that must be included in designing accessible systems. 4) Section 508 identifies the exceptions to 508 accessibility requirements .

(Accessibility guidance is also addressed in FAR Subpart 39).

Systems Design 47

Americans With Disabilities Act - User Accessibility

When acquiring EIT, agencies must ensure that:

Federal employees with disabilities have access to and use of information and data that is comparable to the access and use by Federal employees who are not individuals with disabilities; and;

Members of the public with disabilities seeking information or services from an agency have access to and use of information and data that is comparable to the access to and use of information and data by members of the public who are not individuals with disabilities.

EIT Section 508 Compliant list

 Software applications and operating systems  Video and multimedia products  Self-contained, closed products (e.g., information Web-based information or applications kiosks, calculators, and fax machines)  Telecommunication products

 Desktop and portable computers Systems Design 48

24 11/15/2016

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 49

Why Do Spectrum Management?

• Today, most military systems depend on the spectrum to interoperate. • DoD does not own all of the spectrum in the United States. NOTE: In emergencies, the Federal Emergency Management Agency (FEMA) activates the National Response Framework (NRF) allowing the Federal Communications Commission (FCC) and the National Telecommunications and Information Administration (NTIA) to allocate spectrum back to DoD Mission Systems. • We don’t own the spectrum in foreign countries. • Managing electromagnetic spectrum achieves electromagnetic compatibility for smooth operational capabilities. • Continuous spectrum management is necessary due to continuous competition to use spectrum. • Loss of spectrum means loss of your ability to communicate and interoperate. • Spectrum management is critical to operational mission readiness.

Systems Design 50

25 11/15/2016

Best Practices of Spectrum Management

• Early in your program, use DoDI 4650.01, January 9, 2009, “Policy and Procedures for Management and Use of the Electromagnetic Spectrum,“ to: 1. Identify spectrum-related risks as early as possible via spectrum supportability risk assessments. 2. Review these assessments at acquisition milestones. 3. Manage the risks throughout the system’s lifecycle. • Early in your program, use DoDI 5000.02, January 7, 2015, “Operation of the Defense Acquisition System, “ to: 1. File the Frequency Allocation Application (DD Form 1494) IAW Enclosure 1, Table 2. 2. Create and manage the Spectrum Supportability Risk Assessment IAW Enclosure 1, Table 2. 3. Comply with paragraph 19. Spectrum Supportability, Enclosure 3 • Work with the Defense Spectrum Office (DSO) to help guide you through the complex spectrum development, management and approval processes for all possible deployment scenarios.

Systems Design 51

Commercial Off the Shelf (COTS) and Software Development

COTS Defined • A COTS product is a product sold, leased, or licensed to the general public • Offered by a vendor trying to profit from it **Glue Code—code that does • Supported and evolved by the vendor, not contribute any functionality who retains the intellectual property rights towards meeting the program's requirements, but instead • Available in multiple, identical copies serves solely to "glue together" different parts of code that • Used without modification of the internals would not otherwise be • * Any given version of a COTS component compatible. will reach eventual obsolescence or end of life in which it will no longer be supported by the vendor • Almost always requires **“glue code”(middleware) to integrate into existing software architecture

* Source: Added Sources of Costs in Maintaining COTS-Intensive Systems, Clark, CrossTalk, June 2007 Software Development 52

26 11/15/2016

COTS, Always the Best Solution? Elephant Bungee Wisdom

Universal Truth #9: COTS are not necessarily the best solution. They bring risks + benefits…understand both! Management Emphasis • Investigate the pricing structure • Select established products with large installed bases • Budget for the complete cost of COTS integration and maintenance • “Fly before you buy” + TEST a lot • Adapt your requirements to COTS

Elephant Bungee Jumping #9: Avoiding Diseases that Are Fun to Catch

Software Development 53

COTS—Sources of Added Costs (Compared to custom applications)

• Licensing • Hardware Upgrades • Evaluation of New • Disabling New Features Releases • Early Maintenance • Defect Hunting • Market Watch • Vendor Support • Continuous Funding • Upgrade Ripple Effect

Source: Added Sources of Costs in Maintaining COTS-Intensive Systems, Clark, CrossTalk, June 2007

Software Development 54

27 11/15/2016

Different COTS System Compositions

• COTS Solution Systems - Typical business or standard IT systems - Major COTS component is essentially the system - Has its own architecture - Has internal business logic that must be followed to be used • COTS-Intensive Systems (CIS) - Comprised of many COTS components - Often safety and performance critical systems - User interface and data transmission may be handled by many components - Interact with each other through custom-developed “glue code” using vendor provided application program interfaces and with custom-developed application code

Source: Added Sources of Costs in Maintaining COTS-Intensive Systems, Clark, CrossTalk, June 2007

Software Development 55

COTS-Intensive Systems— Impact on Sustainment

• System obsolescence, technology refresh, and upgrade planning - Each COTS software product life cycle includes updates, refreshes, and obsolescence (i.e., unsupported releases). - Life cycle is not based on the users’ requests or budgetary cycles, but rather on marketplace demands and COTS software vendors’ business plans. • Source code escrow - Source code may be owned by the COTS vendor or the third-party integrator. - Problems can arise when the COTS vendor goes out of business or no longer exists due to a business merger or acquisition. • Vendor license management - During development, licenses may be managed by the system integrator. - The transition of license management tasks to the sustainment organization needs to be jointly planned by the program office and sustainment organization. • Architecture and COTS software interfaces - During system development, third-party integrators/ developers may capitalize on relationships with COTS software vendors to acquire system-specific capabilities. - These capabilities may not be in the official version of the product and there is no guarantee that these “extra” features will be maintained as the product evolves.

Software Development 56

28 11/15/2016

COTS and Modification Elephant Bungee Wisdom

Universal Truth # 8: Never allow developers to modify COTS software products

Management Emphasis • JUST SAY NO! • An absolute prohibition • Never pay a COTS vendor to specifically tailor or modify their product for you

Elephant Bungee Jumping # 8: A Poke In the Eye With A Sharp Stick

Software Development 57

Lesson Overview Lesson Plan Status

• Software Assurance • Interface Management • Software Safety • Modular Open Systems Approach (MOSA) • Open Source Software • Data Management • Accessibility • Spectrum Management • Design Practices Exercise

DAG Chapter 4 Systems Engineering Design Considerations Systems Design 58

29 11/15/2016

Design Exercise

• Each team prepare a short (10 min) presentation for the class. Each team will propose a method to develop / implement DTS2 based on the following: Include these considerations: - Team 1—Legacy (existing) • Accessibility - Team 2—Custom software • SW Assurance - Team 3—Free and Open Source (FOSS) • Interface Management - Team 4—Modular Open Systems Approach (MOSA) • Data management - Team 5—Commercial off the Shelf (COTS) • Spectrum Management

- Address how would your design choice impacts each of the following given the considerations listed: system cost, schedule, performance and risk? - Include an assessment of you ability to meet the KPP’s - Use the available lesson materials or other resources to develop your brief (DAG [Ch 4, 7], AKSS, Google, etc.)

- Read “Design Consideration Exercise DTS 2”— Word document

• Presentations start in 30 minutes. Systems Design 59

Summary Today we learned to:

• Relate reasons why Configuration Management (CM) is critical to the success of Software Intensive Systems. • Identify at least three reasons why interface management and documentation are important to the development of DoD Software Intensive Systems. • Identify the underlying reasons for DoD policy regarding employing an Open Systems Approach (OSA) and its associated principles. • Recognize potential areas of software safety issues and risks when reviewing acquisition documents. • Explain the characteristics and importance of sound data management practices during the development process. • Describe why continuing the spectrum management process throughout the lifecycle of an IT program is critical. • Recognize the benefits of using Open Source Software in your systems design. • Discuss how designing for accessibility impacts software development for DBS and DoD Infrastructure systems

Systems Design 60

30 11/15/2016

Lesson 13 Software Development

Learning Objectives Today you will learn to:

• Identify the key characteristics of the Waterfall, Incremental, Spiral, and Agile Software Development Life Cycle (SDLC) Models • Given a list of software methodologies, match the software methodology best suited for the attributes of a given development project. • Identify the key principles and tenets of an agile software development process • Experiment with forming a self-organizing team to complete a given task • Identify the key components of the Scrum process framework • Given an acquisition scenario, develop estimates for tasks using agile techniques and Story Points • Explain the purpose of key agile software measures • Define the role of DevOps in a software development project

Software Development 62

31 11/15/2016

Lesson Overview Lesson Plan

• Software Development Life Cycle Approaches • Agile Software Development Overview • Scrum - Exercise 1 “Scrum Ball” • Exercise 2 “Story Point Estimation” • Agile Measures • DevOps

Software Development 63

Software Life Cycle Development Approaches

Answer these questions based on the Homework:

• Which process model is best suited for when all the requirements are understood up front and are stable? • Which process model might be best to use on a project with a lot of risks? • Which process model would you choose if the requirements are not well understood and are emerging? • Which process should I consider if the requirements are know up front and immediate functionality is needed?

Software Development 64

32 11/15/2016

Lesson Overview Lesson Plan

• Software Life Cycle Development Approaches • Agile Software Development Overview • Scrum - Exercise 1 “Scrum Ball” • Exercise 2 “Story Point Estimation” • Agile Measures • DevOps

Software Development 65

Why Agile?

Agile projects are 3 times more successful than Waterfall Agile Waterfall

Failed Successful 9% Failed 14% 29% Success ful 42%

Challenged 49% Challeng ed 57%

Successful Challenged Failed Successful Challenged Failed

Successful = delivered on time, on budget, with all required features and functions Challenged = Late, over budget, and/or with less than required features and functions Failed = Cancelled prior to completion or delivered and never used

Source: CHAOS Manifesto 2011, The Standish Group International , Inc. Software Development 66

33 11/15/2016

Why Agile? (Continued)

• The traditional methods aren’t working - MAIS = 38 months from MS B/FFO to FDD • Everything is changing and evolving much quicker than it used to - We need to do it faster • Agile and Lean Methods can speed up the delivery of the software to the field.

Software Development 67

Agile Software Development Definition

• Agile software development is a set of software development methods in which requirements and solutions evolve through collaboration between self- organizing,[1] cross-functional teams. It promotes adaptive planning, evolutionary development, early delivery, and continuous improvement, and it encourages rapid and flexible response to change.[2]

• [1] Collier, Ken W. (2011). Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing. Pearson Education. pp. 121 ff. ISBN 9780321669544. “What is a self-organizing team?”

• [2] "What is Agile Software Development?". Agile Alliance. 8 June 2013. Retrieved 4 April 2015.

Software Development 68

34 11/15/2016

Agile Manifesto

Software Development 69

Agile Software Development Umbrella

Software Development 70

35 11/15/2016

Empirical Process Control

The empirical model of process control provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.

Software Development 71

Agile vs Traditional

36 11/15/2016

Lesson Overview Lesson Plan

Software Development 73

Scrum

• Scrum is an iterative and incremental software process framework • Relies on collaborative, self-organizing, cross-functional, self-managing teams • Empirical approach to managing the process • Responsive to emerging or changing requirements • Focuses on delivering value quickly

Software Development 74

37 11/15/2016

Sprint

• Time-boxed iteration of software development - Typically 1 – 4 weeks in duration - Should not exceed a calendar month • Produces an increment of potentially shippable software • Once Team commits to the sprint they are not interrupted with any changes that would affect them reaching their sprint goal.

Software Development 75

Scrum Framework Overview

38 11/15/2016

Lesson Overview Lesson Plan

• Software Life Cycle Development Approaches • Agile Software Development Overview • Scrum -Exercise 1 “Scrum Ball” • Exercise 2 “Story Point Estimation” • Agile Measures • DevOps

Software Development 77

Exercise: Scrum Ball

This exercise will help us put in practice some of the topics we have been discussing with Agile. We will implement self-organizing teams, practice continuous improvement, time boxed iterations, and utilize empiricism.

Software Development 78

39 11/15/2016

Scrum Ball: Debrief

• How accurate was the first estimate? Did the estimates improve? • What was the team leadership model? • How did you achieve dramatic increase in velocity? • Did every experiment work?

Software Development 79

Scrum Roles

Software Development 80

40 11/15/2016

Scrum Artifacts

Software Development 81

User Story

• Written - “As a , I want so that ” • Conversations - Flesh out the details • Acceptance Criteria - Defines what constitutes when the story is complete • Epic - A large user story that will usually be broken up before being implemented

41 11/15/2016

Example User Stories

• As a maintenance test pilot, I want to be able to view all the readings for a selected flight data parameter graphed over time so that I can review what occurred during the flight • Epic – As a user, I can backup my hard drive - As a power user, I can specify files or folders to backup based on file size, date created and date modified - As a user, I can indicate folders not to backup so that my backup drive isn’t filled up with things I don’t need to save [1] Reference Mike Cohn: Mountain Goat Software

Sprint Planning Meeting

• Who - Team, ScrumMaster. & Product Owner Sprint Goal • Agenda - Discuss the top priority product backlog items Sprint Backlog - Team selects which to do • Why - Know what will be worked on - Understand it enough to do it

“A good plan violently executed now is better than a perfect plan executed next week.” – General George S. Patton

42 11/15/2016

Estimation: Story Points

• Unit of measure for evaluating the relative size of a user story, feature, or work item • Typical Estimation Scale roughly follows the Fibonacci sequence - 0, 1, 2, 3, 5, 8, 13, 20, 40, 100 - Some teams include ½ and ∞ • A story assigned a two should be roughly twice the size of a story assigned a value of one • Unit-less Measurement • Delphi / Expert Judgement method for estimating

Planning Poker

• Each team member has a set of valid Story Point Cards • Each Product Backlog Item (PBI) is discussed briefly, reference story chosen • Discuss each story (2-3 mins) • Each Development Team Member selects a card Card(s) Interpretation matching their estimate 0 Task is already completed • Everybody shows their cards ½ The task is tiny at same time 1, 2, 3 Small tasks • Discuss Outliers (high/lows) 5 & 13 Medium sized tasks • Repeat until team reaches a 20 & 40 Large sized tasks convergence in estimates 100 Very large tasks • Record estimate for the PBI ∞ Huge task ? I have no idea how long this task will take to complete

43 11/15/2016

Planning Poker Scenario

• As the facilitator for a planning poker session, you’ve given each of your six team members cards labeled, 1, 2, 3, 5, 8, and 20. Each team member chooses a card indicating the number of story points tot assign to a particular user story. The resulting values are 3, 5, 8, 8, 13, and 20. What should you encourage the team to do next? a. Use a different set of numbers b. Ask the team members who chose the values of 3 and 20 to explain their choices c. Calculate the average number of points and assign it to the user story d. Decide what story point value to assign to the user story on behalf of the group

Affinity Estimation Variation

• Use columns going from smallest to largest as you move left to right • Identify the smallest candidate PBI and place in far left column • Identify the largest PBI and place in far right column • Compare each PBI to the PBI in a column if smaller go left if Whiteboard, Wall, or larger go right Bulletin board • Once all PBIs are in columns estimate the story points for each column • Record estimate for each PBI

Software Development 88

44 11/15/2016

Daily Scrum

• Development team meets to make the plan for the day Answer 3 questions: These are NOT status for the • ScrumMaster. They are - What did I do yesterday? commitments in front of peers - What am I doing today? - What are my impediments • Time Boxed - 15 Minutes, standup • Scheduled daily for the same time and place • Opportunity to inspect and adapt daily based on the current progress on completing the sprint backlog

Sprint Review

• Demonstrate the work completed during the sprint • Opportunity to Inspect the work product and adapt the plan if needed • Scrum Team and Stakeholders collaborate about what was completed in the sprint • Opportunity to make changes to Product Backlog based on feedback • Chance to discuss the path forward for the next sprint

45 11/15/2016

Sprint Retrospective

• Review the Process and Product looking for ways to improve either • Answer Three Questions - What worked? - What didn’t work? - What will we do differently? • Continuous Process Improvement

Scrum Summary – Key Components of the Scrum Framework

• 4 Activities - Sprint Planning - Daily Scrum - Sprint Review - Sprint Retrospective • 3 Roles - Product Owner - Development Team - ScrumMaster • 3 Artifacts - Product Backlog - Sprint Backlog - “Done” Software 92

46 11/15/2016

Lesson Overview Lesson Plan

Software Development 93

Exercise 2- Estimating Using Story Points

The objective of this exercise is let each team practice using story points to estimate a feature. Each team will create estimates for each of the recipes based on how long it would take for the food to be ready to eat using story points. Braised Corned Beef Brisket Best Ever Meatloaf with Brown Gravy

Valerio’s Pulled Pork Sandwich Chantal’s New York Cheesecake

Key Lime Pie VII Blazing Brisket

Easy Meatloaf Best Hamburger Ever

• The odd teams will estimate the recipes following the Planning Poker steps outlined below. • The even teams will estimate the recipes following the Affinity Estimating steps variation listed below.

Software Development 94

47 11/15/2016

Planning Poker Steps

• Each team member has a set of valid cards • Product owner reads the recipe and it’s discussed briefly • Each team member selects the card that matches their estimate • Everybody shows cards at the same time • Discuss the outliers and why they had their estimate • Re-estimate until you have a convergence • Record your estimate for each recipe

Software Development 95

Affinity Estimating Variant Steps

• Team draws columns on whiteboard • Product owner reads the recipes. • Team identifies the recipe that can be completed the quickest and the one that will take the longest. • The quickest recipe is placed in the far left column and the longest in the far right column • Next the team compares each recipe to the quickest and hardest and selects a column to place it in. Once there is a recipe in another column the team can then evaluate the next one based on that and if it is quicker then go to the left, if it is longer go to the right, and if it is equivalent place it in the same column. • Once all recipes have been assigned to a column the team can use planning poker to determine the story point value for each column. The team can agree to group adjacent columns with the same story point value.

Software Development 96

48 11/15/2016

Lesson Overview Lesson Plan

Software Development 97

Agile Measures Overview

• Measurement is also used extensively used in agile approaches - generally the measures are used internally by the development teams • The measures are generally a smaller subset of measures and the applicable timeframes are generally small • The next few slides: - Introduce measures used in Agile development - Help you to understand Agile measurement visualizations - Identify some team level and program level measures in Agile development

Software Development 98

49 11/15/2016

Integrating Agile Measurement into Your Overall Management Metrics

Agile/Iterative measures for teams leverage the approach of time-boxing (fixing the schedule for the iteration, and varying the target requirements to be fulfilled based on the team’s capacity and capability) • Opposite of traditional acquisition where schedule is based on estimates of what it takes to implement a fixed set of requirements

Agile/Iterative iteration measures are meant primarily for the team’s use • External use of iteration metrics is generally discouraged • Release metrics are generally used for monitoring/management purposes

Software Development 99

Typical Team Measures for Agile Development

Metrics used by and for the development team:

• Kanban Board for Task Tracking • Sprint Burn-down Charts • Release Burn-Up Charts • Velocity

Teams focus on delivering working code

Software Development 100

50 11/15/2016

Burndown Chart

• Tool for visualizing daily progress during a sprint • Ideal trend line graphs hours remaining for tasks in sprint backlog over the length of the Sprint • As team updates hours remaining for the tasks in progress the chart indicates if progress is above or below the trend line

101

Velocity

• Velocity – Is a measure of a team’s rate of progress • The purpose of velocity is to help aid the team in planning the next sprint and forecasting when to expect a subset of features to be ready for shipping • Calculated by summing the story point estimates of the PBIs completed during the Sprint • The average velocity is used to determine how much to plan for the next sprint • NOTE: Velocity is tied to a specific team and should not be used to compare two teams!

102

51 11/15/2016

Cumulative Flow Diagrams

• The green line/area shows how many items have been delivered over time • The yellow line / area shows how many items are in progress • The top part (red) is the backlog or how many items weren’t yet started

Software Development 103

Lesson Overview Lesson Plan

Software Development 104

52 11/15/2016

Silos and Delivery Challenges

Systems Development 105

How to fix these problems? Origin of “DevOps”

• Develop and test against production-like environments

• Iterative and frequent deployments using repeatable and reliable processes

• Continuously monitor and validate operational quality characteristics

Track and Plan everything Version everything Automate everything Test everything Audit and Monitor everything Dashboard everything

Systems Development 106

53 11/15/2016

Agile Benefits Summary

• Agile is not a Silver Bullet, But DoD can benefit greatly from adopting Agile processes - Agile can reduce program risk by frequent releases - Agile can respond to change to ensure that the warfighter is actually getting a product they need - Agile allows the DoD to integrate the latest technological advancements to address an increasingly complex operational environment to allow them to keep pace with the rapid changes in IT • Agile provides a core set of principles to help the DoD get away from cumbersome monolithic systems for IT

Summary Today we learned to:

Software Development 108

54 11/15/2016

Lesson 14 Continuous Process Improvement and Business Process Reengineering

Learning Objectives Today we will learn to:

• Identify the DoD policy that describes the Implementation and Management of the DoD-Wide Continuous Process Improvement Program • Define Continuous Process Improvement (CPI) • Define Business Process Reengineering (BPR)

CPI & BPR 110

55 11/15/2016

Lesson Overview Lesson Plan

• Business Process Reengineering (BPR) • Continuous Process Improvement (CPI) • Principles of Lean • Exercise • Principles of Six Sigma

CPI & BPR 111

Business Process Reengineering (BPR) Defined

• “The fundamental rethinking and radical redesign of business processes to achieve the dramatic improvements in critical contemporary measures of performance, such as cost, quality, service and speed.” - DoDI 2009 5010.43 • The Department defines BPR: a logical methodology for assessing process weaknesses, identifying gaps, and implementing opportunities to streamline and improve the processes to create a solid foundation for success in changes to the full spectrum of operations.” - 2012 DoD Business Process Reengineering Assessment Guidance

CPI & BPR 112

56 11/15/2016

Why BPR?

• Subtitle III of Title 40 of the United States Code (AKA Clinger-Cohen Act) requires BPR. • Section 1072 of the National Defense Authorization Act (NDAA) for Fiscal Year 2010 introduced new requirements into the Department’s investment review process stipulating defense business system (DBS) modernizations may not be certified to obligate funds in excess of $1 million without a determination of whether appropriate business process reengineering (BPR) had been completed. 2012 DoD Business Process Reengineering Assessment Guidance

It’s the Law! CPI & BPR 113

Performing BPR

• The DoD does not mandate a specific BPR methodology.

• The goal of BPR is to ensure that the business processes supported by the DBS have been streamlined, and/or eliminated or unique requirements and interfaces have reduced to the maximum practical extent.

• 2012 DoD BPR Assessment Guidance Standard.

CPI & BPR 114

57 11/15/2016

Performing BPR

• Functional Sponsor and PM are responsible for completing assessment form prior to IRB

• 13 assessment questions requiring tailored objective evidence based upon submission type: - Development - Modernization - Research - Technical Refresh - Sustainment.

Source: Appendix B, 2012 DoD Business Process Reengineering Assessment Guidance CPI & BPR 115

Lesson Overview Lesson Plan Status

• Business Process Reengineering (BPR) • Continuous Process Improvement (CPI) • Principles of Lean • Exercise • Principles of Six Sigma

CPI & BPR 116

58 11/15/2016

CPI Defined

“Continuous Process Improvement—a comprehensive philosophy of operations that is built around the concept that there are always ways in which a process can be improved to better meet the needs of the customer and that an organization should constantly strive to make those improvements.” 2008 Continuous Process Improvement /LSS Guidebook Revision 1

CPI & BPR 117

CPI Policy

“A robust, effective CPI/LSS program will leverage the right tools and methods for problems being addressed, including Lean Six Sigma, theory of constraints, and business process reengineering.” 2009 DoDI 5010.43

CPI & BPR 118

59 11/15/2016

CPI Guidebook

• CPI provides organizations a structured approach for analyzing how they are currently doing work and how they can improve their processes to do the job more efficiently and effectively on an ongoing basis. • Process improvements resulting from applying CPI effectively will greatly benefit the Department, in terms of both improved operations and reduced resource consumption. • Application of tools from Theory of Constraints, Lean and Six sigma will improve both production and service processes.

CPI & BPR 119

Lesson Overview Lesson Plan Status

• Business Process Reengineering (BPR) • Continuous Process Improvement (CPI) • Principles of Lean • Exercise • Principles of Six Sigma

CPI & BPR 120

60 11/15/2016

Lean

A methodology for continuous process improvement, focused on work flow, customer value, and eliminating process waste; unique from traditional process improvement strategies in that its primary focus is on eliminating non-value added activities 2009 DoDI 5010.45

CPI & BPR 121

Key Lean Concepts

Five Principles of Lean Thinking: 1. Specify the value of a process from the customer’s perspective. 2. Identify the value stream for each process 3. ELIMINATE WASTE: Make value-creating steps flow 4. The customer must pull value through the process 5. Continuously pursue perfection

Source: Lean Thinking, Womack and Jones CPI & BPR 122

61 11/15/2016

Eight Sources of Waste

CPI & BPR 123

Lesson Overview Lesson Plan Status

• Business Process Reengineering (BPR) • Continuous Process Improvement (CPI) • Principles of Lean • Exercise • Principles of Six Sigma

CPI & BPR 124

62 11/15/2016

Eight Sources of Waste Exercise

“When you are agile, you get lean” (Pages 1-4 – Student CD) Describe the principle and provide your assessment and at least 1 example • Team 1: Inventory in software development • Team 2: Motion in software development; Waiting in software development. • Team 3: Over-production in software development; • Team 4: Over-processing in software development • Team 5: Transport in software development; Defects in software development; 125

Lesson Overview Lesson Plan Status

• Business Process Reengineering (BPR) • Continuous Process Improvement (CPI) • Principles of Lean • Exercise • Principles of Six Sigma

CPI & BPR 126

63 11/15/2016

Six Sigma

A disciplined, data-driven methodology for process improvement that focuses on satisfying customer requirements while minimizing waste by reducing and controlling process variation. 2009 DoDI 5010.43

Six Sigma evaluates a process in terms of performance, accuracy, and consistency—Reduce Variation of Processes

CPI & BPR 127

Variation

• What Is It? - “Short Answer.” A change in the value of a measured characteristic - “Long Answer.” The difference between things that are produced under conditions that are as nearly alike as it is possible to make them - No two products or characteristics are exactly alike because any process contains many sources of variability • General Principles - No two things are alike - Variation can be measured - Individual outcomes are not predictable - Groups form patterns with definite characteristics

CPI & BPR 128

64 11/15/2016

Variation: Common And Special Causes

If only common causes of variation are present, the output of the process forms a distribution that is stable over time and is predictable

If special causes of variation are present, the process output is not stable over time and is not predictable.

CPI & BPR 129

Six Sigma Methodology

CPI & BPR 130

65 11/15/2016

CPI & BPR Review

• BPR—fundamental rethinking and radical redesign of business processes

• Lean—eliminating non value adding waste from our process

• Six Sigma—reduce variation in key process attributes leading to stable and predictable outcomes

CPI & BPR 131

Summary Today we learned to:

CPI & BPR 132

66 11/15/2016

Lesson 15a Software Quality

Learning Objectives Today we will learn to: • Define software quality • Identify characteristics unique to software that impact quality • Identify characteristics of generic DOD software system domains (e.g. Platform IT, Command and Control, and Defense Business Systems), that might influence how each system is reviewed in a software quality program • Recognize that every IT acquisition program requires a Program Manager approved software quality statement. • Given several process-focused and product-focused software quality assurance methods, describe how each assures quality in a software acquisition • Recognize the benefits of applying Capability Maturity Model Integrated (CMMI) concepts and principles to a DoD SW development project • Given a software acquisition scenario, recognize the preferred method for identifying and tracking defects • Given an acquisition scenario with multiple software related programmatic issues, analyze how each may impact the program’s ability to meet its quality objectives Software Quality Assurance 134

67 11/15/2016

Lesson Overview Lesson Plan

• How is Software Different from Hardware? • What is Software Quality? • Software Domain Considerations • Lesson Exercise Part 1 • Software Quality Assurance Planning and Methods • Lesson Exercise Part 2

Software Quality Assurance 135

What Makes Software Different from Hardware

Software Quality Assurance 136

68 11/15/2016

Lesson Overview Lesson Plan Status

Software Quality Assurance 137

Software Quality

THE ESSENCE: • Definitions for Software Quality Does it do what it is supposed to? - The degree to which a system, component, or process meets specified requirements [IEEE Std 610.12-1990] The Program Manager The degree to which a system, must understand and - define software quality component, or process meets customer for his or her program or user needs or expectations (no undesirable properties) [ISO/IEC 9001} • Other Definitions - Lack of bugs - Adherence to a software quality model (“ilities”)

Software Quality Assurance 138

69 11/15/2016

Software Quality Model Characteristics

• ISO 9126-1 :2001 Software engineering -- Product quality provides a SW quality model that identifies six main quality characteristics:

• Functionality • Efficiency - Suitability - Time behavior - Accurateness - Resource behavior - Interoperability - Analyzability - Compliance • Maintainability - Security - Changeability • Reliability - Stability - Maturity - Testability - Fault tolerance - Adaptability - Recoverability • Portability • Usability - Installability - Understandability - Conformance - Learnability - Replaceability - Operability Software Quality Assurance 139

Lesson Overview Lesson Plan Status

Software Quality Assurance 140

70 11/15/2016

Software Domain Considerations

• Is there a relationship between software quality and software safety? • Is there a relationship between software quality and cybersecurity? C4ISR systems – Platform IT (PIT) systems – Mission C4ISR Security, interoperability Safety, Response time Systems

Defense Business DBS systems – Privacy, interoperability Systems

SW quality assurance practices should be chosen to meet the program’s quality objectives and informed by the risks inherent in the type of system being built. Software Quality Assurance 141

Lesson Overview Lesson Plan Status

• How is Software Different from Hardware? • What is Software Quality? • How do I achieve Software Quality? • Lesson Exercise Part 1 • Software Quality Assurance Planning and Methods • Lesson Exercise Part 2

Software Quality Assurance 142

71 11/15/2016

Quality Exercise 1

• Review the ICM table and identify areas that are applicable to the specific quality characteristics identified in the lesson.

• Identify quality-focused information needs (reference the ICM table in exercise folder) and map the information needs to measurable concepts and potential measures. - Include key aspects of the Software Quality Model Characteristics - Include any safety and cybersecurity quality implications

• Based on your experience, prioritize the information needs in keeping with the common challenges programs experience.

• Summarize your IPT’s results in a briefing using the template provided (whiteboard is preferred if possible)..

Software Quality Assurance 143

Lesson Overview Lesson Plan Status

Software Quality Assurance 144

72 11/15/2016

Software Quality Assurance Methods

Process Assurance Product Assurance • systematic activities • systematic activities providing evidence of the providing evidence that ability of the software the software product process to produce a performs as specified software product fit for use - Desk Checking - Process maturity and - Walk-Throughs compliance - Formal Inspections - Risk Management - Joint Reviews - Independent oversight - Computer-Based Testing - SQA audits and reporting - Product Quality Measures

Software Quality Assurance 145

Quality Program

• A quality program includes a management process that is capable of ensuring the following key activities: - Formulating a quality management plan - Applying software engineering principles - Conducting formal technical reviews - Applying a multi-tiered testing strategy - Enforcing Process adherence - Controlling change and measuring the impact of change - Performing SQA audits, keeping records and reporting • The PM should allow contractors to define and use a preferred quality management process that meets required program support capabilities. • The DOD does not require third party certification or registration of a supplier’s quality system

The PM is ultimately responsible for setting the quality objectives!

Software Quality Assurance 146

73 11/15/2016

Software Quality Assurance Plan

Software Quality Assurance 147

Process Assurance

• Software Engineering - The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software. [IEEE Std 610.12-1990] • Software Cleanroom Engineering - Combines formal methods of requirements and design with statistical usage testing to produce software with nearly none or no defects. - Normally used in systems requiring highly reliable software (space shuttle) • Continuous Process Improvement - A strategic approach for developing a culture of continuous improvement in the areas of reliability, process cycle times, costs in terms of less total resource consumption, quality, and productivity. - Six Sigma: A disciplined approach and methodology for reducing variations in system output. - Lean Six Sigma: A disciplined, data-driven approach and methodology for eliminating defects in any process — from manufacturing to transactional and from product to service. - Theory of Constraints: a management paradigm that views any manageable system as being limited in achieving more of its goals by a very small number of constraints.

Software Quality Assurance 148

74 11/15/2016

Process Assurance: Capability Maturity Model—Integration (CMMI)

• Capability Maturity Model® Integration (CMMI) - Was a collaborative effort sponsored by the Office of the Secretary of Defense/Acquisition and Technology (OSD/A&T) Systems Engineering with participation by government, industry, and the Software Engineering Institute (SEI)*. - Objective is to develop a product suite that provides industry and government with a set of products to support process and product improvement. - It can be used to guide process improvement across a project, a division, or an entire organization. • Benefits of implementing process improvement: - The quality of a system is highly influenced by the quality of the process used to acquire, develop, and maintain it. - Process improvement increases product and service quality as organizations apply it to achieve their business objectives. - Process improvement objectives are aligned with business objectives. - CMMI maturity level can often be a good predictor of whether a software development project will incur cost and schedule overruns.

CMMI Institute http://cmmiinstitute.com/ *CMMI is a spin-off of original SEI activity

Software Quality Assurance 149

Process Assurance: Capability Maturity Model—Integration (CMMI)

CONTINUOUS

Path enables organization to incrementally improve processes corresponding to an individual process area (PA) (or group of process areas) selected by the organization.

STAGED

Path enables organization to improve a set of related processes by incrementally addressing successive sets of process areas.

Software Quality Assurance 150

75 11/15/2016

Process Assurance: Capability Maturity Model—Integration (CMMI)

Level Continuous Representation Staged Representation Capability Levels Maturity Levels Level 0 Incomplete

Level 1 Performed Initial

Level 2 Managed Managed

Level 3 Defined Defined

Level 4 Quantitatively Managed

Level 5 Optimizing

• Capability levels enable your organization to focus its process improvement efforts process area by process area from capability level 0 to capability level 3. • Maturity levels provide a staging of processes for improvement across your organization from maturity level 1 to maturity level 5. Software Quality Assurance 151

Process Assurance: More Quality Process Models and Standards

• ISO 9000, Quality Management and Quality Assurance Standards - Provides guidelines as to which document to use and how to use it. - Use of ISO 9001, 9002 or 9003 depends on business structure • ISO 9001, Quality Systems - Model for Quality Assurance in Design/Development, Production Installation, and Servicing • ISO 9002, Quality Systems - Model for Quality Assurance in Production and Installation • ISO 9003, Quality Systems - Model for Quality Assurance in Final Inspection and Test • ISO 9004, Quality Management and Quality Systems Element - Along with ISO 9000, ISO 9004 is an advisory document.

ISO 9001 gives the requirements that an ISO 9000 - Quality organization must do to manage the processes management://www.iso.org/iso/iso_9000 affecting quality of its products and services Software Quality Assurance 152

76 11/15/2016

Other Quality Standards and Initiatives

• SPICE (SW Process Improvement and Capability Determination) (ISO/IEC 15504) - An international standard for software process assessment. - Derived from process lifecycle standard ISO 12207 and ideas of maturity models like Bootstrap, Trillium and the CMM. • Control Objectives for Information and related Technology (COBIT) - Provides good practices across a domain and process framework. - Practices designed to help optimize IT-enabled investments, ensure service .delivery and provide a measure against which to judge when things do go wrong.. • Information Technology Infrastructure Library (ITIL) - Provides international best practices for IT service management. • Consists of a series of books giving guidance on the provision of quality IT. services, and on the accommodation and environmental facilities needed to support IT. • Practical Software and Systems Measurement (PSM) - Best practices within the software/system acquisition and engineering communities. - Goal is to provide Project Managers with the information needed to meet cost, schedule, and technical objectives on programs.

Software Quality Assurance 153

Product Assurance: Human–based Testing

Software Quality Assurance 154

77 11/15/2016

Product Assurance: Measures

• Low Defect Potentials (< 1 per function point) • High Defect Removal (> 95%) • Unambiguous, Stable Requirements (< 2.5% change) • Explicit Requirements Achieved (> 97.5% achieved) • High User Satisfaction Ratings (> 90% “Excellent”) - Installation - Ease of Learning Various internal DoD and GAO reports and studies have indicated that - Ease of Use establishing meaningful measures and - Functionality using them to manage contributes to - Compatibility program success

Software Quality Assurance 155

Product Assurance: Measures

1. Establish Requirements 4. Analyze Results a) Identify list of possible quality a) Interpret the results requirements b) Identify software quality b) Determine list of quality c) Make software quality requirements predictions c) Assign a direct metric to each d) Ensure compliance with quality requirement requirements 2. Identify Metrics 5. Validate Metrics a) Apply the software quality metrics a) Apply the validation framework methodology b) Perform a cost-benefit analysis b) Apply the validity criteria c) Gain commitment to the metrics c) Apply the validation 3. Implement Metrics procedures a) Define the data collection d) Document results procedures b) Prototype the measurement IEEE Standard for a Software Quality process Metrics Methodology c) Collect the data and compute the IEEE Std 1061-1998 (reaffirmed 2009) metric values

Software Quality Assurance 156

78 11/15/2016

Lesson Overview Lesson Plan Status

Software Quality Assurance 157

Quality Exercise 2 Instructions

Quality Analysis

The current software upgrade project has been in progress for a year now and the project should be well into the Coding and Unit Testing phase. The PM believes that something is amiss and has asked your IPT to perform an in-depth analysis to assess the project’s status from a quality perspective. The PM received from the prime contractor the measurement indicators (without any additional comments or information) and has given them to your IPT to analyze (see Contractor Data Presented to PM). The PM has also identified the program’s specific Quality Attributes and is concerned that the data does not allow for a good quality assessment. The PM wants your IPT to complete the enclosed assessment templates.

Tasks: Within your IPT, review carefully the set of charts to identify problems, assess their impact, project the outcomes, and evaluate alternatives. 1. Compare/contrast the PM’s Quality Attributes with the Software Quality Model Characteristics (ISO 9126-1 :2001 Software engineering) presented in the lesson material and identify any areas you feel the PM should address. 2. Compare and contrast the ICM Quality measurable concepts (with indicators and measures) from Exercise 1 with the PM’s Quality Attributes. Identify the differences between the ICM table and the PM’s Quality Attributes. 3. Assess the data provided by the contractor - does the data provided meet information needs based on the PM’s Quality Attributes? If not, identify the areas where you need additional data to support the PM’s Quality Attributes. 4. What recommendations would you make to the PM based on the insight gained from your analysis? Be sure to include recommendations for strengthening the overall SW quality program. What additional analyses you would perform to assess and strengthen the SW Quality activities.

79 11/15/2016

Summary Today we learned to:

• Define software quality • Identify characteristics unique to software that impact quality • Identify characteristics of generic DOD software system domains (e.g. Platform IT, Command and Control, and Defense Business Systems), that might influence how each system is reviewed in a software quality program • Recognize that every IT acquisition program requires a Program Manager approved software quality statement. • Given several process-focused and product-focused software quality assurance methods, describe how each assures quality in a software acquisition • Recognize the benefits of applying Capability Maturity Model Integrated (CMMI) concepts and principles to a DoD SW development project • Given a software acquisition scenario, recognize the preferred method for identifying and tracking defects • Given an acquisition scenario with multiple software related programmatic issues, analyze how each may impact the program’s ability to meet its quality objectives Software Quality Assurance 159

(Back-up) Integrated Analysis (From Measures Lesson)

• Integrated analysis combines multiple indicators and focuses on the cause and effect relationships inherent between IT performance parameters • These IT performance parameters map directly to the measurement information categories

McGarry, J., Card, D., Jones, C., Layman, B., Clark, E., Dean, J., & Hall, F. (2002). Practical Software Measurement.

80 11/15/2016

Lesson 15b Software Testing

Learning Objectives Today we will learn to:

• Recognize the characteristics of Software Testing. • Recognize the purpose of a Test Readiness Review (TRR). • Identify what Independent Verification and Validation (IV&V) is. • Identify the T&E responsibilities of the IT Program Manager. • Identify the software testing issues and risks. • Recognize the different types of software testing tools. • Recognize the Agile software development T&E challenges. • Identify software test issues and risks in building a Software Test Plan (STP) for a COTS/NDI only system. • Recognize the Software T&E Best Practices. • Recognize the best practices of employing Independent Verification and Validation (IV&V) on Agile Projects.

Software Testing 162

81 11/15/2016

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 163

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 164

82 11/15/2016

Testing Software

Can we test quality into our software???

Software Testing 165

Program Manager’s Responsibilities for Software T&E

The PM is responsible for defining the level of Software Quality for their program. This includes defining the “level of maturity” the software must perform at in order to declare software T&E a success.

Testing must focus on testing the performance of those items the Program Manager (PM) identifies as software quality goals for the success of their program.

Software Testing 166

83 11/15/2016

Software Test and Evaluation Mission

The software T&E mission has two primary objectives:

1) Demonstrate performance of the whole software system 2) Assist in fault detection and correction of faults

There are four (4) primary ways to support the success of the software T&E mission:

1) Your plans should include an incremental test strategy 2) Identify and correct software defects as early as possible in the lifecycle 3) Provide scientific based measures for progress and quality 4) Provide data evaluation to support acquisition decisions

Software Testing 167

Example Typical Costs of Software Fixes*

Lifecycle SW Development Initial $ Errors Errors Relative Cost Activity Spent Introduced Found Requirements 5% 55% 18% 1.0 Analysis Design Activities 25% 30% 10% 1.0-1.5

Testing Activities 60% 10% 50% 1.5-5.0

Documentation 10% ------Post-Deployment Software Support ---* 5% 22% 10-100 (PDSS)

*Once a system is fielded, PDSS costs are typically 50-70% of total system lifecycle costs Software Testing 168 B. Boehm, Software Engineering Economics

84 11/15/2016

Current State of Software T&E

• National Institute of Standards & Technology (NIST) states that inadequate testing methods and tools cost the U.S. economy between $22.2 and $59.5 billion annually.

• Testing methods in general are relatively ineffective. Testing typically only identifies from one-fourth to one-half of defects (found over a typical software lifespan) while other verification methods (e.g., formal inspections) are more effective in finding defects earlier.

• Inadequate and ineffective testing results in approximately 2 to 7 defects undiscovered per thousand lines of code (KLOC) when the software is deployed. This means that major software-reliant systems are being delivered and placed into operation with hundreds or even thousands of residual defects. If you include software vulnerabilities as defects, the defect rates are even more troubling. What are the problems with Software T&E?

https://insights.sei.cmu.edu/sei_blog/2013/04/common-testing-problems-pitfalls-to-prevent-and-mitigate.html Software Testing 169

Known Problems with Software T&E

• Requirements-related testing: Requirements ambiguous, missing, incomplete, incorrect, or unstable.

• Test planning and scheduling: Improper planning, improper test schedules.

• Stakeholder involvement and commitment: Not testing items important to stakeholders; Leadership commitment lacking.

• Management-related testing: Test resource management lacking.

• Test organizational and professionalism: Lack of independence, unclear testing responsibilities, and inadequate testing expertise.

• Test & Evaluation process: T&E and engineering processes poorly integrated.

• Test tools and environments: Over-reliance on manual or COTS testing tools; testing environments inadequate to fully test software.

• Test communication: Test documents inadequate, not maintained, not Software Testing 170 communicated. https://insights.sei.cmu.edu/sei_blog/2013/04/common-testing-problems-pitfalls-to-prevent-and-mitigate.html

85 11/15/2016

Software Test Plan Relationships

Test & Evaluation Master Plan (TEMP) Documents the overall structure and objectives of the T&E program. Includes Developmental Testing: critical operational effectiveness and -substantiate system suitability parameters for [among other TEMP technical performance items]...SOFTWARE and COMPUTER Operational Testing: RESOURCES...it addresses SOFTWARE MATURITY and the degree to which -meet CDD requirements SOFTWARE DESIGN has stabilized.. Defense Acquisition Guidebook

Software Development Plan (SDP) • Coding and Unit Testing ST P • Unit Integration & Test • SI Qualification Testing S D P • SI/HI Integration & Test • System Qualification Testing Software Test Plan (STP) • Software Quality Assurance • ID’s Test Items (SIs, Units) • Corrective Action Process • ID’s Personnel Resources • ID’s Test Environment • Provides Test Schedules Software Requirements • Traces to Requirements Software Testing 171

Example of a Software Testing Priority Classification Scheme

Priority Classification Scheme Priority 1: Prevents Mission Accomplishment or jeopardizes safety or other “critical” requirement Priority 2: Adversely affect Mission Accomplishment or cost, schedule, performance or software support + no work- around exists Priority 3: Adversely affect Mission Accomplishment or cost, schedule, performance or software support + a work- J-STD-016 around exists Priority 4: User/Operator or support inconvenience Classification Priority 5: All other problems Schemes Category Classifications • Project Plans • Operational Concept • System/Software Requirements “Alternate category and priority •Design schemes may be used by developers if approved by the • Code acquirer” • Databases/data files J-STD-016, Annex M • Test Plans, Descriptions, Reports • User, Operator or Support Manuals • Other Software Products

Software Testing 172 Rev 2.2

86 11/15/2016

Types of Software Tests (Human-based and Computer-based)

HUMAN-based COMPUTER-based

• Desk Checking – White Box – Uses source Programmer inspected code knowledge and • Walkthroughs (Informal executes Software Unit Inspections) – tests to ensure correctness Programmer Team Black Box – No knowledge inspected. of internal source code; • FORMAL INSPECTIONS perspective of a Hacker Gray Box – Some knowledge of internal code; FORMAL perspective of a smart INSPECTIONS provide BEST ROI Hacker

Software Testing 173

Types of Software Tests (Human-based and Computer-based)

Human-Based Computer-Based Black-box Gray-box Formal Inspections

White Box

Acceptance Testing occurs after each development phase (yellow and blue boxes) to ensure requirements are met from previous phase. Final Acceptance testing is with the warfighter and is the most important test! Software Testing 174

87 11/15/2016

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 175

Software Development V-Model

Operational Concurrent Requirements Test Operational (user needs) & & Acceptance Project Plans Planning Tests OT&E System Requirements System DT&E Analysis Qualification Testing System Subsystem Design Integration & Testing Software Requirements HI and SI Analysis Integration & Testing Software SI Qualification Design Testing Software Unit Software Unit Integration & Testing Coding Software Unit Testing Software Testing 176

88 11/15/2016

Test Readiness Review (TRR)

Inputs/Entry Criteria Outputs/Exit Criteria • Identified tested requirements (SRS) • Software Test Descriptions are • Traceability of test requirements to defined, verified & baselined SRS & IRS has been established • Testing is consistent with any • All software item test procedures defined incremental approaches have been completed • Test facilities available and OK • Test objectives identified • Tested software under CM • Applicable documentation, • Lower-level software tests done procedures and plans are complete • Software metrics indicate readiness and under control to start testing • Method of documenting and • Software problem report system is dispositioning “test anomalies” is defined and implemented acceptable • Software test baseline established • Development estimates updated • Nontested requirements at the SI Remember: Typically the Software level are identified for later testing Specification Review (SSR) results in approval of the Software Requirements Specification (SRS) and the Interface Requirements Specification (IRS). Software Testing 177

Software OT Maturity Criteria

No Priority I or II problems Software Problems Impact Analysis for Priority III

 Functionality available before OT&E Functionality  DT completed  External interfaces functionally certified

Software Maturity  PMO ID’s unmet critical parameters  Acquisition Executive must certify (+ “must be Management Operational Tester must agree) that: demonstrated • SW requirement & design stable prior to OT&E” • Depth & breadth of testing is OK • DT has demo’ed required functions

 Deficiency reporting process in place CM Process  Software CM system in place  System is baselined  Test Agency has access to CM system

Fixes  All changes completed Software Testing 178 *OSD (OT&E) Memo Subject: Software Maturity Criteria

89 11/15/2016

Independent Verification and Validation (IV&V)

Verification Validation Ensuring that the system Ensuring that the software is well engineered. meets the users’ needs. Answers Question: Answers Question: “Did I build the system right?” “Did I build the right system?”

Software Testing 179

System DT&E Decision Exit Criteria • Test activities (based on the TEMP and Software Test Plan (STP)) • Demonstrate that the system meets all critical technical parameters & identify technological & design risks documented in the Software Requirements Specification (SRS), Interface Requirements Specification (IRS) and Data Design Specification (DDS). • Ready to perform system OT&E • No open priority 1 or 2 problems • Acceptable degrees of: • Requirements traceability / stability (includes Cybersecurity) • Computer resource utilization • Design stability • Breadth & depth of testing • Fault profiles • Reliability & Interoperability • Software Maturity Software Testing 180

90 11/15/2016

Software Evaluations During IOT&E

• Does the software support system performance? Will the system be accepted by our customer? • Is the software: - Mature & reliable? - Usable by typical operators? - Sustainable/maintainable? • Testers Bottom Line… all cases of software testing, reproducible failures facilitate software fixes. When you identify a problem, document it for reproducibility purposes.

Software Testing 181

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 182

91 11/15/2016

Software T&E Consideration: Software Testing Tools

• Automated software testing is critical for testing large and complex DoD software applications. Tool support is very useful for repetitive tasks; the computer doesn’t get bored and will be able to exactly repeat repetitive tests without any mistakes. • Here are some of the benefits of using automated testing tools. Test tools can: - automatically verify key functionality - do automated Regression Testing - test interfaces - test Graphical User Interfaces (GUI) - provide scenario testing models for Mission Critical Threads (MCT) - automate cloud services testing - help test teams run large numbers of tests in a shorter period of time.

Software Testing 183

Software T&E Consideration: Software Testing Tools

• Successful programs use Static and Dynamic Analysis tools in combination.

- The use of Static Analysis and Dynamic Analysis Tools should be put on contract to ensure COTS vendors provide a picture of their code for Government purposes (Security and Product Support).

• Five (5) Automated Test Tool types:

- Test Management Tools - Static Testing Tools - Test Specification Tools - Test execution and logging Tools - Performance and Monitoring Test Software Testing 184

92 11/15/2016

Software T&E Consideration: Software Testing Tools

Example Results of Static Code Analysis

Original Software Code Software Code After Many Changes

Software Testing 185

Software T&E Issues

• Software design issues that may affect the T&E TEMP (strategy) and Software Test Plan (STP): - Size & complexity of the software. Degree of software reuse. Amount of COTS software. - Role the software will play (safety, security risks, activities to be performed by the software, critical or high risk functions, technology maturity) - Number of interfaces / complexity of integration / number of outside systems requiring interoperability - Cost & schedule available for software testing, previous test results - Software development strategy, stability of software

requirements Software Testing 186

93 11/15/2016

Software T&E Trends

• IT trends that may affect T&E (all of these tend to add more complexity to software testing): - Larger & more complex software packages - More emphasis on software reuse - More challenges / more emphasis on cybersecurity - More challenges / more emphasis on interoperability (part of the Net-Ready KPP) - Rapid pace of technology advances - Open Architecture

187 Software Testing

Six (6) DoD Software Domains T&E Emphasis • Mission Systems software: - Focus is on White Box testing (especially on the embedded software applications) - Does the unique hardware/software system function correctly in its intended operating environment? - System safety & mitigation of key risks - Interoperability with other known systems - Reliability usually critical

• C4ISR, DBS, Infrastructure, Cybersecurity and Modeling & Simulation (M&S) software: - Focus is on White, Black and Gray box testing - Does the software function correctly (often in an office or command center environment on varying hardware) - Cybersecurity, to include privacy are key risks - Interoperability through standards compliance - Reliability may not be as critical Software Testing 188

94 11/15/2016

Agile SW Development T&E Challenges

• Agile software development requires: - a more detailed test plan as test events occur more frequently (Agile Time- boxed Development). - intense configuration management of test results due to the frequent nature of testing. • Inadequate Test Coverage and Application Programming Interface (API) Testing : - With each Sprint, use code analysis tools to monitor code changes to ensure adequate test cases. - Use tools that allow testers to test the API who don’t have strong coding skills. • Code Broken Accidentally due to Frequent Builds; - add tests to an automated script so you can use automated testing on a regular basis. - Automated testing tools are required to finish regression test requirements for each test. Software Testing 189

Agile SW Development T&E Challenges (Continued)

• Early Detection of Defects: (Catch errors as early as possible!) - Use static analysis tools to find missing error routines, coding standard deviations, data type mismatches, other errors that can arise due to frequent build and test cycles. • Performance Bottlenecks: - Identify the areas of your code are causing performance issues and how performance is being impacted over time. - Use load testing tools to help identify slow areas and track performance over time to more objectively document performance from release to release.

Software Testing 190

95 11/15/2016

COTS/NDI Software T&E Issues & Risks

• Unit level testing/component • Real-time performance may testing is generally be marginal impossible (Black or Gray • Robustness & reliability Box testing only) lower when compared to • Incomplete documentation custom code • Multiple complex, non- • Higher COTS use in a standard interfaces system generally implies (interoperability is a risk) more difficult system level • Market leverage may not integration testing (4 or exist to force vendor bug more is a critical risk) fixes • Frequent market-driven • Formal requirements releases complicate documents unavailable regression testing (Promote • COTS usage may not match Intelligent Bypass) original design environment

Software Testing 191

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 192

96 11/15/2016

Software T&E Lessons Learned • Formulate test strategy prior to contract award that accommodates • Understand the test--be cost/schedule constraints prepared to prioritize test cases • Test Strategy should be able to: • Be flexible and attuned to end- - Verify all critical software requirements of system of-schedule pressures - Test in a way to isolate faults • Cop an attitude--know when to • Phase testing to focus on: fall on your sword - Qualification Testing • Understand the politics of the - Operational Thread Testing acquisition - Performance/Stress Testing • Ensure you have Separate • Resolve the “requirements” vs. Development, Test and “design” information argument early- Evaluation organizations to on (Watch out for Gold-Plating) provide independent • Plan ahead for 1) Adequate development of software and schedule, 2) Test Regression T&E. Strategy, 3) Timing and format of deliverables and 4) Accommodating incremental builds. From “A Real Life Experience in Testing a 1,000,000+ SLOC Ada Program” STC 1994 193 Software Testing

Arlie Council 16 Best Practices

Product Integrity & Arlie Council 16 Stability (Tester) Software Management Best Practices* 14) Inspect Use Requirements Formal and Design Inspections! 15) Manage Testing as a Integration & Acceptance Continuous Throughout! Process Use 16) Compile & Daily One Example of Smoke Test Software Compile & Management Frequently Smoke “Best Practices” Testing!

*As developed by the Software Program Managers Network (www.spmn.com)

97 11/15/2016

Elephant Bungee Wisdom

Universal Truth #9: COTS are not necessarily the best solution. They bring risks + benefits…understand both!

Management Emphasis • Investigate the pricing structure • Select established products with large installed bases • Budget for the complete cost of COTS integration and maintenance • “Fly before you buy” + TEST a lot • Adapt your requirements to COTS

Elephant Bungee Jumping #9: Avoiding Diseases that Are Fun to Catch

Employing IV&V on Agile Projects: Lessons Learned • Identify the Control Points in the Agile Process. The control points are where decisions need to be made. Decisions on concept documentation, use cases and test results can be independently assessed by the IV&V team to ensure accuracy, completeness, consistency and traceability to the warfighter requirements. • Review the Backlog. IV&V is used to ensure accuracy, completeness, consistency and testability of user stories in the backlog. IV&V assesses the desired change for impact on existing tasks and products. • Manage the Roadmap. IV&V is used to ensure each iteration within a Sprint is in concert with the overall Agile Project Roadmap. • Question Business Value. IV&V will review the Agile Minimum Viable Product (MVP) to ensure that what was produced in a Sprint matches the MVP. The IV&V team will validate each new user story or business requirement to ensure it adds the proper value to the warfighter. • Validate Consistent Practices. The IV&V team can act as the “process cop” to ensure the agreed to repeatable processes in place remain in place from Sprint to Sprint.

196 Software Testing

98 11/15/2016

Class Exercise (Which Team Can Solve the Quickest?)

Short Range Assault Weapon (SRAW) Program

197

Class Exercise

ROUTINE 6-1 Contractor Software Code

198

99 11/15/2016

Class Exercise

SRAW SOFTWARE DESIGN DIAGRAM

199

Class Exercise

SRAW CONTRACTOR TEST REPORT

200

100 11/15/2016

Class Exercise

What type of testing did you do to discover the error?

201

Lesson Overview Lesson Plan

• Software Testing Overview • Software Development and Testing Process • Software T&E Considerations, Issues and Trends • Software Testing Lessons Learned & Best Practices • Summary

Software Testing 202

101 11/15/2016

Summary Today we learned to:

• Recognize the characteristics of Software Testing. Identify defects as early as possible! • Recognize the purpose of a Test Readiness Review (TRR). TRR ensures readiness for the next formal test. • Identify what Independent Verification and Validation (IV&V) is. IV&V means you have an independent team doing independent testing on critical risk areas of your software (e.g., safety). • Identify the T&E responsibilities of the IT Program Manager. The PM is responsible for defining software maturity as part of the Program’s Software Quality definition so we know when T&E is considered successful (done). • Identify the software testing issues and risks. Each software domain has a different architecture. Ensure you plan your tests to address size, complexity, reuse, COTS and integration challenges of that domain.

Software Testing 203

Summary Today we learned to: • Recognize the different types of software testing tools. Use of automated test tools is critical for testing today’s large and complex DoD software applications. Successful programs use Static and Dynamic test tools in combination. • Recognize the Agile software development T&E challenges. T&E events happen more frequently in an Agile development. Use of automated tools and more frequent test plan adjustments are critical to success. • Identify software test issues and risks in building a Software Test Plan (STP) for a COTS/NDI only system. There are a number of risks in testing COTS/NDI, like not being able to see the source code; Ensure you contract for Static and Dynamic analysis results as a deliverable from the vendor if your system requires higher levels of reliability and security.

Software Testing 204

102 11/15/2016

Summary Today we learned to: • Recognize the Software T&E Best Practices. There are a number of Software T&E Best Practices and Lessons Learned. Emphasize Formal Inspections for requirements, design, test plans and source code to ensure accuracy. • Recognize the best practices of employing Independent Verification and Validation (IV&V) on Agile Projects. Identify Control Points, Understand the Backlog status, Understand the software development roadmap, use IV&V to ensure deliverables provide value to the warfighter and validate consistent practices.

Software Testing 205

Lesson 16 Contracting

103 11/15/2016

Learning Objectives Today we will learn to (1 of 2):

• Explain the connection between the acquisition of Information Technology systems and the rules and regulations guiding those acquisitions as presented in the FAR and DFARS. • Explain the relationship among the Contracting Officer, Program Manager, and Contracting Officer's Representative (COR) • Recognize the association of acquisition planning to the contracting strategy. • Recognize the particular aspects of market research unique to IT • Given an IT requirement, choose between a development and a commercial acquisition contracting approach

Contracting 207

Learning Objectives Today we will learn to (2 of 2):

• Identify the elements of Performance-Based Acquisitions (PBA) and Performance-Based Services Acquisitions (PBSA) • Given a scenario, differentiate between a requirement that is performance or outcome-focused versus one that is method or procedure-focused • Recognize different types of data rights • Given a scenario, identify the requirement for a Modular Contracting solution • Given a scenario, evaluate an acquisition strategy that offers optimal opportunity for competitive acquisition

Contracting 208

104 11/15/2016

Lesson Overview Lesson Plan

• Regulations & Guidance • Process Review and Role of the Contracting Officer • Market Research • Commercial Acquisition • Competition • Performance Based Acquisition • Data Rights • Lesson Exercise

Contracting 209

Regulations & Guidance

Federal Acquisition Regulation - http://farsite.hill.af.mil

DFAR Supplement - http://www.acq.osd.mil/dpap/dars/dfar spgi/current/index.html Contracting 210

105 11/15/2016

Lesson Overview Lesson Plan Status

• Regulations & Guidance • Process Review and Contract Administrative Team • Market Research • Commercial Acquisition • Competition • Performance Based Acquisition • Data Rights • Lesson Exercise

Contracting 211

The Contracting Process

Contracting 212

106 11/15/2016

The Contract Administrative Team

• Program Manager—key role in monitoring contractor performance • COR—Frequently the technical expert for the acquisition; evaluates/accepts deliverables • Contracting Officer—Legal authority to sign contract and make changes - Ensures contracts comply with laws/executive orders/regulations - Ensures government as well as the contractor are in compliance with the contract

Contracting 213

COR “Tools of the Trade”

What tools does a COR need to monitor contractor performance?

• Contract • Letter of Appointment • Contract Administration Plan • Quality Assurance Surveillance Plan (QASP)

Contracting 214

107 11/15/2016

The Law Of Agency

The relationship between a principal (Contracting Officer) and their agent (COR) that is created when the principal authorizes the agent to act on behalf of the principal in dealings with a third party (the Contractor).

Contracting 215

Authority Of Agents

Expressed Authority

• Agreed to in expressed & explicit language (e.g., COR Letter of Appointment)

Implied Authority

• Dealings conducted in conformity with general custom

Contracting 216

108 11/15/2016

Acquisition Strategy vs. Plan

Acquisition Strategy Acquisition Plan

Required by DoDI 5000.02, Enclosure 2, paragraph 6a FAR 7.1

Required for All acquisition categories Contracting or procuring for development activities when the total cost of all contract for the acquisition program is estimated at $10M or more Approval Milestone Decision Authority (MDA) CAE or designee IAW Agency FAR supplements Authority Purpose Describes overall strategy for managing the Comprehensive plan for implementing the contracting acquisition program strategy Use Required at program initiation. It should be Integrates the efforts of all personnel responsible for updated for all subsequent milestone and significant aspects of the contractual agreement. The whenever the approved strategy changes. purpose is to ensure that the Government meets its needs in the most effective, economical, and timely manner. Level of Strategy level. Needed by MDA for Execution level….to guide contractual implementation Detail decision-making. and conduct acquisitions Content Prescribed by DoDI 5000.02 and DAG, Prescribed by FAR 7.1; DFARS 207 Chapter 2 & 12 POC PM Person designated as responsible

The Acquisition Plan must be constructed so that it compliments and supports the Acquisition Strategy

Contracting 217

Lesson Overview Lesson Plan Status

Contracting 218

109 11/15/2016

Market Research – How is it used?

Market research information can be used to: • Shape the acquisition strategy • Determine the type and content of the product description or statement of work • Develop the support strategy • Craft the terms and conditions included in the contract • Formulate the evaluation factors used for source selection

Contracting 219

Strategic and Tactical Market Research

Contracting 220

110 11/15/2016

Lesson Overview Lesson Plan Status

Contracting 221

Contracting for Traditional SW Development and Agile SW Development

Traditional Pre-Award Agile Pre-Award • Needs identified • Needs identified • IPT formation • IPT formation • Detailed requirements • Product vision • Product road map

Traditional Post-Award Agile Post-Award • Releases at the end of long, linear • User Stories development phase • Release planning • Linear approach to design, Sprints development and testing • • Performance measurement – • Releases contractor held to standards • Performance measurement – determined pre-award document throughout each Spring

TechFAR Handbook: https://github.com/usds/playbook/blob/gh-pages/_includes/techfar-online.md Contracting 222

111 11/15/2016

Commercial Products & Services

Acquisition processes for Information Resources must:

• Conduct market research to determine whether commercial items or nondevelopmental items (NDI) are available to meet requirements - A NDI is any previously developed item of supply used exclusively for government purposes by a federal agency, a State or local government, or a foreign government with which the US has a mutual defense cooperation agreement. • Acquire commercial items or nondevelopmental items when they are available; and • Require prime contractors and subcontractors at all tiers to incorporate, to the maximum extent practicable, commercial items or nondevelopmental items as components

Reference: FAR Part 12, Acquisition of Commercial Items Contracting 223

Commercial Planning Process

• Define requirements both at a high level and at a detailed engineering level • Identify a candidate commercial or nondevelopmental item that appears to best meet the requirements • Develop a life-cycle support plan for the item • Test and evaluate the item to determine its ability to satisfy the requirements • Plan and implement the acquisition strategy.

Contracting 224

112 11/15/2016

Benefits of Buying Commercial In Class Exercise

1. Read SD-2, Buying Commercial Items and Non- Developmental Items (pp. 4-5) 2. Answer these questions: i. How might buying commercial offer improved scheduling? ii. How might buying commercial produce cost savings? iii. How can buying commercial result in superior performance or higher quality?

Contracting 225

Lesson Overview Lesson Plan Status

Contracting 226

113 11/15/2016

Competition

• Lowers prices • Spurs innovation • Drives quality improvements • Encourages performance improvements by using “best value” source selection criteria • Provides opportunities for capable small businesses or non-traditional sources • Enhances or maintains a strong defense industrial base • Promotes transparency and fairness in the Defense Acquisition System

“Real competition is the single most powerful tool available to the Department to drive productivity.” - USD (AT&L) Better Buying Power, http://bbp.dau.mil/bbp5focus.html Contracting 227

Competition Lowers Prices

How does competition create lower prices?

Contracting 228

114 11/15/2016

Competition Spurs Innovation

How does competition spur innovation?

Contracting 229

Transparency & Fairness

How does competition promote transparency and fairness; why is this important?

Contracting 230

115 11/15/2016

Exceptions to Competition

• Only One Responsible Source and No Other Supplies or Services Will Satisfy Agency Requirements - Not the same as the only source we know about - Must determine a boundary prohibiting all but one contractor from performing – for example proprietary data rights • Unusual and Compelling Urgency (no time to compete fully) - Can’t be due to poor Government planning or potential loss of funding

These 5 Exceptions are the Result of Decisions Outside of or Above the Level of the Program Office or Contracting Officer

• Tailored Competition to Support Industrial Mobilization; Engineering, Developmental, or Research Capability; or Expert Services at a National Level • No Competition as Part of an International Agreement • No Competition Authorized or Required by Statute • Support for Special National Security Issue • Support for Special Public Interest

Contracting 231

Lesson Overview Lesson Plan Status

Contracting 232

116 11/15/2016

Performance Based Acquisition

http://www.dtic.mil/whs/directives/corres/pdf/500074p.pdf Contracting 233

Performance Based Acquisition

The established Government preference to describe a contract requirement in performance based terms For Services • Describe the work in terms of the required results rather than either “how” the work is to be accomplished or the number of hours to be provided • Enable assessment of work performance against measurable performance standards • Rely on the use of measurable performance standards and financial incentives in a competitive environment to encourage competitors to develop and institute innovative and cost-effective methods of performing the work For Hardware or Software Items • Describe the requirement in terms of functionality or resulting output rather than production or manufacturing level characteristics or drawings/specifications • In essence, the what, not the how

Contracting 234

117 11/15/2016

Performance Based Acquisition

Government may write a Performance Work Statement (PWS)

• Describe the work in terms of the required outcomes, results, or even impact • Identifies measurable performance standards by which after contract award the contractor will be judged for contractual acceptance • Contract may include financial incentives to foster higher than required quality, innovative and/or cost-effectiveness

Government may issue solicitation with no PWS but with a Statement of Objectives (SOO)

• SOO describes overarching objectives of the contract in broad terms and outlines constraints that will bound the contractors work • Contractors write/propose a PWS that incorporates their particular methodology to meet SOO objectives • Government selects the best offer and uses successful offerors’ PWS as the contract work statement

Contracting 235

Performance Based Acquisition

For example:

Not Performance Based—“Contractor shall provide two senior IT specialists, as defined in section 4.3.2 of this Statement of Work (SOW), from 0600 to 1800 Monday through Friday and shall perform the tasks as outline in Air Force Instruction 36.001 paragraphs 7–11 through 7–55 to insure optimal functionality of the system”

Performance Based with measurable performance outcome— “Contractor shall insure that the system allows full functional usage by organizational personnel for at least 95% of the time during the hours 0600 to 1800 Monday through Friday.”

Contracting 236

118 11/15/2016

Lesson Overview Lesson Plan Status

Contracting 237

Technical Data, Software and Intellectual Property

• Data Rights, Rights to Software, and Intellectual Property Issues have been the downfall of many programs • Awareness and planning upfront is the best safeguard • Can be very legalistic—consult with the contracting officer and attorneys

Contracting 238

119 11/15/2016

Quick Review of Data Rights

Operations, Maintenance, Installation, Training; Form, Fit, Function; Computer Software Documentation

Contracting 239

Commercial Software Rights

• Related to commercial software • No DFARS clause for commercial software - Rights come from licenses available to the general public - Receive same rights as general public - These licenses may/may not include a section providing rights to government that are not available to the public (such as rights to government funded modifications) • Contractor is not obligated to provide additional rights to the Government - Additional rights can be negotiated - If negotiated, such rights shall be listed/described in a license/agreement which is made part of the contract

Contracting 240

120 11/15/2016

Commercial SW Rights (Continued)

• A presumption exists that a commercial item was developed at private expense - But this presumption can be challenged when warranted • Keep track of commercial software licenses • Know what the license allows - Software use, upgrades, bug fixes, maintenance, modifications • Know what the license does not allow - Restrictions—one or more locations, number of users, named user or site license, only specified applications, no sub-licensing - Reverse engineering prohibition • Caution - Licenses must be governed by/interpreted under applicable federal law—not state law - Cannot indemnify Licensor against future law suits or other legal actions (anti-deficiency violation) - License may not allow contractors to access software

Contracting 241

Specific Needs for Data Rights Varies Acquisition to Acquisition

Every decision about data rights involves tradeoffs • Obtaining more rights generally: - More expensive - Requires unique for-Government type development which takes longer - Assumes ultimately Government will possess more support responsibility • Obtaining less rights generally: - Less immediate expense - May be available immediately - Less control by Government - Supportability continues at will of the contractor or added cost

Contracting 242

121 11/15/2016

Lesson Overview Lesson Plan Status

Contracting 243

Exercise

Contracting Decision Tradeoffs

Contracting 244

122 11/15/2016

Exercise Scenario

The Army intends to use surplus ground vehicles as targets. The requirement is to acquire a single means to create autonomous drones from a variety of types of ground vehicles ranging from: armored personnel carriers, tanks, to future systems yet to be surplus. Cost, simplicity, and ease of system adoption throughout the lifespan to final disposal are program priorities.

Note: There is an IT portion of the system…the guidance system and associated software.

Contracting 245

Exercise Assignment – Formulate a High Level Acquisition Plan

Each team is tasked with quickly formulating a high level acquisition plan for an assigned focus area.

The assigned focus areas are: 1. Maximize competition (Teams 1 and 4) 2. Maximize the use of commercial products and services (Team 2) 3. Maximize performance based acquisition as a strategy (Teams 3 and 5)

Give consideration to the following in the acquisition plan: - The entire system lifecycle (development to disposal) - Structuring data rights - Use of modular contracting - How market research will be conducted

Each team will provide the class with a 5 minute overview of their acquisition plan.

Contracting 246

123 11/15/2016

Summary

Today we learned to (1 of 2):

Contracting 247

Summary

Today we learned to (2 of 2):

Contracting 248

124 11/15/2016

Back Up

Contracting 249

Role of the Contracting Officer

The Contracting Officer is the designated individual with the authority to enter into, administer, and/or terminate contracts and make related determinations and findings.

The contracting officer: • Manages the process to establish and maintain a business relationship (contract) with a private entity (contractor) • Serves as the single authoritative voice between the Government and contractors • Focuses on “protecting” the terms of a contract rather than advocating for Government vs. Contractor • Is responsible for both supporting a Government project/program/customer as well as protecting the integrity of the broader contracting and acquisition process

Contracting 250

125 11/15/2016

Technical Contract Administration—COR

• Contracting Officer’s Representative (COR) has the duty of overseeing technical contract performance and ensuring timely, efficient delivery of purchased items/services - Appointed by the Contracting Officer, in writing - Required to undergo formal training (as determined by the components) prior to assuming duties (DFARS 202.101) - Functions as the “eyes and ears” of the Contracting Officer

Contracting 251

Market Research – What is it?

“The process of collecting and analyzing information about capabilities within the market to satisfy agency needs”

Continuous process gathering data on: • Business and industry trends • Characteristics of products and services • Suppliers’ capabilities • Related business practices

Contracting 252

126 11/15/2016

Market Research – When is it required?

Federal Acquisition Regulation (FAR) Part 10 essentially requires market research: • Before developing new requirements documents for an acquisition • Before soliciting offers • On an ongoing basis

Contracting 253

Market Research – Two Approaches

Strategic—Market Surveillance - Overall Market Developments - Emerging Trends and Capabilities

Tactical—Market Investigation - Specific point in an acquisition to gather information about capabilities, products or services available in the market place

Contracting 254

127 11/15/2016

Competition is a Requirement

• Public law requires that contracting officers promote and provide for competition - Competition must consider price but may be more heavily weighted to other factors such as schedule, technical excellence, or record of past performance - Competition may be limited to a subset of potential offerors, for example a small business set-aside • Only formal competitions conducted by Contracting Officers satisfy the statutory requirements - Users or Program Managers can’t conduct their own “competition” to expedite the process • Utilizing exceptions to the requirement for competition must include a plan to reintroduce competition in future contracts

Contracting 255

Technical Data, Computer Software and Documentation

• The provisions for rights in technical data are contractually separated from rights in computer software/documentation but are treated similarly • In general, rights are categorized based on extent that the Government paid for development - Unlimited Rights: developed exclusively with Government funds - Government Purpose Rights: mixed funding - Limited Rights (technical data)/Restricted (software): exclusively at private expense • Contractor marks deliveries with a legend identifying rights restrictions - Government has a right to review, verify, challenge, and validate contractor asserted restrictions

Contracting 256

128 11/15/2016

Contracting257

258

129 11/15/2016

Modular or Flexible Contracting

• Modular contracting—acquisition of a system for information technology divided into several smaller acquisition increments • Intended to: - Reduce program risk - Incentivize contractor performance - Meeting the Governments need for timely access to rapidly changing technology • Each increment should utilize an appropriate contracting technique that best facilitates the acquisition of subsequent increments • Modularity avoids the potential negative effects of one development-to-production contract by introducing check points and market competition forces key program transition points between module contracts/contractors • Recall: Clinger Cohen Act, element #10: “To the maximum extent practicable, (1) modular contracting has been used, and (2) the program is being implemented in phased, successive blocks…” Contracting 259

Lesson 18 Student Panels

130 11/15/2016

Learning Objectives Today we will learn to:

• Describe emerging and advanced information technologies. • Identify the basic concepts, threats, and best practices associated with cybersecurity in the DoD. • Given acquisition scenarios with several software-reliant program issues, analyze how each issue may affect achieving the program's software quality objectives; Identify quality issues unique to software • Compare and contrast the major DoD software development paradigms and recognize new development paradigms and trends • Track how the DoD information technology systems engineering process supports DoD Acquisition Management processes • Describe the keys to successful software sustainment and support. • Recognize the benefits of consuming cloud services; Recognize public, private, community and hybrid cloud deployment models (NIST); Recognize some DoD Concerns of Using Cloud Services

Student Panels 261

Lesson Overview Lesson Plan

• Panel Rules and Possible Approaches • Potential Panel Topics

Student Panels 262

131 11/15/2016

Student-led Discussion Panels

Software Development / Software / Systems Cybersecurity Quality Engineering

Cloud Computing Software Support Emerging Technology

More information is available in the Student panel directory on the Student Compact Disk (CD)

You can also get information in the references folders of each class (also on your CD) Introduction 263

Student Panel Rules

• Rule Number 1: Format is your call - Within bounds of Good Taste and Propriety! - Related to topics covered in class or at least associated with Information Technology (IT) or Software Intensive System (SIS) areas of interest. - You can replace an existing topic with one of your choice provided it meets the requirements (min of 4 team members); however, it has to replace another topic. - 30-minute maximum time. • Rule Number 2: Coordinated Effort - Minimum of 4, Maximum of 6 per team. - Class time will be made available throughout the two weeks for preparation. • Rule Number 3: Opportunity - Make this a Learning/Teaching experience for the entire class.

Introduction 264

132 11/15/2016

Possible Ways to Approach Student Panel Discussion

• Standard Presentation format • Individual Panel member throws out a topic for discussion and then elicits class responses • Panel members present topic pros and cons and then involve class in discussion • Role play—e.g., One panel member is PM, another is an cybersecurity professional—then discuss • Point/counterpoint format • Game Format (like "Jeopardy", or "Are You Smarter than a Fifth Grader?", or "Who Wants to be a Millionaire?")

Introduction 265

Lesson Overview Lesson Plan Status

• Panel Rules and Possible Approaches • Potential Panel Topics

Student Panels 266

133 11/15/2016

Emerging Technology

• Unclassified info-brief about projects on which you are currently working • Discussion of other emerging IT related capabilities and their impact to the DoD

Next-Gen Robotics Biochips

Big Data Quantum Computing

Wearable User Interfaces 3D/4D Printing

Student Panels 267

Cybersecurity

• Biggest Cyber problem • Cyber Policy and guidance within DoD • Types of Cyber Attacks • State of DoD Cyber • Access Control compared to private sector • On-line Cyber Resources • Confidentiality, Integrity, Availability (CIA)—what • Number 1 security threat does DoD value the most? • Cyber espionage / terrorism Least? • Insider threat • Software Assurance—What • Cyber Tools is it? Is it utilized as needed • Recent Cyber Trends within DoD or is it an • Effectiveness of common afterthought? security tactics • Software vulnerabilities • Cyber Best Practices • Emerging Cyber Technologies

Student Panels 268

134 11/15/2016

Cloud Computing

Cloud Policy and guidance • Biggest Cloud problem within • DoD • Cloud Resources • State of DoD Cloud compared • Cloud Tools to private sector • Recent Cloud Trends • Emerging Cloud Technologies • Cloud Best Practices • Cloud providers • Limitations in transitioning to • Cloud transition strategies Cloud • Benefits of transitioning to • Personal experiences with Cloud Cloud implementations • Cloud terminology • Mission-related considerations with transitioning to Cloud

Student Panels 269

Software Quality

• Defining and measuring • Software quality Processes software quality (metrics) - Lean Six Sigma • Software quality initiatives - Continuous Process - Practical Software and improvement Systems Measurement - Software engineering (PSM) - Software Cleanroom - Relating developer process Engineering maturity to software quality • Software Quality factors - ISO 9000 series (ilities—reliability, portability, - Other quality initiatives …) • Software quality best practices • Preventing common types of software errors

Student Panels 270

135 11/15/2016

Software Development

• Software development • Software Assurance paradigms and the relative - Vulnerabilities success or failure: - Best Practices - Rapid Application • Capability Maturity Model Development (RAD) Integrated (CMMI) Agile - - As a predictor of quality Extreme Programming - As a predictor of schedule (XP), Scrum, Crystal, - and cost overruns Feature Driven Development, Open • COTS Software benefits Source Software and challenges Development, … • Data Management • Data Rights

Student Panels 271

Software Support

• Software Support best • Software reuse and practices software support • Software maintenance and • System transition critical maintenance process success factors • The Software Support Plan • Software deployment and other planning strategies considerations • System disposal issues • The importance of support • The Software Transition planning throughout the Plan acquisition lifecycle • The Software Installation • Commercial Off The Shelf Plan (COTS) Software support considerations

Student Panels 272

136 11/15/2016

Software / Systems Engineering

• Systems Engineering and • Systems Engineering and Software engineering technical reviews • Applying the systems • Architectures engineering process to software • Systems engineering technical processes • Systems engineering technical - Stakeholders Requirements management processes Definition - Decision Analysis - Requirements Analysis - Technical Planning - Architectural Design - Technical Assessment - Implementation - Requirements Management - Integration - Risk Management - Verification - Configuration Management - Validation - Technical Data Management - Transition - Interface Management

Student Panels 273

Summary Today we will learned to:

Student Panels 274

137 11/15/2016

Lesson 19 Software Support

Learning Objectives Today we will learn to:

• Identify when software (SW) support activities occur throughout the acquisition life cycle • Analyze software sustainment planning considerations. • Review the criteria and related areas necessary for transitioning software from development to sustainment. • Describe the rationale for software sustainment • Understand software sustainment concepts related to hardware and software sustainment and maintenance. • Explain why early sustainment planning is essential for effective sustainment. • Identify challenges with sustaining COTS reliant systems • Explain the elements of the DoD computer and software disposal process.

Software Support 276

138 11/15/2016

Lesson Overview Lesson Plan

• Software Lifecycle Planning • Lesson Exercise 1 • Sustainment and Support • Disposal • Lesson Exercise 2

Software Support 277

Lesson Overview Homework review

Team discussions from Camtasia: • Team 1 -- Explain the difference and relationship between design for support and support the design • Team 2 – Discuss the primary planning considerations for software sustainment that should be addressed in program IPTs • Team 3 – Identify and discuss some of the categories that should be assessed when considering software for reuse • Team 4 – Identify and discuss the key documents that influence software sustainment • Team 5 – Identify and discuss the deployment methodologies

Software Support 278

139 11/15/2016

The Software Iceberg: More to Manage than Meets the Eye

Must plan to manage more than just this

Many considerations when conducting software sustainment planning. SW development environment needs to be replicated. Software Support 279

Software Support Planning and the Acquisition Life Cycle

(LCSP)

(LCSP) (LCSP) (LCSP) Software Support 280

140 11/15/2016

Software Development Planning: Product Support Planning Requirements

1. What are the projected costs to fix/update the Software Item (SI) to be developed in O&S? 2. Is the software architecture designed with open interfaces to ensure plug and play updating with new module? 3. Are data rights in place to ensure we can update/fix bugs? 4. What tools will the SSA need to fix/update the software? 5. How is the developed data managed and stored so the SSA knows how to manage and store data during O&S? 6. Is the code being developed using secure coding practices? Static and Dynamic code analysis (NDAA) 7. What is the proposed Security Architecture of the SI being built? Software Support 281

Software Development Planning: Mechanisms Overview

• Business Case Analyses help identify pros/cons of various software development/sustainment COAs. • 2 BCA versions related to IT management: - IT BCA - Required to support IT investment decisions for the Investment Review Board (IRB) - Goal is to justify ROI - DOD CIO Memo 23 Oct 2014 - PS BCA - Required to support each MS decision and every 5 years after deployment - Goal is to optimize holistic sustainment approach - USD AT&L WSAR 2009, NDAA 2010 sec. 805 - Can support selection of Software Support Activity (SSA) - May also consider warranty implications (CBA) for COTS sustainment Software Support 282

141 11/15/2016

Software Deployment—Strategies

Lifecycle Planning 283

Deployment Strategies Cost, Effort, and Risk

Method Cost User Effort Team Effort Risk

Parallel High High Low Low

Pilot Medium Low Medium Medium

Phased Medium Medium Medium Medium

Immediate Low Medium Low High

“When you deploy your system, expect the unexpected!”

Lifecycle Planning 284

142 11/15/2016

Software Transition: Criteria

Necessary activities prior to transitioning to sustainment: • Source of Repair Assignment Process (SORAP) • Test & Evaluation • Stable Software Baseline The Software Transition Plan • Complete Documentation identifies the resources needed to support delivered software • Authority to Operate and describes the developer’s • Software Transition Plan plans for transitioning delivered software to the • Staffing and Training Plan support agency (MER/MP&TP)

Sustaining Software Intensive Systems, Lapham, CMU/SEI-2006-TN-007

Software Support 285

Software Transition: Activities

What must be considered for transition?

• Support database(s) • Development environment infrastructure • Software Test Lab/ Software Integration Lab • Release processes/procedures • Staffing • Operations and Maintenance training • System documentation • Stable software baseline • ATO

Software Support 286

143 11/15/2016

Software Support Activity (SSA)

• Assumes the role of providing post-deployment software support for modifications or upgrades made to a system's software following the system's initial fielding. • The SSA organization typically compiles updates into formal software releases to avoid disrupting the fielded system. • Software Maintenance activities performed by a SSA are the same as those carried out during the development effort that led to the first fielding. They are tailored, as appropriate, to reflect the effort required to implement each change package, update pertinent documentation, verify the changes, and

distribute the changes to users. Software Support 287

Software Support Activity (SSA)

• Due to the scope and nature of SSA activities, and given the ‘7 factors’, they need to be involved, early identification and incorporation of SSAs into planning is a key enabler to successful sustainment. • Considerations/Lessons Learned include: - Identifying the sustainment strategy early: What functionality/authority will the SSA have, and what information will they require? - Use an early PS BCA to identify candidate SSAs - Working with the SSA to identify their needs, ensure appropriate development contracts/deliverables are in place. - Establish SSA performance expectations – measures/metrics - Use SLA/MOA/MOU to articulate the SSA relationship. - SSA can help create the Software Transition Plan (STrP) to discuss how they will assume control from the developer.

Software Support 288

144 11/15/2016

Lesson Overview Lesson Plan Status

• Software Lifecycle Planning • Lesson Exercise 1 • Sustainment and Support • Disposal • Lesson Exercise 2

Software Support 289

Software Support Exercise 1 CCB and Open ECPs

Just before the program was scheduled for a Configuration Control Board meeting the engineers found that more than 150 open ECPs (Engineering Change Proposals) exist. Many are overdue for incorporation, and some ECPs implement a change of an earlier ECP. Replacing system components and corresponding code as a result of these ECPs could impact system performance. The program needs to understand the magnitude of these outstanding ECPs and the risk before PDR. What’s the plan? How should the user be involved?

Task: Determine the root cause for this issue. Prepare your recommendation for resolving the outstanding issues. Identify potential impacts to deployment and sustainment activities.

Software Support 290

145 11/15/2016

Lesson Overview Lesson Plan Status

• Software Lifecycle Planning • Lesson Exercise 1 • Sustainment and Support • Disposal • Lesson Exercise 2

Software Support 291

Software Sustainment Key Considerations: Rationale

• Rationale for software sustainment: - Software provides an increasing proportion of functionality in our weapon systems, growing at an exponential rate. - Software Maintenance is growing faster than new software development - ~1996, software development was roughly 18Million LOC, while Maintenance was ~20Million LOC (111%) - ~2011, software development was roughly 38 Million LOC, while Maintenance was ~54Million LOC (142%)

Software Support 292

146 11/15/2016

Software Sustainment Key Considerations: Sustainment vs Maintenance

• Software Sustainment - Comprehensive requirements to support, maintain and operate the software capabilities of a system. - Includes processes, procedures, people material and information required to support. • Software Maintenance - One facet of software sustainment. - Software failures not subject to the ‘laws of physics’ - Software maintenance is essentially development, creating a new version (baseline). • Hardware Maintenance - Failures are subject to the ‘laws of physics’ - Hardware maintenance typically returns a failed system to its previously established baseline. • Implications - Software maintenance requires a ‘redevelopment’ environment - Software maintenance drives additional considerations inherent to a new version (baseline), including configuration management/control, updated manuals and training, distribution process (release management), etc.

Software Support 293

Software Sustainment Key Considerations: Rationale

• There are 8 drivers of software change throughout sustainment - Defect corrections - Responding to Threats - Policy and doctrine - Safety - Enhance Interoperability - Hardware changes - Technology insertion - Functional changes based on user requests

Software Support 294

147 11/15/2016

Software Sustainment Key Considerations: Types of Software Maintenance

Traditional ‘Maintenance’ (fixing ‘bugs’) • Corrective Maintenance - Reactive modification to correct discovered problems • Adaptive Maintenance - Modification to maintain usability in a changed environment • Perfective Maintenance - Provide functional enhancement to users • Preventative Maintenance Perfective, Adaptive, and Preventative Enhance maintainability - (80%) are considered “software evolution”

In 1995, it was estimated that $70Billion was spent maintaining 10Billion LOC in the U.S. As seen in other acquisition areas, ~70% of SW costs are in sustainment. In 1995, sustainment costs were ~$7 per year per KLOC. WhatLifecycle might Planning it be today?295

Software Maintenance Activities

Software Sustainment entails developing a maintenance capability that mirrors developmental environment. Software maintenance activities include: • Understanding Requirement • Understanding Existing Code • Implementing Change • Checkout & Deployment

However, poor documentation of existing code can dramatically increase the effort required to ‘understand the product’ – often up to 50% of the entire sustainment effort. The effect is: • Increased sustainment costs, • Longer schedule requirements, or • Reduced testing and debugging (higher failure rates post-deployment). “Pay me now or pay me later…” Lifecycle Planning 296

148 11/15/2016

Keys to Successful Sustainment (1 of 2)

• Strive for Commonality - Ultimate goal is to reach consensus on a common sustainment solution and, thereby, minimize the incidence of multiple system/software configurations. • Apply Industrial Engineering Practices to Software - Parameters like the frequency and quantity of software changes, the number of versions, and the time required for development, validation, and distribution must be considered early on. • Engage Customers - Each stage of sustainment should engage users and relevant subject matter experts. - Ensure adequate training is provided when needed - Use a closed loop system to capture customer feedback (e.g. Help Desk metrics) • Adopt a Holistic Approach to Sustainment - The effect of any software change should be evaluated in terms of net worth provided to the warfighter.

Software Support 297

Keys to Successful Sustainment (2 of 2)

• Develop Highly Maintainable Systems and Software - Enhance ability for SSA to ‘understand the product’ by reducing complexity and ensuring documentation is accurate. - Ensure documentation is kept up to date - Leverage an open and scalable architecture to increase maintainability. - Allows for expansion with minimal impact to unchanged elements. • Manage Off-the-Shelf Software - Up-front savings of COTS frequently offset by risk/expense later in the product life cycle. - This wreaks havoc on the program without appropriate management. • Plan for the Unexpected - Develop realistic and detailed scenarios to minimize surprises. - Plan for unexpected growth – the ‘1% rule’. • Analyze and Refine the Software Sustainment Business Case - Address business case analysis sustainment, annual sustainment total ownership cost estimates, and software cost estimation practices that support these analyses.

Software Support 298

149 11/15/2016

COTS–Intensive Systems— Impact on Sustainment (1 of 2)

Software Support 299

COTS–Intensive Systems— Impact on Sustainment (2 of 2)

• Choosing a government SSA/Product Support Provider - Data rights and licensing considerations with COTS items - Service Level Agreements/MOU/MOA – (GOTS?) - Functional authority (Help Desk, Configuration Management/Change Control, Information Assurance, etc) - Performance measures reporting/frequency • ‘Modified’ COTS - Discouraged, creates unique configurations that marginalize COTS benefits. • Counterfeit/Grey Market/Malicious software considerations - DODI 4140.67 Counterfeit Prevention Policy, Apr 2013 - CLL032/CLL062 - DODI 5200.39 Critical Program Information Protection within DOD • Warranty considerations - BCA to consider ROI for extended warranties Software Support 300

150 11/15/2016

Lesson Overview Lesson Plan Status

• Software Lifecycle Planning • Lesson Exercise 1 • Sustainment and Support • Disposal • Lesson Exercise 2

Software Support 301

IT Disposal

IT/ Software Disposal Process–Purpose (per ISO/IEC 12207:2008, Systems and Software Engineering) • To end the existence of a system’s software entity. • Ends active support by the operation and maintenance organization, or deactivates, disassembles and removes the affected software products, consigning them to a final condition and leaving the environment in an acceptable condition. • Destroys or stores system software elements and related products in a sound manner, in accordance with legislation, agreements, organizational constraints and stakeholder requirements. • Where required, it maintains records that may be monitored. Objective is to retire a system's existing software products or services while preserving the integrity of organizational operations. Software Support 302

151 11/15/2016

Impacts of Improper Disposal

September 2009

Some Defense Department organizations haven't scrubbed data from information technology equipment before disposing of the hardware, resulting in the possible release of information that could be used for identity theft, or releasing other sensitive DoD information, according to an Inspector General audit.

An investigation by DoD's IG also found that one organization had lost track of one unclassified computer entirely, the report said.

Software Support 303

IT Disposal: Considerations (1 of 2)

• Hardware - Consider residual data that can be reconstructed – sanitize! • Storage Media - Consider residual magnetic, optical electrical or other data representations – sanitize! • Software (must also consider data remanance) - Overwriting - Degaussing - Destruction (Source: IPSE Guidebook, 2011; Computer Resources Support section) NIST Special Publication 800-88 also contains information regarding security and privacy in cloud computing Software Support 304

152 11/15/2016

IT Disposal: Considerations (2 of 2)

• US Policy Guidance - Reuse within own agency/organization - Post notice on the Defense Information Systems Agency (DISA) surplus equipment list - Transfer useful equipment to schools - Dispose of as other surplus equipment • Environmental considerations - Specific hazards: lithium batteries, Cathode Ray Tubes (CRTs) - Volume of solid waste - Other Environmental, Safety and Occupational Health (ESOH) impacts • Secure data archiving considerations - Determine what data must be archived, and for how long. - Who will archive data – SSA? - PM responsibility DAG 4.2.3.1.7.3 Software Support 305

IT Disposal Process—Outcomes

As a result of successful implementation of the IT/Software Disposal Process: • a software disposal strategy is defined; • disposal constraints are provided as inputs to requirements; • the system's software elements are destroyed or stored; • the environment is left in an agreed-upon state; and • records allowing knowledge retention of disposal actions and any analysis of long-term impacts are available.

Software Support 306

153 11/15/2016

Lesson Overview Lesson Plan Status

• Software Lifecycle Planning • Lesson Exercise 1 • Sustainment and Support • Disposal • Lesson Exercise 2

Software Support 307

Software Support Exercise 2 Software Support Activity (SSA) Decision

• The initial Joint Training and Maintenance System (JTAMS) Increment 1 software support strategy was to use the developer, JUGGERNAUT, to provide software support. Due to declining budgets the Program Management Office (PMO) is considering changing the support strategy for JTAMS Increment 2 software. It has been proposed that the PMO use support provided by an existing DoD SSA.

• What are the pros and cons of moving to support to a government SSA? • What factors should be considered when making that decision?

Software Support 308

154 11/15/2016

Summary Today we learned to:

Software Support 309

Lesson 21 DoD Cloud Computing

155 11/15/2016

Learning Objectives Today we will learn to:

• Identify the basic terms of Cloud Computing • Recognize some DoD Concerns of Using Cloud Services • Summarize some Program concerns when purchasing cloud services from a vendor. • Identify the advancements in technology that enabled the rise of cloud computing. • Recognize the five essential characteristics of a cloud service. • Identify the Three Cloud Service Models defined by NIST. • Recognize characteristics of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). • Recognize public, private, community and hybrid cloud deployment models (NIST). • Recognize the steps for obtaining Cloud services. • Describe the problems with Legacy software applications and Cloud.

Software Quality Assurance 311

In-Class Quiz

Team • True or False: According to the DoD Chief Information Officer (CIO), DoD components are required to use the Defense Information Systems Agency (DISA) to 1 acquire cloud services.

• The ______provided cloud services must be considered as part of the Team Enterprise IT Business Case Analysis (BCA) performed by the Component for cloud 2 services.

Team • The ______is intended to give cloud providers a stable 3 security requirement, and to help DoD cloud customers move more rapidly and securely into the cloud.

• Which of the following is NOT a benefit of Cloud Computing per the DoD Cloud Team Computing Strategy? De-coupled from private sector innovation; Enables improved 4 asset utilization; Allows for near-instantaneous increases and reductions in capacity; Shifts focus from asset ownership to service management

Team • According to the DoD Cloud Computing Strategy, what are the three areas DoD can 5 benefit from by moving to cloud computing?

DoD Cloud Computing 312

156 11/15/2016

Lesson Overview Lesson Plan

• Cloud Laws, Policies, Guidance and Standards HOMEWORK • Cloud Basics and Benefits • Cloud Computing Definition • Concerns with using Cloud • Using the Cloud (Assessment & Authorization) • Exercise

DoD Cloud Computing 313

Trademark Information

• Names, products, and services referenced within this lesson may be the trade names, trademarks, or service marks of their respective owners. References to commercial vendors and their products or services are provided strictly as a convenience to our students, and do not constitute or imply endorsement by DoD or DAU.

DoD Cloud Computing 314

157 11/15/2016

Lesson Overview Lesson Plan Status

• Cloud Laws, Policies, Guidance and Standards • Cloud Basics and Benefits • Cloud Computing Definition • Concerns with using Cloud • Using the Cloud (Assessment & Authorization) • Exercise

DoD Cloud Computing 315

Official DoD Definition of Cloud Computing

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

NIST Special Publication 800-145

DoD Cloud Computing 316

158 11/15/2016

The Composition of the Cloud

• The “Cloud” is composed of - five essential characteristics, - three service models, - four deployment models

NIST Special Publication 800-145

DoD Cloud Computing 317

5 Essential Cloud Characteristics

• According to the NIST Special Publication 800-145, the Cloud model is composed of five essential characteristics:

• On-demand self-service • Broad network access • Resource pooling - Location independence • Rapid elasticity

• Measured service NIST Special Publication 800-145

DoD Cloud Computing 318

159 11/15/2016

The 3 Cloud Service Models

• Infrastructure as a Service (IaaS) - Compute, storage, and networking capability • Platform as a Service (PaaS) - Deploy customer-created applications to a cloud • Software as a Service (SaaS) - Use provider’s applications over a network

• To be considered “cloud” the Cloud Service Models must be deployed on top of cloud infrastructure that has the key characteristics

DoD Cloud Computing 319

Infrastructure as a Service (IaaS)

• Provisioning processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. • The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls).

DoD Cloud Computing 320

160 11/15/2016

Platform as a Service (PaaS)

• Deployed onto the cloud infrastructure consumer‐created or acquired applications created using programming languages, libraries, services, and tools supported by the provider. • The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application‐hosting environment.

DoD Cloud Computing 321

Software as a Service (SaaS)

• Using the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through either a thin client interface, such as a web browser (e.g., web‐based email), or a program interface. • The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user‐specific application configuration settings.

DoD Cloud Computing 322

161 11/15/2016

Management Responsibilities with the 3 Cloud Service Models

• Cloud Services offers a way for the DoD to lower costs, improve performance, increase utilization and security, and take advantage of commercial innovation

DoD Cloud Computing 323

Pizza as a Service

DoD Cloud Computing 324

162 11/15/2016

The 4 Cloud Deployment Models

• Cloud services can be deployed in different ways depending on the customer’s specific needs, such as security, privacy, and cost.

1. Public cloud

2. Private cloud

3. Community cloud

4. Hybrid cloud NIST Special Publication 800-145

DoD Cloud Computing 325

Public Cloud Deployment Model

• Public cloud infrastructures operate in a multi-tenant environment whose resources are allocated for the general public. • Public clouds tend to be large and provide economies of scale for their customers. • Security and privacy concerns are heightened because any individual or organization can potentially access the same cloud infrastructure. • Only DoD information that has been approved for public release should be placed on a public facing website.

DoD Cloud Computing 326

163 11/15/2016

Private Cloud Deployment Model

• Private cloud infrastructures are operated only for an individual organization (Single Tenant). • The organization can leverage the scalability and performance aspects of cloud computing, but the infrastructure is isolated from that of other organizations, improving security and privacy. • Because of their specialized nature, private clouds could potentially be as costly as dedicated data centers. • For example, the DoD has a Private Cloud, milCloud, which is operated by DISA.

DoD Cloud Computing 327

Community Cloud Deployment Model

• Community cloud infrastructures are private clouds provisioned for a specific community of interest with shared concerns, such as a government-only cloud. • The Community cloud infrastructure is provisioned for exclusive use by a specific community of consumers from organizations that have shared concerns (e.g., mission, security requirements, policy, and compliance considerations). • Community clouds may be owned, managed, and operated by one or more of the organizations in the community, a third party, or some combination of them, and it may exist on or off premises. • Amazon GovCloud is an example of a Community Cloud that is available to Federal, State and Local Governments.

DoD Cloud Computing 328

164 11/15/2016

Hybrid Cloud Deployment Model

• Hybrid cloud infrastructures are combinations of any two or more of the other cloud deployment models. • This model will be the most prevalent model for the DoD given its strategy to aggressively pursue the competitive acquisition and use of commercial cloud service offerings and understanding that “one cloud’ will not meet all the unique requirements of the DoD. • One example of Hybrid Cloud is used in the Development – Test – Production software lifecycle.

DoD Cloud Computing 329

Lesson Overview Lesson Plan Status

DoD Cloud Computing 330

165 11/15/2016

DoD’s Concerns of Using Cloud Services

• Data Security - Location of DoD data - Comingling of DoD data with other customer’s data - Physical security of data center - “Noisy Neighbor” • Latency - Network congestion/bandwidth availability - Remote cloud data centers • Unanticipated costs - Network upgrades to maintain performance (increased bandwidth demands) - Strict security requirements (e.g. Private vs Public) • Cybersecurity: Protecting the Defense Information Systems Network (DISN) - The DISN is a critical infrastructure to the DoD Mission

DoD Cloud Computing 331

DISN, DoDIN; what’s the diff?

• Joint Publication 1-02 states: - Defense Information Systems Network (DISN) - The integrated network, centrally managed and configured by the DISA to provide dedicated point-to-point, switched voice and data, imagery, and video teleconferencing services for all Department of Defense activities. - Department of Defense Information Network (DoDIN) - The set of information capabilities, and associated processes for collecting, processing, storing, disseminating, and managing information on-demand to warfighters, policy makers, and support personnel, whether interconnected or stand-alone, including owned and leased communications and computing systems and services, software (including applications), data, security services, other associated services, and national security systems The DISN is the protected networks which include NIPRNet, SIPRNet, or other DISN- based mission partner/Community of Interest networks DoD Cloud Computing332

166 11/15/2016

Cybersecurity is a Concern when using Cloud Services

• With respect to Cloud Computing, “Mission” refers to the information systems and function for which a DoD entity acquires or uses a Cloud Service • The Mission Owner must consider Risk to Data (referred to as Information Impact Level) and Risk to the DISN

• Risk to Data - Loss of Confidentiality, Integrity and Availability (CIA) • Risk to DISN - Loss of CIA of Data on DISN - Loss of Availability of DISN

DoD Cloud Computing 333

Lesson Overview Lesson Plan Status

DoD Cloud Computing 334

167 11/15/2016

Cloud Service Providers and Offerings

• Types of Cloud Services - Commercial - DoD - Non-DoD (i.e., Federal, DHS) • Cloud Service Provider (CSP) - A company or organization that offers some component of cloud computing (i.e., IaaS, PaaS, or SaaS) to other businesses, organizations or individuals. • Cloud Service Offering (CSO) - The deployed cloud computing service(s) (i.e., IaaS, PaaS, or SaaS)

DoD Cloud Computing 335

Using the Cloud

• The DoD Chief Information Officer’s memo from December 2014 identified 5 activities when acquiring cloud services 1. Perform an IT business case analysis 2. Apply the DoD Cloud Computing Security Requirements Guide 3. Use commercial cloud services that have a DoD Provisional Authorization and obtain a Component Authority to Operate 4. Use an approved DoD Boundary Cloud Access Point (BCAP) and Cyber Security Service Provider (CSSP) to protect sensitive data 5. Apply the Defense Federal Acquisition Regulation Supplement Interim Rule to commercial cloud contracts

DoD Cloud Computing 336

168 11/15/2016

Activity 1 - Performing the IT Business Case Analysis (BCA)

• Keep in mind that a BCA is not a requirements validation process. The purposes of the BCA are as follows: - Ensure a consistent approach in IT investment analysis. - Facilitate comparison of alternatives. - Clearly define expected costs, benefits, operational impacts, and risk. • The major components of a BCA are: - Cost and economic viability - Requirement satisfaction/completeness - Operational benefit (qualitative) - Risk assessment - Conclusions and recommendations - Balance cost effectiveness with operational benefit - Funding type and sources

DoD Cloud Computing 337

Activity 1 - Performing the IT BCA

• Each use of cloud services must complete an Enterprise IT Business Case Analysis (BCA) • The BCA must be approved by the Component CIO, or designee, with a copy submitted to the DoD CIO - Follow Component direction on completing the BCA • DISA provided services must be considered as an Alternative in the BCA Life Cycle Cost Comparison (dollars in millions) To Lowest Prior FY15 FY16 FY17 FY18 FY19 FY20 FY21 LCCE Complete LCC$ Alternative 1 $_ $_ $_ $_ $_ $_ $_ $_ Alternative 2  Alternative 3

DoD Cloud Computing 338

169 11/15/2016

Activity 2 - Apply the DoD Cloud Computing SRG

• All DoD data is important, but not all data needs to be equally protected • Information Impact Levels (IILs) consider the potential impact should the confidentiality and integrity of the information be compromised

DoD Cloud Computing 339

Federal Risk and Authorization Management Program (FedRAMP)

• For cloud products and services used by the Federal Government, FedRAMP is a program that provides a standardized approach to: - Security assessment - Authorization - Continuous monitoring • OMB policy requires Federal departments and agencies to use FedRAMP approved Cloud Service Providers (CSPs) and share Agency ATOs with the FedRAMP Secure Repository - “Do Once, Use Many Times” - https://www.fedramp.gov/marketplace/compliant- systems/

DoD Cloud Computing 340

170 11/15/2016

FedRAMP+ and DoD Provisional Authorization

• FedRAMP+ is the concept used in order to meet and assure DoD’s critical mission requirements - Leverages FedRAMP assessment - Adds specific security controls and requirements • DoD Provisional Authorization is an acceptance of risk based on an evaluation of the CSP’s Cloud Service Offering (CSO) and the potential for risk introduced to the DISN • DoD PAs are granted by DISA to the CSP for a CSO, not for a CSP - If a CSP’s CSO (e.g., SaaS) leverages another CSP’s CSO (e.g., IaaS) then the DoD PA for the former includes inherited compliance for the latter.

DoD Cloud Computing 341

Activity 3 – Use Commercial CSPs with DoD PAs and Obtain an Authority to Operate

• Each CSO must be granted a DoD PA in order to host DoD mission systems • CSOs possessing a DoD PA are listed in the DoD Cloud Service Catalog • The responsible Authorizing Official leverages the DoD PA information, supplemented with an assessment of the risks within the Mission Owner’s responsibility, in granting an Authorization to Operate (ATO) • Authorizing Officials use the Risk Management Framework to issue an ATO

DoD Cloud Computing 342

171 11/15/2016

Activity 4 – Use a DoD BCAP and CSSP (1 of 2)

• A DoD Boundary Cloud Access Point (BCAP) is a system of network boundary protection and monitoring devices, otherwise known as an Information Assurance stack, through which CSP infrastructure and networks will connect to the DISN • With Controlled Unclassified Information data (IIL 4 & 5), a BCAP is required between the DISN and the CSO • The BCAP is used to protect the DISN, and systems, information and users residing on the DISN from attacks that may be launched from within a compromised CSO; facilitate protected connections between users on a DoD network and systems/applications on the CSO

DoD Cloud Computing 343

Activity 4 – Use a DoD BCAP and CSSP (2 of 2)

• DoD BCAPs will provide the following generalized functions: - Intrusion Detection/Intrusion Protection - Data Loss Prevention - Full Packet Capture - Network Routing/Switching - Network Access Control to CSPs - Next Generation Firewall - Application Firewall • The Cyber Security Service Provider (CSSP) provides cyber security services and Command and Control direction addressing the protection of the network, detection of threats and response to incidents • DoD PMs must ensure that CSSP processes are in place and functional prior to any transition to or use of a CSO

DoD Cloud Computing 344

172 11/15/2016

Activity 5 – Apply the DFARS Interim Rule for Cloud Services

• DoD issued an interim rule amending the DFARS to implement a section of the FYs 13 & 15 National Defense Authorization Acts - Require contractor reporting on network penetrations - Implements DoD policy on the purchase of cloud computing services • DFARS, Subpart 239.76 Cloud Computing - Policy and Responsibilities - Required storage of data within the US or outlying areas - Solicitation provision and contract clauses (252.239-7010)

DoD Cloud Computing 345

Required Storage of Data within the US

• The contractor shall maintain within the United States or outlying areas all government data that is not physically located on DoD premises, unless the contractor receives written notification from the contracting officer to use another location. • The contractor shall provide the government with a list of the physical locations which may contain government data within 20 days. Updates are required on a quarterly basis.

DoD Cloud Computing 346

173 11/15/2016

Storing Data in Non-US Locations

• The U.S. government restricts the transfer of sensitive or classified data (such as sensitive technology information and information that could potentially affect operational security) to locations outside of the control of U.S. companies or the U.S. government • There are specific rules for the locations of data processing centers based on the IIL of the data: - IIL 2 and 4 must be hosted at locations in the U.S., U.S. territories, or on DoD premises per the Status of Forces Agreement (SOFA) unless the location is authorized by the AO - IIL 5 must be hosted at locations in the U.S., U.S. territories, or on DoD premises per the SOFA - IIL 6 must be hosted at locations authorized for classified processing

DoD Cloud Computing 347

Additional Considerations for Using the Cloud

• The DoD Program Manager needs to understand and perform additional activities when acquiring cloud services 1. Consider key skills needed for a successful deployment

2. Protect DoD Equities in cloud contracts and Service Level Agreements

3. Complete Cloud Service Offering funding reporting responsibilities, e.g., SNaP-IT, Budget 300 Exhibits 53A/C

4. Plan for Close-Out and Transition

DoD Cloud Computing 348

174 11/15/2016

Program concerns when purchasing commercial cloud services

• Cloud Service Provider Maturity • Deployment Model Considerations, Physical/Virtual Separation Requirements, “noisy neighbor” • Encryption (at rest & in transition) • CSP Personnel Requirements • Physical Access • Legacy Software Interoperability

Problems with legacy software applications and the cloud

• Legacy software applications were not designed to be virtualized • Redesigning legacy software applications to use cloud services can be cost prohibitive • Legacy software applications that are tightly integrated with a computer’s operating system are extremely difficult to migrate to the cloud • Software that is encapsulated from the operating system has a better chance of migrating to the cloud - Encapsulation means there is no direct dependency on any one operating system

DoD Cloud Computing 350

175 11/15/2016

Lesson Overview Lesson Plan Status • Cloud Laws, Policies, Guidance and Standards • Cloud Basics and Benefits • Cloud Computing Definition • Concerns with using Cloud • Using the Cloud (Assessment & Authorization) • Exercise

DoD Cloud Computing 351

Summary Today we learned to:

DoD Cloud Computing 352

176