Technology Infusion of CodeSonar into the Space Network Ground Segment (Technical Briefing) Markland J. Benson / NASA GSFC 2009 Software Assurance Symposium Agenda Agenda Preliminaries (2) Environment (5) Goals (2) Approach (3) Applicability (2) Baseline (3) Findings (7) Conclusions (5) Wrap-Up (2) Environment-1 Agenda • Space network software must Preliminaries (2) support 24x7x365 operations with Environment (5) a high level of integrity, Goals (2) confidentiality, and availability Approach (3) • Current staff consists of 50 FTE to Applicability (2) sustain and enhance software Baseline (3) Programmer 30 Findings (7) • Approximately one-third of softwaTersete r 5 Conclusions (5) effort goes into discrepancy workC-M 5 Wrap-Up (2) SysAdmin 8 off and two-thirds to enhanced Management 2 capabilities Environment-2 Agenda • Scope: ground software systems that Preliminaries (2) control a satellite fleet and provide Environment (5) near earth communication services Goals (2) Approach (3) • Prioritized Responsibilities Applicability (2) Baseline (3) • Investigate operational issues Findings (7) • Resolve urgent operational issues Conclusions (5) Wrap-Up (2) • Provide enhanced capabilities for customers and operations • Resolve routine operational issues Environment-3 Volume IV Book 1 Local Operating Procedure 1 Agenda Capability Board Approved Change Preliminaries (2) Maturity Model Requirements Requirements Inspection Requirements Environment (5) Inspection Integrated, Six Checklist Goals (2) Design Preliminary Design Preliminary Inspection Inspection Design Approach (3) Sigma, and Checklist NASA standards Applicability (2) Detailed Design Detailed Inspection Design Baseline (3) and Static Analysis Tool Output Source Code Source Code Findings (7) requirements Inspection Conclusions (5) Source Code SCM Test are applied in Inspection Programmer Test Baseline Wrap-Up (2) software Checklist Software Test SCM Staging Software Test sustaining Plan Baseline Operational engineering Operational Test Test Plan Operational Baseline Environment-4 Agenda • Hours of loss due to software is 7% of Preliminaries (2) overall Environment (5) Goals (2) • The 7% slice of overall loss equates to Approach (3) 27% of the loss internal to the Space Applicability (2) Network [that which we directly control] Baseline (3) Findings (7) External Hardware Software Operations Conclusions (5) Wrap-Up (2) 74% 4% 7% 14% Environment-5 Agenda • Overall trend of software loss is down Preliminaries (2) Software Loss % Linear (Software Loss %) Environment (5) 60% Goals (2) 50% Approach (3) 40% Applicability (2) 30% Baseline (3) 20% Findings (7) 10% Conclusions (5) 0% 2003 2004 2005 2006 2007 2008 Wrap-Up (2) Percent internal loss due to software • Improvements in 2006 attributed to introduction of formal inspections Goals-1 Agenda • High availability and proficiency of Preliminaries (2) service requirements drive the need to Environment (5) reduce system defects Goals (2) Approach (3) Goal: Eliminate defects existing in the current Applicability (2) baseline software Baseline (3) • High demand for discrepancy Findings (7) resolution and new capabilities drive Conclusions (5) Wrap-Up (2) the need to produce software more quickly Goal: Eliminate defects earlier in the software development lifecycle (reduce rework) Goals-2 Agenda • Infuse automated source code Preliminaries (2) analysis technology Environment (5) Goals (2) • Provide for a uniform analysis toolset Approach (3) across languages and platforms Applicability (2) • Apply technology to systems that have Baseline (3) Findings (7) higher than average contribution to Conclusions (5) service loss Wrap-Up (2) • Minimize engineer time required to apply technology Approach-1 Agenda • GrammaTech’s CodeSonar product Preliminaries (2) provides mature analysis toolset for C Environment (5) and C++ (~50% of current code base) Goals (2) with new capability to cover Ada Approach (3) Applicability (2) (~30% of current code base) Baseline (3) Findings (7) Conclusions (5) Wrap-Up (2) • Software engineers use CodeSonar results as an input to the existing source code inspection process Approach-2 Agenda • Collect baseline information from the Preliminaries (2) sustaining engineering processes Environment (5) Goals (2) • Apply static analysis tool to a subset of Approach (3) the software baseline Applicability (2) • Review findings from the tool to Baseline (3) Findings (7) eliminate false positives and estimate Conclusions (5) future review time to be added to Wrap-Up (2) inspection process • Assess costs and benefits of larger deployment of analysis tools Approach-3 Agenda Apply study resources to Computer Preliminaries (2) Software Components (CSCs) known Environment (5) to be more troublesome than others Goals (2) Approach (3) Applicability (2) SW CSCI Hours Lost % Loss DR count % DRs KSLOC %KSLOC Baseline (3) CSCI A 6.126 37% 28 6% 200 2% Findings (7) CSCI B 0.910 5% 33 7% 64 1% Conclusions (5) Others 9.748 58% 398 87% 7865 97% Wrap-Up (2) Total 16.784 100% 459 100% 8129 100% Applicability-1 Agenda • The study is focused on large scale Preliminaries (2) software developed using formal Environment (5) processes Goals (2) Approach (3) • The systems studied are mission Applicability (2) critical in nature but some use Baseline (3) commodity computer systems Findings (7) Conclusions (5) • Linux, Windows, and VxWorks Wrap-Up (2) operating systems are represented • The application domain of the software is communications and spacecraft control systems Applicabilty-2 Agenda • If you have... Preliminaries (2) Environment (5) • in-house maintained software... Goals (2) • using a general purpose language... Approach (3) • with formal development process... Applicability (2) Baseline (3) • where failures lead to injury or Findings (7) significant financial loss... Conclusions (5) • ...this study has results that are Wrap-Up (2) directly meaningful to you. (even if #4 is not true for you, lower cost analyzers likely would be of benefit) Baseline -1 (What is SLOC anyway?) Agenda • SLOC definitions are inconsistent… Preliminaries (2) • David A. Wheeler’s SLOCCount tool Environment (5) Goals (2) and definition is used here: Approach (3) Physical SLOC is defined as follows: ``a physical source line of code (SLOC) is a line ending in a newline or end-of-file Applicability (2) marker, and which contains at least one non-whitespace non- Baseline (3) comment character.'' Comment delimiters (characters other than newlines starting and ending a comment) are considered Findings (7) comment characters. Data lines only including whitespace Conclusions (5) (e.g., lines with only tabs and spaces in multiline strings) are not included. Wrap-Up (2) • Non-Comment Source Lines (NCSL) == Physical SLOC • (Total) SLOC includes comments, blanks, and NCSL Baseline-2 Agenda • Productivity of software engineers is 0.16 Preliminaries (2) source lines of code (SLOC) per hour to Environment (5) perform requirements elicitation, design, Goals (2) coding, inspections, unit test, and Level 2 Approach (3) test Applicability (2) • Comparative industry productivity value is Baseline (3) 0.6 SLOC per hour by the German Findings (7) Aerospace Center Conclusions (5) Wrap-Up (2) • Difference can be attributed somewhat to evolving a product as opposed to new product development but productivity improvement is one of the goals Baseline-3 Agenda • In 2008, 204 software change Preliminaries (2) requests (CRs) were completed Environment (5) Goals (2) • Each CR produced an average of 209 Approach (3) new or modified lines of code Applicability (2) (changes to database values not Baseline (3) Findings (7) counted as SLOC modifications) Conclusions (5) • Formal inspections caused the Wrap-Up (2) removal of 1,255 defects from the code; 374 of the defects were classified as major (~30%) Findings-1 Agenda • For the 439 KSLOC (310 KNCSL) of Preliminaries (2) C/C++ code analyzed in 1,245 files, Environment (5) 1,011 findings were produced Goals (2) Approach (3) • All of 1,011 findings were reviewed Applicability (2) with an average review time of ~7 Baseline (3) minutes each All Findings Findings (7) 600 Conclusions (5) 500 Wrap-Up (2) 400 300 200 100 0 True Requires False Vendor Positive Research Positive Software Findings-2 Agenda • Requires Research findings were Preliminaries (2) classified as True or False Positive Environment (5) based on finding category Goals (2) Approach (3) • Vendor Software findings were Applicability (2) classified as False Positive Baseline (3) Findings (7) True Positive 600 Conclusions (5) 500 Wrap-Up (2) 400 300 200 100 0 Routine Urgent Findings-3 Agenda • Definition: An Urgent (a.k.a. major) Preliminaries (2) defect is one assessed as directly Environment (5) impacting availability or proficiency Goals (2) Approach (3) • Note: The large number of findings Applicability (2) and corresponding review time is not Baseline (3) expected to be repeated. Previously Findings (7) reviewed findings are filtered and only Conclusions (5) Wrap-Up (2) displayed if specifically requested Findings-4 Agenda Findings were not uniformly distributed Preliminaries (2) as a whole or proportionally among Environment (5) CSCs Goals (2) Approach (3) Component K-NCSL Defects / K-NCSL Applicability (2) CSCI B 121 1.8634 Baseline (3) Findings (7) CSCI A - CSC B 139 1.1571 Conclusions (5) CSCI A - CSC C 22 3.7156 Wrap-Up (2) CSCI A - CSC D 29 4.1332 NCSL = non-comment source lines Findings-5 200 Agenda 180 Preliminaries (2) 160 Environment (5) 140 Goals (2) 120 Approach (3) 100 80