Software Engineering Technology

The Qualification of Software Development Tools From the DO-178B Certification Perspective

Dr. Andrew J. Kornecki Dr. Janusz Zalewski Embry Riddle Aeronautical University Florida Gulf Coast University

Software development tools are in wide use among safety-critical system developers. Examples of such use include aviation, auto- motive, space, nuclear, railroad, medical, and military applications. However, verification of tool output to ensure safety, man- dated in highly regulated industries, requires enormous effort. If a tool is qualified, this effort can be reduced or even eliminat- ed. The Radio Technical Commission for Aeronautics Document Order-178B and related documents provide guidelines by which to qualify these tools. However, current regulations, business models, and industry practice make this goal difficult to accomplish. This article discusses the qualification of development tools and the potential impact of this process on the aviation industry.

oftware development tools are com- (SC-167) in 1989 with the objective of context of a specific airborne system.” It is puter programs that help developers updating the DO-178A by focusing on the certification authority that decides on createS other programs. Such tools have five areas: documentation integration and the outcome of the qualification process. been in use since the early days of com- production, system issues, software devel- Moreover, qualification, if claimed, is a puting to improve the efficiency of the opment, software verification, and soft- requirement in getting a system certified. development process by automating mun- ware configuration management and soft- dane translation operations and bringing ware quality assurance. The resulting doc- Types of Software the level of abstraction closer to the appli- ument, DO-178B, provides guidelines for Development Tools cation engineer. Nowadays, development applicants developing software-intensive DO-178B differentiates between verifica- tools are used in a variety of safety-critical airborne systems [1, 2]. It discusses objec- tion tools that cannot introduce errors but may applications, including the aviation, auto- tives that need to be met to show that the fail to detect them and development tools motive, space, nuclear, railroad, medical, software development process provides whose output is part of airborne software and and military industries, and contribute to specified levels of safety assurance. It also thus can introduce errors. There is a signifi- the risks associated with using respective describes the processes and means of cant amount of effort involved to qualify products. Despite these risks to society, compliance. a verification tool, and much more to development tools are rarely qualified in a Systems are categorized by DO-178B qualify a development tool. However, sense comparable to product certification as meeting safety assurance levels A numerous development tools have been in regulated industries. The objective of through E based on their criticality in sup- used successfully in many certified pro- this article is to look at the current state of porting safe aircraft flight. The level A sys- jects without being qualified. To define a the tool qualification process, identify the tem is the most critical: The failure of subject matter more narrowly, we need to issues, and propose recommendations for such a system could result in a catastroph- take a closer look at the entire domain of potential improvement, focusing on the ic failure condition for the aircraft. The software development tools. aviation industry. level E system is the least critical: Such a The landscape of modern software System Certification Versus system has no effect on the operational development tools is very broad, as illus- capability of the aircraft or pilot workload. trated in Figure 1 (see page 20). Following Software Tool Qualification Although the RTCA DO-178B is the lead- the traditional model of the development Certification of airborne equipment is typ- ing source of guidelines for software process from requirements to implemen- ically achieved through the Federal developers engaged in such system con- tation, we can identify the following: Aviation Administration (FAA) authoriza- struction, two other documents have criti- • The requirements category that includes tion of a type certificate (the entire air- cal bearing on the subject. RTCA DO- tools used early in the life cycle to craft), supplemental type certificate (new 248B [3] clarifies some of the misinterpre- identify and specify the software equipment in a specific aircraft), or a tech- tation of the DO-178B. The FAA Order requirements. nical standard order (minimum perfor- 8110.49 compiles a variety of guidelines • The design category that includes tools mance standard for materials, parts, and related to the use of software in airborne allowing developers to create architec- appliances used on civil aircraft). A special systems. Chapter 9 is specifically dedicated tural and detailed design of the soft- committee (SC-145) of the Radio Tech- to tool qualification [4]. ware in a notation of their choice sup- nical Commission for Aeronautics (RTCA) A key component of the updated ver- ported by the tool; often in this cate- convened in 1980 to establish guidelines sion of DO-178B is the concept of tool gory, tools translate the model to for developing airborne systems.The qualification elaborated in Section 12. source code. report “Software Considerations in Air- Qualification is a supplementary process • The implementation category that borne Systems and Equipment Certifica- that the applicant may elect to follow in includes all support required to trans- tion” was published in January 1982 as the the course of certifying an airborne sys- late the computer code and transfer it RTCA Document Order (DO)-178 (and tem. According to the definition given in to the target computer. revised as DO-178A in 1985). DO-178B, tool qualification is defined as, As illustrated in Figure 1, three other Due to rapid advances in technology, “The process necessary to obtain certifica- categories of tools can be identified: those the RTCA established a new committee tion credit for a software tool within the related to analysis, testing, and target.

April 2006 www.stsc.hill.af.mil 19 Software Engineering Technology

e.g.: a Tool Configuration Management Index, Structural Rhapsody Typically With Design RoseRT Code Generator Tool Development Data, Tool Verification Functionality Tool STOOD Records, Tool Quality Assurance Records, Requirements Artisan Tool Integrated Tool Configuration Management Records, or/and e.g.: Development e.g.: Functional SCADE Environment etc. These requirements are also described Reqtify Design Matlab (IDE) in [4]. Tool qualification data are approved BEACON DOORS Tool e.g.: SpecTRM Sildex Tornado only in the context of the overall software DOME Multi development for the specific system Analysis Implementation where the intention to use the tool is stat- Tool Tool ed in the PSAC. The tool itself does not receive a separate qualification stamp of e.g.: Testing RapidRMA Tool approval. Therefore, using the tool on TimeWiz another system/project requires a separate e.g.: Target qualification, although some qualification CodeTest (with RTOS) TestRT credits may be reused. VectorCast e.g.: Surveys requesting which tools are used Insure++ VxWorks QNX by industry were conducted at two national OSE conferences: the 2002 FAA National Tool Categories Integrity LynxOS Software Conference and the 2004 Embry Riddle Aeronautical University/FAA Soft- Figure 1: Software Tool Categories ware Tool Forum. In addition, two follow- Figure 1:Software Tool Categories up e-mail solicitations were sent to more However, for this article, we will focus pri- producer) is to transform an input artifact than 500 professionals working on air- marily on the tools used in the design into output, thus creating another software borne systems. These surveys and solicita- phase, the central component of the soft- artifact. The current process mandates ver- tions resulted in a relatively small sample of ware development life cycle. They reflect ification after each transformation. If this Can responses that did not provide a base for two divertseool vie inwsertpoints error on real-time, safety- transformation has an impact on the final statistically significant results. The com- NO critical systemsinto a irdevelopment,borne which result airborne product, the producer needs to be ments included industry discouragement NO QUALIFICATION from differentso fdevelopers’tware? backgrounds: qualified, but only if the transformation regarding the rigor of development tool • Control engineers consider a system to output wouldNEC notESS beA verifiedRY and the trans- YES qualification, and a justified perception of be a dynamic model consisting of well- formation leads to elimination, reduction, the extensive cost of qualification. defined blocks of specific functionality or automation of any of the DO-178B Potential solutions to assist in commer- (logic, arithmetic,W dynamic).ill The func- processes. The conditions under which a the tool’s output cial off-the-shelf (COTS) development tional paradigm of the model is the development tool requires qualification are NOT be verified tool qualification included extensive ven- basis for system simulation and analysis presented TinOO FigureL M U2 S[4].T (as specified in NO dor collaboration and using alternate means of its behavior.DO Subsequently,-178B)? the model SoftwareBE QdevelopmentUALIFIED tool qualifica- allowed in DO-178B. The limited feedback can be translated automatically into an tion is attempted only as an integral com- shows that there has been interest in qual- equivalent code, typically without any ponent of a specific application program YES ifying software development tools classi- additional developer’s involvement.Are requiring the FAA’s certification. The soft- fied in the function-based/block-oriented • Software engineers, on thepro othercesse shand,of ware tools to be used are referenced with- category, which cannot be said about are familiar with theD conceptsO-178B e oflimin oper-ated, in the Plan for Software Aspects of reduced, or automated structure-based/object-oriented tools. ating systems, programming languages, CertificationNO (PSAC) and the Software by use of A short list of qualified development software development methodologies, Accomplishment Summary documents of tool? tools includes code generators (Gener- and notations. TheYES graphic notations the original certification project. If devel- ation Automatique de Logiciel Avionique, Fig(classes,ure 2:C packages,onditio states,ns Wh transitions,en a Softopmentware D toolev equalificalopmenttion isTool requir Requireed, the s Qualification events) allow developers to represent applicant should present for review the Graphical Processing Utility, Virtual the structure and behavior of the target Tool Operational Requirements (TOR) – a Application Prototyping System Code system software as a set of components document describing tool functionality, Generator, Safety Critical Avionic that can be translated into programming environment, installation, operation man- Development Environment Qualifiable constructs (data structures, objects, ual, development process, and expected Code Generator) and configuration- functions, etc.) using the automatic code responses (also in abnormal conditions). scheduling table generators (Universal generation functionality of the tool. Two documents must be submitted and Table Builder Tool, Configuration Table Consequently, the software design approved: a Tool Qualification Plan, and a Generation Tool), most of them being in- tools, which assist developers in translat- Tool Accomplishment Summary as de- house products. According to several ing the software requirements into source scribed in [4]. To make an argument for informal exchanges with industry, many of code, can be categorized into two groups: qualification, the applicant must demon- the modern COTS software development (a) a function-based, block-oriented approach strate correctness, consistency, and com- suites actually have been used in the cre- applied by control and system engineers, pleteness of the TOR and show that the ation of software artifacts on certified and (b) a structure-based, object-oriented tool complies with its TOR. This demon- projects without going through the quali- approach applied by computer scientists stration may involve a trial period during fication process. and software engineers. which a verification of the tool output is Problems With Development performed and tool-related problems are The Qualification Process analyzed, recorded, and corrected. Tool Qualification A typical use for a design tool (software Other data required for review include It is clear that qualification of develop-

20 CROSSTALK The Journal of Defense Software Engineering April 2006 The Qualification of Software Development Tools From the DO-178B Certification Perspective ment tools is an option rarely exercised in nificant difference between tool software ed to the translation process. The transla- the airborne software industry. In fact, and application software. Applications run tion component is hidden deep inside the one could argue that qualification of on a target computer while tools operate tool, which causes problems with tool development tools is not a viable option. on a general-purpose workstation, typical- qualification. Typically, there is no access Current interpretation of applicable ly closely interacting with a COTS operat- to a COTS tool’s life-cycle data, which guidelines makes development tool quali- ing system and conventional programming describe the tool’s requirements, design, fication a proposition that is not practical environment. Considering this, several and code. Unless the tool has been devel- from a managerial viewpoint, and not easy DO-178B objectives are not applicable to oped in-house, the qualification efforts from a technical viewpoint. tool software and thus cannot be met. may be doomed. There is also no general agreement on Managerial Viewpoint what metrics would allow developers to Potential Solutions The first group of problems is of a regu- carry an independent tool assessment [6]. The qualification of a stand-alone devel- latory and managerial nature. The major One often-repeated statement regard- opment tool is not feasible in the strict hurdle is the current state of regulations ing development tool qualification is the sense of existing guidelines. Such concepts and guidelines. The secondary obstacle is requirement that “only deterministic tools as component-based software, software the business model and lack of incentives, can be qualified.” The DO-178B refers to reuse, and service history should be in particular the prohibitive cost of tool determinism as “… tools which produce explored [7] to identify the feasibility of qualification. The existing tools, often the same output for the same input data such qualification. The issues of tool ver- used in certification projects, do not have when operating in the same environment.” sion control and the precise definition of appropriate data to support arguments The definition does not take into account operational environment, constraints, and about meeting the objectives of DO- how the output is generated. By this defin- e.glimitations.: are the basis for starting discus- Structural Rhapsody Typically With 178B. The applicant team’s intent is to cer- ition, one may interpret that it is not sion about solutionsCode Gener toat otoolr qualification. Design RoseRT Functionality tify the product rather than expand effort required to provide proof on theTool internalSTOOTheD availability of extensive tool software Requirements Artisan and qualify the tool. The tool vendor does behavior of a Tooltool. An example of this can development data, often scarceInte forgra teCOTSd not see the business advantage of qualify- be memory use for a tool runningor/and on the productse.g.: , may be a challengeDeve lotopment ever e.g.: Functional SCADE Environment ing a tool while disclosing proprietary host worReqtikstationfy in a multitasking, multi-Design accomplishMatlab COTS tool qualification(IDE) [8]. BEACON e.g.: information to potential competitors. user, networkedDOORS environment. The prob-Tool It could be conceivable to create an SpecTRM Sildex Tornado Development tool qualification lem is toD OdefineME what the object code for a independent lab dedicated to toolMu ltiqualifi- cation and encourage commercial vendors requires close collaboration between the tool is. Does it includeAn athelysi soperating system Implementation tool vendor and the applicant. This is the (OS) of the host workstaTooltion? A tool clear- to submit their product forTool assessment. A reason why in-house tools are more likely ly needs to make explicit calls to the OS similar approach is known from other e.g.: areasTestin ofg verification and validation [9, 10]. to be qualified. Internal trade studies [5] routines, and any verificationRapidRMA of these Tool have shown that the cost of development would require full visibilityT imeof Wtheiz host’s OS Another idea would be to require certified tool qualification is significantly higher and related high assurance of its operation. producte.g .:applicants to discloseTarget information CodeTest (with RTOS) than the cost of verification tool qualifica- The main function of a software regardingTestRT the development tool use and tion. The use of qualified verification development tool is to transform, i.e., qualificaVectortionCast efforte. gby.: creating an FAA- sponsorInsureed++ databaseVxWo rkfors DO-178B certi- tools can result in fast savings on the first translate an input artifact into output. This QNX program where they are introduced. In is why the qualification, if applicable, fied products. ThisOSE could face serious Tool Categories Integrity contrast, the use of qualified development should be focused on this translation objections from industrLynxOS y due to an appre- tools may require several programs to component of the tool functionality. hensiveness to disclose any information, make up the cost. However, modern, complex software which may result in the loss of commer- The intellectual property rights may developmentFigure 1 :toolsSof twprovideare Tool a variety Cate ofgoriecials advantage. It would be possible to need to be waived by the vendor to achieve other functions that are not directly relat- research a potential for development tool qualification. The tool cannot be qualified as standalone, but only within the scope of Figure 2: Conditions When a Software Development Tool Requires Qualification a particular certification project. The tools Can that could be considered for qualification tool insert error into airborne NO are very simple: typically in-house created NO utilities where the applicant holds all intel- software? QUALIFICATION lectual property rights, maintains all tool NECESSARY YES development data, and can reuse the tool software artifacts on consecutive projects. Will The qualification is accomplished within the tool’s output the specific certification project and thus is NOT be verified TOOL MUST (as specified in not clearly visible from the outside as devel- NO BE QUALIFIED opment tool qualification. DO-178B)?

Technical Viewpoint YES Are The second group of problems is related processes of to technical aspects. According to the DO- DO-178B eliminated, 178B interpretation, the development tool reduced, or automated NO needs to be qualified to the same level of by use of scrutiny as the appropriate application it is tool? helping to develop. However, there is a sig- YES Figure 2:Conditions When a Software Development Tool Requires Qua April 2006 www.stsc.hill.af.mil 21 Software Engineering Technology qualification using an approach different Washington, D.C.: RTCA, 10 Dec. 7. Lougee, H. “DO-178B Certified Soft- than the one outlined in Section 12.2 of 2001 . html>. pressing need to come up with methods to “Software Approval Guidelines.” FAA 8. Zalewski, J., W. Ehrenberger, F. audit a tool that is independent of the spe- Order 8110.49. Washington, D.C.: Saglietti, J. Gorski, and A. Kornecki. cific program and applications using it. FAA, 2003 (Chapter 9 replaces FAA “Safety of Computer Control Systems: This would require updating the guidelines Notice N8110.91 of 2001) . Century.” CrossTalk Mar. 2001 application program using the tool. A 5. Potter, Bill. “Use of the MathWorks . opment tools and to keep them current as tified Code.” Slide No. 13. Honeywell, 10. Adams, M., et al. “Conformance technology advances would be useful. May 2004 . Products.” Advanced Multi-Micro- history and how to address what has to be 6. Kornecki A., and J. Zalewski. Criteria processor Bus Architectures. Ed. J. done for incremental tool changes would for Software Tools Evaluation in the Zalewski. Los Alamitos, CA.: IEEE also be needed. These and other issues Development of Safety-Critical Real- Computer Society Press, 1995: 392-399. have been discussed at the recent Tools Time Systems. Proc. of PSAM-7/ Euro- 11. Embry Riddle Aeronautical Universi- Forum [11]. The RTCA convened another pean Safety and Reliability Conference, ty/FAA Software Tools Forum, special committee (SC-205) with a charge Berlin, Germany, 14-18 June 2004. Embry Riddle Aeronautical University, to recommend modifications to the exist- London: Springer-Verlag, 2004 pdf>. toolsforum.html>. changes may be forthcoming. Acknowledgement About the Authors The presented work was supported in part by the Aviation Airworthiness Center of Andrew J. Kornecki, Janusz Zalewski, Ph.D., Excellence under contract DTFA-0301 Ph.D., is a professor at is a professor of comput- C00048 sponsored by the FAA. Findings the Department of Com- er science at Florida Gulf contained herein are not necessarily those puter and Software Engi- Coast University. Prior to of the FAA. Additional support was received from the Florida Space Grant neering, Embry Riddle this, he worked for various Consortium under Grant No. UCF01- Aeronautical University. nuclear research institu- E000029751. He has more than 20 years of research tions, including the Data Acquisition and teaching experience in areas of real- Group of Superconducting Super Collider References time computer systems. Kornecki con- and Computer Safety and Reliability 1. Radio Technical Commission for tributed to research on intelligent simula- Center at Lawrence Livermore National Aeronautics, Inc. “RTCA DO-178B, tion training systems, safety-critical soft- Laboratory. He also worked on projects Software Considerations in Airborne ware systems, and served as a visiting and consulted for a number of private Systems and Equipment Certifica- researcher with the Federal Aviation companies, including Lockheed Martin, tion.” Advisory Circular. Washington, Administration (FAA). He has been con- Harris, and Boeing. Zalewski served as a D.C.: RTCA, 1 Dec. 1992 . safety-critical software in medical and avi- for Information Processing Working 2. Federal Aviation Administration. ation industries and for the FAA Certi- Group 5.4 on Industrial Software Quality, “RTCA Inc., Document RTCA/DO- fication Services. Recently, he has been and of an International Federation of 178B.” Advisory Circular No. 20-115B. engaged in work on certification issues Automatic Control Technical Committee Washington, D.C.: U.S. Department of and assessment of development tools for on Safety of Computer Control Systems. Transportation, Nov. 1993 . Embry Riddle Aeronautical University Florida Gulf Coast University 3. Radio Technical Commission for 600 Clyde Morris BLVD 10501 FGCU BLVD Aeronautics, Inc. “RTCA DO-248B, Daytona Beach, FL 32114 Fort Myers, FL 33965 Final Report for Clarification of DO- Phone: (386) 226-6888 Phone: (239) 590-7317 178B ‘Software Considerations in Fax: (386) 226-6678 Fax: (239) 590-7330 Airborne Systems and Equipment E-mail: [email protected] E-mail: [email protected] Certification’.” Advisory Circular.

22 CROSSTALK The Journal of Defense Software Engineering April 2006