The ’s Review of Emerging Technologies 6œÊ£™Ê œÊ£ÊUÊÓ䣣

HighHigh ConfidenceConfidence SoftwarSoftwaree andand SystemsSystems Letter from the Guest Editor For years the National Security Agency (NSA) has pursued research in high confidence and systems (HCSS) technologies to improve the assurance of security critical algorithms, protocols, software, and hardware. Along the way, NSA has been a leader in the development of a national, collaborative community of HCSS researchers and sponsors, some of whom are represented in this issue of The Next Wave (TNW). HCSS research has primarily focused upon developing foundational technology and techniques, yielding components and systems that are “correct by construction.” HCSS research has also been aimed at creating analytic techniques to assess and improve the quality of existing code and specifications. Over the years, HCSS research projects have delivered significant advances within both developmental and analytic areas, and yet substantial questions remain unanswered: ฀ ฀฀฀฀฀฀฀฀฀฀฀฀฀฀฀ can one obtain high assurance that security has been achieved? ฀ ฀฀฀฀฀฀฀฀฀฀฀฀฀฀฀ or worse, are of unknown provenance?

This issue of TNW provides a glimpse into the multi-faceted research strategy gaining traction within and beyond the HCSS community—a strategy that attempts to tackle tough questions such as those identified above. Each facet of the strategy, whether preventive or analytic, will require better evidence—evidence capable of supporting an objective assessment that the system in question meets specified requirements. In short, the need for evidence-based assurance is the core tenet of each approach discussed here. Additionally, each article in this issue highlights the strong overlap between preventive and analytic methods, with an emphasis on the early application of analytic methods in the development process. When used at the earliest stages in the process, analytic methods guide development choices, thereby lessening engineering risks. In closing, it would be irresponsible to publish this issue of TNW without explicitly acknowledging the one person I consider to be the heart of the HCSS community within the United States—Dr. Helen Gill from the National Science Foundation. Dr. Gill has worked tirelessly within this community, giving of her time, her talent, and her wisdom. Dr. Gill exemplifies the ฀฀฀฀฀฀฀฀฀฀฀฀฀฀

William B. Martin, Chief, High Confidence Software and Systems Division

The Next Wave is published to disseminate technical advancements and research activities in telecomm- unications and information technologies. Mentions of company names or commercial products do not imply endorsement by the US Government. CONTENTS

FEATURES

4 A Letter from Sir Tony Hoare

5 Empowering the Experts: High-Assurance, High-Perlormance, High-Level Design with Cryptol

14 A High-Assurance Methodology for the Development of Security Software

22 Correct by Construction: Advanced Software Engineering

32 Verified Software in the World

34 Software for Dependable Systems: Sufficient Evidence?

44 Critical Code: Software Producibility for Defense

50 Cyber-Physical Systems A Letter from Sir Tony Hoare

I heartily welcome this special issue of Theoretical research now can use the experimental The Next Wave. It gives a realistic picture of the method as a means of differentiating, selecting, advancing state of the art in the specification, and improving the relevant theories for solution of design, implementation, and certification of high existing and future problems. confidence systems. The articles in this issue concentrate on This topic has interested me since the 1960s, advances in tools and experiments. They explicitly when I first encountered an article by Bob Floyd outline the remaining deficiencies and difficulties, on Assigning Meanings to Programs. At that time, but I hope that they give sufficient evidence to I judged this was a topic highly suited to pure encourage a wider range of pioneering applications, academic research, a career on which I was just leading at a sensible rate towards general adoption embarking. Like other scientific investigators, we of computer-assisted programming methods, both hoped to enlarge scientific understanding of what by software engineers and by their customers. ~ computer programs do, and how and why they work. We hoped to test the range of applicability of scientific theory by experimental verification of real programs. We were driven by ideals of total program correctness, and total certainty achieved by mathematical proof. About the author As in other mature branches of science (e.g., In 1980, Sir Tony Hoare received the ACM physics, chemistry, and most recently biology), Turing Award for his "fundamental contributions the fundamental research has now reached a point to the definition and design of programming where it can be applied in engineering practice. As languages," and in 2000 he was awarded the in other branches of engineering, the key to this Kyoto Prize for his "pioneering and fundamental technology transfer has been the availability of contributions to software science." These two powerful programme analysis and theorem-roving awards represent the top international accolades tools. They are based soundly on scientific theory, available to a computer scientist. Also in 2000 he but conceal this fact to an appropriate degree from was knighted by Her Majesty the Queen for services their users. The tools are now subject to continuous to education and computer science. Sir Tony is now improvement in the light of realistic academic Emeritus Professor at Oxford University, and works and industrial experiments, and by exploiting as a Senior Researcher at Microsoft Research in the increasing performance of algorithms for Cambridge. logical and mathematical reasoning by computer.

4 A Letter from Sir Tony Hoare Empowering the Experts: High-Assurance, High-Performance, High-Level Design with Cryptol

domain-specific language (DSL) is a programming language targeted at producing solutions in a given problem domain by enabling subject­ A matter experts to design solutions in terms they are familiar with and at a level of abstraction that makes most sense to them. In addition, a good DSL opens the way for powerful tool support: simulations for design exploration; automatic testing and generation of test harnesses; generation of highly specialized code for multiple targets; and generation of formal evidence for correctness, safety, and security properties.

Cryptographer as designer or has insufficient throughput, or... , the into a test harness, and blasted with test You are a highly skilled crypto­ hardware designer iteratively tweaks the vectors. If it works, great. Otherwise, the design until it is "good enough." grapher charged with designing a custom, search begins to find the error. But how do you know if it still state-of-the-art solution for And what if a security hole; for works the way you intended? protecting mission-critical information. example, a malicious counter or a back There are explicit and competing Typically, the design is fabricated door; was introduced? Would you even requirements for the implementation­ (if it is an ASIC-application-specific know? throughput, size, power utilization, ) or loaded into an FPGA operation temperature, etc.-that may (field-programmable gate array), placed There must be a better way. affect the implementation. You produce a design and want to see how it matches up with the Creating a crypto algorithm requires implementation requirements. How ski/ls In math AND programming would you proceed? Figure 1: Traditionally, the crypto developer must be highly Typically, you find an expert trained and expert at balancing hardware designer who translates your a myriad of often conflicting requirements. algorithm into VHDL (a hardware description language), and then runs proprietary tools to characterize the Image Source: Galois, Inc. implementation. If it uses too much power, The Next Wave• Vol 19 No 1 • 2011 5 From Section 3.1 of the AES definition [2]: The input and output for the AES algorithm each consist of sequences of 1 28 bits ... The Cipher Key for the AES algorithm is a sequence of 128, 192 or 256 bits. Other input, output and Cipher Key lengths are not permitted by this standard. In Cryptol: Cryptol: A better way {k}{k >= 2, 4 >= k} => {[128],[64*k]} -> [128] The Cryptol specification language Image Source: Galois, Inc. was designed for the National Security Figure 2: The constraints and requirements from the Advanced Encryption Standard Agency (NSA) as a public standard for (AES) [2] can be translated directly into Cryptol types, as shown above. The colored specifying cryptographic algorithms [ l]. text shows the linkage between English constraint and Cryptol type. The Cryptol tools provide a development path for cryptographic modules across the entire software process, from specification des : ([64],[56]) ·> [64]; and implementation to verification and des (pt, key) = permute (FP, last) certification. Cryptol tools significantly where { pt'= permute (IP, pt); reduce overall life-cycle costs by iv = [I round (Ir, key, rnd) addressing the key cost drivers in the 11 rnd <· [O .. 15] II Ir <·[(split pt')]# iv deployment of cryptography. IJ; last= join (swap (iv@ 15)); Rapid design cycle swap [ab] = [b a]; }; Cryptol specifications are fully executable, allowing designers round : ([2][32], [56], [4]) ·> [2][32]; ••• round([I r], key, rnd) = [r (IAf(r, kx))] to experiment with their programs where { incrementally as their designs evolve. kx = expand(key, rnd); f(r,k) = permute(PP, SBox(kApermute(EP, r))); The Cryptol tools support a refinement }; methodology that bridges the conceptual gap between specification and low-level Image Source: Galois, Inc. implementation, thereby reducing time to market. For example, Cryptol allows Figure 3: The Data Encryption Standard (DES) algorithm is a block cipher that engineers and mathematicians to program uses a 56-bit symmetric key. The diagram above is taken from the Standard [3]. Cryptol uses parallel stream comprehensions to interleave data and lazy evaluation cryptographic algorithms on FPGAs as if to encapsulate multiple computational stages in a single statement. Colors and they were writing software. shapes are used to help relate the program text to the diagram. Details of the language can be found in [4] and at www.cryptol.net. Reusable specification The Cryptol tools provide a predicates constrain the range of values platform-neutral specification language Design: The Cryptol the quantified size variables can accept; that generates implementations on multi­ language here, k must be between 2 and 4. To the ple platforms. Cryptol tools can generate Cryptol [l] is a pure functional right of the =>, we see the actual type. software implementations, hardware language built on top of a polymorphic The function has two inputs: a 128- implementations, and formal models for type system that has been extended with bit word containing the plaintext and a verification, all from a single Cryptol size polymorphism and arithmetic type 64*k-bit wide key. The function outputs program. predicates designed to capture constraints that arise naturally in cryptographic another 128-bit word, the ciphertext. Note Accelerated certification specifications. the precise correspondence of the type to the English description in the standard. A Cryptol reference specification Figure 2 shows an excerpt from becomes the formal documentation for the AES specification [2] that describes Figure 3 shows a Cryptol code the cryptographic module, eliminating the generator inputs and outputs, and the snippet-a specification for the core of the need for separate and voluminous corresponding Cryptol definition. The the DES algorithm. Note the compact English descriptions. In addition, Cryptol text to the left of=> ([128},[64*k]) in the mathematical function notation and the verification tools show functional Cryptol definition describes quantified definition of sequence structures and bit sizes. The Cryptol Reference Manual [4] equivalence between the specification type variables and predicates on them. In and the implementation at various stages this case, the type is size polymorphic, has many more examples as well as a of the toolchain. relying on the size variable k. The detailed description of the language.

6 Empowering the Experts: High-Assurance, High-Performance, High-Level Design with Cryptol FEATURE

Implement: operations are to be applied. But there High-level exploration of The Cryptol FPGA the resemblance ends. Sequences, which the design space Type 1 cryptographic devices appear repeatedly in the mathematical Good design is always at the root of protect information of national descriptions of crypto algorithms, great performance. One of the key factors importance. The information assurance have many different instantiations as in Cryptol’s performance results is its standards for such products are hardware. At one extreme, the sequence ability to explore the implementation correspondingly high. In addition, crypto can be spread out in space as side-by- design space at a very high level. A modernization requirements mandate field side parallelism. At the other extreme, Cryptol developer can experiment with programmability, and various operational the sequence can be laid out in time as many different microarchitectures in the requirements call for a reduced space, consecutive values held in a register, or course of a few days, covering ground weight, and power footprint. over many registers in a pipeline. Many that would otherwise take weeks or FPGAs offer a compelling platform combinations of these are also possible. months using traditional methods. A to address these needs. They are field The Cryptol FPGA generator uses a variety of implementation approaches can updatable by design, offer tremendous wide variety of engineering heuristics to be modeled and characterized quickly. performance potential, and have fewer pick an appropriate translation of a Cryptol For example, at the Cryptol level, nonrecurring engineering costs than function to an FPGA configuration that a straightforward idiom identifies pipe- traditional ASIC designs. will make effective and efficient use of lined functional units in hardware. Recall However, FPGA development the silicon. The user can also provide the specification for DES shown in Figure still requires the considerable time and pragmas (compiler commands) about 3. The designer has created a pipelined talents of skilled hardware designers, space/time mappings, thereby guiding the version of the round function by hand which increases development time translation process without compromising by factoring the high-level Cryptol and costs. Mainstream design tools the integrity of the original specification. specification, as shown in Figure 4. The supplied by FPGA vendors have more The declarative quality of Cryptol, Cryptol FPGA generator produces an in common with VLSI (very-large- which makes Cryptol a good specification efficient pipelined circuit, also shown in scale integration) design tools than with language, also plays a key role in the Figure 4 on page 8. modern programming environments. effectiveness of automatic generation High-level design exploration pro- These design tools automatically limit of FPGA cores. In contrast, the inherent vides a profound advantage in the devel- the user population to designers trained in sequentiality of mainstream program- opment of high-performance algorithms VLSI design. ming languages makes them a poor match (or in algorithms meeting other design The Cryptol FPGA generator for the highly parallel nature of FPGAs. constraints). The key is the speed with introduces a new design flow that allows which the developer is able to iterate the Creating high-performance engineers and mathematicians to program design, the bottleneck of hardware design. designs cryptographic algorithms on FPGAs A crypto developer can produce rapid de- in a high-level language incorporating The Cryptol FPGA generator sign iterations using the Cryptol Toolkit, concepts and constructs familiar to produces cores whose throughput and effectively increasing productivity by up cryptologists. The vision is that instead area usage have been comparable to to an order of magnitude over traditional of demanding low-level hardware design (and in some cases better than) hand- VHDL development. knowledge, users are able to express their coded VHDL/Verilog. For example, an designs and programs at a much higher implementation of 128-bit AES for the Trust: The Cryptol level of abstraction and take advantage Xilinx Virtex 4 FPGA has been generated verification framework of powerful automated mechanisms with clock rates in excess of 200 MHz The FPGA generator uses semantic for generating, placing, and routing the (which translates to throughput of better models to establish the correctness of circuits. than 25 Gbps) using only 6912 slices (25 the process. To gain final assurance, In some ways, the mathematics percent of the slices on the chip) and 100 Cryptol developer Galois provides an behind a cryptographic specification is Block RAMs (62 percent of the available automatic equivalence checker to prove like a hardware description. Both give Block RAMs). Theoretical results based that the actual code that will run on the unambiguous specification of how bits on Xilinx tools indicate that 500 MHz (65 FPGA is equivalent to the reference are to be handled and how bit-level Gbps) is achievable by these cores. implementation.

The Next Wave „ Vol 19 No 1 „ 2011 7 round : [inf]([2][32],[56]) -> [inf]([2][32],[56]); round dataO = data3 where { The Cryptol equivalence checker data1 =[zero]# [I (expand key" permute(EP, r) , [I r], key) utilizes state-of-the-art SAT (Boolean II ([Ir], key)<- dataO satisfiability) and SMT (satisfiability IJ; data2 =[zero]# [I (SBox(kx) , [Ir], key) modulo theories) solvers as proof engines, II (kx, [Ir], key)<- data1 together with custom heuristics and IJ; techniques. For example, the equivalence data3 = [zero]# [I ([r (I "permute(PP, sb))], key) II (sb, [Ir], key)<- data2 checker can show the equivalence of an IJ; AES specification written in Cryptol }; with an unrolled, pipelined VHDL dataO data1 data2 data3 implementation of AES generated from Cryptol and passed through the Xilinx toolchain all the way to place and route.

Two classes of problems Image Source: Galois, Inc. Cryptol's verification framework has been designed to address equivalence­

Figure 4: The code snippet above shows a new implementation of the DES and safety-checking problems. round function, shown in Figure 3 on page 6. A flow diagram is included, with The equivalence-checking problem colors showing the correspondence between code and diagram element. This version uses sequence comprehensions that can be performed in parallel asks whether two functions,f and g, agree and introduces extra variables that translate into registers and pipelined on all inputs. Typically, f is a reference operations in the VHDL implementation. implementation of some algorithm, following a standard textbook-style description, and g is a version optimized for time and/or space for a particular target platform. The equivalence­ Refine spec Create an FPGA far a specific implementation from the checking framework allows a developer target target specification to formally prove that f and g are semantically equivalent, ensuring that the often very complicated and extensive optimizations performed during synthesis ------·· have not introduced bugs. Note that the final implementation g does not need to

...... / be in Cryptol-an important use case of Key .. the verification framework is to verify that - CllJl!OllOo~ - FPOAvendartoolS third-party algorithm implementations • Cryptolfite:s (typically in VHDL) are functionally .. FormilMo!lelS Image Source: Galois, Inc . Oemeraled mes equivalent to their high-level Cryptol -+ Input to tool versions. In this case, Cryptol acts as a - - ~ 1nr:iutto de:slg,~r hardware/software verification tool [5]. The safety-checking problem is Figure 5: Verification can be performed at various points during the translation, about run-time exceptions. Given a which allows for high-assurance refinement during development. Note that the major compiler phases [the flow through the top line) remain outside the function f, we would like to know if f s trusted-code base for verification. Trust in the down-arrows, representing execution can perform operations such as translators from various intermediate forms to formal models, along with the division by zero or index out of bounds. off-the-shelf equivalence checkers themselves, is all that is needed. These checks are essential for increasing the reliability of Cryptol-generated implementations, since they eliminate the

8 Empowering the Experts: High-Assurance, High-Performance, High-Level Design with Cryptol FEATURE

need for sophisticated run-time exception transformations to increase speed and/or check points, Cryptol provides the user handling mechanisms. reduce space usage. Behind the scenes, with a high-assurance development The Cryptol toolset comes with a the Cryptol toolchain translates Cryptol to environment, ensuring that the applied push-button equivalence/safety checking a custom signal-processing intermediate transformations preserve semantic framework to answer these questions representation (SPIR), which acts as a equivalence. The final piece of the puzzle automatically for a large subset of the bridge between Cryptol and FPGA-based for end-to-end verification is generating Cryptol language [6]. Cryptol uses off- target platforms. The SPIR representation an AIG for the bit file generated by the the-shelf SAT/SMT solvers such as allows for easy experimentation with Xilinx tools, as represented by the dashed ABC [7] or Yices [8] as the underlying high-level design changes, because it line in Figure 5. At this time, the format of equivalence-checking engine, translating remains fully executable while also this file remains proprietary. Cryptol specifications to appropri- providing essential timing/space usage Verification for the cryptography ate inputs for these tools automatically. statistics without going through the domain: Why this works However, the use of these external tools computationally expensive synthesis remains transparent to the users, who tasks. Cryptol’s formal verification framework clearly benefits from recent only interact with Cryptol as the main Once the programmer is happy with advances in SAT/SMT solving. However, verification tool. the design, Cryptol translates the code to it is also important to recognize that the Of course, equivalence checking VHDL, which is further fed to third-party properties of cryptographic algorithms applies not only to handwritten programs synthesis tools. Figure 5 shows the flow but also to generated code. Cryptol’s for the Xilinx toolchain, taking the VHDL make applications of automated formal synthesis tools perform extensive and through synthesis, place and route, and methods particularly successful. This often very complicated transformations bit-file generation steps. In practice, is especially true for symmetric key to turn Cryptol programs into hardware these steps might need to be repeated, encryption algorithms that rely heavily primitives available on target FPGA using feedback from the synthesis tools, on low-level bit manipulations instead platforms. The formal verification until the implementation satisfies the of the high-level mathematical functions framework of Cryptol allows equivalence requirements. The overall approach aims employed by public-key cryptography. checking between Cryptol and netlist at greatly reducing the number of such In particular, symmetric-key representations that are generated by repetitions by providing early feedback cryptographic algorithms almost never various parts of the compiler, as we will to the user, at the SPIR level. The final perform control flow based on input data, explain shortly. Therefore, any potential outcome is a binary file that can be in order to avoid attacks based on timing. bugs in the compiler itself are also caught downloaded onto a Xilinx FPGA board, The series of operations performed are by the same verification framework. This completing the design process. typically “fixed,” without any dependence is a crucial aspect of the system: proving Cryptol’s verification flow is on the actual input values. Similarly, the the Cryptol compiler correct would be an interleaved with the design process. As loops used in these algorithms almost extremely challenging if not impossible depicted in Figure 5, Cryptol provides always have fixed bounds; typically these task. Instead, Cryptol provides a verifying custom translators at various points in bounds arise from the number of rounds compiler that generates code along with a the translation process to generate formal specified by the underlying algorithm. formal proof that the output is functionally models in terms of AIG (andinverter- Techniques like SAT-sweeping [10] are equivalent to the input. graph) representations [9]. In particular, especially effective on cryptoalgorithm Design and verification flow the user can generate AIG representations verification, since simulation-based Figure 5 provides a high-level from the reference (unoptimized) Cryptol node-equivalence guesses are likely to overview of a typical Cryptol development specification, from the target (optimized) be quite accurate for algorithms that and verification flow. Starting with a Cryptol specification, from the SPIR rely heavily on shuffling input bits. Cryptol reference specification, the representa tion, from the post synthesis Obviously, these properties do not make designer iteratively refines the program circuit description, and from the final formal verification trivial for this class of and “runs” it at the Cryptol command (post-place-and-route) circuit description. crypto algorithms; rather, they make the line. These refinements typically By successive equivalence checking of use of such techniques highly feasible in include various pipelining and structural the formal models generated at these practice [11].

The Next Wave „ Vol 19 No 1 „ 2011 9 Verify: Evaluating Proving security properties. Not third-party VHDL all properties of interest can be cast as implementations functional equivalence problems. This The process of verification in is especially true for cryptography. For Cryptol typically begins with instance, if we are handed an alleged understanding the high-level interface of VHDL implementation of AES, in the VHDL implementation under study. addition to knowing that it implements Through Cryptol’s foreign-function AES correctly, we would like to be interface, the base interface to the VHDL is sure that it does not contain any “extra simply imported using Cryptol’s “extern” circuitry” to leak the key. In general, declaration capability. Then the required we would like to show that an end -matching code is written in cannot gain any information from an Cryptol, mainly implementing the proper implementation that cannot be obtained use of control signals. This process makes from a reference specification. the external implementation available at Reducing the size of the trusted the Cryptol command prompt, enabling code base. Cryptol’s formal verification the user to call it on specific values, pass it system relies on the correctness of the through previously generated test vectors, Cryptol compiler’s front-end components essentially making the external definition (i.e., the parser, the type system, etc.), the behave just like any other Cryptol symbolic simulator, and the translators function. This facility greatly increases productivity, since it unifies software and to SAT/SMT solvers. Note that Cryptol’s hardware under one common interface. internal compiler passes, optimizations, Once the reference specification and and code generators (i.e., the typical the Cryptol/VHDL hybrid expose the compiler back-end components) are not same interface, the user generates formal in the trusted code base. While Cryptol’s models for both of them, and checks for trusted code base is only a fraction of the equivalence. entire Cryptol tool suite, it is nevertheless a large chunk of code from the open- Challenges ahead source language, Increasing the coverage of formal Haskell. Reducing the footprint of this methods. Cryptol’s formal verification trusted code base, and/or increasing framework works on a relatively large assurance in these components of the subset of Cryptol [6]. The main limitation is in verifying algorithms for all time, i.e., system, is an ongoing challenge. programs that receive and produce infinite streams of data. Currently, Cryptol can verify such algorithms only up to a fixed number of clock cycles, effectively introducing a time bound. While this

restriction is irrelevant for most block- based crypto algorithms, it does not generalize to stream ciphers in general. The introduction of induction capabilities in the equivalence checker or the use of hybrid methods combining manual top- level proofs with fully automated SAT/ SMT-based sub proofs might provide a feasible alternative for handling such problems.

10 Empowering the Experts: High-Assurance, High-Performance, High-Level Design with Cryptol FEATURE

Q: What can YOU do with Cryptol? Q: What can YOU do with Cryptol? A: Create a crypto algorithm and A: Produce and refine a family Q: What can YOU do with Cryptol? generate test vectors. of designs. A: Gain assurance about your design.

"... an experienced Cryptol programmer given A team of developers from Rockwell Collins, Van der Waerden's theorem states that for any a new crypto program specification and a so~ Inc. and Galois, Inc. has successfully produced positive integers rand k there exists a positive copy of test vectors can be expected to learn high-speed embedded Cryptographic Equipment integer N such that if the integers (1 2 ... NJ the algorithm and have a fully functional and Applications [CEAsJ, automatically generated are colored, each with one of r different colors, then there are at least k integers in arithmetic verified Cryptol model in a few days to a week. " from high-level specifications. An algorithm progression all of the same color. For any rand core generated from a Cryptol specification for 'The AIM crypto engine software engineers k, the smallest such N is the van der Waerden AES-256 running in Electronic Codebook mode at General Dynamics C4 Systems use the number W[r,k]. demonstrated throughput in excess of 16 Cryptol modeling language as part of their Gbps. These high-speed CEA implementations Van de Waerden numbers are difficult to Software Engineering Institute CMM® Level comprise a mixture of software and VHDL, compute. In 2007, Dr. Michal Kouril of the 5 development process. Cryptol provides four University of Cincinnati established that and target a compact new embedded platform basic benefits leading to the certification of W[2,6)=1132 [i.e., 1132 is the smallest designed by Rockwell Collins. Notably, almost no crypto equipment. First, Cryptol allows the integer N such that every 2-coloring of (1 2 traditional /ow-level interface code was required design engineer to rapidly express an algorithm ... NJ contains a monochromatic arithmetic in order to implement these high-performance in a common mathematical notation, which progression of length 6) [19]. The most recent CEAs. In addition, automated formal methods is fully executable on the Cryptol , previous result, W[2,5)=178, was discovered prove that algorithm implementations faithfully some 30 years earlier. Kouril computed W[2,6) providing verification that the algorithm is implement their high-level specifications. using a special SAT-solver and clever techniques completely understood. Second, the Cryptol Significantly, the Rockwell Collins/Galois team to bound the search and employed FPGAs to notation for the various components of the was able to design, implement, simulate, speed up the search. algorithm are used to annotate the AIM micro integrate, analyze, and test a complex CEA on sequencer code which provides much greater Kouril wrote VHDL to program the FPGAs. the new hardware in less than 3 months. readability of that extremely dense assembly In order to convince himself that the FPGA ensemble was doing what he expected, he also language. Third, component testing of AIM AES-256, ECB mode, Clockrate Resources Throughput Virtex-4 technology expressed his algorithm in Cryptol, generated code, from small snippets through major Implementation (MHz) (slices) (Gbps/ formal models for both the Cryptol specification subroutines is greatly facilitated with Cryptol characteristics second) and the VHDL implementation, and verified that Optimized for generated test vectors derived from end-to­ 127.5 2690 16.3 high throughput the two were equivalent! end test vectors provided in algorithm source Optimized to minimize 135.1 849 1.2 specifications. Finally, Cryptol models are resource usage Why not let Cryptol generate the solution? Handwritten, 102.0 2535 0.9 So far no one has found a way to prove evolving to directly support the certification minimal size unsatisfiability of W[r, k] directly without an effort ... " extensive search. The reliance on search makes the problem hard; and although people have found ways to generate long partitions without a monochromatic arithmetic progression [20], the true test that there are no longer partitions is currently only possible using a search.

• ·-- ...... ~ / ,: . Cryf)tol ' , )I( • ' referenc ' \ \ I specificati I I

I '-,_ - : _· - __)' ·~...... ~ . Image Source: Galois, Inc. Image Source: Galois, Inc. ~/ \'---[§J- -­Image Source: Galois, Inc.

The Next Wave• Vol 19 No 1 • 2011 11 Q: What can YOU do with Cryptol? Q: What can YOU do with Cryptol? A: Gain assurance about someone A: Teach and learn about cryptography, satisfiability theory, .... else's design. "Cryptol was quite an experience. We began with simple sequences such as [1 2 3 4] and by Skein [12] is a suite of cryptographic hash applying '@' and '!' to our list of numbers, we learned the priority/position of each number: when algorithms targeted at the NIST SHA-3 using@, the order is zero based, [Dth 1st 2nd 3rd], and when using !, the order is reversed, [3rd competition [13]. At its core, Skein uses a 2nd 1st Dth]. Each number or element contains a certain numbers of bits: 1 [Db1) contains one tweakable block cipher named Threefish. The bit, 2 [Db1 DJ is two bits, 3 [Db11) is also two bits and 4 [Db1 DD) is three bits. unique block iteration [UBI) chaining mode Once the group grasped the concept of bits, we moved on to shifting and permuting sequences defines the mode of operation by the repeated using split, join, splitBy, groupBy, take, drop, reverse, and transpose. We then applied these fun­ application of the block cipher function. damentals we had learned about Cryptol to interact with its interpreter and to explore some of the Galois developed and published a Cryptol concepts we had learned earlier in the year, such as Pascal's Triangle, the Fibonacci sequence, the specification for Skein [14]. We have verified two sum of a series of odds, even, etc. Once that was complete, and given that Cryptol's intended use independently developed VHDL implementations is cryptography, we used Cryptol to encrypt plaintext and decrypt ciphertext for a range of classes of Skein against our specification for one 256- of cryptographic algorithms, to include classic [substitution and transposition) and modern [sym­ bit input block, generating a 256-bit hash value. metric and asymmetric) cryptographic systems.

The first verification was performed against We concluded our study of Cryptol by looking into Q: What can YOU do with Cryptol? Men Long's implementation [15]. Long propositional logic and satisfiability, and ultimately A: Make a MILS FPGA. implemented only the underlying Threefish at a satisfiability solver that could be called from The Cryptol Development Toolkit from encryption and the XOR of input data; we within the Cryptol interpreter. In our examination Galois provides a tool flow that puts FPGA modified our reference specification to of propositional logic, we were initially forced implementation into the hands of mainline match. The AIG generated from the Cryptol to prove our satisfying assumptions by hand developers, improving both productivity and specification had 118,156 AND-gates; the through the construction of small truth tables assurance, without sacrificing performance. VHDL version was more than five times as with assignments of values with the goal of having large, with 653,963 AND-gates. Equivalence the formula evaluate to 'true', that is, they were The Xilinx Single Chip Cryptographic [SCC] checking took about an hour to complete on satisfied. To extend these concepts we utilized technology enables Multiple Independent commodity hardware using ABC [7]. the automated satisfiability solver that we could Levels of Security [MILS] in a single chip. call from the Cryptol interpreter. One application These two technologies fit seamlessly into a In this work, we encountered a problem where we were able to represent a problem single development flow. with Long's VHDL code that rotated a 64-bit within Cryptol and to utilize the satisfiability solver signal a variable distance. The code was given was in solving Sudoku puzzles. It was an amazing The combined solution can address different meanings by GHDL [16], simili [17]. experience and I will continue to play around with high-grade cryptographic application and the Xilinx synthesis tools. We removed Cryptol and the satisfiability solver because it was requirements [redundancy, performance, the ambiguity by replacing it with the standard so very intriguing." red/black data, and multiple levels of library function rotate_left. Thus, the Cryptol security on a single chip) as well as high verification path identified an otherwise assurance development needs [high­ undetected ambiguity bug. level designs, automatic generation of implementation from design, automatically­ The second verification was performed against generated equivalence evidence), and has Stefan Tillich's full Skein implementation [18]. the potential to significantly reduce the time The AIG sizes in this case were 301,085 AND­ of costs of developing Type-1 cryptographic gates for the reference Cryptol versus 900,239 applications. AND-gates for the VHDL implementation: about MILS/S)'Slem design three times larger. Equivalence checking was decisions ma de here completed in about 18 hours, again using ABC.

Key ____.. lnput totool

_ _.., Feedback to designer

- Oyptol t- Image Source: Galois, Inc. - FPGA Vendor tools Image Source Galo1s, Inc .. Datalilesused asEvaluation!CertificationevidencB

Data Tiles (not used inforWILS e 11 aluation)

- Path from algorithm to MILS FPGA

12 Empowering the Experts: High-Assurance, High-Performance, High-Level Design with Cryptol FEATURE

References for equivalence checking and functional method to construct lower bounds for van property verification. IEEE Trans. on der Waerden numbers. The Electronic [l] Lewis JR, Martin B. Cryptol: CAD of Integrated Circuits and Systems. Journal of Combinatorics. 2007; 14( 1):R6. High-assurance, retargetable crypto 2002;21(12):1377-1394. Available at: Available at: http://www.combinatorics. development and validation. In: doi: 10. l 109/TCAD.2002.804386 org/Volume_l 4/v 14i1 toe.html Proceedings of Military Communications Conference 2003 (MILCOM 2003); [11] Smith EW, Dill DL. Automatic Further reading Oct 2003; Monterey (CA). p. 820- formal verification of block cipher 825. Available at: doi: 10.1109/ implementations. In: Proceedings of the Hardin DS, Browning SA. HSE final MILCOM.2003.1290218 2008 International Conference on Formal report, available from the authors. Methods in Computer-Aided Design [2] Announcing the AES. NIST; Nov Hardin DS, editor. Design and verification (FMCAD '08); Nov 2008; Portland 2001. FIPS Publication 197. Available of microprocessor systems for high­ (OR). p. 1-7 Available at: doi: 10.1109/ at: http://csrc.nist.gov/publications/Pubs assurance applications. 1st ed. Springer; FMCAD.2008.ECP.10 PIPS.html 15 Mar 2010. ISBN-10: 1441915382 [12] Ferguson N, Lucks S, Schneier B, [3] Data Encryption Standard (DES). Lewis JR, Hoffman C, Browning SA, Whiting D, Bellare M, Kohno T, Callas J, NIST; Oct 1999. FIPS Publication Walker J. The Skein hash function family. Martin WB. A complete design flow 46-3. Available at: http://csrc.nist.gov/ 2009. Available at: http: //www.skein­ for MILS in a single high-assurance publications/PubsFIPS.html hash.info FPGA. In: Proceedings of the Second [4] Galois, Inc. The Cryptol reference Annual European Reconfigurable Radio [13] NIST's Cryptographic hash manual. Available at: http://www.cryptol. Technologies (ERRT) Workshop; Jun algorithm competition. 2008. Available net at: http://csrc.nist.gov/groups/ST /hash/ 2010. Mainz, Germany. Available at: [5] L. Erkok L, Carlsson M, Wick A. sha-3 http:/!groups. winnforum.org/p/cm/ld/ Hardware/software co-verification of fid=98 [14] Finne S. A Cryptol implementation cryptographic algorithms using Cryptol. of Skein. Galois, Inc.; 23 Jan 2009. McLean M, Moore J. FPGA-based In: Proceedings of Formal Methods Available at: http://corp.galois.com/blog/ single chip cryptographic solution. in Computer Aided Design (FMCAD month/january-2009 Military Embedded Systems. '09); Nov 2009; Austin, (TX). p. 188-191. Available at: doi: 10.1109/ [15] Long M. Implementing Skein Mar 2007. Available at: http:// FMCAD.2009.5351121 hash function on Xilinx Virtex-5 FPGA w ww. mil-embedded. com/articles/ id/?2069 [6] Erkok L, Matthews J. Pragmatic platform. 02 Feb 2009. Available at: http://www.skein-hash.info/downloads equivalence and safety checking in Cryptol. In: Proceedings of Programming [16] GHDL simulator version 0.26. Languages meets Program Verification Available at: http://ghdl.free.fr/ (PLPV'09); Jan 2009; Savannah (GA). p. [17] Symphony EDA. VHDL Simili 73-81. Available at: http://portal.acm.org/ simulator version 3.1. Available at: http:// citation.cfm ?id= 1481860 www.symphonyeda.com/products.htm [7] Mishchenko A. Berkeley Logic [18] Tillich S. Hardware implementation Synthesis and Verification Group. ABC: of the SHA-3 candidate Skein. Report System for sequential synthesis and 2009/159; Apr 2009. Cryptology ePrint verification, release 70930. Available at: Archive. Available at: http://eprint.iacr. http://www.eecs.berkeley.edu/-alanmi/ org/2009/159 abc [19] Kouril M, Paul JL. The van [8] Yices: An SMT Solver. Available at: der Waerden number W(2, 6) is http://yices.csl.sri.com/ 1132. Experimental Mathematics. [9] Biere A. The AIGER And-Inverter 2008:17(1):53-61. Available at: http:// Graph (AIG) format, version 20071012. www.expmath.org/expmath/contents. Available at: http://fmv.jku.at/aiger/ html [ 10] Kuehlmann A, Paruthi V, Krohm F, [20] Herwig PR, Heule MJH, van Ganai MK. Robust Boolean reasoning Lambalgen PM, van Maaren H. A new

The Next Wave· Vol 19 No 1 • 2011 13 A High-Assurance Methodology for the Development of Security Software

1. Introduction which very high assurance of correctness, The critical HSCC design goals are reliability, and security or safety is high reliability and achieving very high Security systems require especially essential. In this article, we use the High assurance of functional correctness and high levels of assurance of correctness, Speed Crypto Controller (HSCC) project essential security properties. As a result, reliability, and security. Researchers to illustrate a development methodology project responsibility for implementing in the National Information Assurance which we believe is useful in producing the data accelerators and the crypto cores Research Laboratory (now Trusted both high quality software and the remained with the hardware engineering Systems Research) at the National assurance evidence to support evaluation. organization while responsibility for the Security Agency (NSA) with the HSCC was passed to the High Confidence In order to study advanced high­ assistance of engineers at Rockwell Software and Systems (HCSS) Division. Collins conducted a project to exercise, speed electronics technology, hardware Because of the research mission evaluate, and enhance a methodology research engineers in the NIARL started of the HCSS division, the project had a project to build a prototype high­ for developing high-assurance software two main goals. The first goal was to de­ speed encryption system. The system for an embedded system controller. liver a working controller. The second architecture they arrived at is shown in In this approach, researchers captured goal was to exercise, evaluate, and try to Figure 1. system requirements precisely and enhance a strong unambiguously through functional In this design, the data accelerators methodology. Since HSCC is a security specifications using the Z (pronounced handle input/output functions, data system, the methodology has to support "zed") formal specification notation. formatting, and enforcement of some a full range of development aspects from Rockwell Collins then implemented these security policy rules. The encrypt core and requirements through very rigorous requirements using an integrated, model­ decrypt core perform the actual encryption evaluation by independent evaluators. In based software development approach. and decryption. These six subsystem addition to being rigorous, it should also The development effort was supported by blocks are in the high-speed data paths. be cost-effective in time and money. a suite of tools that provides automated The control block manages the subsystem Given the project goals and code generation and support for formal blocks but lies outside the high-speed the limited resources of our research verification. The specific system is a data path. An important consequence of organization, we in the HCSS division prototype high-speed encryption system, this architecture is that the HSCC does needed an industrial partner. We found although the controller could be adapted not need to be implemented using any the ideal partner in Rockwell Collins. for use in a variety of critical systems in exotic high-speed electronics technology. One reason for teaming with Rockwell

14 A High-Assurance Methodology for the Development of Security So~ware FEATURE

Collins was their capability with the with the security evaluation work done by use clear helpful names and well-chosen AAMP7G microprocessor and high- other NSA personnel. abbreviations. We have a house style for assurance FPGA development. The The approach we chose for the notational details such as capitalization. AAMP7G supports strict time and space HSCC project was for HCSS researchers The important point is that both writers partitioning in hardware, and has received to take the lead in writing control DQG UHDGHUV RI = EHQH¿W JUHDWO\ IURP DQ16$0,/6FHUWL¿FDWLRQEDVHGLQSDUW software requirements in the form of D FRQVLVWHQW VW\OH 7KH VSHFL¿F GHWDLOV on a formal proof of correctness of its IXQFWLRQDO VSHFL¿FDWLRQV LQ = 5RFNZHOO of the style are not nearly as important VHSDUDWLRQNHUQHOPLFURFRGHDVVSHFL¿HG &ROOLQVZRXOGWDNHWKHVHVSHFL¿FDWLRQVDV as the fact that there is a set of standard by the EAL-7 level of the Common input into their established development FRQYHQWLRQV ,Q RXU ¿QLVKHG GRFXPHQWV &ULWHULD >@ 7KH IRUPDO YHUL¿FDWLRQ RI process. They would look for we adhered strictly to the principle that the AAMP7G partitioning system was opportunities to strengthen the process, HYHU\ = SDUDJUDSK ZDV LPPHGLDWHO\ conducted using the ACL2 theorem including the support for evaluation, or preceded by an accurate natural language prover and culminated in the proof of a save time and money by taking advantage translation. theorem that the AAMP7G partitioning RIWKHIRUPDOVSHFL¿FDWLRQV Since the HSCC project was to microcode implements a high-level produce the controller for a crypto security policy [2]. 2. Z specification work system, we had to describe, at a suitable Perhaps more important than Over the last ten years, HCSS level of abstraction, the main work of their hardware capabilities, Rockwell researchers have worked with other the system. On the outbound data path Collins has a solid approach to RUJDQL]DWLRQV XVLQJ = LQ VXSSRUW RI D WKLV LQFOXGHV DFFHSWLQJ ¿OWHULQJ DQG software development. It features an variety of development projects. We use formatting unsecured data in the Red integrated, model-based development WKH =(9(6 >@ VXSSRUW WRRO DQG KDYH Ingress data accelerator; encrypting in the suite of tools—a toolchain—with a found it quite suitable for our needs. encrypt core; and formatting and sending IRFXV RQ SURYLGLQJ D GRPDLQVSHFL¿F Based on our experience, we chose to use secure data out in the Black Egress data modeling environment that abstracts =WRZULWHIXQFWLRQDOVSHFL¿FDWLRQVRQWKLV accelerator. The inbound data path is a the implementation details, promotes high-assurance controller project. mirror image with a decrypt core. architectural level design, and provides On this project we tried to follow From this basic system analysis we automated transformations between good habits acquired over the years. We could see what control data structures the problem domain formalisms and think carefully about names and try to had to be provided by the controller to the target platform. The tools simplify code development and facilitate the application of automated formal analysis RI_DA BE_DA EncryptCore tools. In addition, the toolchain is capable (Red Ingress Data Accelerator) (Black Egress Data RedIngress BlackEgress Status EncryptDB Status Status of interfacing directly to a simulation PolicerDB PolicerDB environment, providing another level of assurance of design correctness. Control Block For their part, HCSS researchers

KDYH H[SHULHQFH LQ WKH = VSHFL¿FDWLRQ FPGA_Info RedTEK_DB ControlsDB ODQJXDJH >@ 7KH\ KDYH ZULWWHQ = outControlSAs inControlSAs IXQFWLRQDO VSHFL¿FDWLRQV DQG GHVLJQ Security NetworkConfig ControlPending descriptions for several internal ProtocolInfo DB development projects [4]. In these projects, [5,6] HCSS researchers played the role of customers and read

DQG FRPPHQWHG RQ GUDIW VSHFL¿FDWLRQV RE_DA BI_DA DecryptCore DQG GHVLJQV LQ = ZULWWHQ E\ 3UD[LV (Red Engress Data Accelerator) (Black Ingress Data RedEgress BlackIngress Status DecryptDB Status Status High Integrity Systems. In addition to PolicerDB PolicerDB experience in the requirements stage of development, HCSS people are familiar Figure 1: High-Speed Crypto System functional block diagram

The Next Wave „ Vol 19 No 1 „ 2011 15 0 0 RoutingTable 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D D D D D D D D routingRecords: D RoutingRecord D knownDestAddresses: D NETWORK ADDRESS D maxNumRoutingRecords: D DDDDDDDDDDDOOOOD 0 # routingRecords 0 maxNumRoutingRecords D knownDestAddresses = D Ill rt: routingRecords Ill rt.destinationAddressesD 0 0 D kda: knownDestAddresses; rri, rr2: routingRecords formatting and control processing) were D Ill kda D rr1.destinationAddresses D kda D rr2.destinationAddresses implemented in hand-coded SPARK. D D rr1 = rr2 This code includes information flow 0 D rr1 , rr2: routingRecords annotations to enable use of the Praxis D Ill rr .destinationEncryptorAddress = rr .destinationEncryptorAddress 1 2 toolchain and to provide assurance of D D rr1 = rr2 OOOOODOOOODOOOOOOOOOOODOOOOOOOOOOOOOOOOO correctness. Database transactions were de­ Figure 2: Z specification of the routing table database signed and developed using the Rockwell properly manage the system. Basically, graphical modeling languages that can be Collins MBD toolchain, Gryphon [8]. the system had to match each incoming executed and analyzed before the actual Simulink/Stateflow models were created piece of user data with the right crypto­ system is built. The use of such modeling for each database transaction. Each model graphic algorithm and key material. languages allows the developers to create was then tested via simulation in the Re­ Secondary functions such as managing a model of the system, execute it on their actis tool to discover and correct obvi­ and updating key material were handled desktops, analyze it with automated tools, ous errors. When complete, the Gryphon next. We had to define a system control and use it to automatically generate code framework is used to translate the model protocol to convey system management and test cases. into the Prover tool. Gryphon supports messages back and forth between the several back-end formal analysis tools, controller and the other subsystems. After 3.1 HSCC software including Prover, NuSMV and ACL2; for specifying this basic functionality of the development using MBD this project, Prover was deemed to have system and the controller, we worked Software for the HSCC system was the best combination of performance and on the functional description of the developed in two parts. Some code was automation. Prover is used lo exhaus­ subsystems. hand coded by a human guided by the Z tively verify each transaction preserves By way of example, the Z schema spec and general engineering knowledge. properties (derived from Z specifications) that specifies the controller's routing table Other code was generated using portions about the database it is acting upon. The is shown in Figure 2. The specification of the tool chain in Figure 3. Simulink model proven to be correct was describes the contents of the database, System software (drivers and then used to generate SPARK-compliant the maximum size of the database, and interrupt/trap handling) and portions of Ada95 for use on the target. Figure 3 il­ further constraints on the data (e.g., no the high-level application code (message lustrates the process flow. duplicate addresses). Customer Modeling Test Formal Automated Formally The work described in this paper is Specification and Generation Specification Formal Verified Simulation and Language Analysis Model Property part of an ongoing research program. An Verification early version of a system specification was written over a period of about 18 months. It consisted of 185 pages ofZ and English. Z schema Using that document, specifications for the six subordinate subsystems and a ', ~---~ Various model checkers lower level communication protocol, and/or theorem provers 11 msy be used. Prover was totaling 290 pages, were written in about II the tool of choice for HSCC. II eight months. Finally, the revised High Figure 3: II Speed Crypto (HSC) System Control II Model-based II Specification, Version 2.0, 27 January development II process flow formally II verified 2010, containing 263 pages, was written I I model I I ~ in approximately seven months. I [ au

VHDL implementation 16 A High-Assurance Methodology for the Development of Security So~ware FEATURE

The HSCC software development integers and real numbers, through the table. The inputs (at left) are the routing process relies on a several tools: use of integrated decision procedures for table (Rt_Tbl) and the destination en- 6LPXOLQNŠ 6WDWHÀRZŠ DQG real and integer arithmetic. cryptor address (Dest_Encr_Addr) for MATLAB® are products of The By leveraging its existing Gryphon which to search. The output (at right) is MathWorks, Inc. [9] Simulink was chosen translator framework, Rockwell Collins the Boolean value (Found) resulting from for development because it is the standard designed and implemented a toolchain the search. The rectangular block in the model-based development environment capable of automatically generating center is a Simulink subsystem block that at Rockwell Collins and has extensive SPARK-compliant Ada95 source code implements the database lookup. existing tool support, including support IURP6LPXOLQN6WDWHÀRZPRGHOV Typically, a transaction model for formal analysis. ZLOOFRQWDLQD6WDWHÀRZFKDUWLQVLGHWKH 3.2 Transaction development Reactis® [10], a product of 6LPXOLQNPRGHO6WDWHÀRZLVZHOOVXLWHG Reactive Systems, Inc., is an automated 6LPXOLQN6WDWHÀRZPRGHOVDUHXVHG to the implementation of the database test generation tool that uses a Sim- as the common starting point for both the operations. The screenshot in Figure 5 XOLQN6WDWHÀRZPRGHODVLQSXWDQGDXWR implementation and analysis. Each model shows the contents of the Simulink sub- JHQHUDWHV WHVW FRGH IRU WKH YHUL¿FDWLRQ corresponds to a single database transac- system block depicted in Figure 4. The of the model. The test suites may be tion. Model inputs correspond to SPARK heavy vertical bar at the left is a Simulink used in testing of the implementation for procedure “in” parameters and outputs bus selector. Simulink bus objects are behavioral conformance to the model, as correspond to “out” parameters. Note the roughly analogous to a record in Ada or well as for model testing and debugging. database object used by each transaction SPARK. (The Reactis tool does not allow model may appear as both an input and an Gryphon [8] refers to the Rockwell EXVREMHFWVDVLQSXWVWR6WDWHÀRZFKDUWV so a bus selector is used to separate the Collins tool suite that automatically RXWSXWLIWKHGDWDEDVHLVPRGL¿HGE\WKH component parts of the bus object into translates from two popular commercial transaction. In this case, the database ob- modeling languages, Simulink/ ject access appears as an “in-out” param- VHSDUDWH LQSXWV WR WKH 6WDWHÀRZ FKDUW  The large rounded rectangle block is a 6WDWHÀRZŠ DQG 6&$'(Œ >@ LQWR eter in the generated code. For each data- several back-end analysis tools, including base, one model must be created to initial- 6WDWHÀRZFKDUW model-checkers and theorem provers. ize the data object, in addition to models As stated earlier, a model must be Gryphon also supports code generation to perform necessary transactions (add, built for each transaction in each database. into SPARK/Ada and C. Gryphon uses delete, lookup) on the database. Addi- In the case of the routing table, these are: WKH/XVWUHIRUPDOVSHFL¿FDWLRQODQJXDJH tional models are required for the formal Init – procedure to initialize the routing as its internal representation and has analysis to model invariants on the data- table data structure (called upon reset) been used at Rockwell Collins on several base object. This topic will be covered in Add – database transaction to add a VLJQL¿FDQW IRUPDO YHUL¿FDWLRQ HIIRUWV more detail in subsequent sections. routing record to the routing table involving Simulink models. The screenshot in Figure 4 shows a Delete – database transaction to remove a routing record from the routing table Prover [12] is a best-of-breed sample Simulink model that contains the commercial model-checking tool for Dest_Encr_Addr_Found lookup func- Dest_Encr_Addr_Found – database query to determine existence of analysis of the behavior of software and tion performed on the routing table. This destination encryptor address hard-ware models. Prover can analyze function performs a lookup in the routing Get_Dest_Addr_List – database lookup ERWK¿QLWHVWDWHPRGHOVDQGLQ¿QLWHVWDWH WDEOHWRGHWHUPLQHLIWKHVSHFL¿HGGHVWL- to return list of addresses mapped to an models, that is, models with unbounded nation encryptor address is found in the encryptor address

Figure 4: Destination Encryptor Address Found model Figure 5: Stateflow chart inside the model

num_routing_records_in 1 addr_count_list_in Rt_Tbl Rt_Tbl Found 1 dest_addr_map_in Dest_Encr_Addr Found 1 dest_addr_valid_list_in found 1 2 Dest_Encr_Addr_Found Rt_Tbl dest_addr_list_in Found Dest_Encr_Addr dest_addr_revmap_in

dest_encr_addr_list_in

dest_encr_addr L 2 Routing Table_dest_encr_addr_found Dest_Encr_Addr

The Next Wave „ Vol 19 No 1 „ 2011 17 procedure In it ( Rt Thl:out Routing_Table_Types.Routing_Table_'r;yp

I nit procedure _n..dd ( 'RLTbl RLTbLout

'DesLAddr

Respons ' DesLEncr_Addr

Add procedure Delete ( 'RLTbl Routing_Table _ouH> 3.4.1 Proof strategy ' DesLAddr:_in

Respons ' DesL Encryptoc_Addc_Found The proof strategy employed for Delete the data invariants is induction over the procedure Dest Encr Addr Found ( 'RLTbl Rt Tbl :in sequence of transactions that are per­ Faun DesLEncr_ Addr Found : out Boolean ) formed. We first verify that the Simulink Dest_Encr:._Addr_ Found procedure models responsible for initializing each 'RLTbl ls_Vali Rt Tbl database establish the data invariant for Boolean ; DesLAddr Dest_Encr_Add Dest Encr Addr nut Routing_Table_ Types.Address _Type ) that database. This step provides the Get_Dest_Encr_Addr basis for our induction. We then prove ls_Vali every transaction that modifies a data­ DesLAdd!:.._ListP.

LJesLAdd!:.._Coun base maintains the invariant for that Get_DesLAddr_List database. More concretely, on the "init"

Figure 6: Transaction models and associated SPARK signatures models we use the model checker to de­ termine whether or not the data invariants Get_Dest_Encr_Addr - database lookup cate destination encryptor addresses exist hold on the model outputs. For the other to return encryptor address mapped to a in the data structure and sets the output transactions the proof strategy is to assume destination address Boolean values accordingly. Note that the the invariants in the input "pre" database Figure 6 shows the interfaces number of Boolean outputs in the model (prior to performing the transaction), and provided by each model, alongside the is determined by the internal representa­ then use the model checker to determine generated SPARK procedure signature. tion of the routing table data structure, whether the invariants hold in the output "post" database (resulting from and that the condition in which all four 3.3 Invariant modeling performing the transaction). Boolean outputs are "false" indicates that To perform formal analysis on the both invariants hold. We prove all the invariants required transaction models, it is first necessary by the Z specification and also additional to model any invariants on the data 3.4 Formal verification invariants involving implementation structures. These invariants are taken In order to perform the formal details related to realizing the Z databases directly from the Z specification. As an verification of a database transaction, we in Simulink/Stateftow. For example, example, the invariants shown in Figure need to establish two kinds of properties: a linked-list representation is used for 7 appear in the Z specification for the 1) data invariants over the databases (as many of the finite sets described in the routing table. defined by the Z schemas defining each Z document. In this case, additional This specification indicates that no database) and 2) transaction requirements invariants establish that the linked list is duplicate destination addresses or dupli­ that ensure that the operation performed a faithful representation of the finite set. cate encryptor addresses may appear in by a model matches the Z schema for The transaction requirements for the routing table. These invariants are that transaction. The necessary models each operation are specified as additional checked by the no_dups model (shown include both the transaction model and properties that must hold on the "post" in Figure 8). Given a routing table input any invariant models associated with the database. For example, when deleting an [Rt_Tbl), the model checks that no dupli- relevant database(s). element, these properties ensure that the

Figure 7: Z specification invariant sample Figure 8: Sample invariant model

dup_dest_addr_map_entry f------1•(D 0 0 kda: knownDestAddresses; 1-r1, rr2: routingRecords dup_dest_ addr_map_entry D DD kda D rr .destinationAddresses D kda D 1-r .destinationAddresses 1 2 dup_dest_encr_addr_list_entry f------l•(D D D rr1 = rr2 dup_dest_encr_ addr_I ist_entry ITJ1~~..-+i Rt_Tbl 0 0 rr , rr : routingRecords 1 2 Rt_Tbl dup_dest_addr_list_entry >------1•(I) D DD rr1.destinationEnc1yptor Address = rr2.destinationEncryptorAddress dup_dest_ addr_I ist_entry D D rr = rr 1 2 dup_dest_addr_revmap_entry f------1•IT) dup_ dest_addr_revmap_ entry no_dups

18 A High-Assurance Methodology for the Development of Security So~ware FEATURE

element in question has been removed accurately determine the necessary “pre” through traditional software testing from the database. and “post” conditions for each database methods. With all these components in )RUPDOYHUL¿FDWLRQUHVXOWV transaction. our software development approach, we summary The use of a model-based approach have exercised a viable methodology to deliver high-assurance software with a 7KH IRUPDO YHUL¿FDWLRQ HIIRUW IRU to transaction development provides PXFK JUHDWHU OHYHO RI FRQ¿GHQFH WKDQ the project as a whole resulted in the proof early simulation capabilities, leading to software developed through traditional of some 840 properties for the HSCC earlier discovery of errors in both the approaches. databases, of which 140 were written by VSHFL¿FDWLRQ DQG LQ WKH LPSOHPHQWDWLRQ The use of automated code generation The use of SPARK information WKH YHUL¿FDWLRQ WHDP DQG WKH UHPDLQGHU removes the possibility of human coding (mainly well-formedness checks) ÀRZ DQQRWDWLRQV IRU $GD FRGH DW errors. The application of automated the system level provides assurance automatically generated by the Gryphon model checkers provides a proof of the system code is properly routing IUDPHZRUN 9HUL¿FDWLRQ UHTXLUHG OHVV correctness at a level unattainable information to each of the devices WKDQ ¿YH SHUFHQW RI WRWDO SURMHFW HIIRUW over the course of seven calendar months.

3.5 Code generation Code generation is performed after a transaction is proven to satisfy all of its invariant properties. Code generation for this project is accomplished through the use of a translation tool, developed during the program, that leverages the existing Gryphon framework to generate SPARK- compliant Ada95 source code for use on the AAMP7G, including the automatic generation of SPARK annotations. Model-based development is used with increasing frequency in the development of aircraft avionics. By using a model-based development All of the transactions are compiled approach, developers can detect errors early, avoiding more expensive fixes into single Ada95 package for use by later on. the system programmer. The procedures in the package declaration are shown in Model-based development was used successfully to develop the ADGS- Figure 6. 2100 Adaptive Display and Guidance System (ADGS) Window Manager. In modern aircraft critical status information is provided to pilots through 4. Conclusion computerized display panels like those shown. The ADGS-2100 is a Rockwell Our experiences developing the Collins product that provides the heads-up and heads-down displays and HSCC system have shown that the display management software for next-generation commercial aircraft. methodology described in this paper is The system ensures that data from different applications is routed to the a viable process for the development correct display panel, and in the case of a component failure decides which of high-assurance software for use in information is most important and routes that inaformation to the correct display panel. The displays are essential to the safe flight of an aircraft since cryptographic systems. they provide critical flight information to the flight crew. NSA-provided specifications Rockwell Collins has developed tools that translate models used to ZULWWHQ LQ WKH = IRUPDO QRWDWLRQ SURYHG develop systems like the ADGS-2100 to a suite of analysis tools. Verification to be superior to those written in English throughout a design process—while a design is still changing—leads to language in producing a complete earlier error detection. During the ADGS-2100 development project, 563 and unambiguous set of software properties were developed and checked and 98 errors were found and UHTXLUHPHQWV 8VLQJ WKHVH VSHFL¿FDWLRQV corrected in early versions of the model where they are much easier to fix. as the main development artifact, Rockwell Collins was able to quickly and

The Next Wave „ Vol 19 No 1 „ 2011 19 in the HSCC architecture. Hardware enforced (AAMP7G partitioning) red/ Auditing Queries: Does Information produced by black separation serves as the final Domain A always ftow through sentinel in preventing unintended red/ a guard condition before MLS passing to Domain ? Informa tion Flow black communication. In our judgment, ,,_ Contract : -· ... -- the methodology described in this paper is sturdy enough to support full EAL-7 Checking I Inference Partition of Flows by oiticality level certification of a production encryptor based on this research prototype. :;i

Information Flow Visualization

Developing and certifying systems with multiple levels of security (MLS) has proven to be extremely challenging. Despite the widespread use of so­ phisticated integrated development environments (IDEs) with analysis and verification tools for conventional software development, IDEs that provide dedicated support for specification and certification of MLS systems have yet to emerge.

Researchers at Kansas State University are moving to fill this void by developing an IDE called Chispa. Chispa is a visualization, analysis, and verification tool designed to evaluate MLS systems against associated in­ formation assurance requirements. For program development, Chispa uses SPARK, a safety-critical subset of Ada developed by Praxis High Integrity Systems and distributed by AdaCore. SPARK is used by various organiza­ tions, including Rockwell Collins and the National Security Agency (NSA), to engineer information assurance systems such as cryptographic controllers, network guards, and key management systems.

Chispa uses static analyses to automatically discover information flows in source code. A variety of visualizations are provided to help developers determine if these flows conform to desired MLS policies. System and pro­ cedure parameters can be tagged with security policy levels (Top Secret, Secret, Unclassified). Chispa uses its flow analysis to propagate this infor­ mation to all program statements and to color each statement to indicate the security level of associated data. Chispa includes a software contract language that makes it easy for developers to specify formally the condi­ tions under which information from one data component or security domain is allowed to flow to another. Chispa uses advanced automated deduction techniques to check that procedure and system implementations correctly follow their information flow contracts. Quality assurance teams as well as evaluators for certification authorities can use Chispa's analysis and visual­ ization capabilities to improve the effectiveness of audits and code reviews and to pose automated "what if?" queries related to system assurance.

An early version of Chispa is being used to develop components of the high-speed cryptography engine project at Rockwell Collins.

20 A High-Assurance Methodology for the Development of Security So~ware FEATURE

&RQIHUHQFH RI = 8VHUV =80¶  $SU 0387964805 1997; Reading, UK. p. 72–86. Available +DUGLQ'6HGLWRU'HVLJQDQGYHUL¿FDWLRQ at: http://citeseerx.ist.psu.edu/viewdoc/ of microprocessor systems for high- summary?doi=10.1.1.50.4825 assurance applications. 1st ed. Springer; 15 [8] Whalen M, Cofer D, Miller S, Krogh Mar 2010. ISBN-10: 1441915382 References B, Storm W. Integration of formal analysis into a model-based software de-velopment Harrison J. Handbook of practical [1] Hardin D. Invited tutorial: Consider- process. In: Proceedings of the 12th logic and automated reasoning. 1st ed. DWLRQV LQ WKH GHVLJQ DQG YHUL¿FDWLRQ RI International Workshop on Industrial Cambridge University Press; 13 Apr 2009. microprocessors for safety-critical and Critical Systems (FMICS 2007), Jul 2007; ISBN-10: 0521899574 security-critical applications. In: Cimatti Berlin, Germany. p. 68–84. Available at: A, Jones R, editors. Proceedings of the Jackson D. Software abstractions: Logic, http://www.msse.umn.edu/publications/ language, and analysis. The MIT Press; 24 Eighth International Conference on Formal Integration-Formal-Analysis-Model- Mar 2006. ISBN-10: 0262101141 Methods in Computer-Aided Design Based-Software-D (FMCAD 2008); Nov 2008; Portland Kaufmann M, Manolios P, Moore JS. [9] The Mathworks, Inc. Simulink product (OR). p. 1–7. Available at: doi: 10.1109/ description. Available at: http://www. Computer-aided reasoning: An approach. FMCAD.2008.ECP.5 mathworks.com/products/simulink/ 1st ed. Springer; 31 Jul 2000. ISBN-10: [2] Greve D, Richards R, Wilding M. description1.html 0792377443 A summary of intrinsic partitioning [10] Reactive Systems, Inc. Reactis product Kozen DC. Automata and comput- YHUL¿FDWLRQ ,Q )LIWK ,QWHUQDWLRQDO description. Available at: http://www. ability. Springer; Aug 1997. ISBN-10: Workshop on the ACL2 Theorem Prover reactive-systems.com 0387949070 and Its Applications (ACL2-2004); Nov 2004; Austin (TX). Available at: http:// [11] Esterel Technologies, Inc. SCADE 0DQQD=:DOGLQJHU57KHGHGXFWLYH www.cs.utexas.edu/users/moore/acl2/ Suite product description. Available at: foundations of computer programming. workshop-2004/ http://www.esterel-technologies.com/ Addison-Wesley Professional; 10 Mar products/scade-suite >@6SLYH\-07KH=1RWDWLRQ$5HIHUHQFH 1993. ISBN-10: 0201548860 [12] Prover Technologies, Inc. Prover SL/ Manual. 2nd ed. Prentice Hall International 0DUHN9,QWURGXFWLRQWRPDWKHPDWLFVRI DE plug-in product description. Available Series in Computer Science; 1992. Available at: http://www.prover.com/products/prover_ VDWLV¿DELOLW\VWHG&KDSPDQDQG+DOO at: http://spivey.oriel.ox.ac.uk/~mike/zrm/ plugin CRC; 22 Sep 2009. ISBN-10: 1439801673 [4] Johnson R. Engineering protection Pierce BC. Basic category theory for com- software for the Tokeneer ID station (TIS). Further reading puter scientists. 1st ed. The MIT Press; The Next Wave. 2006;15(2):21–25, 28–31. $SW .5 2OGHURJ (5 9HUL¿FDWLRQ RI 7Aug 1991. ISBN-10: 0262660717 [5]. AdaCore. The Tokeneer Project. sequential and concurrent programs. 3rd Pierce BC. Types and programming lan- Available at: http://www.adacore.com/ ed. Springer; 28 Oct 2010. ISBN-10: guages. 1st ed. The MIT Press; 1 Feb 2002. home/products/sparkpro/tokeneer/ 1848827448 ISBN-10: 0262162098 [6] Barnes J, Chapman R, Johnson Barnes J. High integrity software: The R, Widmaier J, Cooper D, Everett W. SPARK approach to safety and security. Thompson S. Haskell, the craft of function- Engineering the Tokeneer enclave Addison-Wesley Professional; 25 Apr 2003. al programming. 2nd ed. Addison-Wesley ISBN-10: 0321136160 protection software. In: Proceedings Professional; 8 Apr 1999. ISBN-10: of the 1st International Symposium on Brooks FP Jr. The mythical man-month: 0201342758 Essays on software engineering. Addison- Secure Software Engineering (ISSSE); Wiedijk F, editor. The seventeen provers Wesley Professional; 12 Aug 1995, 0DU  $UOLQJWRQ 9$  $YDLODEOH DW of the world: Foreword by Dana S. Scott. http://www.altran-praxis.com/downloads anniversary edition. ISBN-10: 0201835959 1st ed. Springer; 16 Mar 2006. ISBN-10: /SPARK/technicalReferences/issse2006 Clarke EM Jr, Grumberg O, Peled DA. 3540307044 tokeneer.pdf Model checking. The MIT Press; 7 Jan >@ 6DDOWLQN 0 7KH =(9(6 V\VWHP ,Q 1999. ISBN-10: 0262032708 Bowen JP, Hinchey MG, Till D, editors. Gries D. The science of programming. Proceedings of the 10th International Springer; 1 Feb 1987. ISBN-10:

The Next Wave „ Vol 19 No 1 „ 2011 21 Correct by Construction: Advanced Software Engineering

ver SO years have passed since the introduction of and we still cannot get software right. Why does correct Osoftware elude us? First, software systems, maybe the most complex creation of mankind, exceed an individual's capacity to understand. Many different software engineering techniques have emerged over the years to address this complexity, for example structured and object-oriented programming, but failure-prone software persists. Second, subsequent changes to software obscure the author's original intent. In fact, no robust processes or techniques have emerged in practice to document design decisions so maintainers and developers can readily understand the implications of subsequent software changes. However, recent research in correct-by-construction techniques may help. By using formal specifications and automated synthesis we can make correctness claims about these systems and their evolution via an enhanced software engineering process that utilizes formally-described design knowledge. We will never obtain perfect assurance of correctness or security, but we can realize major improvements over current practice. Formal methods are defined in this paper as traditionally applied in the information assurance domain and in correct-by­ construction processes. A particular correct-by-construction (CxC) methodology, which uses the Specware tool, is then described. Specware supports the production of high-assurance1 code. A programmer using Specware does not directly write or modify code. Instead, the technology creates code systematically and automatically from the programmer's input (the formal specification) and guidance (formally applied design decisions). In conclusion, new CxC techniques that have impacted real-world problems are noted as well as a description of how they could help resolve information assurance problems.

1 "High-Assurance" in this paper means that the system meets its specification as expressed in the formal language. This includes functional correctness as well as other types of expressible properties.

22 Correct by Construction: Advanced So~ware Engineering FEATURE

1. CxC foundations $YDULHW\RIWRROVKDYHHPHUJHGWKDW methodsFDQPHDQVLPSO\DJRRGVRIWZDUH Many areas of computer science have prompted industry and academia HQJLQHHULQJ SURFHVV $OWKRXJK )0 research provide the foundation for WR H[SHULPHQW ZLWK &[& PHWKRGRORJLHV UHTXLUH JRRG SURFHVVHV JRRG SURFHVVHV WKLV ZRUN ZLWK &[& PHWKRGV DUWL¿FLDO NQRZOHGJH UHSUHVHQWDWLRQ WHFKQLTXHV E\ WKHPVHOYHV GR QRW VDWLVI\ WKH )0 LQWHOOLJHQFH SURJUDPPLQJ ODQJXDJHV DQG UHZULWLQJ ORJLF IURP WKH DUWL¿FLDO GH¿QLWLRQ QRU GR WKH\ JXDUDQWHH JRRG IRUPDOPHWKRGVDQGVRIWZDUHHQJLQHHULQJ LQWHOOLJHQFH FRPPXQLW\ FRPSLOHU UHVXOWV$)0EDVHGVRIWZDUHHQJLQHHULQJ 7KH86$LU)RUFH5RPH$LU'HYHORSPHQW HQKDQFHPHQWV DQG VHPDQWLFDOO\ ZHOO SURFHVVFDQDFKLHYHDTXDOLWDWLYHO\PRUH Center (now Rome Laboratory) provided GH¿QHGODQJXDJHVIURPWKHSURJUDPPLQJ UREXVWVROXWLRQ the impetus for CxC research in 1983 ODQJXDJHFRPPXQLW\UHDVRQLQJWHFKQLTXHV The FM process can be depicted E\ VSRQVRULQJ WKH .QRZOHGJH%DVHG IURP WKH IRUPDO PHWKRGV FRPPXQLW\ DV D WULDG 7KH )RUPDO 0HWKRGV 7ULDG and software process improvements and 6RIWZDUH $VVLVWDQW UHSRUW >@ ZKLFK )LJXUH   UHSUHVHQWV D SURFHVV IRU EHFDPH WKH EDVLV IRU WKHLU .QRZOHGJH VXSSRUWWRROVIURPWKHVRIWZDUHHQJLQHHULQJ PRYLQJIURPUHTXLUHPHQWVWRDVROXWLRQ FRPPXQLW\ 7KLV SDSHU GHVFULEHV RQH %DVHG 6RIWZDUH (QJLQHHULQJ .%6(  $WWKHWRSRIWKHWULDQJOHLVDUHDOZRUOG DSSURDFKWR&[&HQJLQHHULQJ SURJUDP .%6( LV EDVHG RQ FDSWXULQJ SUREOHP GH¿QHG E\ UHTXLUHPHQWV 7R DOO GHVLJQ GHFLVLRQV LQ D UHXVDEOH DQG 2. Advanced software DUULYH DW D ZRUNDEOH VROXWLRQ WR WKHVH FKHFNDEOHIRUP+RZHYHUWKHFRPSOH[LW\ engineering UHTXLUHPHQWV WKH SUREOHP DQG VROXWLRQ of the captured information approaches The Specware software PXVW EH GHVFULEHG LQ GHWDLO 7KHUHIRUH WKHFRPSOH[LW\RIWKHVRIWZDUHLWVHOI GHYHORSPHQW HQYLURQPHQW SURYLGHV D WKHSUREOHPGHVFULSWLRQDQGWKHVROXWLRQ 5HODWLYHO\ QHZ DUHDV RI FRPSXWHU description comprise the foundation of JRRG H[DPSOH RI KRZ &[& VRIWZDUH VFLHQFH DUH SURYLGLQJ WKH VWUXFWXUH DQG GHYHORSPHQWLQFRUSRUDWHVIRUPDOPHWKRGV WKHWULDQJOH SRZHU WR KDQGOH WKLV FRPSOH[LW\ DQG LQZD\VWKDWFDQEHQH¿WWKHLQIRUPDWLRQ $UURZVDURXQGWKHWULDQJOHUHSUHVHQW DFKLHYH .%6(¶V JRDO E\ XVLQJ &[& DVVXUDQFH ,$ FRPPXQLW\,QDYDULHW\RI WKHSURFHVVHVXVHGWRJHWIURPHDFKSRLQW PHWKRGRORJLHV ,QGXVWU\ XVHV WKH WHUP DSSOLFDWLRQV6SHFZDUHKDVDOUHDG\SURYHQ WRWKHQH[W7KHDUURZVSRLQWIRUZDUGWR &[& WR PHDQ PHWKRGV WKDW UDQJH IURP WR EH D SRZHUIXO WRRO IRU VSHFLI\LQJ WUDFH WKH PDLQ ÀRZ EXW IHHGEDFN DQG JRRGSURFHVVZLWKVRPHIRUPDOVXSSRUW GHVLJQLQJ DQG GHYHORSLQJ FRGH 6XFK UHLWHUDWLRQ DUH FHQWUDO WR WKH SURFHVV to automated construction of the software &[& WHFKQRORJLHV KDYH WKH SRWHQWLDO WR 5HWXUQLQJ WR D SUHYLRXV VWHS ZRXOG IURPVSHFL¿FDWLRQV>@6RPHH[DPSOHV H[SDQGWKHWUXVWZRUWKLQHVVRI,$GRPDLQ RFFXUIRUH[DPSOHZKHQLQFRQVLVWHQWRU RI VFLHQFH VXSSRUWLQJ &[& LQFOXGH DSSOLFDWLRQV:KDWGR PRGHOEDVHG VRIWZDUH HQJLQHHULQJ DQG ZHPHDQE\³)RUPDO FRUUHFWQHVVSUHVHUYLQJWUDQVIRUPDWLRQV Methods”? Real World Problem 6SHFZDUH WKH &[& V\VWHP XVHG IRUWKHH[DPSOHLQWKLVSDSHUOLHVWRZDUG 2.1 Definition of Requirements Solution WKHDXWRPDWHGHQGRIWKH&[&VSHFWUXP formal methods SURYLGLQJ DQ HPHUJLQJ FDSDELOLW\ WR )RUPDOPHWKRGV JHQHUDWH FRUUHFW LPSOHPHQWDWLRQV IURP (FM) are used to GHYHORS D VROXWLRQ WR Specify Interpret VRIWZDUH VSHFL¿FDWLRQV $OWKRXJK Formal VRPHRQH XVLQJ 6SHFZDUH GRHV QRW D SUREOHP WKURXJK QHHG WR NQRZ FDWHJRU\ WKHRU\ &7 ²D D SUHVFULSWLYH SUR Methods XQLI\LQJ FRQFHSW LQ PDWKHPDWLFV &7 FHVV %\ DSSO\LQJ Triad SURYLGHV WKH IRXQGDWLRQ IRU 6SHFZDUH¶V PDWKHPDWLFDO ULJRU Problem Solution DELOLW\ WR VWUXFWXUH WKH NQRZOHGJH EDVH D SUREOHP FDQ Description Description Analyze LQVXFKDZD\WRPDNHFRPSLOLQJVPDOO be studied with XQGHUVWDQGDEOH VRIWZDUH DUWLIDFWV LQWR SUHFLVLRQ,QLQGXVWU\ Figure 1: The Formal Methods Triad FRPSOH[RQHVSUDFWLFDO the term formal

The Next Wave „ Vol 19 No 1 „ 2011 23 Software lifecycle cost reductions due to a CxC development process In a recent study, Kestrel researchers examined the suite of documentation required for certifying Type 1 devices, and the possibilities to extend Specware’s correct-by-construction (CxC) development process to auto-generate certification documents. Our thesis is that LQFRPSOHWH UHTXLUHPHQWV DUH GLVFRYHUHG &HQWHU GHYHORSHG ZKDW LV FRPPRQO\ by using automated tools to generate both GXULQJ WKH IRUPDO SUREOHP VSHFL¿FDWLRQ referred to as the Rainbow Series of the software and significant portions of its SKDVH 'LVFRYHULQJ HUURUV HDUO\ LQ WKH VWDQGDUGV EHJLQQLQJ ZLWK WKH 7UXVWHG certification documentation, a CxC approach will dramatically lower lifecycle costs, including SURFHVVFDQ\LHOGVLJQL¿FDQWFRVWVDYLQJV &RPSXWHU 6\VWHP (YDOXDWLRQ &ULWHULD the cost of recertification. Furthermore, by LQWKHORQJUXQ 7&6(&  YROXPH EHWWHU NQRZQ DV WKH speeding up the recertification process, a CxC 7KH DUURZ SRLQWLQJ IURP 5HDO 2UDQJH %RRN >@ ,Q  WKH 1DWLRQDO approach facilitates the evolution process, 6HFXULW\ $JHQF\¶V 1DWLRQDO &RPSXWHU resulting in higher quality products over the :RUOG 3UREOHP WR 3UREOHP 'HVFULSWLRQ lifecycle. LVODEHOHG6SHFLI\EHFDXVHWKLVVWHSRIWKH 6HFXULW\ &HQWHU 1&6&  UHSODFHG WKH To quantify these claims, we first SURFHVVXVHVIRUPDOODQJXDJHWRGHVFULEH 2UDQJH%RRNZLWKWKH&RPPRQ&ULWHULD estimated the cost reductions that arise from WKH SUREOHP 7KH PRUH H[SUHVVLYH WKH >@ 'HVSLWH WKH FRPSXWHU VHFXULW\ a CxC process independently from certification ODQJXDJH XVHG WKH PRUH FRPSOHWH WKH FRPPXQLW\¶V HDUO\ H[FLWHPHQW DQG costs. The dominant factor seems to be the size reduction in formal specifications relative DQDO\VLVFDQEH7KH$QDO\]HVWHSPRYHV VXEVHTXHQW GLVDSSRLQWPHQWV UHJDUGLQJ to executable code. This size reduction varies WKH SURFHVV IURP D SUREOHP GHVFULSWLRQ )0 IRUPDO PHWKRGV PD\ VWLOO KDYH D considerably over projects, but a ballpark figure WR D VROXWLRQ GHVFULSWLRQ 0HWKRGV WKDW IXWXUHLQEXLOGLQJKLJKDVVXUDQFHV\VWHPV of 4-5x is consistent with the JavaCard project and related efforts. A 4-5x reduction in size of VXSSRUW WKLV DQDO\VLV ZLWK PDWKHPDWLFDO 7KH 'R' KDV LQLWLDWHG VLJQL¿FDQW the formal text usually correlates with a similar ULJRU HJ )0  DUH XVHG )LQDOO\ WR HIIRUWV WR LQFRUSRUDWH )0 LQ WKH GHVLJQ reduction in development and evolution costs. DUULYHDWDVROXWLRQZHPDSWKHUHVXOWRI DQG HYDOXDWLRQ RI LQIRUPDWLRQ VHFXULW\ Consequently we estimate that, independent WKH DQDO\VLV EDFN WR LWV PHDQLQJ LQ WKH V\VWHPV0RVWQRWDEO\)0ZRXOGDSSO\ of certification costs, a CxC process should reduce lifecycle costs by roughly 75-80 percent. SUREOHPGRPDLQ7KLVLVWKH,QWHUSUHWVWHS DW WKH 'HVLJQ 3KDVH RI WKH GHYHORSPHQW Second, we estimated the cost reduction due LQWKHSURFHVV DQG ZRXOG IRFXV RQ ZKHWKHU WKH GHVLJQ to extensions of the CxC process that allow $VD)0SURFHVVH[DPSOHFRQVLGHU KDV WKH GHVLUHG VHFXULW\ SURSHUWLHV auto-generation of certification documents as RU DW WKH (YDOXDWLRQ 3KDVH ZKHUH WKH a by-product of the code generation process. WKH UHDO ZRUOG SUREOHP RI GHWHUPLQLQJ For each of the thirteen documents required KRZ PXFK ZLQG D VXVSHQVLRQ EULGJH LPSOHPHQWLQJFRGHZRXOGEHDQDO\]HGIRU for certifying Type 1 devices, we estimated FDQ ZLWKVWDQG 7KH SUREOHP FDQ EH VHFXULW\YXOQHUDELOLWLHV7KHVHDSSURDFKHV that the average cost savings vary from a high GHVFULEHG E\ D VHW RI LQWHJUDO HTXDWLRQV KDYH EHHQ LPSOHPHQWHGSULPDULO\ LQ WKH of 75 percent for Formal Security Policy Model (FSPM) documents to a low of 20 percent for UHSUHVHQWLQJ D SURSHUW\ RI WKH EULGJH UHVHDUFKFRPPXQLW\DQGUDUHO\LQSURGXFW a Security Verification Plan and Procedures DIIHFWHGE\ZLQG6ROYLQJWKHVHHTXDWLRQV GHYHORSPHQWXQWLOUHFHQWO\ (SVP) documents. Assuming roughly equal SURGXFHV D VROXWLRQ GHVFULSWLRQ 7KH 2.2.1 Formal methods at design time weight to each of the 13 documents, we EULGJHHQJLQHHUZRXOGVHOHFWDSSURSULDWH estimated an average overall cost reduction of )RUPDO PHWKRGV DUH DSSOLHG GXULQJ LQWHJUDO FDOFXOXV HTXDWLRQV²WRROV IURP about 59 percent per certification application WKH GHVLJQ SKDVH E\ GHYHORSLQJ IRUPDO due to using CxC methods. KLVHQJLQHHULQJGRPDLQNQRZOHGJH²DQG VSHFL¿FDWLRQVRIWKHV\VWHPDQGWKHVHFXULW\ These two estimates can be combined in LQVHUW WKH EULGJH¶V VSHFL¿F UHTXLUHPHQWV SROLF\ W\SLFDOO\ DW D YHU\ KLJK OHYHO RI a variety of ways. For example, if we assume LQWR WKHP DQG GHULYH D VROXWLRQ ,I WKH that certification costs are roughly the same DEVWUDFWLRQ 7KH SRLQW RI WKH DEVWUDFW FDOFXODWLRQV IURP WKH IRUPDO DQDO\VLV as development costs, then CxC brings about VSHFL¿FDWLRQ LV WR GH¿QH WKH ³ZKDW´ WKH a 70 percent reduction in lifecycle costs GHWHUPLQHGWKDWZLQGVVXI¿FLHQWWRFDXVH V\VWHP VKRXOG GR DQG QRW WKH ³KRZ´ LW (evolution plus certification); that is, a CxC WKH EULGJH WR FROODSVH RFFXU IUHTXHQWO\ VKRXOGGRLWEHFDXVHWKH³KRZ´QRUPDOO\ process will produce a certified product for 30 WKHLQWHUSUHWDWLRQZRXOGSUREDEO\OHDGWR percent of the cost of a conventional process. ELDVHV WKH V\VWHP WRZDUG SDUWLFXODU FRQGHPQLQJWKHEULGJH2IFRXUVHQRRQH If we assume, as is the case in aerospace LPSOHPHQWDWLRQV WKXV SRWHQWLDOO\ applications, that the cost of certification is ZRXOGEXLOGDEULGJHEHIRUHDQDO\]LQJLWV SUHFOXGLQJWKHEHVWLPSOHPHQWDWLRQV about 7x development costs, then we obtain GHVLJQEXWVRIWZDUHLVURXWLQHO\EXLOWDQG an estimate of 63 percent cost reduction for 7R EHJLQ WKLV SURFHVV WKH V\VWHP WKHQDQDO\]HGDIWHUZDUG a CxC process. This leads us to conclude that GHYHORSHU FUHDWHV D V\VWHP VHFXULW\ a CxC process will produce a certified product 2.2 The use of formal methods for roughly 30-40 percent of the cost of a SROLF\ D VHW RI UHTXLUHPHQWV  DQG D conventional process. This estimate does not in the DoD V\VWHP VSHFL¿FDWLRQ 7KHVH UHTXLUHPHQWV account for the possibility that some forms of The US Department of Defense DUH H[SUHVVHG PRVW RIWHQ LQ QDWXUDO certification become unnecessary because of 'R' KDVDORQJKLVWRU\RIDSSO\LQJ)0 ODQJXDJH (DUO\ RQ GHYHORSHUV FUHDWHG the strong form of evidence provided by a CxC process. ,QWKHVWKH'R'&RPSXWHU6HFXULW\ IRUPDO VSHFL¿FDWLRQV PDQXDOO\ ZLWK OLWWOH

24 Correct by Construction: Advanced Software Engineering FEATURE

DXWRPDWHGVXSSRUW7KH\ZRXOGRIWHQUHXVH 5DUHO\ XQWLO UHFHQWO\ KDV DQ\RQH JAVA Card Runtime Environment FRPSRQHQWV RI H[LVWLQJ VSHFL¿FDWLRQV DWWHPSWHGWRLWHUDWHUH¿QLQJDVSHFL¿FDWLRQ DGDSWLQJ WKHP LQ PXFK WKH VDPH ZD\ DV E\ DGGLQJ GHWDLO GRZQ WR WKH FRGH OHYHO We used Specware to formally SURJUDPPHUV UHXVH FRGH /DWHU WKHRUHP DQG LQ PRVW FDVHV WKH\ GLG WKLV RQ VPDOO specify a real-world operating SURYLQJ IUDPHZRUNV HYROYHG WR KHOS VOLFHVRIWKHV\VWHP7KHFRVWRISURGXFLQJ system, the Java Card Runtime VWDQGDUGL]H DQG DXWRPDWH SURFHVVHV RI FRGHIURPWKLVSURFHVVZDVSURKLELWLYH Environment (JCRE). The JCRE consists of a JAVA virtual machine (VM) and system ZULWLQJV\VWHPVSHFL¿FDWLRQVDQGVHFXULW\ 7KLV SURFHVV RI IRUPDOO\ SURYHQ libraries (e.g., for I/O and cryptography), UHTXLUHPHQWV 7KH GLIIHUHQW WKHRUHP VSHFL¿FDWLRQVRQFHWRRFRVWO\DQGODERU along with card management capabilities SURYLQJ WRROV HPSOR\ XQLTXH YDULDQWV RI LQWHQVLYH KDV LPSURYHG (YHQ VR WKH according to the Global Platform IRUPDOORJLF LHDVSHFL¿FDWLRQODQJXDJH  EHQH¿WV RI IRUPDO VSHFL¿FDWLRQ ZLWKRXW Standard. The formal specification is HDFK KDYLQJ LWV RZQ VWUHQJWKV DQG FRPSOHWHSURRIVKDYHSURYLGHGVXI¿FLHQW about 30,000 lines long and over 6,000 ZHDNQHVVHV(DUO\WRROVDQGODQJXDJHVXVHG YDOXHWREHUHTXLUHGIRU&RPPRQ&ULWHULD consistency proofs of it have been mechanically verified so far. A desktop IRUVSHFLI\LQJV\VWHPUHTXLUHPHQWV VRPH ($/UDWHGV\VWHPV DUHVWLOOXVHG LQFOXGHG(+'0*\SV\,QD simulator (reference implementation) 2.2.2 Code-based analysis has been generated by refinement from -R/DUFK$&/= ]HG 147+0*9( the formal specification; the correctness 7KH1DWLRQDO6HFXULW\$JHQF\ 16$  3966'9=(9(6DQGWKH/DUFK3URYHU of the refinements is currently being 6HFXULW\ SROLF\ ZRXOG WKHQ EH GHVFULEHG KDV DOVR DSSOLHG )0 WKURXJK FRGHEDVHG mechanically verified. A C implementation XVLQJWKHVDPHIRUPDOODQJXDJH DQDO\VLVPDLQO\LQVXSSRUWRIHYDOXDWLRQV for a commercial chip has been manually *LYHQ WKH GHVFULSWLRQV IRU WKH RI LQIRUPDWLRQ DVVXUDQFH V\VWHPV &RGH derived from the formal specification; V\VWHPUHTXLUHPHQWVDQGVHFXULW\SROLF\ EDVHGDQDO\VLVGLIIHUVIURPWKHGHVLJQSKDVH a new version of this implementation is currently being generated via automated WKHGHYHORSHUZKHQDEOHWRXQGHUVWDQG )0SURFHVVLQWKDWLQVWHDGRIGHYHORSLQJ DIRUPDOGHVFULSWLRQRIZKDWWKHVRIWZDUH refinements, with mechanical proofs. We WKHP ZRXOG ³SURYH´ WKDW WKH V\VWHP anticipate that this will be the highest VKRXOG GR FRGHEDVHG DQDO\VLV DWWHPSWV VSHFL¿FDWLRQ HQIRUFHV WKH VHFXULW\ level of assurance yet achieved, and that to discover and prove properties about the SROLF\ 8VXDOO\ WKHVH SURRIV UHTXLUHG it will reduce the cost and increase the EXLOGLQJ DQ LQIUDVWUXFWXUH RI OHPPDV VRIWZDUHFRGHLWVHOI)RUH[DPSOHWKHXVHU confidence of a Type 1 certification. WKDWUDUHO\FRXOGEHUHXVHG7KHYHUL¿HG PLJKWLGHQWLI\SRLQWVZLWKLQWKHVRIWZDUH Specware has also been used to study KLJKOHYHOGHVLJQPD\RUPD\QRWUHÀHFW ZKHUH SURSHUWLHV RI LQWHUHVW PXVW KROG the extension of (standard) JCRE with WKH DFWXDO LPSOHPHQWDWLRQ DOWKRXJK DQG WKHQ DQQRWDWH WKH FRGH ZLWK VW\OL]HG MILS and MLS separation. The study has LPSOHPHQWDWLRQV EXLOW IURP YHUL¿HG FRPPHQWV IRUPDO ODQJXDJH  DERXW WKHVH been carried out on a formal specification of an idealized subset of the JCRE. GHVLJQV KDYH D PXFK JUHDWHU FKDQFH RI SURSHUWLHV 7KH GHYHORSHU WKHQ DSSOLHV Separation policies have been formally PHHWLQJWKHLUUHTXLUHPHQWV D WRRO WKDW XQGHUVWDQGV WKH VHPDQWLFV RI specified, along with run-time monitors WKH ODQJXDJH DQG JHQHUDWHV YHUL¿FDWLRQ An improvement to this method to enforce the policies. The monitors conditions based on the code and the IRU DFKLHYLQJ FRQ¿GHQFH WKDW WKH V\VWHP have been formally proved to guarantee VSHFL¿FDWLRQJXDUDQWHHGWKHVHFXULW\SROLF\ DQQRWDWLRQV7KLVSURFHVVRXWSXWVDVHWRI the policies. The monitors and the formal ZRXOGEHWRDGGGHWDLOWRWKHVSHFL¿FDWLRQV ORJLFDOVWDWHPHQWVWKDWWKHQFDQEHXVHGWR proofs are currently being extended from DQGUHLWHUDWHWKHSURFHVV7KLVRIFRXUVH YDOLGDWHWKHVHXVHUGHVLUHGSURSHUWLHV the idealized to the complete JCRE. UHTXLUHGVRPHNLQGRIGHPRQVWUDWLRQWKDW ,Q WKH DQDO\VLV SURFHVV ZH DWWHPSW See http://www.kestrel.edu/java the more concrete system and security WR¿QGDQGSURYHWKHWDUJHWHGSURSHUWLHV for more information. SROLF\ VSHFL¿FDWLRQV LQGHHG UHSUHVHQWHG RIWKHVRIWZDUH$6$7VROYHU D%RROHDQ WKHLUFRUUHVSRQGLQJDEVWUDFWVSHFL¿FDWLRQV ³6DWLV¿DELOLW\´ RU 6$7 VROYHU XVHV $WVRPHSRLQWXVXDOO\OHDYLQJDVLJQL¿FDQW VSHFLDOL]HG SURFHGXUHV WR DWWHPSW WR JDSEHWZHHQWKHPRVWGHWDLOHGVSHFL¿FDWLRQ VDWLVI\ WHVW FRQGLWLRQV  RU DQ $73 DQ DQG FRGH LWHUDWLRQ ZRXOG VWRS DQG WKH “automated theorem prover” derives the VSHFL¿FDWLRQ ZRXOG EHFRPH WKH EDVLV IRU WUXWKRIWKHVSHFL¿HGFRQGLWLRQVIURPPRUH WKHFRGH2IFRXUVHZHNQRZWKDWQRFRGH EDVLFIDFWV LVDSSOLHGWRWKHVHYHUL¿FDWLRQ ZRXOG HYHU EH ZULWWHQ EHIRUH FRPSOHWLQJ FRQGLWLRQV:HJHWDQ\RIWKUHHSRVVLEOH WKHVSHFL¿FDWLRQ RXWSXWVIURPWKLVDQDO\VLVWKHFRQGLWLRQV

The Next Wave „ Vol 19 No 1 „ 2011 25 FDQKROGWKHFRQGLWLRQVGRQRWKROGRULQ DOWHUQDWLYH ZRXOG EH WR VLPXOWDQHRXVO\ JUHDWO\HQKDQFHVWKHEHQH¿WVRI&[&IRU VRPHFDVHVWKHDQDO\VLVFDQQRWGHWHUPLQH FRQGXFW WKH DQDO\VLV ZKLOH FUHDWLQJ WKH UHXVH DQG UHFHUWL¿FDWLRQ²DQ DGYDQWDJH HLWKHU ZD\ LI WKH FRQGLWLRQV KROG RU QRW VSHFL¿FDWLRQVDUWLIDFWV %\ PDLQWDLQLQJ LQGXVWU\LVVWDUWLQJWRUHFRJQL]H>@ ZLWK WKH FRPSXWDWLRQDO UHVRXUFHV JLYHQ RU HQIRUFLQJ WKH SURSHUWLHV RI FRQFHUQ )LJXUH  LOOXVWUDWHV WKH &[& PRGHO 7KH¿UVWWZRFRQGLWLRQVWHOOXVVRPHWKLQJ WKURXJK WKH GHYHORSPHQW SURFHVV LWVHOI XVHG IRU RXU VRIWZDUH GHYHORSPHQW XVHIXODERXWWKHFRGH GHVLJQHUURUVFRXOGEHFDXJKWHDUO\ZKHQ SURMHFW$WWKHKLJKHVWOHYHORIDEVWUDFWLRQ 7KHVRXUFHDQGRUELQDU\FRGHOHYHO UHSDLULVOHVVFRVWO\/HWXVORRNDWWKH&[& WKH UHTXLUHPHQWV H[SUHVVHG LQ IRUPDO WKHGHWDLOUHTXLUHGWRSHUIRUPWKHDQDO\VLV DSSURDFKIRUDVRIWZDUHGHYHORSPHQW ODQJXDJH SOXV DQ\ H[LVWLQJ GRPDLQ FDQ EH FRVW SURKLELWLYH +RZHYHU DV 7KHGHYHORSHUW\SLFDOO\VWDUWVZLWK NQRZOHGJH VSHFL¿FDWLRQV DUH XVHG WR EHWWHU SURYHUV DQG VROYHUV EHFRPH VRPHQRWLRQRIZKDWWKHV\VWHPVKRXOGGR FUHDWH D IRUPDO VSHFL¿FDWLRQ RU PRGHO DYDLODEOH DQG ZLWK VKHHU FRPSXWLQJ ,GHDOO\ FUHDWLYH SURJUDPPLQJ HQHUJ\ LV 7KLV VWHS LV UHSUHVHQWHG LQ WKH GLDJUDP SRZHU LQFUHDVHV WKHVH PHWKRGV DUH GLUHFWHGDWFDSWXULQJV\VWHPUHTXLUHPHQWV E\WKH6SHFLI\DUURZ0HUHO\IRUPDOL]LQJ EHFRPLQJPRUHSUDFWLFDO:HDOUHDG\VHH LQDVSHFL¿FDWLRQ7KLVLVKDUGZRUN&[& WKH UHTXLUHPHQWV LQ WKLV ZD\ FDQ H[SRVH LQGXVWU\ XVLQJ FRGH DQDO\]HUV EDVHG RQ with automation moves the work from LQFRQVLVWHQFLHV DQG DPELJXLWLHV HDUO\ LQ )0VXFKDV&RGH+DZN-DYD3DWK¿QGHU ORZOHYHO SURJUDPPLQJ WR KLJKOHYHO WKH SURFHVV VDYLQJ PRQH\ DQG WLPH$W DQG(6&-DYD SUREOHPGHYHORSPHQW ORZHUOHYHOVRIDEVWUDFWLRQLQWKH&RPSXWH VWHS LQ WKH GLDJUDP WKH LPSOHPHQWDWLRQ 7KHUHLVKRZHYHUDWOHDVWRQH&[& $&[&DSSURDFKVROYHVWKHSUREOHP FDQEHGHULYHGWKURXJKDSRVLWDQGSURYH approach that combines the best of both LQFUHPHQWDOO\E\GHYHORSLQJUHTXLUHPHQWV DSSURDFKRUDWUDQVIRUPDWLYHDSSURDFK GHVLJQYHUL¿FDWLRQDQGFRGHDQDO\VLV DQGGHULYLQJVDWLVIDFWRU\LPSOHPHQWDWLRQV ,Q WKH SRVLWDQGSURYH DSSURDFK 2.3 A correct-by-construction ZLWKHYHULQFUHDVLQJOHYHOVRIUH¿QHPHQW SURJUDPPHUV FUHDWH WKH ORZOHYHO approach 7KLVLVGRQHE\XVLQJ)0WRDXWRPDWLFDOO\ VSHFL¿FDWLRQ DQG WKHQ SURSRVH D 7KH IRUPDO PHWKRGV GHVFULEHG FRPSXWH FRGH IURP VSHFL¿FDWLRQV &[& PDWKHPDWLFDO DUJXPHQW ZLWK YDU\LQJ SUHYLRXVO\WU\WRDVFHUWDLQVRPHWKLQJDIWHU LQGXVWULDOSUDFWLFHVYDU\E\WKHGHJUHHRI GHJUHHV RI IRUPDOLW\ WKDW ZLOO SURYH WKHIDFW,QWKHGHVLJQSKDVH)0DUHXVHG DXWRPDWLRQ RI WKH FRPSXWH SURFHVV $Q ZKHWKHU RU QRW WKH VROXWLRQ VSHFL¿FDWLRQ WR YHULI\ WKH VSHFL¿FDWLRQV DIWHU ZULWLQJ LGHDO &[& DSSURDFK ZRXOG FRPSOHWHO\ REH\V WKH SUREOHP VSHFL¿FDWLRQ$ FRGH WKHPDQGLQFRGHDQDO\VLVWKHDQDO\VLVLV DXWRPDWH WKLV SURFHVV $OWKRXJK IXOO JHQHUDWRU LV XVHG WR JHQHUDWH WKH FRGH DSSOLHG WR FRGH DOUHDG\ GHYHORSHG 7KH DXWRPDWLRQ LV QRW WKH RQO\ ZD\ WR JR LW IURPWKHORZOHYHOVROXWLRQVSHFL¿FDWLRQ :LWK D WUDQVIRUPDWLYH DSSURDFK Software System Development WKHXVHUWDNHVWKHKLJKOHYHOVSHFL¿FDWLRQ RI WKH V\VWHP DQG LWHUDWLYHO\ DSSOLHV WUDQVIRUPDWLRQV WKDW DSSO\ FRPSXWHU System & Security Software VFLHQFH DQG SUREOHP GRPDLQ NQRZOHGJH Requirements Implementation WR JHW DQ HI¿FLHQW DQG FRUUHFW ORZOHYHO H[HFXWDEOH VSHFL¿FDWLRQ 7KLV SURFHVV WHUPLQDWHV DIWHU WUDQVIRUPLQJ WKH HQWLUH Specify Generate SUREOHP VSHFL¿FDWLRQ LQWR DQ H[HFXWDEOH CxC VSHFL¿FDWLRQ 5HPHPEHU WKDW WKH LQLWLDO VSHFL¿FDWLRQ GH¿QHG ³ZKDW´ WKH V\VWHP Triad VKRXOGGR(DFKWUDQVIRUPDWLRQDGGVGHWDLO DQGPDNHVGHFLVLRQVWKDWEULQJLWFORVHUWRD SDUWLFXODU³KRZ´WKHV\VWHPVKRXOGGRLW,Q Problem Formal Solution Formal Specification Specification DGGLWLRQHDFKWUDQVIRUPDWLRQSUHVHUYHVWKH SURSHUWLHVDQGIXQFWLRQDOLW\RIWKHVRXUFH Compute VSHFL¿FDWLRQ LQ WKH WDUJHW VSHFL¿FDWLRQ WKHUHVXOWRIWKHWUDQVIRUPDWLRQ 2QFHWKH Figure 2: The CxC Triad “how” has been determined and expressed

26 Correct by Construction: Advanced Software Engineering FEATURE

DV D IRUPDO VSHFL¿FDWLRQ WKH QH[W VWHS :H FRQWUDFWHG IRU WKH GHYHORSPHQW RI D AutoSmart LV WR JHQHUDWH VRIWZDUH LQ VRPH WDUJHW UREXVW -&5( DQG UREXVW DSSOHWV UXQQLQJ SURJUDPPLQJ ODQJXDJH DV LQ WKH SRVLW RQDVSHFL¿FKDUGZDUHSODWIRUP The AutoSmart (automatic generator of smart card applets) tool is an example DQGSURYHDSSURDFK  7R FUHDWH WKH -&5( D IRUPDO of a domain-specific CxC generator. It 7KH LQLWLDO KLJKOHYHO IRUPDO VSHFL¿FDWLRQ ZDV GHYHORSHG DQG D features a specification language tailored to the smart card domain, with constructs GHVFULSWLRQFUHDWHGIURPWKHUHTXLUHPHQWV SRVLWDQGSURYH DSSURDFK ZDV XVHG WR to conveniently capture concepts LQWKH6SHFLI\VWHSPXVWVWLOOEHYDOLGDWHG UH¿QH WKH FRGH 7KH FRPSOH[LW\ RI WKH like personal identification numbers, DJDLQVW WKH RULJLQDO UHTXLUHPHQWV DQG LQLWLDOVSHFL¿FDWLRQHYROYHGDVVWDQGDUGV cryptography, ISO 7816 I/O exchanges, VSHFL¿FDWLRQVLQWKHVWDQGDUGZD\$QRWKHU ZHUH DGGHG ,QFUHPHQWDO FKDQJHV ZHUH and so on. AutoSmart performs several consistency checks on the applet ZD\ WR ORRN DW WKLV &[& VLPSO\ WULHV WR HDVLHU WR LPSOHPHQW XVLQJ 6SHFZDUH specifications, including a security analysis DXWRPDWH WKH )0 SURFHVV IRU ,1)26(& WKDQ LQ VWDQGDUG VRIWZDUH GHYHORSPHQW that flags potential leaks of confidential V\VWHPV E\ LWHUDWLQJ WKH SURFHVV DOO WKH SURFHVVHV (YHQ WKRXJK WKH 6SHFZDUH information like private and secret ZD\GRZQWRFRGH7KLVEHJVWKHTXHVWLRQ keys. AutoSmart compiles the applet WRROV XVHG DW WKH WLPH ZHUH SULPLWLYH specifications to Java Card code, which ³'RHVVXFKDWRROH[LVW"´ FRPSDUHGZLWKODWHUYHUVLRQVWKH\ZHUH can be compiled and loaded into a Java 2.4 Specware, a VXI¿FLHQWO\UREXVWWRSURGXFHDZRUNLQJ Card. Along with the code, AutoSmart also generates documentation for FIPS correct-by-construction tool -&5( FRPSOHWH ZLWK FU\SWRJUDSK\ DQG 140-2 certification as well as informal @:LWKWKLVWRROZH &UHDWLQJ DSSOHWV LQYROYHG WKH tables of commands and internal data). AutoSmart is currently being extended FDQEXLOGVSHFL¿FDWLRQVIRUUHTXLUHPHQWV GHYHORSPHQW RI D GRPDLQVSHFL¿F with the capability to generate a machine- FRPELQH VPDOO VSHFL¿FDWLRQV LQWR ODUJHU ODQJXDJH 6PDUW6ODQJ  DQG D checkable formal proof of the correctness RQHV LPSOHPHQW GHVLJQ GHFLVLRQV E\ FRUUHVSRQGLQJ FRPSLOHU $XWR6PDUW of the generated Java Card code with respect to the input specifications. This UH¿QLQJVSHFL¿FDWLRQVDQGJHQHUDWHFRGH SURGXFHG ZLWK 6SHFZDUH  8VLQJ “credible compiler” capability enables IURP H[HFXWDEOH VSHFL¿FDWLRQV DOO WKH 6PDUW6ODQJ D GHYHORSHU FDQ VSHFLI\ trust in the correctness of the code to be ZKLOH SURYLGLQJ IRU WKH SURRI WKDW WKH DQ DSSOHW PRUH HDVLO\ WKDQ E\ ZULWLQJ shifted from the AutoSmart tool to a much smaller and simpler proof checker, in the GHULYHG VSHFL¿FDWLRQV DQG FRGH HQIRUFH -DYD &DUG FRGH GLUHFWO\ 7KH FRPSLOHU spirit of proof-carrying code. WKHUHTXLUHPHQWV JHQHUDWHV ERWK WKH -DYD &DUG FRGH IRU See http://www.kestrel.edu/jcapp 6SHFZDUH >@ XVHV D WKH DSSOHW DQG D SURRI WKDW WKH FRGH lets for more information. YHUVLRQ RI &KXUFK¶V KLJKHURUGHU ORJLF IRU LWV VSHFL¿FDWLRQ ODQJXDJH 7KLV ODQJXDJH ERUURZV IHDWXUHV IURP IXQFWLRQDO Software System Development SURJUDPPLQJ DQG KLJKO\ H[SUHVVLYH DXWRPDWHGWKHRUHPSURYHU $73 ODQJXDJHV System & Security Software DQGLVWKXVXVHDEOHE\PDQ\LQWKHFRPSXWHU Requirements Implementation VFLHQFH DQG WKHRUHP SURYLQJ GLVFLSOLQHV Researchers have used Specware for D ZLGH UDQJH RI VSHFL¿FDWLRQV IURP D Specify and IXOOEORZQ RSHUDWLQJ V\VWHP -DYD &DUG Translate and Compose Generate 5XQWLPH (QYLURQPHQW RU -&5(  DQG D Formal PDWKHPDWLFDOO\ DVVXUHG VHSDUDWLRQ NHUQHO Methods WKH 0DWKHPDWLFDOO\ $QDO\]HG 6HSDUDWLRQ Triad .HUQHO >0$6.@ SDUW RI WKH $GYDQFHG ,1)26(& 0RGXOH FKLS  WR ORZOHYHO Problem Formal Solution Formal DOJRULWKPVDQGGDWDVWUXFWXUHV Specification Specification In Metaslang Refine and In Metaslang 2.4.1 An NSA security token Transform $W 16$ ZH XVHG 6SHFZDUH WR VXFFHVVIXOO\EXLOGDUREXVWVHFXULW\WRNHQ Figure 3: The Specware Triad

The Next Wave „ Vol 19 No 1 „ 2011 27 Synthesis of propositional satisfiability solvers

Dramatic improvements to propositional satisfiability [SAT) solvers were made during the last two decades [e.g., 1,2]. We used Specware to demonstrate the implements the specification. In addition, the compiler enforces properties stated in the automated generation of fast SAT solvers. applet specification and produces reports to meet certification requirements. The main result was that we were able to Kestrel Institute succeeded in producing a working JCRE on the chip running recapitulate many of the key design features of a modern SAT solver using mechanized robust applets. Smart card developers are now considering the formal JCRE specification representations of abstract and reusable and the Specware toolset. However, given that Specware is not a commercial tool, design knowledge. Starting with a formal there is some resistance to its use. Commercial support is critical for wider adoption of specification of the SAT problem (find a satisfying assignment for a given set of this technology. propositional clauses, if any), the overall algorithmic structure of a Davis-Putnam­ 2.4.2 Using Specware Logemann-Loveland (DPLL) SAT solver Building specifications with Specware is no harder than programming. And was calculated from the global search and because high-level specifications should only stipulate "what" is to be done, not "how" to constraint propagation algorithm paradigms [3,4]. Performance of the correct, but high­ do it, we can understand these top-level specifications much more easily than code. Thus level algorithm was improved by applying we can make our changes in the high-level specifications and avoid the more complex problem-independent transformations for low-level specifications by redoing the refinements. Most likely, the original design expression simplification, finite differencing [5], and data type refinement. Applying these choices will apply with minimal changes and the low-level specifications regenerated. algorithm design tactics and transformations To illustrate this process, we take an example from the Specware tutorial [4]. in different ways resulted in a family tree of SAT algorithm variants, including some novel 2.4.2.1 Specification non-DPLL variants. In the Specware tutorial, the problem for which we want to specify and generate This project, together with previous a solution is determining the first match of a word within a message, where a word is work on scheduling applications [6], provided evidence that it is feasible to generate list of symbols and a message consists of a list of symbols and wilds (a wild matches customized high-performance solvers for all characters). For example, the word "ABCD" would match at the first position of particular problems. Key features of state­ the message "AB*D***" and "BAD" would match at second position. of-the-art SAT solvers, such as conflict­ resolution and learning, can be applied Here is a typical specification: mechanically to other problems. References WordMatching = spec [1] Moskewicz MW, Madigan CF, Zhao Y, Zhang import Words L, Malik S. Chaff: Engineering an efficient sat solver. In: Proceedings of the 38th Conference import Messages on Design Automation [DAC 'D1]; Jun 2DD1 ; Las import SymbolMatching Vegas [NV] p. 53D-535. Available at http:// www.princeton.edu/-chaff/publication/DAG 2DD1v56.pdf op word_matches_at?(wrd: Word, msg: Message, pos: Nat): Boolean= [2] Marek V. Introduction to mathematics of satisfiability. 1st ed. Chapman and Hall/CRC; pos + length wrd <= length msg && 22 Sep 2DD9. ISBN-1 D 14398D1673 (fa(i:Nat) i symb_matches?(wrd@i, msg@(pos+i))) [3] Smith DR. Structure and design of global endspec search algorithms. Kestrel Institute; Nov 1987. Technical report number: KES.U.87.12. [4] Westfold S, Smith D. Synthesis of efficient In this specification, two conditions are necessary for a word match: (1) there is constraint satisfaction programs. Knowledge Engineering Review. 2DD1; 16(1 ):69-84. enough room left to contain the word in the message, and (2) all the symbols in the word match their corresponding positions in the message. Notice that this is not saying [5] Paige R, Koenig S. Finite differencing of computable expressions. ACM Transactions how to check these conditions; just what it means to match a word with a segment of on Programming Languages and Systems. a message. Also notice that the specification for what symbol matching means (two Jul 1982;4(3) 4D2-454 Available at doi identical characters or one is a wild) is not in this specification at all, but in an included 1 D.1145/357172.357177 specification named "SymbolMatching." So the game in specification is to build up a [6] Smith DR, Parra EA, Westfold SJ. Synthesis of planning and scheduling software. In: collection of specifications and compose them to say what is desired. A. Tate, editor. Proceedings of Advanced Planning Technology [ARPI 'D6]; May 2DD6; 2.4.2.2 Design Edinburgh, Scotland. p. 226-234. Available at: Design involves the composition of a series of refinements to get an executable http://www.aaai.org/Library/ARPl/arpi96 contents.php specification. Each design decision is formally captured and is available for reuse for

28 Correct by Construction: Advanced So~ware Engineering FEATURE

both exploring possible implementations of the current specification and for reuse in maintenance and enhancements to be other developments of similar systems. Consider how hard (and costly) it would be to done at the specification level rather than explore the design space using standard practice. With CxC, design space exploration in code, thus precluding a number of becomes doable. errors produced by code changes that have The key steps in design are to specify/generate the "how" and define/generate unforeseen side effects. The user's goal in mappings between the "what" and "how" specifications. The mappings constitute the Design phase is to complete his "how" property-preserving refinement. If we get the "how" right with respect to the "what," specification in terms of one of these we will be able to prove all obligations that Specware generates from the mapping collections. Once done, a program can be constrnct. Essentially with the properties proved, we ensure that the definitions in synthesized from the specification. source specification are theorems in the target specification. In other words our "how" 2.4.2.4 Proofprocessing does correctly "what" we want. All specification, specification Design is done iteratively until arriving at a specification for "how" to compute the composition, and specification refinement "what" from the high-level specification. (Specware has as a main goal the capabilities constructs may require proofs to establish to derive/generate low-level specifications, which is the intended mode of operation. and maintain properties through the But this capability is still in its infancy.) Here is a specification of "how" to see if a entire process. Proof obligations are word matches a segment of a message: generated when the user requests them and packaged for distribution to a proof op word_matches_at?(wrd: Word, msg: Message, pos: Nat): Boolean= tool. The main proof tool available in if pos + length wrd > length msg the Specware environment is Isabelle/ HOL [12], which has powerful automated then false proof methods and integrated expert user else word_matches_aux?(wrd, removePrefix(msg, pos)) guidance. Isabelle has a large user base, including industrial use. The first test checks if there is room for the word at this position. If there is Since the Isabelle theorem prover room, an auxiliary function is called to check from the current position if the word is sound, proofs completed with the tool matches symbol for symbol with the rest of the message. The "what" specification, the provide assurance that the constructed one above, needs to be related to this "how" specification. This is done by mapping code is correct with respect to the between the two specifications. In this example, a transformation is used to automate high-level specification. The Isabelle/ the construction of the refinement relating the word matching specifications: HOL language is close to the Specware language, but with a few quirks. Isabelle is WordMatching_Ref= difficult to use to prove some obligations, as are all other proof tools now in use. morphism MatchingSpecs#WordMatching -> But, even though it requires effort to use, MatchingRefinements#WordMatching {} the CxC process with Specware allows for the complete proof of the system This mapping, called a morphism, maps the symbols in WordMatching to refinement to code, yielding extremely corresponding elements in the target specification. In this example, all the names in the high-assurance software. high-level (source) specification (MatchingSpecs#WordMatching) are also in the low­ 2.4.3 Correct-by-construction level specification, so the task is done (otherwise the name to name mapping would be successes explicit within the {} ). The CxC approach has demonstrated 2.4.2.3 Code synthesis practical successes. Praxis used CxC to Specware can synthesize code in several different target programming languages. develop almost error-free code from a Currently, there are several collections of low-level executable specifications formal specification of an enclave access mapped directly to a program language. The most robust collection supports the Lisp system called Tokeneer [13,14]. With programming language-a natural fit to higher-order logic specifications. Much less­ Tokeneer, a user presents a token and mature collections exist for C and Java. The code synthesis is automatic, allowing biometric input, and then is either allowed

The Next Wave • Vol 19 No 1 • 2011 29 or denied access based on a database of PDFKLQH GLDJUDPV 6XSSRVH \RX KDG PHWULFVWRNHQLGVDQGXVHULQIRUPDWLRQ OLEUDULHVRIUHXVDEOHVSHFL¿FDWLRQVIRU\RXU 7KHSXEOLVKHGUHVXOWVFDXVHGDVWLULQWKH SUREOHPGRPDLQIRUVWDQGDUGDOJRULWKPV IRUPDO PHWKRGV UHVHDUFK FRPPXQLW\ DQG SURFHGXUHV DQG IRU SODWIRUP SURYLGLQJ D PXFK QHHGHG UHDO ZRUOG DZDUH LPSOHPHQWDWLRQV &RQVLGHU WKDW H[DPSOH .HVWUHO GHYHORSHG DQ DOWHUQDWH WKH PHUH SURFHVV RI H[SORULQJ D GHVLJQ VSHFL¿FDWLRQ RI WKH 7RNHQHHU V\VWHP VSDFH DQG FKRRVLQJ DQ LPSOHPHQWDWLRQ >@XVLQJWUDFHVRIHYHQWVPRUHDEVWUDFW ZRXOGUHVXOWLQDPDWKHPDWLFDOO\SUHFLVH WKDQ WKH VWDWH PDFKLQH IRUPDOLVP XVHG LPSOHPHQWDWLRQ RSWLPL]HG IRU \RXU LQ 3UD[LV¶V VSHFL¿FDWLRQV WKXV SURYLQJ SODWIRUP ,Q DGGLWLRQ DXWRPDWH WKH DGGLWLRQDOSURSHUWLHV UH¿QHPHQW SURFHVV ZKLOH VWLOO DOORZLQJ $QRWKHU 6SHFZDUH VXFFHVV WKLV XVHUFRQWUROWRWDNHDGYDQWDJHRIKXPDQ WLPHIURPLQGXVWU\LVWKH0DWKHPDWLFDOO\ H[SHUWLVH :H FODLP WKDW VXFK D V\VWHP $QDO\]HG 6HSDUDWLRQ .HUQHO 0$6.  ZRXOG UHYROXWLRQL]H RXU VRIWZDUH >@ 7KLV VSHFL¿FDWLRQ UHVXOWHG LQ GHYHORSPHQWSUDFWLFHV PRUH HDVLO\ HYDOXDWHG NHUQHO FRGH IRU WKH $GYDQFHG ,1)26(& 0RGXOH FKLS ‡8VLQJ &[& ZRXOG LPSURYH WLPH WR GHYHORSHG E\ *( WKHQ 0RWRUROD $OVR PDUNHW9DULDWLRQVRIH[LVWLQJV\VWHPVDUH .HVWUHO GHPRQVWUDWHG WKH EHQH¿WV RI HDVLO\ REWDLQHG WKURXJK PLQRU FKDQJHV IRUPDO VSHFL¿FDWLRQ UHXVH IRU ,$53$ WR VSHFL¿FDWLRQV DQG WKH UHSOD\ RI WKH 7KH\JHQHUDWHGDQ³LGHDOL]HG´-&5(IRU UH¿QHPHQWWRFRGHSURFHVV VLQJOH WKUHDGHG 0XOWLSOH ,QGHSHQGHQW ‡8VLQJ &[& ZRXOG LPSURYH WLPH DQG /HYHOVRI6HFXULW\ 0,/6 DQG0XOWLSOH FRVW WR FHUWLI\$OO GHVLJQ GHFLVLRQV DUH /HYHOVRI6HFXULW\ 0/6 DQGIRUPXOWL FDSWXUHGKHQFHVRPHFULWHULDUHTXLUHPHQWV WKUHDGHG 0,/6 DQG 0/6 VHSDUDWLRQ IRU FDQ EH JHQHUDWHG DXWRPDWLFDOO\ UDWKHU VPDUWFDUGV1RWRQO\ZHUHWKH\DEOHWR WKDQE\PDQXDOO\VHDUFKLQJWKHFRGH UHXVHDORWRIWKHRULJLQDOVSHFL¿FDWLRQV EXW WKH\ JHQHUDWHG UXQWLPH PRQLWRUV ‡8VLQJ &[& ZRXOG LPSURYH WLPH IURPWKHVHVSHFL¿FDWLRQVDQGSURYHGWKDW and cost to maintain and recertify: WKH\HQIRUFHWKHVHSDUDWLRQSURSHUWLHV $XWRPDWLRQ RI WHVWLQJ DQG FHUWL¿FDWLRQ 6$7 VROYHUV KDYH DOVR EHHQ HYLGHQFH JHQHUDWLRQ PDGH SRVVLEOH LQ VXFFHVVIXOO\ JHQHUDWHG XVLQJ &[& WKLV HQYLURQPHQW ZRXOG VROYH WKH DJH WRROV )RUPDO GHVFULSWLRQV RI WKH 6$7 ROG SUREOHP RI PDLQWDLQLQJ FRQVLVWHQF\ DOJRULWKPVDQG6$7GDWDVWUXFWXUHVH[LVW EHWZHHQ VSHFL¿FDWLRQ DQG FRGH EHFDXVH LQ6SHFZDUHVRWKDWYDULRXVYHUVLRQVRI FKDQJHVDUHPDGHDWWKHVSHFL¿FDWLRQOHYHO VROYHUV FDQ EH JHQHUDWHG DXWRPDWLFDOO\ DQGWKHFRGHLVUHJHQHUDWHGNHHSLQJWKH XVLQJ UHXVDEOH UH¿QHPHQW VFULSWV :LWK WZRLQV\QF PRUHUHVHDUFKHYHQPRUHVXFFHVVHVFRXOG ‡8VLQJ&[&ZRXOGLQFUHDVHWKHGHJUHH EHDGGHGWRWKHOLVW RIDVVXUDQFHWRDTXDOLWDWLYHO\KLJKHUOHYHO 3. An extreme CxC vision 6SHFZDUH LV D VWHS LQ WKH ULJKW :KHUH PLJKW &[& WDNH VRIWZDUH direction and provides evidence that GHYHORSPHQW"6XSSRVH\RXKDGDV\VWHP VXFKWRROVDUHSRVVLEOH:HQHHGWRSXVK GHYHORSPHQW HQYLURQPHQW ZKHUH \RX &[& SURFHVVHV WKDW KDYH PDWKHPDWLFDO FRXOG WDNH \RXU V\VWHP UHTXLUHPHQWV SUHFLVLRQLIZHHYHUKRSHWRJHWDKDQGOH DQG SURGXFH IRUPDO VSHFL¿FDWLRQV IURP RQ WKH FRPSOH[ VRIWZDUH DQG KDUGZDUH VWDQGDUG FRQVWUXFWV OLNH 80/ DQG VWDWH V\VWHPVRIWRGD\

30 Correct by Construction: Advanced Software Engineering FEATURE

References and Further Reading:

[1] Trusted computer system evaluation criteria. Department of Defense Standard 5200.28-STD; Dec 1985 Available at: http://csrc.nist.gov/publications/history/dod85.pdf

[2] Common criteria for technology security evaluation, version 3.1.

Part 1: Introduction and general model, Revision 1, Sep 2006

Part 2: Security functional components, Revision 2, Sep 2007

Part 3: Security assurance components, Revision 2, Sep 2007

Available at: http://www.niap-ccevs.org/cc-scheme/cc_docs/

[3] Specware 4.2 User Manual. Kestrel Institute; 2009. Available at: http://www.specware.org/doc.html

[4] Specware 4.2 Tutorial. Kestrel Institute; 2009. Available at: http://www.specware.org/doc.html

[5] Specware 4.2 Language Manual. Kestrel Institute; 2009. Available at: http://www.specware.org/doc.html

[6] Specware to Isabelle Interface Manual. Kestrel Institute; 2009. Available at: http://www.specware.org/doc.html

[7] Specware 4.2 Quick Reference. Kestrel Institute; 2009. Available at: http://www.specware.org/doc.html

[8] Kestrel Institute (home page). Palo Alto (CA). Available at: http://www.kestrel.edu

[9] Isabelle generic proof assistant (home page). Cambridge University, UK; 20 Sep 2010 Available at: http://www.cl.cam.ac.uk/research/hvg/Isabelle/index.html

[10] Martin W, White P, Taylor FS, Goldberg A. Formal construction of the Mathematically Analyzed Separation Kernel. In: Proceedings of the 15th IEEE International Conference on Automated Software Engineering (ASE’00); Sep 2000; Grenoble, France. p. 133–141. Available at: doi: 10.1109/ASE.2000.873658

[11] Anton J, Coglio A, McDonald J. Tokeneer. Kestrel Institute technical report; released at the 2006 High Confidence Software and Systems Conference.

[12] Green C, Luckham D, Balzer R, Cheatham T, Rich C. Report on a knowledge-based software assistant. Kestrel Institute; 15 Jun 1983. Technical report number: KES.U.83.2. Available at: http://www.kestrel.edu/home/publications/

[13] AdaCore. The Tokeneer Project. Available at: http://www.adacore.com/home/products/sparkpro/ tokeneer/

[14] Barnes J, Chapman R, Johnson R, Widmaier J, Cooper D, Everett W. Engineering the Tokeneer enclave protection software. In: Proceedings of the 1st International Symposium on Secure Software Engineering (ISSSE); Mar 2006; Arlington (VA). Available at: http://www.altran-praxis.com/downloads/SPARK/ technicalReferences/issse2006tokeneer.pdf

[15] Escher Technologies Ltd. (home page). Aldershot, UK. Available at: http://www.eschertech.com/index.php

The Next Wave „ Vol 19 No 1 „ 2011 31 Verified Software • the World

oftware is a critical component of the technological infrastructure. Many physical and electronic devices are controlled by software, which offers unparalleled sophistication and Sflexibility over coding in hardware. However, software is also a source of vulnerability. Unreliable software can be a significant cost in the development of software-based systems. Software bugs can be exploited to breach security and propagate malware. Software unreliability has been estimated to cost nearly one percent of the GDP to the United States economy. The technical challenges of developing and maintaining software are only growing in complexity with the advent of cyber-physical systems, service-oriented architectures, and multicore processors.

Software development can be made highly rigorous. The theoretical understanding of software and hardware models has existed for decades, but recent dramatic advances in the technologies of software specification, design, and analysis make it feasible to carefully and productively examine large code bases for errors. Interactions between the software and the physical and biological world, as well as with human operators can be analyzed in this manner. The technologies for software analysis can also be used to find security vulnerabilities and to identify strategies for safe parallelization. The Verified Software Initiative (VSI) is an ambitious fifteen-year, cooperative, international project directed at the scientific challenges of large-scale software verification. VSI is aimed at bringing formal scientific methods for software design into wider use so that software is viewed as the most trusted component in a system. The research agenda is directed at developing a comprehensive theory of program correctness that is supported hy a coherent suite of novel and powerful tools for designing, debugging, composing, and verifying software. The theory and tools must be validated on a wide range of examples and used to train a new generation of software engineers in the construction of trustworthy software. The need for verification technology is most acute in systems that are required to be reliable, resilient, and secure in an uncertain and hostile environment. Such systems include those from avionics, automotive control, process control, power distribution, health care, and electronic voting. These systems exhibit complex interaction between the software components and the physical world. The slightest flaw in the software can expose security vulnerabilities or lead to catastrophic system failure. Verification approaches the construction of software through the use of rigorous formal models. These models have a mathematical meaning that is captured using formal logic. Such models can be used to capture requirements, emulate the operating environment, formulate specifications, craft designs, decompose system functionality into modules, interpret and annotate programs, generate test cases, and verify component and system properties. The use of mathematical models also facilitates the use of highly automated tools. These tools can be used to identify the presence of flaws through the systematic generation of test cases, proof obligations, interface assumptions, and security vulnerabilities. They can also establish the absence of certain kinds of errors through analysis, exploration, and proof. Finally, such tools can be used as design aids to decompose problems, derive new solutions, and compose existing solutions.

32 Verified So~ware in the World FEATURE

9HUL¿FDWLRQ WHFKQRORJ\ KDV EHHQ LPSURYLQJ repair system state through constraint solving. rapidly in both scale and functionality. A range of Seamless integration between different tools is UREXVWDQGPDWXUHWHFKQLTXHVIRUVWDWLFDQGG\QDPLF QHHGHGWRJHQHUDWHUXQWLPHFKHFNVDQGPRQLWRUV analysis, state space exploration, constraint solving, test cases, counterexamples, conjectures, scenarios, automated and interactive proof generation, and test abstractions, and proofs. A formal integrated case generation are now available and in use. These GHYHORSPHQW HQYLURQPHQW IRU YHUL¿HG VRIWZDUH WHFKQLTXHVFDQEHDSSOLHGWRPRGHOVDQGSURJUDPV can be used to construct an assurance case for 9HUL¿FDWLRQWHFKQRORJLHVQHHGWREHV\VWHPDWLFDOO\ FHUWL¿FDWLRQWKURXJKDV\VWHPDWLFDUJXPHQWIRUWKH woven into the software development process. VDIHW\DQGVHFXULW\RIWKHV\VWHP9HUL¿FDWLRQDOORZV 7KH VXFFHVVIXO PDLQVWUHDPLQJ RI YHUL¿FDWLRQ the assurance argument to be decomposed along the WHFKQRORJ\ UHTXLUHV D VHDPOHVV LQWHJUDWLRQ RI WKH lines of components and service layers, each with its LQGLYLGXDO WHFKQLTXHV VXSSRUWHG E\ DQ DPELWLRXV own reusable assurance case. Software is expected agenda of experimental work. Tool construction to operate in a safe, secure, and predictable manner and experiments must be supported by novel in a world of physical uncertainty and virtual theoretical insights leading to accurate and tractable YXOQHUDELOLW\ 3RZHUIXO YHUL¿FDWLRQ WHFKQRORJ\ PRGHOVDVZHOODVVFDODEOHDQGHI¿FLHQWDOJRULWKPV will be needed to economically develop, validate, 9HUL¿FDWLRQWHFKQRORJ\KDVDUDSLGO\JURZLQJUDQJH and maintain software that is not only reliable, but RI DSSOLFDWLRQV 7HFKQLTXHV OLNH PRGHO FKHFNLQJ manifestly trustworthy. and constraint solving are being used to model physical and biological systems and to generate plans, schedules, and optimizations. They are also XVHGWR¿QJHUSULQWVHFXULW\WKUHDWVVXFKDVZRUPV and viruses and to check hardware and software HTXLYDOHQFH WR JXDUG DJDLQVW WKH LQVHUWLRQ RI malicious code. $ FRPSUHKHQVLYH IUDPHZRUN IRU YHUL¿HG software development can address a number of challenges in software engineering. At the UHTXLUHPHQWV OHYHO LW SURYLGHV D FRQYHQLHQW modeling framework for describing discrete and continuous behavior, time and resource constraints, fault models, and security policies. These formal models can be analyzed for anomalies and putative properties, and also used for generating test cases. ,Q WKH GHVLJQ SKDVH YHUL¿FDWLRQ WHFKQRORJ\ FDQ be used to verify algorithms and architectures; decompose the system into modules; establish the DEVHQFHRIXQLQWHQGHGLQIRUPDWLRQÀRZVEHWZHHQ software components; support semantic service discovery and composition; and facilitate resilient system operation in the face of device, platform, or operator failure. During the implementation phase, various integrated tools for synthesis and analysis can be used to generate and optimize FRGH HVWDEOLVK WKH DEVHQFH RI UXQWLPH HUURUV UDFH FRQGLWLRQV DQG LQIRUPDWLRQ ÀRZV LGHQWLI\ interface properties; compose software modules; schedule tasks on multicore processors; and even

The Next Wave Q Vol 19 No 1 Q 2011 33 Software for Dependable Systems: Sufficient Evidence?

34 Software for Dependable Systems: Sufficient Evidence? FEATURE

Note: The following article is the introduction to the National Academy of Science (NAS) report, Software for 'HSHQGDEOH6\VWHPV6XI¿FLHQW(YLGHQFH")XOOFRSLHVRIWKHUHSRUW IUHH3')GRZQORDGDQGERRNSXUFKDVH  DUHDYDLODEOHWKURXJKWKH1DWLRQDO$FDGHP\3UHVVDWKWWSZZZQDSHGXFDWDORJSKS"UHFRUGBLG 

ow can software and the systems that rely on it be made dependable in a cost-effective manner, and how can one obtain Hassurance that dependability has been achieved? Rather than focusing narrowly on the question of software or system certification per se, this report adopts a broader perspective.

A system is dependable when it can assumed to be the result of errors of usage; be depended on to produce the UDWKHUWKHUROHRIGHVLJQÀDZVVKRXOGEH consequences for which it was designed, considered as well as the role of the human and no adverse effects, in its intended operator. As a consequence, a systems HQYLURQPHQW 7KLV PHDQV ¿UVW DQG engineering approach — which views foremost, that the term dependability has the software as one engineered artifact no useful meaning for a given system in a larger system of many components, until these consequences and the intended some engineered and some given, and the environment are made explicit by a pursuit of dependability as a balancing of clear prioritization of the requirements FRVWV DQG EHQH¿WV DQG D SULRULWL]DWLRQ RI of the system and an articulation of risks — is vital. environmental assumptions. The effects of software are felt in the physical, human, 8QIRUWXQDWHO\LWLVGLI¿FXOWWRDVVHVV and organizational environment in which WKH GHSHQGDELOLW\ RI VRIWZDUH 7KH ¿HOG it operates, so dependability should be of software engineering suffers from understood in that context and cannot be a pervasive lack of evidence about the reduced easily to local properties, such as incidence and severity of software failures; resilience to crashing or conformance to about the dependability of existing software a protocol. Humans who interact with the V\VWHPVDERXWWKHHI¿FDF\RIH[LVWLQJDQG software should be viewed not as external proposed development methods; about the and beyond the boundary of the software EHQH¿WVRIFHUWL¿FDWLRQVFKHPHVDQGVRRQ engineer’s concerns but as an integral part There are many anecdotal reports, which— of the system. Failures involving human although often useful for indicating areas operators should not automatically be of concern or highlighting promising

The Next Wave n Vol 19 No 1 n 2011 35 avenues of research—do little to establish a sound and complete basis for making policy decisions regarding dependability. Moreover, there is sometimes an implicit assumption that adhering to particular process strictures guarantees certain levels of dependability. The committee [NAS &RPPLWWHH RQ &HUWL¿DEO\ 'HSHQGDEOH Software Systems] regards claims of extraordinary dependability that are sometimes made on this basis for the most dependability casH ZLOO LQÀXHQFH PDQ\ Assessment critical of systems as unsubstantiated, aspects of the development, including the Society is increasingly dependent on DQGSHUKDSVLUUHVSRQVLEOH7KLVGLI¿FXOW\ choice of programming language and the software. Software failures can cause or regarding the lack of evidence for system software architecture, and simplicity will contribute to serious accidents that result dependability leads to two conclusions, be key. For high levels of dependability, LQGHDWKLQMXU\VLJQL¿FDQWHQYLURQPHQWDO UHÀHFWHGLQWKHFRPPLWWHH¶V¿QGLQJVDQG the evidence provided by testing alone recommendations below: (1) that better GDPDJH RU PDMRU ¿QDQFLDO ORVV 6XFK ZLOO UDUHO\ VXI¿FH DQG ZLOO KDYH WR EH accidents have already occurred, and, evidence is needed, so that approaches augmented by analysis. The ability to without intervention, the increasingly aimed at improving the dependability make independence arguments that allow pervasive use of software—especially in of software can be objectively assessed, global properties to be inferred from arenas such as transportation, health care, and (2) that, for now, the pursuit of dependability in software systems should an analysis of a relatively small part of and the broader infrastructure—may make focus on the construction and evaluation the system will be essential. Rigorous them more frequent and more serious. In of evidence. processes will be needed to ensure that the future, more pervasive deployment of software in the civic infrastructure could The committee thus subscribes to the chain of evidence for dependability lead to more catastrophic failures unless the view that software is “guilty until claims is preserved. improvements are made. proven innocent,” and that the burden of The committee also recognized proof falls on the developer to convince the importance of adopting the practices Software, according to a popular WKHFHUWL¿HURUUHJXODWRUWKDWWKHVRIWZDUH that are already known and used by the view, fails because of bugs: errors in the code that cause a program to fail to meet its is dependable. This approach is not novel best developers; this summary gives a and is becoming standard in the world sample of such practices in more detail VSHFL¿FDWLRQ,QIDFWRQO\DWLQ\SURSRUWLRQ of failures can be attributed to bugs. As is of systems safety, in which an explicit below. Some of these (such as systematic safety case (and not merely adherence well known to software engineers, by far FRQ¿JXUDWLRQPDQDJHPHQWDQGDXWRPDWHG to good practice) is usually required. the largest class of problems arises from regression testing) are relatively easy to Similarly, a software system should be errors made in the eliciting, recording, adopt; others (such as constructing hazard regarded as dependable only if it has a and analysis of requirements. A second analyses and threat models, exploiting credible dependability case, the elements major class of problems arises from poor formal notations when appropriate, and of which are described below. Meeting human factors design. The two classes applying static analysis to code) will the burden of proof for dependability are related; bad user interfaces usually will be challenging. The demand for require new training for many developers. UHÀHFW DQ LQDGHTXDWH XQGHUVWDQGLQJ RI credible evidence will, in practice, make However valuable, though, these practices the user’s domain and the absence of a it infeasible to develop highly dependable are in themselves no silver bullet, and new coherent and well-articulated conceptual systems in a cost-effective way without techniques and methods will be required model. Security vulnerabilities are some radical changes in priorities. If in order to build future software systems to some extent an exception to this very high dependability is to be achieved to the level of dependability that will be observation: The overwhelming majority at reasonable cost, the needs of the required. of security vulnerabilities reported in

36 Software for Dependable Systems: Sufficient Evidence? FEATURE

software products—and exploited to of particular development approaches, software system dependability and attack the users of such products—are at tools, or techniques. Moreover, the report that suggest a different approach to the implementation level. The prevalence leaves to the developers and procurers of the development and assessment of of code-related problems, however, is individual systems the question of what dependable software. Due to a lack of a direct consequence of higher-level level of dependability is appropriate, and VXI¿FLHQW GDWD WR VXSSRUW RU FRQWUDGLFW decisions to use programming languages, what costs are worth incurring to achieve it. any particular approach, a software design methods, and libraries that admit Nonetheless, the evidence available system may not be declared “dependable” these problems. to the committee did support several based on the method by which it In systems where software failure qualitative conclusions. First, developing was constructed. Rather, it should be FRXOGKDYHVLJQL¿FDQWKXPDQRU¿QDQFLDO software to meet even existing UHJDUGHG DV GHSHQGDEOH² FHUWL¿DEO\ costs, it is crucial that software be GHSHQGDELOLW\ FULWHULD LV GLI¿FXOW DQG dependable—only when adequate dependable—that it can be depended costly. Large software projects fail at a evidence has been marshaled in support upon to function as expected and to not high rate, and the cost of projects that do of an argument for dependability that can cause or contribute to adverse events in succeed in delivering highly dependable be independently assessed. The goal of software is often exorbitant. Second, the environment in which it operates. FHUWL¿DEO\ GHSHQGDEOH VRIWZDUH FDQQRW the quality of software produced by the Improvements in dependability would therefore be achieved by mandating industry is extremely variable, and there allow such systems to be used more particular processes and approaches, is inadequate oversight in some critical ZLGHO\ DQG ZLWK JUHDWHU FRQ¿GHQFH regardless of their effectiveness in certain DUHDV 7RGD\¶V FHUWL¿FDWLRQ UHJLPHV DQG IRU WKH EHQH¿W RI VRFLHW\ 0RUHRYHU situations. Instead, software developers consensus standards have a mixed record. software itself has great potential to bring should marshal evidence to justify an Some are largely ineffective, and some improvements in safety in many areas. explicit dependability claim that makes are counterproductive. They share a Complete and reliable data about clear which properties in the real world heavy reliance on testing, which cannot software-related system failures or the system is intended to establish. Such SURYLGH VXI¿FLHQW HYLGHQFH IRU WKH KLJK WKH HI¿FDF\ RI SDUWLFXODU VRIWZDUH evidence forms a dependability case, levels of dependability required in many development approaches are hard to come critical applications. and creating a dependability case is the E\PDNLQJREMHFWLYHVFLHQWL¿FHYDOXDWLRQ cornerstone of the committee’s approach $¿QDOREVHUYDWLRQLVWKDWWKHFXOWXUH GLI¿FXOW0RUHRYHUWKHODFNRIV\VWHPDWLF of an organization in which software is WR GHYHORSLQJ FHUWL¿DEO\ GHSHQGDEOH reporting of software-related system software systems. produced can have a dramatic effect on failures is a serious problem that makes its quality and dependability. It seems LWPRUHGLI¿FXOWWRHYDOXDWHWKHULVNVDQG Explicit claims, evidence, likely that the excellent record of avionics costs of such failures and to measure and expertise software is due in large part to a safety the effectiveness of proposed policies or The committee’s proposed approach culture in that industry that encourages interventions. can be summarized in “the three Es”— meticulous attention to detail, high explicit claims, evidence, and expertise: This lack of evidence has two direct aversion to risk, and realistic assessment consequences for this report. First, it has ‡ ([SOLFLW FODLPV No system can be of software, staff, and process. Indeed, informed the key recommendations in this “dependable” in all respects and under PXFK RI WKH EHQH¿W RI VWDQGDUGV VXFK all conditions. So to be useful, a claim report regarding the need for evidence to be as DO-178B, Software Considerations of dependability must be explicit. It at the core of dependable software system in Airborne Systems and Equipment development; for data collection efforts must articulate precisely the properties &HUWL¿FDWLRQ PD\ EH GXH WR WKH VDIHW\ the system is expected to exhibit and to be established; and for transparency culture that their strictures induce. and openness to be encouraged. Second, the assumptions about the system’s it has tempered the committee’s desire Toward certifiably environment upon which the claim is to provide prescriptive guidance: The dependable software contingent. The claim should also indicate approach recommended is therefore The focus of this report is a set of explicitly the level of dependability largely free of endorsements or criticisms fundamental principles that underlie claimed, preferably in quantitative terms.

The Next Wave n Vol 19 No 1 n 2011 37 Different properties may be assured to development, in the domain under Systems engineering approach different levels of dependability. consideration, and in the broader Complementing “the three Es” are ‡(YLGHQFHFor a system to be regarded systems context, among other things—is several systems engineering ideas that necessary to achieve dependable systems. as dependable, concrete evidence provide an essential foundation for the Flexibility is an important advantage of must be present that substantiates the building of dependable software systems: dependability claim. This evidence will the proposed approach; in particular the ‡6\VWHPVWKLQNLQJ(QJLQHHULQJ¿HOGV take the form of a dependability case developer is not required to follow any with long experience in building complex arguing that the required properties particular process or use any particular systems (for example, aerospace, follow from the combination of the PHWKRG RU WHFKQRORJ\ 7KLV ÀH[LELOLW\ chemical, and nuclear engineering) properties of the system itself (that is, the allows experts freedom to employ new have developed approaches based on implementation) and the environmental techniques and to tailor the approach to “systems thinking.” These approaches assumptions. Because testing alone the system’s application and domain. But focus on properties of the system as a LV XVXDOO\ LQVXI¿FLHQW WR HVWDEOLVK the requirement to produce evidence is properties, the case will typically combine highly demanding and likely to stretch whole and on the interactions among its evidence from testing with evidence today’s best practices to their limit. It will components, especially those interactions from analysis. In addition, the case will therefore be essential that developers are (often neglected) between a component inevitably involve appeals to the process familiar with best practices and deviate being constructed and the components of by which the software was developed— from them only for good reason. its environment. As software has come to for example, to argue that the software These prescriptions shape any be deployed in—indeed has enabled— GHSOR\HGLQWKH¿HOGLVWKHVDPHVRIWZDUH particular development approach only in increasingly complex systems, the system that was subjected to analysis or testing. outline and give considerable freedom aspect has come to dominate in questions ‡ ([SHUWLVH Expertise—in software to developers in their choice of methods, of software dependability. languages, tools, and processes. This ‡ 6RIWZDUH DV D V\VWHP FRPSRQHQW approach is not, of course, a silver bullet. Dependability is not an intrinsic There are no easy solutions to the problem property of software. The committee of developing dependable software, and strongly endorses the perspective of there will always be systems that cannot be systems engineering, which views the built to the required level of dependability software as one engineered artifact in even using the latest methods. But, the a larger system of many components, approach recommended is aimed at some engineered and some given, and SURGXFLQJFHUWL¿DEO\GHSHQGDEOHV\VWHPV views the pursuit of dependability as today, and the committee believes it holds a balancing of costs and benefits and promise for developing the systems that a prioritization of risks. A software will be needed in the future. component that may be dependable in In the overall context of engineering, the context of one system might not be the basic tenets of the proposed approach dependable in the context of another. are not controversial, so it may be a surprise to some that the approach is ‡ +XPDQV DV FRPSRQHQWV 3HRSOH² not already commonplace. Nor are the the operators and users (and even elements of the approach novel; they have the developers and maintainers) of a been applied successfully for more than system—may also be viewed as system a decade. Nevertheless, this approach components. If a system meets its would require radical changes for most dependability criteria only if people act software development organizations in certain ways, then those people should and is likely to demand expertise that is be regarded as part of the system, and an currently in short supply. estimate of the probability that they will

38 Software for Dependable Systems: Sufficient Evidence? FEATURE

behave as required should be part of the profound implication. Any component individual components much smaller than evidence for dependability. for which compelling evidence of the system as a whole, which can preserve ‡5HDOZRUOGSURSHUWLHV7KHSURSHUWLHV dependability has been amassed at these properties despite failures in the of interest to the user of a system are reasonable cost will likely be small by rest of the system. Independence can be typically located in the physical world: the standards of most modern software established in the overall design of the that a radiotherapy machine deliver a V\VWHPV (YHU\ FULWLFDO VSHFL¿FDWLRQ system, with the support of architectural certain dose, that a telephone transmit property, therefore, will have to be mechanisms. Its effect is to dramatically a sound wave faithfully, that a printer assured by one, or at most a few, small reduce the cost of constructing a make appropriate ink marks on paper, components. Sometimes it will not be dependability case for a property, since and so on. The software, on the other possible to separate concerns so cleanly, only a relatively small part of the system KDQG LV W\SLFDOO\ VSHFL¿HG LQ WHUPV RI and in that case, the dependability case needs to be considered. properties at its interfaces, which usually may be less credible or more expensive Appropriate simplicity and involve phenomena that are not of direct to produce. independence cannot be accomplished interest to the user: that the radiotherapy As a result, one key to achieving without addressing the challenges of machine, telephone, or printer send or dependability at reasonable cost is a “interactive complexity” and “tight receive certain signals at certain ports, serious and sustained commitment to coupling.” Both interactive complexity, with the inputs related to the outputs simplicity, including simplicity of critical where components may interact in according to some rules. It is important, functions and simplicity in system unanticipated ways, and tight coupling, therefore, to distinguish the requirements interactions. This commitment is often wherein a single fault cannot be isolated of a software system, which represent the mark of true expertise. An awareness but brings about other faults that cascade these properties in the physical world, of the need for simplicity usually comes through the system, are correlated IURP WKH VSHFL¿FDWLRQ RI D VRIWZDUH only with bitter experience and the with the likelihood of system failure. system, which characterizes the behavior of the software system at its interface with the environment. When the “Testing is indispensable, software system is itself only and no software system can be regarded as dependable one component of a larger LILWKDVQRWEHHQH[WHQVLYHO\WHVWHG´ system, the other components in the system (including perhaps, as explained above, humility gained from years of practice. Software-intensive systems tend to have the people who work with the system) There is no alternative to simplicity. both attributes. Careful attention should will be viewed as part of the environment. Advances in technology or development therefore be paid to the risks of interactive The dependability properties of a software methods will not make simplicity complexity and tight coupling and the system, therefore, should be expressed redundant; on the contrary, they will advantages of modularity, isolation, as requirements, and the dependability give it greater leverage. To achieve high and redundancy. The interdependences case should demonstrate how these levels of dependability in the foreseeable among components of critical software properties follow from the combination future, striving for simplicity is likely to systems should be analyzed to ensure that RIWKHVSHFL¿FDWLRQDQGWKHHQYLURQPHQWDO be by far the most cost-effective of all there is no fault propagation path from assumptions. interventions. Simplicity is not easy or less critical components to more critical Coping with complexity cheap, but its rewards far outweigh its costs. components, that modes of failure are The need for evidence of The most important form well understood, and that failures are GHSHQGDELOLW\ DQG WKH GLI¿FXOW\ RI of simplicity is that produced by localized to the greatest extent possible. producing such evidence for complex independence, in which particular The reduction of interactive complexity systems have a straightforward but system-level properties are guaranteed by and tight coupling can contribute not

The Next Wave n Vol 19 No 1 n 2011 39 only to the improvement of system tested, even if its correctness has been modules in aggregate achieve appropriate dependability but also to the development proven mathematically. Testing may system-level effects. These analyses will of evidence and analysis in the service of ¿QG ÀDZV WKDW HOXGH DQDO\VLV EHFDXVH sometimes involve informal argument a dependability case. it exercises the system in its entirety, that is carefully reviewed; sometimes whereas analysis must typically make mechanical inference (as performed, for Rigorous process and preserving assumptions about the execution H[DPSOHE\³W\SHFKHFNHUV´WKDWFRQ¿UP the chain of evidence platform, the external environment, and that memory is used in a consistent way Generating a dependability case operator responses, any of which may and that boundaries between modules are after the fact, when a development is turn out to be unwarranted. At the same respected); and, sometimes, formal proof. largely complete, might be possible time, it is important to realize that testing Indeed, the dependability case for even in theory. But in practice, at least with DORQHLVUDUHO\VXI¿FLHQWWRHVWDEOLVKKLJK a relatively simple system will usually today’s technology, the costs of doing so levels of dependability. It is erroneous require all of these kinds of analysis, and would be high, and it will be practical to to believe that a rigorous development WKH\ZLOOQHHGWREH¿WWHGWRJHWKHULQWRD develop a dependability case only if the process, in which testing and code review coherent whole. system is built with its construction in DUHWKHRQO\YHUL¿FDWLRQWHFKQLTXHVXVHG mind. Each step in developing the software Traditional software development MXVWL¿HV FODLPV RI H[WUDRUGLQDULO\ KLJK needs to preserve the chain of evidence on methods rely on human inspection and OHYHOVRIGHSHQGDELOLW\6RPHFHUWL¿FDWLRQ which will be based the argument that the WHVWLQJ IRU YDOLGDWLRQ DQG YHUL¿FDWLRQ schemes, for example, associate higher resulting system is dependable. Formal methods also use testing, but safety integrity levels with more they employ notations and languages At the start, the domain and burdensome process prescriptions and that are amenable to rigorous analysis, environmental assumptions and the imply that following the processes and they exploit mechanical tools required properties of the system should recommended for the highest integrity for reasoning about the properties of be made explicit; they should be expressed levels will ensure that the failure rate is UHTXLUHPHQWVVSHFL¿FDWLRQVGHVLJQVDQG unambiguously and in a form that permits minuscule. In the absence of a carefully systematic analysis to ensure that there FRGH 3UDFWLWLRQHUV KDYH EHHQ VNHSWLFDO constructed dependability case, such about the practicality of formal methods. DUH QR XQUHVROYDEOH FRQÀLFWV EHWZHHQ FRQ¿GHQFHLVPLVSODFHG the required properties. Each subsequent Increasingly, however, there is evidence Because testing alone will not be stage of development should preserve that formal methods can yield systems VXI¿FLHQW IRU WKH IRUHVHHDEOH IXWXUH the evidence chain—that these properties of very high dependability in a cost- the dependability claim will also have been carried forward without being effective manner, at least for small- to require evidence produced by analysis. corrupted—so each form in which the medium-sized critical systems. Although Moreover, because analysis links the requirements, design, or implementation formal methods are typically more software artifacts directly to the claimed is expressed should support analysis expensive to apply when only low levels properties, the analysis component of the to permit checking that the required of dependability are required, the cost of dependability case will usually contribute properties have been preserved. What traditional methods rises rapidly with the FRQ¿GHQFH DW D ORZHU FRVW WKDQ WHVWLQJ LV VXI¿FLHQW ZLOO YDU\ ZLWK WKH UHTXLUHG level of dependability and often becomes for the highest levels of dependability. A dependability, but preserving the evidence prohibitive. When a highly dependable dependability case will generally require chain necessitates that the checks are system is required, therefore, a formal many forms of analysis, including (1) the carried out in a disciplined way, following approach may be the most cost effective. validation of environmental assumptions, a documented procedure, and leaving use models, and fault models; (2) the Certification, auditable records. analysis of fault tolerance measures transparency, The roles of testing, analysis, against fault models; (3) schedulability and accountability and formal methods analysis for temporal behaviors; (4) $ YDULHW\ RI FHUWL¿FDWLRQ UHJLPHV Testing is indispensable, and no security analysis against attack models; exist for software in particular application software system can be regarded as   YHUL¿FDWLRQ RI FRGH DJDLQVW PRGXOH domains. For example, the Federal dependable if it has not been extensively VSHFL¿FDWLRQV DQG   FKHFNLQJ WKDW $YLDWLRQ$XWKRULW\ )$$  LWVHOI FHUWL¿HV

40 Software for Dependable Systems: Sufficient Evidence? FEATURE

QHZDLUFUDIW DQGDLUWUDI¿FPDQDJHPHQW  When choosing suppliers and explicit who is accountable for any failure systems that include software, and this products, customers and users can make to achieve it. Such accountability can be FHUWL¿FDWLRQ LV WKHQ UHOLHG RQ E\ WKH informed judgments only if the claims made explicit in the purchase contract, customers who buy and use the aircraft; are credible. Such claims are unlikely to RUDVSDUWRIFHUWL¿FDWLRQRIWKHVRIWZDUH the National Information Assurance be credible if the evidence underlying or as part of a professional licensing 3DUWQHUVKLS 1,$3  OLFHQVHV WKLUGSDUW\ them is not transparent. Economists have scheme, or in other ways. Since no single laboratories to assess security software established that if consumers cannot solution will suit all the circumstances products for conformance to the Common reliably observe quality before they buy, LQZKLFKFHUWL¿DEO\GHSHQGDEOHVRIWZDUH Criteria. Some large organizations have VHOOHUV PD\ JHW OLWWOH HFRQRPLF EHQH¿W systems are deployed, accountability their own regimes for certifying that from providing higher quality than their regimes should be tailored to particular the software products they buy meet the competitors, and overall quality can circumstances. At present, it is common organization’s quality criteria, and many decline. Sellers are concerned about for software developers to disclaim, software product manufacturers have future sales, and “reputation effects” so far as possible, all liability for their own criteria that each version of compel them to strive to maintain a defects in their products, to a greater their product must pass before release. minimum level of quality. If consumers extent than customers and society )HZ LI DQ\ H[LVWLQJ FHUWL¿FDWLRQ rely heavily on branding, though, it expect from manufacturers in other regimes encompass the combination EHFRPHVPRUHGLI¿FXOWIRUQHZ¿UPVWR industries. Clearly, no software should of characteristics recommended in this enter the market, and quality innovations be considered dependable if it is supplied report—namely, explicit dependability spread more slowly. with a disclaimer that withholds the claims, evidence for those claims, and Those claiming dependability for manufacturer’s commitment to provide a a rigorous argument that demonstrates their software should therefore make warranty or other remedies for software WKDW WKH HYLGHQFH LV VXI¿FLHQW WR available the details of their claims, that fails to meet its dependability claims. establish the validity of the claims. To criteria, and evidence. To assess the Determining the appropriate scale of establish that a system is dependable credibility of such details effectively, an remedies, however, was beyond the scope will involve inspection and analysis of evaluator should be able to calibrate not of this study and would require a careful the dependability claim and the evidence only the technical claims and evidence DQDO\VLV RI EHQH¿WV DQG FRVWV WDNLQJ offered in its support. Where the customer but also the organization that produced into account not only the legal issues but for the system is not able to carry out that them, because the integrity of the also the state of software engineering, work itself (for lack of time or lack of evidence chain is vital and cannot easily the various submarkets for software, expertise) it may need to involve a third be assessed without supporting data. This the economic impact, and the effect on party whose judgment it can rely on to suggests that in some cases data of a more innovation. be independent of commercial pressures general nature should be made available, Key findings and IURP WKH YHQGRU &HUWL¿FDWLRQ FDQ WDNH LQFOXGLQJ WKH TXDOL¿FDWLRQV RI WKH recommendations PDQ\IRUPVIURPVHOIFHUWL¿FDWLRQE\WKH personnel involved in the development; 3UHVHQWHGEHORZDUHWKHFRPPLWWHH¶V supplier at one extreme, to independent the track record of the organization in ¿QGLQJV DQG UHFRPPHQGDWLRQV HDFK WKLUGSDUW\ FHUWL¿FDWLRQ E\ D OLFHQVHG providing dependable software; and of which is discussed in more detail in FHUWL¿FDWLRQ DXWKRULW\ DW WKH RWKHU 1R the process by which the software was Chapter 4. (The full report is available at: VLQJOHFHUWL¿FDWLRQUHJLPHLVVXLWDEOHIRU developed. The willingness of a supplier http://www.nap.edu/catalog.php?record_ all circumstances, so a suitable scheme to provide such data, and the clarity and id=11923 ) should be chosen for each circumstance. integrity of the data that the supplier Industry groups and professional societies provides, will be a strong indication of its Findings should consider developing model attitude to dependability. Improvements in software FHUWL¿FDWLRQVFKHPHVDSSURSULDWHWRWKHLU Where there is a need to deploy development are needed to keep pace domains, taking account of the detailed VRIWZDUH WKDW VDWLV¿HV D SDUWLFXODU with societal demands for software. recommendations in this report. dependability claim, it should always be Avoidable software failures have already

The Next Wave n Vol 19 No 1 n 2011 41 been responsible for loss of life and for ‡ ([SORLW VLPSOLFLW\ If dependability LQVXI¿FLHQWWKHGHSHQGDELOLW\FODLPZLOO major economic losses. The quality is to be achieved at reasonable cost, require, in addition, evidence produced of software produced by the industry simplicity should become a key goal, and by other modes of analysis. Security is extremely variable, and there is developers and customers must be willing FHUWL¿FDWLRQ LQ SDUWLFXODU VKRXOG JR inadequate oversight in several critical to accept the compromises it entails. beyond functional testing of the security areas. More pervasive deployment of Make a dependability case for a components of a system and assess the software in the civic infrastructure may given system and context: evidence, effectiveness of measures the developer lead to catastrophic failures unless explicitness, and expertise. A software took to prevent the introduction of improvements are made. Software has system should be regarded as dependable security vulnerabilities. WKH SRWHQWLDO WR EULQJ GUDPDWLF EHQH¿WV RQO\LIVXI¿FLHQWHYLGHQFHRILWVH[SOLFLWO\ Include security considerations to society, but it will not be possible to articulated properties is presented to in the dependability case. Security UHDOL]H WKHVH EHQH¿WV²HVSHFLDOO\ LQ substantiate the dependability claim. vulnerabilities can undermine the case critical applications—unless software This approach gives considerable leeway made for dependability properties becomes more dependable. to developers to use whatever practices by violating assumptions about how More data is needed about are best suited to the problem at hand. components behave, about their software failures and the efficacy of In practice the challenges of developing interactions, or about the expected behavior of users. The dependability development approaches. Assessment GHSHQGDEOHVRIWZDUHDUHVXI¿FLHQWO\JUHDW case must therefore account explicitly of the state of the software industry, the that developers will need considerable for security risks that might compromise risks posed by software, and progress expertise, and they will have to justify its other aspects. It is also important to made is currently hampered by the lack any deviations from best practices. HQVXUH WKDW VHFXULW\ FHUWL¿FDWLRQV JLYH of a coherent source of information about Demand more transparency, so meaningful assurance of resistance to software failures. that customers and users can make more DWWDFN1HZVHFXULW\FHUWL¿FDWLRQUHJLPHV Recommendations to builders informed judgments about dependability. DUH QHHGHG WKDW FDQ SURYLGH FRQ¿GHQFH and users of software Customers and users can make informed WKDWPRVWDWWDFNVDJDLQVWFHUWL¿HGSURGXFWV Make the most of effective judgments when choosing suppliers or systems will fail. Such regimes can be software development technologies and products only if the claims, criteria, EXLOW E\ DSSO\LQJ WKH RWKHU ¿QGLQJV DQG and formal methods. A variety of and evidence for dependability are recommendations of this report, with an modern technologies—in particular, safe transparent. emphasis on the role of the environment— programming languages, static analysis Make use of but do not rely solely in particular, the assumptions made about (analysis of software and source code on process and testing. Testing will be an the potential actions of a hostile attacker done without actually executing the essential component of a dependability and the likelihood that new classes of program), and formal methods— are FDVH EXW ZLOO QRW LQ JHQHUDO VXI¿FH vulnerabilities will be discovered and OLNHO\WRUHGXFHWKHFRVWDQGGLI¿FXOW\RI because even the largest test suites new attacks developed to exploit them. producing dependable software. typically used will not exercise enough Demand accountability and make it Follow proven principles for paths to provide evidence that the software explicit. Where there is a need to deploy software development. The committee’s LV FRUUHFW QRU ZLOO LW KDYH VXI¿FLHQW FHUWL¿DEO\GHSHQGDEOHVRIWZDUHLWVKRXOG proposed approach also includes VWDWLVWLFDO VLJQL¿FDQFH IRU WKH OHYHOV RI always be made explicit who or what is adherence to the following principles: FRQ¿GHQFH XVXDOO\ GHVLUHG 5LJRURXV accountable, professionally and legally, ‡7DNHDV\VWHPVSHUVSHFWLYH Here the process is essential for preserving the for any failure to achieve the declared dependability of software is viewed not chain of dependability evidence but is not dependability. in terms of intrinsic properties (such as per se evidence of dependability. Recommendations to agencies the incidence of bugs in the code) but %DVH FHUWL¿FDWLRQ RQ LQVSHFWLRQ and organizations that support rather in terms of the system as a whole, and analysis of the dependability claim software education and research including interactions among people, and the evidence offered in its support. The committee was not constituted process, and technology. Because testing and process alone are or charged to recommend budget levels

42 Software for Dependable Systems: Sufficient Evidence? FEATURE

or to assess trade-offs between software Software Systems, convened under the dependability and other priorities. auspices of the NRC’s Computer Science However, it believes that the increasing and Telecommunications Board. The importance of software to society and committee consisted of 13 experts from the extraordinary challenge currently industry and academia specializing in faced in producing software of adequate diverse aspects of systems dependability dependability provide a strong rationale including software engineering, software for investment in education and research testing and evaluation, software initiatives. dependability, embedded systems, 3ODFH JUHDWHU HPSKDVLV RQ human-computer interaction, systems dependability—and its fundamental engineering, systems architecture, underpinnings—in the high school, accident theory, standards setting, undergraduate, and graduate education of avionics, medicine, economics, security, software developers. Many practitioners and regulatory policy. Committee chair do not have an adequate appreciation of the 'DQLHO-DFNVRQ, a professor of Computer software dependability issues discussed Science at MIT; committee member in this report, are not aware of the most Martyn Thomas, visiting professor effective development practices available of software engineering at Oxford today, or are not capable of applying them University; and Lynette Millett, senior appropriately. Wider implementation of VWDIIRI¿FHUDWWKH15&HGLWHGWKHUHSRUW the committee’s recommended approach, Discussions initiated by the which goes beyond today’s state of +LJK &RQ¿GHQFH 6RIWZDUH DQG the practice, implies a need for further Systems Coordinating Group (HCSS education and training activities. CG) of the National Science and Federal agencies that support Technology Council’s Networking and information technology research and Information Technology Research and development should give priority to basic 'HYHORSPHQW 1,75'  6XEFRPPLWWHH research to further software-enabled with the NRC’s Computer Science and system dependability, emphasizing a Telecommunications Board resulted systems perspective and evidence. In in this study on the current state of keeping with this report’s approach, such FHUWL¿FDWLRQ LQ GHSHQGDEOH V\VWHPV research should emphasize a systems Funding for the study was obtained from perspective and “the three Es” (explicit HCSS CG member agencies. claims, evidence, and expertise) and should be informed by a systems view that attaches more importance to those advances that are likely to have an impact in a world of large systems interacting with other systems and operators in a complex physical environment and organizational context. About the report This report was authored by the National Research Council’s (NRC) &RPPLWWHH RQ &HUWL¿DEO\ 'HSHQGDEOH

The Next Wave n Vol 19 No 1 n 2011 43 • Critical Code • Software Producibility for Defense A Short Summary

he rapid growth in the role of software in defense systems is significant and parallels the growing role of software in a broad range of application domains, T ranging from financial services and health care to telecommunications, logistics, and transportation. This growth is reflected in recent macroeconomic studies, which suggest that in the US and Europe 20 percent to 25 percent of overall economic growth and nearly 40 percent of the increase in overall economic productivity since 1995 are attributed to information and communications technology. It is also reflected in individual systems. For example, in modern automobiles, the portion of system functions performed in software is now 40 percent and approaching 50 percent. In the DoD, the growth has been even more profound-in military aircraft, for example, the percentage of system functions performed by software has risen to more than 80 percent. This growth of software in role and significance is a natural outcome of its special engineering characteristics: software is uniquely unbounded and flexible, having relative­ ly few intrinsic limits on the degree to which it can be scaled in complexity and capability. This is because software is an abstract and purely synthetic medium that, for the most part, lacks fundamental physical limits and natural constraints. For example, unlike physical hardware, software can be delivered and upgraded electronically and remotely, greatly facilitating rapid adaptation to changes in adversary threats, mission priorities, technology, and other aspects of the operating environment. The principal constraint on what can be accomplished is the human intellectual capacity to understand problems and systems, to build tools to manage them, and to provide assurance-all at ever­ greater levels of scale and complexity. The extent of the DoD code in service has been increasing by more than an order of magnitude every decade, and a similar growth pattern has been exhibited within individual, long-lived military systems. In addition to this growth in size, there is a cor­ responding growth in overall systems capability, complexity, interconnectedness, and agility. This growth is enabled by the increasing power of software languages, tools, and practices, as well as by a significant growth in the dependence of DoD systems on increasingly complex, diverse, and geographically distributed supply chains. These supply chains include not only custom components developed for specific mission purposes, but also commercial and open-source ecosystems and components, such as the widely used infrastructures for web services, mobile devices, and graphical user interaction. Because of the rapid growth in significance of software capability to the DoD overall, the Director of Defense Research and Engineering (now Assistant Secretary of Defense for Research and Engineering) requested the National Research Council (NRG) Committee for Advancing Software-Intensive Systems Producibility to undertake a study to address the challenges of defense software producibility, identifying the principal challenges and developing recommendations regarding both improvements to practice and priorities for research. The NRG committee just released its final report, titled Critical Code: Software Producibility for Defense. Full copies of the report (free PDF download and book purchase), along with related prior reports, are available through the National Academy Press at http://www.nap.edu/catalog.php?record_id=12979. This article summarizes the principal findings and recommendations of that report.

44 Critical Code: So~ware Producibility for Defense FEATURE

The necessity of sustaining paper by John Backus describing the document structure very explicitly into software innovation ¿UVW)RUWUDQFRPSLOHU7KHWLWOHLQFOXGHG numbers and arrays. A person reading the words “automatic programming.” program text would see only numerical An initial question is whether software is indeed a strategic building The point of this phrase, with respect and array operations because that was material, worthy of special attention. This to Backus’s great accomplishment, the limit of what could be explicitly question has been addressed periodically is that there was a much more direct expressed in the notation. This meant by the Defense Science Board (DSB) correspondence between his high-level that programmers needed to keep track, since 1985—a 2007 DSB report, for programming notation—the earliest in their heads or in documentation, of the example, stated that “in the Department )RUWUDQ FRGH²DQG SXUH PDWKHPDWLFDO nature of this representational encoding. of Defense, the transformational effects thinking than had been the case with It also meant that testers and evaluators RI LQIRUPDWLRQ WHFKQRORJ\ ,7²GH¿QHG the early machine-level code. One can needed to assess programs through this here broadly to include all forms of FRQVWUXHWKDWLWZDVLPDJLQHGWKDW)RUWUDQ (hopefully) same layer of interpretation. computing and communications), joined enabled mathematicians to express As languages have evolved with a culture of information sharing, their thoughts directly to computers, LQFOXGLQJ PRUH PRGHUQ )RUWUDQ called Net-Centricity, constitute a seemingly without the intervention of versions), these additional structures powerful force multiplier. The DoD SURJUDPPHUV 7KH HDUO\ )RUWUDQ ZDV can be much more directly expressed— has become increasingly dependent for indeed an extraordinary and historical characters and strings, most obviously, mission-critical functionality upon highly breakthrough. But we know that, in the are intrinsic in nearly all modern interconnected, globally sourced IT of end, those mathematicians of 50 years languages. It is interesting, however, that dramatically varying quality, reliability, ago soon evolved into programmers—as the claim of “automatic programming” and trustworthiness.” a direct consequence of their growing continues to reappear from time to time Despite the strength of this statement, ambitions for computing applications. as major steps are made in improved every few years speculation surfaces Just a few years after the Backus abstractions, for example related to that perhaps software and information SDSHU )RUWUDQ ZDV XVHG WR VXSSRUW OLVW data manipulation (the so-called 4GLs). technology may be approaching a plateau processing applications, typesetting These developments move us forward, of capability and performance and that applications, compilers for other but ironically they do not actually get strategic attention to these technologies is languages, and other applications whose us closer to “eliminating programmers” consequently not merited. The committee abstractions required some considerable or otherwise emerging at some plateau emphasizes that this continues to be a programming sophistication (and of capability and near-commodity false and dangerous speculation—the representational gerrymandering) to be status. Instead, new software-manifest capability and the complexity of hardware UHSUHVHQWHG HIIHFWLYHO\ DV HDUO\ )RUWUDQ capabilities are constantly emerging— and software systems are both rising at an data structures—arrays and numeric for example, techniques for machine- accelerating rate, with no end in sight. values. Any program that manipulated learning algorithms and highly parallel It is instructive, in this regard, to textual data, for example, needed to data-intensive analytics—that continue consider the publication in 1958—more encode the text characters, textual strings, to demand considerable intellectual effort than a half century ago—of the landmark and any overarching paragraph and on the part of programmers.

The Next Wave Q Vol 19 No 1 Q 2011 45 The profound fact is that software on mainstream commercial and open made within an engineering process— capability is bounded primarily by source components, supply chains, and the risks are high when the outcomes of our intellectual abilities—our human software ecosystems, it nonetheless has immediate project commitments are both capability both to create new abstractions special needs in its mission systems that FRQVHTXHQWLDO DQG GLI¿FXOW WR SUHGLFW appropriate for application domains are driven by the growing role of software Engineering risks can relate to many and to manifest those abstractions in in systems overall. The committee different kinds of engineering decisions— languages, models, tools, and practices. recommends that the DoD regularly PRVW VLJQL¿FDQWO\ DUFKLWHFWXUH TXDOLW\ As our understanding advances, so can XQGHUWDNH DQ LGHQWL¿FDWLRQ RI DUHDV attributes, functional characteristics, and our software capability advance with us. of technological need where the DoD infrastructure choices. has “leading demand” and where As a consequence of this seeming When well managed, incremental accelerated progress is needed. XQERXQGHGQHVVWKHFRPPLWWHH¿QGVWKDW practices can enable innovative technological leadership in software is a Three goals for software- engineering to be accomplished without a key driver of overall capability leadership intensive development necessarily consequent increase of overall programmatic risk. (Programmatic risk in systems—and that at the core of the 7KHFRPPLWWHHLGHQWL¿HGWKUHHDUHDV relates to the successful completion ability to achieve integration and maintain where improvements in practice would of engineering projects with respect mission agility is the ability of the DoD PDWHULDOO\EHQH¿WWKHDELOLW\RIWKH'R' to expectations and priorities for cost, to produce and evolve software. The to develop, sustain, and assure software- schedule, capability, quality, and other committee recommends that, to avoid intensive systems of all kinds. Each of attributes.) This is because incremental loss of leadership, the DoD take active these areas is the subject of a chapter in practices enable engineering ULVNV WR EH LGHQWL¿HG HDUO\ “...to avoid loss of leadership, and mitigated promptly. the DoD [should] take active steps to become more fully engaged Incremental practices are in the innovative processes related to software producibility.” enabled through the use of diverse techniques such steps to become more fully engaged the Critical Code report. (These three as modeling, simulation, in the innovative processes related to areas of practice correspond to Chapters prototyping, and other means for early software producibility. In particular, 2, 3, and 4. Chapter 1 of the report focuses validation—coupled with extensions to earned-value models that measure and WKHFRPPLWWHH¿QGVWKDWLQGXVWU\GHVSLWH on the necessary role of DoD in software the extraordinary pace of innovation we innovation. Chapter 5 summarizes the give credit for the accumulating body of evidence in support of feasibility. are now witnessing, will not produce research agenda related to software Incremental approaches include iterative software innovations in areas of defense producibility.) The three areas of practice approaches, staged acquisition, evidence- VLJQL¿FDQFH DW D UDWH IDVW HQRXJK WR are summarized below: based systems engineering, and other allow the DoD to fully meet its software- Practice improvement 1: methods that explicitly acknowledge related requirements and remain ahead of Process and measurement engineering risk and its mitigation. potential adversaries. Advances related to process and 7KH FRPPLWWHH ¿QGV WKDW A loss of leadership could threaten measurement would facilitate broader incremental and iterative methods are the ability of the DoD to manifest and more effective use of incremental RI IXQGDPHQWDO VLJQL¿FDQFH WR 'R' world-leading capability, and also to iterative development, particularly in for innovative, software-intensive achieve adequate levels of assurance the arms-length contracting situations engineering in the DoD, and they can for the diversely sourced software it common in DoD. be managed more effectively through intends to deploy. This is an important Incremental development practices improvements in practices and supporting part of the rationale for the committee HQDEOH FRQWLQXRXV LGHQWL¿FDWLRQ DQG tools. The committee recommends a recommendation that the DoD reengage mitigation of engineering risks during diverse set of improvements related directly in the innovation processes. a systems development process. to advanced incremental development 7KH FRPPLWWHH DOVR ¿QGV WKDW Engineering risks pertain to the practice, supporting tools, and earned- although the DoD relies fundamentally consequences of particular choices to be value models.

46 Critical Code: Software Producibility for Defense FEATURE

Practice improvement 2: quality attributes, architecture decisions system characteristics. The weights Architecture may dominate functional capability given the various attributes are typically Advances related to architecture FKRLFHV LQ RYHUDOO VLJQL¿FDQFH 7KH determined on the basis of models of practice would facilitate the early focus committee also notes that architecture hazards associated with the operational on systems architecture that is essential practice in many areas of industry is context, including potential threats. The particularly for systems with demanding VXI¿FLHQWO\ PDWXUH IRU 'R' WR DGRSW process of achieving software assurance, requirements related to quality attributes, The committee recommends that DoD regardless of sector, is generally recognized LQWHUOLQNLQJDQGSODQQHGÀH[LELOLW\ more aggressively assert architectural to account for approximately half the total development cost for major projects. Software architecture models the leadership, with an early focus on structures of a system that comprises architecture being essential for systems In addition to overall cost, DoD software components, the externally with innovative functionality or faces several particular challenges for visible properties of those components, demanding quality requirements. DVVXUDQFH )LUVW WKHUH LV RIWHQ DQ DUPV length relationship between a contractor and the relationships among the Practice improvement 3: development team and government components. Good architecture entails Assurance and security a minimum of engineering commitment VWDNHKROGHUVPDNLQJLWGLI¿FXOWWRGHYHORS Advances related to assurance and that yields a maximum value. In particular, and share the information necessary to security would facilitate achievement of architecture design is an engineering making assurance judgments. This can mission assurance for systems at greater activity that is separate, for example, lead to approaches that overly focus on degrees of scale and complexity, and post hoc acceptance evaluation, rather IURP HFRV\VWHPV FHUWL¿FDWLRQ DQG RWKHU in the presence of rich supply chains standards-related policy setting. than on the emerging practice of “building and architectural ecosystems that are in” evidence in support of an overall )RU FRPSOH[ LQQRYDWLYH V\VWHPV increasingly commonplace in modern assurance case. Second, modern systems DUFKLWHFWXUH GH¿QLWLRQ HPERGLHV software engineering. draw on components from diverse SODQQLQJ IRU ÀH[LELOLW\²GH¿QLQJ DQG Assurance is a human judgment sources. This implies that supply-chain encapsulating areas where innovation regarding not just functionality, but DQG FRQ¿JXUDWLRQUHODWHG DWWDFNV PXVW and change are anticipated. Architecture also diverse quality attributes related be contemplated, with “attack surfaces” GH¿QLWLRQ DOVR PRVW VWURQJO\ LQÀXHQFHV to reliability, security, safety, and other existing within an overall application, diverse quality attributes, ranging from availability and performance to security and isolation. Additionally, architecture embodies planning for the interlinking of systems and for product line development, enabling encapsulation of individual innovative elements of a system. )RU PDQ\ LQQRYDWLYH V\VWHPV therefore, it may be more effective to consider architecture and quality attributes EHIRUH PDNLQJ VSHFL¿F FRPPLWPHQWV to functionality. Because architecture includes the earliest and, often, the most important design decisions—those HQJLQHHULQJ FRVWV WKDW DUH PRVW GLI¿FXOW to change later—early architectural commitment (and validation) can yield better project outcomes with less programmatic risk. 7KHFRPPLWWHH¿QGVWKDWLQKLJKO\ complex systems with emphasis on

The Next Wave Q Vol 19 No 1 Q 2011 47 The committee notes that traditional approaches based purely on testing and inspection, no matter how extensive, are RIWHQ LQVXI¿FLHQWO\ HIIHFWLYH IRU PRGHUQ software systems. It emphasizes that evaluation practices that focus primarily on post hoc acceptance evaluation are not RQO\YHU\FRVWO\EXWDUHRIWHQLQVXI¿FLHQW to justify useful assurance judgments. That is, quality and security must be built in, and not “tested in”—with the consequence that evidence production in support of assurance must be integrated into software development. 7KHFRPPLWWHH¿QGVWKDWDVVXUDQFH is facilitated by advances in diverse aspects of software engineering practice and technology, including modeling, analysis, tools and environments, traceability DQG FRQ¿JXUDWLRQ PDQDJHPHQW and not just at its perimeter. This has The Defense Science Board found programming languages, and process the consequence that evaluative and in 2007 that “it is an essential requirement VXSSRUW 7KH FRPPLWWHH DOVR ¿QGV WKDW preventive approaches ideally must be that the United States maintain advanced after many years of slow progress, integrated throughout a complex supply capability for ‘test and evaluation’ of IT recent advances have enabled more chain. A particular challenge is managing products. Reputation-based or trust-based rapid improvement in assurance-related opaque, or “black box,” components in a credentialing of software (‘provenance’) techniques and tools. This is already system—this issue is addressed in the full needs to be augmented by direct, artifact- evident in the most advanced commercial report. Third, the growing role of DoD focused means to support acceptance development practice. The committee VRIWZDUHLQZDU¿JKWLQJLQSURWHFWLRQRI HYDOXDWLRQ´ 7KLV LV D VLJQL¿FDQW DOVR ¿QGV WKDW VLPXOWDQHRXV FUHDWLRQ RI national assets, and in the safeguarding challenge, due to the rapid advance of assurance-related evidence with ongoing of human lives creates a diminishing software technology generally and also development has high potential to tolerance for faulty assurance judgments. the increasing pace by which potential improve the overall assurance of systems. Indeed, the Defense Science Board The committee recommends enhancing adversaries are advancing their capability. notes that there are profound risks incentives for preventive software This, coupled with the observations associated with the increasing reliance assurance practices and production of above regarding software innovation, is on modern software-intensive systems: assurance-related evidence throughout an important part of the rationale for the “this growing dependency is a source of the software lifecycle and through committee recommendation that the DoD weakness exacerbated by the mounting the software supply chain. This actively and directly address its software size, complexity, and interconnectedness includes both contractor and in-house producibility needs. RI LWV VRIWZDUH SURJUDPV´ )RXUWK development efforts. losing the lead in the ability to evaluate In the full report, the committee software and to prevent attacks can confer addressed a broad range of issues related The challenge of DoD advantage to adversaries with respect to software assurance, including evidence- software expertise to both offense and defense. It can also based approaches, evaluation practices, The committee also took up force us to overly “dumb down” systems, and security-motivated challenges related the issue of software expertise that is restricting functionality or performance to WRFRQ¿JXUDWLRQLQWHJULW\ SDUWLFXODUO\LQ VSHFL¿FDOO\ DOLJQHG ZLWK 'R' LQWHUHVWV a level such that assurance judgments can the presence of dynamism) and separation The committee found that DoD has a be more readily achieved. (including isolation and sandboxing). growing need for software expertise,

48 Critical Code: Software Producibility for Defense FEATURE

but that it is not able to meet this need (3) Component-based development, and Development (NITRD) program. The through intrinsic resources. This need including architectural designs for NITRD program provides a framework is essential for the DoD to be a smart particular domains for diverse federal agencies to coordinate software customer and program manager, 9DOLGDWLRQYHUL¿FDWLRQDQGDQDO\VLV R&D in areas related to networking and particularly for larger-scale innovative of design and code. Goals include: information technology. The framework software-intensive projects. In particular, (1) Effective evaluation for critical quality includes two areas that primarily relate to access to DoD-aligned expertise is attributes; (2) Components in large software producibility, which are Software important for the DoD to be able to take heterogeneous systems; (3) Preventive Design and Productivity (SDP) and effective action in the three areas of methods to achieve assurance, including +LJK &RQ¿GHQFH 6RIWZDUH DQG 6\VWHPV SUDFWLFHWKDWDUHLGHQWL¿HGDERYH$FFHVV process improvement, architectural (HCSS). There is also a third area, Cyber to DoD-aligned expertise has been an building blocks, programming languages, Security and Information Assurance area of ongoing challenge to the DoD, coding practice, etc. (CSIA) that encompasses some activities with recommendations made by various 3. Process support and economic related to software producibility. panels and committees since the 1980s. models for assurance. Goals include: The committee undertook a The need to reinvigorate (1) Enhanced process support for assured longitudinal study of sponsored R&D DoD software engineering software development, (2) Models for EXGJHWV DV LGHQWL¿HG LQ 1,75' UHSRUWV research evidence production in software supply ZLWK VSHFL¿F IRFXV RQ 6'3 DQG +&66 chains, (3) Application of economic In addition to recommending It found that while NITRD overall has principles to process decision-making improvements to the three areas of grown over the past decade, there has practice, as outlined above, the committee 4. Requirements. Goals include: EHHQ D VLJQL¿FDQW UHGXFWLRQ LQ ERWK (1) Expressive models, supporting tools LGHQWL¿HG VHYHQ DUHDV RI VXSSRUWLQJ overall and DoD-sponsored R&D in SDP research for consideration by science for functional and quality attributes; and HCSS. The committee recommends and technology program managers (2) Improved support for traceability and that DoD take immediate action to (managing 6.1, 6.2, and 6.3a funds and early validation reinvigorate its investment in software HTXLYDOHQW 7KHVHDUHDVDUHLGHQWL¿HGRQ 5. Language, modeling, coding, and producibility research, with focus in the basis of four criteria: (1) Advances tools. Goals include: (1) Expressive WKHVHYHQLGHQWL¿HGDUHDV ZRXOG\LHOGVLJQL¿FDQWSRWHQWLDOYDOXHIRU programming languages for emerging DoD software producibility. (2) A well- challenges, (2) Exploit modern managed research program would result concurrency: shared-memory and scalable in feasible progress. (3) The goals are not distributed, (3) Developer productivity DGGUHVVHG VXI¿FLHQWO\ E\ RWKHU IHGHUDO for new development and evolution agencies. (4) The pace of development 6. Cyber-physical systems. Goals in industry or research labs would be include: (1) New conventional RWKHUZLVHLQVXI¿FLHQW architectures for control systems, In each of the seven areas, the (2) Improved architectures for embedded FRPPLWWHH LGHQWL¿HG VSHFL¿F JRDOV IRU applications research and technology development 7. Human-system interaction. Goals that, in its judgment, could feasibly meet include: (1) Engineering practices for the four criteria. The areas and, for each, systems in which humans play critical WKH LGHQWL¿HG JRDOV DUH VXPPDUL]HG roles. (This area is elaborated in a below. (Details are in the full report.) separate NRC report.) 1. Architecture modeling and 8QGHUWKHDXVSLFHVRIWKH2I¿FHRI architectural analysis. Goals include: Science and Technology Policy (OSTP) (1) Early validation for architecture and the National Science and Technology decisions; (2) Architecture-aware systems Council (NSTC), there is a National management, including: Rich supply &RRUGLQDWLRQ 2I¿FH IRU WKH 1HWZRUNLQJ chains, ecosystems, and infrastructure; and Information Technology Research

The Next Wave Q Vol 19 No 1 Q 2011 49 Cyber-Physical Systems (CPS]

ver its brief history, most of the computer science and engineering field has focused on systems (e.g., the Internet and Web) that enable humans through information, communication, Oand knowledge. Just as the first wave of desktop and high-performance computing technology revolutionized the way people interact with information and with each other, the second wave will revolutionize the way humans interact with their physical environment.

Our vision is one of fundamentally engineered physical systems-as well as scales. They will need to be predictive, cyber-physical systems that exhibit their interactions with human participants. reactive to conditions and external events deeply integrated computational and What will such future systems with predictable and accurate timing, and physical capability, interacting with be like? Every system action will be receptive to coordination and (private) humans through many new modalities. In engineered to exploit both cyber and negotiation. Control loops may need to this future, the ability to interact with, and physical capability, deeply integrated be closed at various levels and scales. expand capabilities of, the physical world throughout the system. Systems will Topologies may adapt and reconfigure. through computational means will be the interact with humans in entirely new Cyber-physical systems (CPS) will have key technological multiplier. Individual ways, sharing authority. They may be to be fault tolerant and recoverable, precursors are seen in the control of highly tailored to the requirements and satisfying potentially very high inherently unstable systems such as.flying needs of individual users and uses, hence availability and timeliness requirements. wings and other extreme-performance highly heterogeneous. These systems CPS is a vision then for developing aircraft, automobiles with hybrid gas­ will be extensively, even ubiquitously, a scientific and engineering foundation electric or hydrogen-electric car engines networked. The majority of the systems for routinely building cyber-enabled and enhanced vehicle stability systems, will be configured from cooperating engineered systems in which cyber fully autonomous urban driving, medical components that interoperate through capability is deeply embedded at all devices for deep brain stimulation, and a complex mechanical, electrical, scales, yet which remain safe, secure, prostheses that allow brain activity to biological, and/or chemical system, and dependable-"systems you can bet control physical objects. A rich field of coupled with a physical environment such your life on." The CPS challenge spans innovative research is envisioned that as a human. Many (perhaps most) systems essentially every engineering domain. It can advance human progress through will be safety-, life-, or mission-critical will require the integration of knowledge the tensor product of cyber (computing, and must be highly dependable, available, and engineering principles across communication, and control) technology and secure. They will exhibit complex many computational and engineering and the dynamics of natural and dynamics at many spatial and temporal research disciplines (computing,

50 Cyber-Physical Systems [CPS] FEATURE

networking, control, human interaction, evolvability. Yet, wide design margins modifications on the status quo will not learning theory, as well as electrical, both limit performance and may vanish in work!” Don Winter, Vice President for mechanical, chemical, biomedical, nano- the face of changing usage patterns. This Engineering and Information Technology, bioengineering, and other engineering lack of design discipline induces extreme Boeing Phantom Works, in a hearing disciplines) to develop a “new CPS risk in technology-impoverished sectors before the House Science Committee, science.” such as the electric power industry. called for “a national strategy in which The objective of an initiative would Impact/need for the CPS long-term CPS technology needs are be to establish unified foundations and initiative addressed by combined government and technologies, and exemplars for rigorous A new foundation is required for joint engineering of the cyber, physical, corporate investment.” future CPS. The existing science and and human aspects of systems. This A focused initiative in CPS is engineering base does not support the objective includes science and technology needed that would seek to maximize routine, efficient, and robust design for the engineering of cyber and physical human capability and well-being through and development of these inherently components that must be integrated to computationally enabled engineered and complex systems. Such complex systems constitute such systems. Additionally, physical systems. The goal would be to must possess trustworthy qualities that this objective includes the cyber- usher in a new era of CPS for which we are lacking in much of today’s cyber physical characterization of complex have end-to-end science and engineering infrastructures. Today we can produce environments and human action, within (at great cost and effort) exceptionally which such systems must operate and to principles. The extent to which such complicated systems. We lack, however, which they contribute. In contrast with advances are achieved will determine the scientific and engineering foundations today’s artisanal approach, our objective (and can transform) the course of US to securely, safely, and systematically is to build foundations, tools, and highly innovation; advancement of consumer understand, build, manage, and adapt CPS capable infrastructure for rigorous design health, safety, and security; and gov- that remain reliable as they interact across and engineering of 21st century systems ernment agency mission effectiveness. internal subsystems, with each other, with that are truly cyber-physical. human users, and with highly complex Today, CPS grand challenges are and uncertain physical environments. being articulated in many sectors (for The design complexity of x-by- example, net-zero energy buildings, a wire for complex systems already is smart grid, energy management systems outstripping safe engineering design and for petroleum-free energy, zero-fatality implementation. Also, the opportunities and zero-crash highway and vehicle for mischief in this generation of systems, zero-prototype manufacturing, technology will make today’s Internet and the wireless and highly automated security problems pale by comparison. The operating room of the future). These consequence is inefficient, unsound, and heavily computation-, control-, and potentially dangerous design outcomes, as communication-centric systems call for well as tedious, costly, and failure-prone a new, unified systems science and new design cycles. Certification is estimated engineering technologies imagined by to consume 50 percent of the resources the CPS initiative. In a keynote address required to develop new, safety-critical on the challenges of design automation systems in the aviation industry. Similar for emerging vehicle technologies, estimates are predicted for the medical Scott Staley, Chief Engineer, Hybrid & and automotive domains. Over-design Fuel Cell Technology Development for currently is the only path to safety and Ford Motor Company argued the need successful system certification, leading to to abandon ad hoc experimental design a mindset of optimizing for a narrow task approaches and find more rigorous instead of encouraging adaptability and methods, saying, “…incremental

The Next Wave „ Vol 19 No 1 „ 2011 51 CAREERS AT THE N ATIONAL S ECURITY A GENCY

Make a critical difference with what you know.

You already know that intelligence is vital to national security. But here’s something you may not know.

The National Security Agency is the only Intelligence Community agency that generates intelligence from foreign signals and protects U.S. systems from prying eyes.

If you have the professional skills or technical expertise to support this important mission, then explore NSA. At NSA you can experience a variety of opportunities throughout your career as you work on real-world challenges with the latest technology. You’ll also be able to maintain a good balance between work and family life, as well as enjoy a collaborative work environment with flexible hours.

You won’t find this kind of experience anywhere else. KNOWINGMATTERS

Excellent Career Opportunities in the Following Fields:

 Computer/Electrical Engineering  Cryptanalysis  Computer Science  Signals Analysis  Information Assurance  Business Management  Mathematics  Finance & Accounting  Foreign Language  Paid Internships,  Intelligence Analysis Scholarships, and Co-op >> Plus other opportunities

WHERE INTELLIGENCE GOES TO WORK®

Get the free App for your camera phone at gettag.mobi and then launch the App and aim it at this tag. Search: NSACareers WATCH THE VIDEO

U.S. citizenship is required. NSA is an Equal Opportunity Employer. All applicants for employment are considered without regard to race, color, religion, sex, national origin, age, marital status, disability, sexual orientation, or status as a parent. iN52281