Firmware Forensics: Best Practices in Embedded Software Source Code Discovery

Firmware Forensics: Best Practices in Embedded Software Source Code Discovery | Michael Barr

By Michael Barr

Software has become ubiquitous. From microwave 2. Ask for the Bug List ovens to electronic throttle controls, it is now embedded into the fabric of our lives through billions At least in cases of alleged product liability, the of new products each year. Whether for infringement software development team’s defect database—or of intellectual property rights or product liability, when “bug list”--should be routinely requested and supplied products controlled by software are the subject of in discovery. litigation, it is imperative to analyze the embedded software, also known as firmware, properly and Very nearly every piece of software ever written has thoroughly. both known and unknown defects. During product development, thorough software teams will maintain This article enumerates six best practices for embedded a defect database--or “bug list”—to record problems software source code discovery and the rationale for observed in the software as well as track the status of each. The relative value and importance of the bug resolution. This list of fixed and known bugs—or individual practices will vary by type of litigation, so the the company’s lack of such a list—is germane to issues recommendations are presented in the order that is of software quality and provides helpful guidance to most readable. experts during the source code review process.

TIP: It is also recommended that a request be made for copies of software design documents, 1. Get the Source Code Early coding standards, build logs and associated tool outputs, testing logs, and other artifacts of the Any serious litigation involving embedded software embedded software design and development will require an expert review of the firmware’s human- process. readable source code. The source code should be requested early in the discovery process to 3. Insist on an Executable streamline the forensic review process, which could result in significant savings in time and money for In software parlance, the “executable” program is the both experts and litigators. binary version of the program that’s actually executed in the product. The machine-readable executable is In our experience, the owners of source code often constructed from a set of human-readable source code resist requests for its production. However, files using software build tools such as compilers and procedures limiting access to the source code by only linkers. a list of named and pre-approved experts with access under some sort of physical security--such as on a non- TIP: It is important to recognize that one set of networked computer with no removable storage -- source code files may be capable of producing tend to be agreed upon or ordered by a judge. multiple executables based on tool configuration and options.

Sometimes it is easy to extract the executable directly from the product for expert examination—in which case the expert should engage in this step. For experts.barrgroup.com | Copyright Barr Group. All rights reserved. Firmware Forensics: Best Practices in Embedded Software Source Code Discovery | Michael Barr

instance, software running on Microsoft Windows provided—reproduce the software build tools setup consists of an executable file with the extension .EXE, and compile the produced source code. which is easily extracted. However, the executable To do this it is necessary to have a copy of the programs in most embedded systems are difficult, at development team’s detailed build settings, such as best, to extract. make files, preprocessor defines, and linker control TIP: If the expert is able to extract an executable files. If the build process completes and produces an from one or more exemplars of the product, an executable, it is certain the other party has provided a automated comparison should always be made complete copy of the source code. between the installed and produced binary files. TIP: Experts may also need to be provided Executables can make a large impact in litigation. certaub third-party header files or libraries Though not human-readable, an executable program utilized by the developers. may provide valuable information to an expert If the executable as built matches the executable as reviewer. For example, one common technique is to produced—or ideally, extracted from the product—bit extract the human-readable “strings” within the by binary bit, it is certain that the other party has executable. The strings in an executable program provided a true and correct version of the source code. include information such as on-screen messages to the Unfortunately, trying to prove this part may take user (e.g., “Press the ‘?’ button for help.”). In a longer than just completing a build. The build could fail copyright infringement case once consulted by Barr to produce the desired proof for a variety of reasons. Group, several strings in the defendant’s executable To get exactly the same output executable, it is helpfully contained a phrase similar to “Copyright necessary to use all of the following: Plaintiff”—a simple yet important string that had a major impact on the case. • Precisely the same version of the compiler, It may also be possible to reverse engineer or linker, and each other build tool as the disassemble an executable file into a more human- original developers • readable form. Disassembly could be important in Precisely the same configuration of each of cases of alleged patent infringement. For example, in those tools situations that appear to be an infringement of a • Precisely the same set of build instructions method claim in the source code, the code in question Even a slight variation in just one of these details will may not actually be part of the executable in the generally produce an executable that doesn’t match product as used by customers, but rather, unused the other binary image at all—just as the wrong code. version of the source code would.

4. Reproduce the Development Environment 5. Try for the Version Control Repository

The dichotomy between source code and executable Embedded software source code is never created in an code and the inability of even most software experts instant. All software is developed one layer at a time to make sense of binary code can create problems in over a period of months or years in the same way that the factual landscape of litigation. For example, a bridge and the attached roadways exist in numerous suppose that the source code provided by a defendant interim configurations during construction. The was inadvertently incomplete in that it was missing version control repository for a software program is two or three source code files. In complex systems, like a series of time-lapse photos tracking the day-by- even an expert reviewer looking at the source code day changes in the construction of the bridge. But might not know about the absent files. And no expert there is one considerable difference: it is possible to go can find the bugs or infringing code in a missing file. back to one of those source code snapshots and Fortunately, there is a reliable way for an expert to rebuild the executable of that particular version. This confirm that all of the source code has been becomes critically important when multiple software versions will be deployed over a number of years.

It may not be possible to gain access to the entire Reason 2: In embedded systems, only one or more version control repository that holds all of the past features of the hardware are enabled or active when versions of a company’s firmware source code through the hardware is set to a particular configuration. discovery. For example, a judge may only allow the For instance, consider an embedded system with a plaintiff to choose one calendar date and to then network interface, such as an Ethernet jack that is only receive a snapshot of the defendant’s source code powered when a cable is mechanically inserted. In this from that specific date. If the plaintiff is lucky, evidence scenario, some or all of the software required to send of their intellectual property in that specific snapshot and receive messages over this network may not be will be found. Unfortunately, the observed absence of executed until a cable is inserted. their intellectual property from that one specific snapshot does not prove that the alleged theft did not Tip: To perform a thorough analysis of a system, happen earlier or later in time. firmware testing should be done on the There are some problems with the examination of an hardware as configured in exemplars of the units entire version control repository. For complex at issue. Therefore, it is useful to ask for systems, it may be difficult to make sense of the hardware during discovery if you are not able to repository’s structure. Or, if the structure can be acquire exemplars in other ways. understood, performing a thorough review of the Reason 3: There are times when a bug is not visible in major and minor versions of the various source code the software itself. Such a bug may result from a files may take multiple times longer than reviewing just combination of hardware and software behaviors or one moment in time. At first glance, many of those files multi-processor interactions. may appear the same or similar in every version—but subtle differences may be important to making a case. For example, one motor control system in a case To streamline the review process for high volumes of previously supported by experts from Barr Group had code with multiple versions, it may be necessary to a dangerous race condition. The bug was the result of obtain a chronological report provided by a bug list an unforeseen mismatch between the hardware and/or other production documents which describe reaction time and the software reaction time around a the source code at various points in time. sequence of commands to the motor.

6. Remember the Hardware 7. Hire the Right Experts

Embedded software is always written with the The best practices presented here are meant to hardware platform in mind. In embedded systems, establish the critical importance of making certain hardware and software are complexly intertwined and specific requests early in the legal discovery process. can be one of the most important considerations of all They are by no means the only types of analysis that when analyzing firmware. Therefore, the expert must should be performed on the source code. This and keep the hardware-software interaction in perspective other types of technical analysis should be well throughout the analysis process for the following understood by any expert witness or litigation reasons: consultant with the proper background. For cases involving embedded software, an software expert Reason 1: It is only possible to properly reverse should typically have a degree in electrical engineer or disassemble an executable program once engineering, computer engineering, or computer the specific microprocessor (e.g., Pentium, PowerPC, science plus years of relevant experience designing or ARM) is known. embedded systems and programming in the relevant language(s).

Barr Group provides testifying expert witnesses and software source code analysis teams to support complex litigation, including litigation involving product liability and infringement of intellectual property such as patents and software copyrights. CONTACT US experts.barrgroup.com | Copyright Barr Group. All rights reserved.