Prototype of Fault Adaptive Embedded Software for Large-Scale Real-Time Systems Derek Messie, Mina Jung, and Jae C. Oh Department of Electrical Engineering and Computer Science Syracuse University Syracuse, NY 13244 [email protected], [email protected], [email protected] Shweta Shetty and Steven Nordstrom Institute for Software Integrated Systems Vanderbilt University Nashville, TN 37235 [email protected], [email protected] Michael Haney High Energy Physics University of Illinois at Urbana-Champaign Urbana, IL 61801 [email protected] Abstract gies based on rules capturing every possible system state. Instead, a distributed reactive approach is imple- This paper describes a comprehensive prototype of mented using the tools and methodologies developed by large-scale fault adaptive embedded software devel- the Real-Time Embedded Systems group. oped for the proposed Fermilab BTeV high energy physics 1. Introduction experiment. Lightweight self-optimizing agents em- bedded within Level 1 of the prototype are responsible We describe in detail a prototype for the data ac- for proactive and reactive monitoring and mitiga- quisition and analysis components for the trigger- tion based on specified layers of competence. The ing and data acquisition system for the proposed arXiv:cs/0504109v1 [cs.SE] 29 Apr 2005 agents are self-protecting, detecting cascading fail- BTeV (http://www.btev.fnal.gov/) system, a par- ures using a distributed approach. Adaptive, reconfig- ticle accelerator-based High Energy Physics (HEP) urable, and mobile objects for reliablility are designed experiment system at the Fermi National Labora- to be self-configuring to adapt automatically to dynam- tory. This system consists of a very large number of ically changing environments. These objects provide Digital Signal Processors (DSPs) and General Pur- a self-healing layer with the ability to discover, diag- pose Processors apart from other hardware compo- nose, and react to discontinuities in real-time processing. nents like Field Programmable Gate Arrays (FPGAs) A generic modeling environment was developed to facil- and pixel detectors and sensors. In order to build soft- itate design and implementation of hardware resource ware for the upcoming BTeV hardware, we formed specifications, application data flow, and failure miti- a Real-Time Embedded Systems (RTES) collabora- gation strategies. Level 1 of the planned BTeV trigger tion, whose responsibility is to develop lower-level system alone will consist of 2500 DSPs, so the num- real-time embedded intelligent software to ensure ber of components and intractable fault scenarios in- system integrity, fault-tolerance, as well as intelli- volved make it impossible to design an ‘expert system’ gent diagnosis and recovery to process data gener- that applies traditional centralized mitigative strate- ated by collisions of physics particles in extremely high data-rate environments (approximately 1.5 Ter- per describes the design and development of the pro- abytes per second). Given the complexity of the totype in detail. system, the goal is to develop tools and method- The rest of this paper is organized as follows. Section ologies that are self-* (self-configuring, self-healing, 2 provides some background on the BTeV experiment self-optimizing, self-protecting) as possible. and the RTES collaboration. The RTES system devel- The BTeV trigger system has three levels, namely opment environment is then presented in Section 3, in- Level 1 (L1), Level 2 (L2), and Level 3 (L3). L1 con- cluding an overview of VLAs and ARMOR. The vari- sists of approximately 2,500 DSPs that process data ous system modeling tools developed within the collab- collected from sensors. L2 and L3 are approximately oration, along with an explanation of how each is used 2,500 Linux machines for processing the data passed for design and implementation is also detailed. through L1 processors. In all three levels, processing Section 4 describes the prototype that was presented the data collected from sensors is the most important at SuperComputing 2003 (SC2003). Design motivation work, which is carried out by High Energy Physics is discussed, followed by software and hardware spec- (HEP) applications1. Due to the high-speed data rate ifications. The embedded VLA design and implemen- and enormous amount of data, the system has to be dy- tation for the prototype is detailed, along with an ex- namically fault-adaptive and self-correcting. planation of the Experimental Physics and Industrial Very Lightweight Agents (VLAs) [14] are embed- Control System (EPICS) used to inject system faults ded within L1 as simple software entities which can and monitor VLA mitigation and overall system be- be implemented in a few dozen lines of assembly lan- havior. Lessons learned are also provided. guage, and take advantage of the exception-signaling Finally, future efforts planned for the next phase and interrupt-handling mechanisms present in most of prototype development are described, followed by DSP kernels to expose errors in the kernel behavior. a conclusion. VLAs consist of a proactive part and a reactive part to provide fault tolerance in the form of intelligent er- 2. Background and Motivation ror detection, diagnosis, and recovery. The proactive 2.1. RTES/BTeV part of VLAs can further be divided into a mandatory part and an optional part. BTeV is a proposed particle accelerator-based High When the VLA detects (e.g., by monitoring DSP ex- Energy Physics (HEP) experiment currently under de- ception signals) an error condition, it may take fault velopment at Fermi National Accelerator Laboratory. mitigative action directly, or notify appropriate higher The goal is to study charge-parity violation, mixing, level components, which may take appropriate actions and rare decays of particles known as beauty and such as disabling the execution thread or discarding charm hadrons, in order to learn more about matter- the current data item. A similar mechanism will be ex- antimatter asymmetries that exist in the universe to- plored for the monitoring and reporting of deadlines, day [11]. When approved, the BTeV experiment will be traffic, processor loads, etc. sponsored by the Department of Energy. The fault tolerance and performance-oriented ser- vices offered at L2/L3 will be encapsulated in intelli- gent active entities (agents) called ARMORs (Adap- tive, Reconfigurable, and Mobile Objects for Reliabil- ity) [9]. ARMORs are, by design, highly flexible pro- cesses, which can be customized to meet the runtime needs of the system. A prototype for the BTeV L1 trigger system has been built on DSP boards consisting of 16 Texas In- strument DSPs. The prototype includes L1 VLAs, AR- MORs, and the Experimental Physics and Industrial Control System (EPICS). It exhibits several fault adap- tiveness and tolerance behaviors. The prototype pro- vided us a great opportunity to realize the ideas and concepts to a real-working hardware platform. This pa- Figure 1: BTeV pixel detector layout. 1 HEP applications are also called physics applications (PAs) The BTeV experiment will operate in conjunction pression is performed, it is expected that the resulting with a particle accelerator where the collision of pro- data rate will be approximately 200 Megabytes per sec- tons with anti-protons can be recorded and examined ond. The events that are actually accepted within this for detached secondary vertices from charm and beauty system occur very infrequently, and the cost of oper- hadron decays [7]. The layout for the BTeV detector is ating this environment is high. The extremely large shown in Figure 1. streams of data resulting from the BTeV environment The experiment uses approximately 30 planar sili- must be processed real-time with highly resilient adap- con pixel detectors to record interactions between col- tive fault tolerant systems. For these reasons, a Real- liding protons and antiprotons in the presence of a large Time Embedded Systems Collaboration (RTES) was magnetic field. The pixel detectors, along with read- formed with the purpose of designing real-time embed- out sensors are embedded in the accelerator, which are ded intelligent software to ensure data integrity and connected to specialized field-programmable gate ar- fault-tolerance within this data acquisition system. The rays (FPGAs). The FPGAs are connected to approxi- collaboration includes team members from Fermi Lab, mately 2,500 digital signal processors (DSPs). Syracuse University, Vanderbilt University, University The measurements of the interactions resulting from of Illinois at Urbana-Champaign, and the University of the collision of protons and antiprotons are carried via Pittsburgh. custom circuitry hardware to localized processors that reconstruct the 3-dimensional crossing data from the 3. RTES System Development Environ- silicon detector in order to examine the trajectories for ment detached secondary vertices [13]. These detached ver- An overview of the BTeV system design and run- tices are indicators of the likely presence of beauty or time framework is shown in Figure 2. There are four charm decays. primary components, including very lightweight em- BTeV will operate at a luminosity of 2x1032cm−2s−1 bedded fault mitigation agents (VLAs), adaptive, re- corresponding to about 6 interactions per 2.53 MHz configurable, mobile objects for reliability (ARMOR), beam crossing rate [11]. Average event sizes will be a generic modeling environment (GME), and a system around 200 Kilobytes after zero-suppression of data is operator interface (EPICS). performed on-the-fly by front-end detector electronics. EPICS (http://www.aps.anl.gov/epics) provides an Every beam crossing will be processed, which translates interface for injecting faults into the system, which al- into the extremely high data rate of approximately 1.5 lows for evaluation of the effect of individual fault sce- Terabytes of data every second, from a total of 20x106 narios on the BTeV environment. It provides a way for data channels. operators to monitor and control overall system behav- A three tier hierarchical trigger architecture will be ior. Details and screenshots of the EPICS interface are used to handle this high rate.