Title page - CLEO Baltimore, May 9, 2007 Software Engineering Processes Used to Develop the NIF Integrated Computer Control System* A. Peter Ludwigsen
2007 International Conference on Accelerators and Large Experimental Physics Control Systems, Knoxville, TN, October 14, 2007 Lawrence Livermore National Laboratory, USA
1 * This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. NIF is a stadium-sized facility that will contain • 192-beam, 1.8-Megajoule, 500-Terawatt, ultraviolet laser system • 10-meter diameter target chamber with room for nearly 100 diagnostics
2 National Ignition Facility
“Quads” (4 beams) and “Bundles” (8 beams) are the basic building blocks of the NIF
3 NIF’s 192 energetic laser beams will compress small deuterium-tritium fusion targets to conditions where they will ignite and burn, and achieve energy gain
4 The Integrated Computer Control System (ICCS) robustly operates a complex physics facility
•• ControlControl 60,00060,000 pointspoints ¾¾ 1,500 processes 1,500 processes AutomaticAutomatic controlcontrol ofof 192-beam192-beam shotsshots ¾¾ 850 computers 850 computers areare overseenoverseen byby 1414 operatoroperator stationsstations ¾¾ 140,000140,000 distributeddistributed objectsobjects
•• AlignAlign andand firefire 192192 laserslasers ¾¾ << 5050 micronsmicrons onon targettarget ¾¾ << 3030 psecpsec timing timing
•• AutomaticAutomatic shotshot controlscontrols ¾¾ << 4-hours4-hours shot-to-shotshot-to-shot ¾¾ Hands-offHands-off operationoperation
•• AssureAssure ¾¾ SituationalSituational awarenessawareness ¾¾ MachineMachine protectionprotection
5 ICCS is required to automatically align, fire and diagnose laser shots every 4 hours
Fire pre- Fire main amplifier amplifiers S Amplifier cooling S S h h h o o o t Archive Shot Preparation t Verify t Archive
30-m 20-m 2-s
4 hours
Automated Shot Cycle • Acquire campaign shot goals from laser physics model • Perform automatic alignment and wavefront correction • Configure diagnostics and laser performance settings • Conduct countdown (software:4-min and timing:2-sec) • Assess shot outcome and archive shot data
6 Summary of Key Technical Challenges
Key Challenges Solutions
Scale from single bundle to full NIF Partitioned into 24 independent bundles
Align 192 beams on target in 30 Robust parallel image processing of minutes machine vision sensors
Shots automated with model-based Fire laser shots in 4-hour cycles control software
Incorporated shot verification and Assure machine protection situational awareness features
Provide flexible controls capable of Created versatile plug-in software supporting NIF for decades framework using CORBA distribution
7 Software engineering and computer science were key to building NIF’s large-scale control system
• Software engineering — Open distributed architecture based on industry tools — Formalized design process — Configuration Management — Product Integration — Formal Verification Testing
• Computer science — Application frameworks developed in-house — Standardized coding techniques — Extensive peer reviews
Goal: Build team with the necessary experience and engineering rigor
8 A segmented and layered architecture was implemented to decompose the control system
9 ICCS software framework is an object-oriented toolkit to build large-scale control systems
10 Open software architecture supports multiple environments and simplifies technology evolution
Category Tool / Environment Usage
- Sun (Solaris) - Servers, FEPs w/o hardware Computers / - PowerPC VME (VxWorks) - FEPs with hardware Operating - X86 (WinXP, LINUX) - Consoles, image processing System - X86 PC104 (WinCE) - Target diagnostic controllers
- Java - GUIs, frameworks, commissioning tools - Ada - FEPs, servers and supervisors Languages - C - Embedded controllers - IDL - On-line data and image analysis - XML - Workflow models and scripts CORBA - JacORB - Java applications (open source) Distribution - ORBexpress - Ada applications - Eclipse - Java IDE (open source) - IBM Rational - Solaris, VXworks Ada IDE Software - Ada Core Technology - Windows, LINUX Ada IDE Development - ILOG - Graphics builder tools - UML - Object modeling Database - Oracle - Data-driven configuration, archives
AsAs technologytechnology evolves,evolves, CORBACORBA allowsallows ICCSICCS toto migratemigrate withwith itit
11 ICCS architecture permits each bundle to be built and commissioned independently of the others
A partitioned architecture helped ICCS eliminate scaling risks
12 Data-driven architecture adapts quickly in the field to suit operating conditions
• Flexible and responsive — Nominal values deployed along with new capabilities — Data modified as components are commissioned — Operators update equipment calibration
• Separate database instances map to the environment Development: Schema and initial data Integration: Validates data and schema Formal Test: Final verification Production: On-line use Aux. Facilities: Factory and maintenance facility support
Software modeling and data-driven automation behavior supports flexible point-of-use commissioning and optimization
13 Shot Director Supervisor
14 15 Rigorous quality controls assure delivery of reliable software to the facility
The QC process resolves software problems early, with more than 90% of problems found prior to commissioning or shot operations
16 Incremental development strategy delivered functionality, while also iteratively managing risk
• Target high-level project milestones — Major release planned every 12 months — Over 60 minor releases and patches deployed per year
• Release content driven by — Project goals and identified capabilities — Defect resolution — Improvements requested by operation staff — Framework enhancements — Perceived risks
• Strict development lifecycle followed — Managers plan release and collect detailed requirements — Developers design, code, unit test and integrate — Separate test team performs offline and online tests
Goal: Support the overall project schedule while maintaining a manageable release content
17 Software metrics demonstrate the team’s adaptability to deliver flexible releases
Average Average Release Average Deployments Average Development Integration Type Changes per Year Test Time Time Time
Major 540 1 1 y 1 m 1.5 m
Minor 469 3 12 w 2.5 w 4 w
Service 57 11 4 w 1 w 2.5 w Pack
Patch 7 50 4 d <1 d 2 d
Script & Several Hundreds NA NA NA Database per Day
18 Releases are actively managed by closely monitoring the change tracking system
Tracking System Database — Releases contain subsystems that are individually tracked — Subsystem leads estimate the work in hours — Change states tracked: – Assigned, Completed, Desk-checked, Integrated, Released
Active Management — Planning – Work assigned to each subsystem is analyzed – High levels of change requests indicate increased testing – Unrealistic developer workloads are replanned
— Release – Progress is tracked on a daily basis – Trends predict completion dates – Progress is easily understood – Release content continues to be adjusted as needed
The tool maintains data on past release performance, which is used to flag potentially unreasonable expectations
19 Status of Software Change Requests (SCR) predicts the release completion date
900
800
700
600
500 Integrated SCRs
400 Assigned Completed
300
200 Predicted to Desk Checked Formal Test 100 Integration Predicted Start to NIF 0
c c c n b b b b r r y y y n l l p p p t t t e e a an a a a a u u u Jul g g g e c c c De Mar S -D -D - -J -Fe -Fe -M - -M -M -J 2-J 9-J -Au -Au -Se -Se - -O -O 1-Jan8-Jan 9 5-Fe 9 6 5 2 9-Mar6-Mar2-Apr9-Apr 7-May4-M 4 1-Jun8-Jun5-Jun 16-Jul23-Jul30- 6-Aug3-Au 3 1-Oct8-Oct 2 9 11 18 25 15-Jan22-J 2 12-Fe1 2 1 1 2 16-Apr23-Apr30-Ap 1 21 28 1 1 2 1 20 27 10 17 24-Sep 15-O2 2
20 Peer reviews enforce design and coding standards during the development phase
• Requirements Reviews — Held with hardware and operations Subject Matter Experts — Assures developers and testers understand work
• Design Reviews — Mandatory for new systems, or for complex changes
• Code Reviews — Desk Checks performed by another developer – Does code meet requirements? – Does code implement the design properly? — Walkthrough Inspections – Team review of critical code or key interfaces
¾ NIF’s 100% Desk Check policy catches 15% of total defects
Goal: Find defects earlier in the process, preferably before testing when cost to repair is lowest
21 Efficient developer testing is a combination of unit testing followed by formal product integration
• Unit Tests confirm code changes — Informal peer review — Developed to specific standards and practices — Automated tests are being developed
• Integration Tests ensure products work together — Tests interfaces — Verifies proposed database changes — Applies various test levels – Change verification – Regression checks – Shot sequences – Scaling and loading tests
22 Software releases are formally integrated in an “extreme programming” mode
• Led by the Product Integration Manager
• Integration is started before all coding is completed — Gets the team into test and fix mode
• Daily integration goals are published — Lower-level software functions verified first — Shot sequences evaluated last — Goals updated based on previous day’s results
• Extreme programming works for NIF — All developers on call (via pagers) — Development code base used for integration — Defects fixed on the spot where possible — Integration defects carefully prioritized
Goal: Produce a final integrated product with good quality
23 Independent verification demonstrates readiness
• Off-line Release Verification Testing — Verifies software for delivery to NIF – Critical software controls are validated – Issues reviewed with Responsible Individuals – All testing is documented — Performed in an environment similar to NIF – Mimics NIF hardware in a low-cost test environment — Initial operator training on new features — Operator checklists revised to conform to the new software
• On-line Deployment Verification — Verifies software performs correctly with NIF hardware — Verifies build configuration and production database changes — Final operator shot training
All software releases (and patches) undergo independent offline testing
24 Offline test bed uses real devices where practical, along with software emulators for scaling tests
25 Test bed simulators mimic operation of large- scale NIF Hardware
Target Positioner in NIF
Target Positioner Simulator
26 Code configuration is managed by a dedicated staff of specialists
• Code base managed across multiple environments — Support mixed languages, development tools, and platforms — Several releases are simultaneously active
• Automated tools validate build accuracy — Automatic regression checker identifies back-revisions — Automatic platform checker identifies version mismatches across platform code bases
• Online data/file changes are checked frequently to maintain conformity — Online data changeable only through a formal permit and approval process
Software and data configuration management are essential to produce predictable and traceable releases
27 The ICCS team includes diverse skills in software, controls, QC and laser technology
ICCS Staff Composition, 100 •• SoftwareSoftware skillsskills —— Object-orientedObject-oriented —— DatabaseDatabase Hardware, 21 —— UserUser interfacesinterfaces —— Real-timeReal-time —— ImageImage processingprocessing Database, 5 Software, 50 —— BuildBuild managementmanagement —— TestTest engineeringengineering Test, 13 —— QualityQuality assuranceassurance
QA/CM, 3 •• HardwareHardware skillsskills Managers, 7 —— MotionMotion controlscontrols —— TimingTiming systemssystems —— DataData acquisitionacquisition •• TeamTeam averagesaverages 2020 yearsyears inin fieldfield —— ComputersComputers && •• KeyKey experienceexperience drawndrawn fromfrom otherother physicsphysics networksnetworks projectsprojects andand relevantrelevant industriesindustries suchsuch asas —— ManufacturingManufacturing aerospaceaerospace
28 Software development of 1.8 million lines of code is greater than 85% complete
2000
1800 192 Beams
Target 1600 Multi-Cluster 1400 Multi-Bundle 1200 Automation 1000
800 Scripted Shots
600 Manual Shots Thousand Source Lines of Code 400 Main Laser
Injection Laser 200 Labs 0
8 9 0 0 1 1 2 3 3 4 5 6 6 7 7 8 8 9 9 -98 0 0 -01 -02 0 -03 -0 -05 -0 0 0 0 - t -99 - -02 n -04 n n - - c ct- u Oct-97Feb Jun-98O Feb Jun-9 Oct-99Feb Jun-00O Feb-0 Jun-0 Oct Feb Ju Oct- Feb-0 Jun-0 Oct Feb Ju Oct-04Feb-0 Jun-05Oct Feb-0 J Oct-06Feb Jun-07Oct- Feb-0 Jun-0 Oct-08Feb
29 ICCS successfully supported over 400 shots while the software was under construction
30 The control system scales and operates all bundles in parallel 31 The focus now is on automated shots with target-area systems including final optics, positioners and diagnostics
32 33 Large control systems can be developed on time with expectation of consistent reliability
• Architecture — Segment, partition, layer, and (open) distribution — Common frameworks, design patterns and code templates ¾ Address constraints and scaling concerns
• Development practices — Careful planning under authorized change control — Configuration management of code base and data — Diverse, experienced, and specialized staff ¾ Resolve risks as well as urgent issues
• Quality controls — Peer reviews and unit testing — Intensive product integration — Independent formal testing ¾ Measure quality to guide corrective actions
NIF continues on track for project completion in 2009, followed by National Ignition Campaign experiments beginning in 2010
34 35