Int'l Conf. Frontiers in Education: CS and CE | FECS'15 | 149

Agile Systems Integration Process

Jason D. Winningham, David J. Coe, and Jeffrey H. Kulick Department of Electrical and Computer Engineering University of Alabama in Huntsville, Huntsville, Alabama, USA

requirements and ending in the verification and validation, Abstract - This paper presents an adaptation of an Agile that the code meets the requirements and the requirements development process specifically tailored towards software are the correct one for the system. However, as we all systems integration projects. The proposed Agile Systems know, system requirements change as a system evolves. Integration Process has been piloted in a senior-level Over the multi-year development cycle of a weapon system, software engineering studio course using an HO gauge for example, technology evolves and threats emerge that model of a Positive Train Control system as the cyber- were unforeseen at the time the program started. physical target hardware. The software developed by the students included five components: train scheduling, interactive display control, walk-about tablet manual controller, track geometry and connectivity input and safety monitoring software. Approximately 30 students were divided into ten teams with two teams competing to build each of the five components. This paper discusses what aspects of the proposed Agile systems integration process worked successfully and several important lessons learned to improve future performance.

Keywords: Agile, systems integration, positive train

control, agile in an educational environment

1 Introduction The classical approach to software development is Figure 1 – Classic Waterfall Process [1] called the Waterfall process, which is shown in Figure 1 A modified Waterfall model, called the spiral model below [1]. This paradigm is often used in industries that has evolved to allow for a cyclic evolution of complex have large complex systems that have to be sustained over products [4]. As shown in Figure 2, each phase is run to many years such as aircraft, weapons systems, and cyber- completion with a delivered product following which the physical systems. Oftentimes the process is proscribed by a requirements are re-evaluated, the design re-considered and certification organization such as the Federal Aviation new requirements and design elements added. During the Administration (FAA) for aircraft systems. The currently multiple cycles of the spiral process full-fledged artifacts are followed processes, DO-178C calls for explicit produced including requirements, design, code and test, and identification of the phases and artifacts of each stage and integration test documents. formal reviews either by the FAA or it’s representative While these processes are appropriate and even called a DER (Designated Engineering Representative) [2]. required by many industries, the new more responsive Agile Each stage of development has formally identified approach has been developed for applications that do not documents that must be produced as part of the states: have the need for detailed artifacts or reviews. Agile requirements document, design document, code and unit test development processes are often used by small teams of documents, integration documents. developers for short lifetime or short turnaround products At the production of each artifact, the quality of the such as cell phone applications or internal research and artifact is assessed. A formal review is performed including development projects. In an Agile process such as Scrum, a requirements review, design review, unit test review, requirements are replaced by user stories and the integration test, and independent verification and validation development cycle is very short, called sprints. Often these review. The goal of these quality control steps is to identify sprints are just a few weeks long. At each sprint the defects as early as possible since the defect removal cost development team consults with the customer, identifies part increases exponentially the longer the defect remains in the of the product that can be produced in the next sprint, and product [3]. develops and tests the code for that sprint. A key element of The advantage of the Waterfall approach is that much Agile processes is the delivery of an implemented and tested thought must be given to each stage, starting with the partial product at the end of each sprint. 150 Int'l Conf. Frontiers in Education: CS and CE | FECS'15 |

of developers. Below we first present a description of the senior software studio course, the cyber-physical system utilized during the pilot study, and the Agile systems integration process as employed in the pilot study.

2 Agile senior software studio course Our senior-level software engineering studio course was chosen to pilot the Agile systems integration process. This one semester course was originally created to provide an immersive team software development experience for undergraduate computer engineering students at the University of Alabama in Huntsville (UAH). In this course, teams of three to four students design, implement, and test a non-trivial software product. As the course was initially formulated, students employed a Waterfall methodology that included the preparation of several milestone documents including the Software Requirements Specification (SRS), Software Design Document (SDD), Figure 2 – Spiral development process [4] and Software Test Plan (STP) in addition to development and testing of the software itself. After several offerings of One huge advantage of the Agile approach is that the course, it became clear that one semester was too little defective or misunderstood requirements are very quickly time for some students to develop a high quality non-trivial identified since the intended interpretation of the product using a Waterfall methodology. requirements are revealed within a short period. There is no In early 2011, a more Agile process derived from question that defective or conflicting requirements are Scrum was adopted for use in the course. In developing this rapidly identified and remediated. process, it was important to retain some elements of a As development of large cyber-physical systems have Waterfall process such as traceability and test coverage evolved, which have traditionally used a Waterfall process, analysis since many graduates from UAH remain in north developers have been using modeling, simulation, and Alabama developing safety-critical systems such as hardware in the loop for systems of systems integration to helicopters, missiles, and radar systems. It was also deal with the misalignment of development times and important to reduce documentation demands so that students delivery dates for completed subsystems. So it is not had more time to implement and test their products. uncommon for a complex system to be built completely At the beginning of the course, students are divided virtually, instantiated in a hardware-in-the-loop framework into teams of three to four students. Every student on the (HWIL), and have component parts replaced piece by piece team develops his or her own complete proposal for a team- (when available). Even after the component parts are sized project that satisfies the list of project constraints, available, virtual implementations may continue to be used which include development of an open source product, use due to limited availability of hardware or software of real-time interactive graphics, the use of socket-based components. Thus complex missile systems may have networking, interaction with an SQL database, and started with some legacy hardware, simulation models of mandatory use of Subversion repository. Given the threats, sensor behavior, command and control components, constraints, students typically pursue development of a and synthetic threat model generators. As the system video game or simulation. The project proposal includes the progresses, components of the HWIL system will be following sections. replaced with real hardware/software as part of the system integration activities. Parts, such as threat simulations may • Project Overview remain in their initial form as Agile developed components • Market Research so that they may be rapidly adjusted to model evolving threats and sensor capabilities. So there is a need for • Fundamental Requirements systems of systems integration with both fully Agile • Storyboard developed components and mixed systems with traditional • Derived Requirements waterfall and Agile developed subsystems. o High-level use cases In this framework the University of Alabama in o Non-functional requirements Huntsville evolved an existing agile development senior software design studio into a systems of systems integration • Traceability experience using Agile-developed parts by multiple teams Int'l Conf. Frontiers in Education: CS and CE | FECS'15 | 151

After receiving feedback from the instructor on the document deliverables are updated and completed to match proposals, each team must select the project they want to the as-built product. pursue from the set of the project proposals produced by While the implemented and tested source code within members of that team. the Subversion repository was the primary course The modified Scrum process includes the following deliverable, students were also required to deliver a set of five steps, which are further explained below. requirements, which consisted of the set of as-implemented • Sprint planning use cases (updated from those in the sprint plan), as well as a test report containing the acceptance tests developed for • Sprint execution each sprint (from the sprint plan) and the test results (from • End of sprint live demonstration the sprint retrospective). Finally, as a measure of the • Sprint retrospective thoroughness of their testing efforts, students are also required to deliver a test coverage report that summarizes • Peer evaluation the amount of delivered code per module that remains During the semester, students have time to execute four to untested. five sprints. In the sprint plan, students must include detailed use 3 The cyber-physical system cases only for those use cases selected for the sprint backlog. Use cases not selected for the sprint backlog For the pilot systems of systems integration study, remain on the product backlog as high-level (unelaborated) students were asked to develop train scheduling, display use cases. Additionally, students must include a detailed set control, walk-about tablet manual controller, and safety- of acceptance tests for all use cases on the sprint backlog monitoring software for our newly enhanced Positive Train (test-driven development). Note that by requiring detailed Control (PTC) test bed. use cases instead of user stories, the use cases may serve as both requirements and as test scripts for conducting the end of sprint acceptance tests. At the end of each sprint, students must conduct a live demonstration of the new functionality implemented. For the live demo, the team must check out their source code from the Subversion repository, compile it, and execute the set of acceptance tests along with any additional tests suggested by the instructor or customer. Responsibility for conducting the demonstration rotates through the team members so that everyone must demonstrate competence and participation in the project. The traditional sprint retrospective provides an evaluation of the process as executed during that sprint. The retrospective for the modified Scrum process has been augmented with additional information including a labor report itemizing the time spent by each team member and productivity statistics such as source lines of code produced and use cases attempted versus completed. It also includes Subversion repository statistics gathered using the StatSVN utility to indicate who is or is not contributing to the project [5]. Finally, the retrospective includes a summary of the acceptance test results (pass/fail). At the end of each sprint, each student completes a

peer evaluation form (derived from [6]) in which they must assess the performance of each team member on four Figure 2 – Block diagram of system criteria: Quality of Work, Quantity of Work, Level of Teamwork, and Timeliness of Work. The performance The components of the system as identified in Figure 2 scale ranges from 1 (little or no contribution) to 5 include: (exceptional). Written comments are required to explain 1- a train scheduling system that computes routes for any below average assessment of 1 or 2. After completion trains to follow and issues commands to trains and of the source code development sprints, a final switches as the train progresses down the route; “documentation” sprint is performed during which 152 Int'l Conf. Frontiers in Education: CS and CE | FECS'15 |

2- a safety monitoring system that implements what is transponders located on each train [9]. The re-integration of tradiditionally called vital logic and is concerned fine grain localization into the enhanced PTC test bed has with local safety issues such as trains heading been left to the next offering of the software studio course. towards a collision; 3- a display system that displays for hte user the state of the tracks, train occupancy, train routes, switch settings, and safety violations. 4- a track layout system that allows the entry into the train control system layouts of tracks, switches, and detection sections, and 5- a tablet based controller for manual remote control of the layout's locomitives and switches.

As previously presented [7-10], the UAH Software Figure 4 – Panoramic photograph of PTC test bed [9]. Safety and Security Engineering Laboratory has developed HO gauge model train cyber-physical systems of increasing As a safety precaution, the operating rules of real train complexity to facilitate our teaching and research programs systems require that only one train may occupy a given in software safety and security. Our PTC test bed models block of track, except under certain special circumstances. real world positive train control systems currently being Figure 5 below shows one of the two Arduino boards that deployed nationwide in the United States. A key feature of comprise the safety system. Each block of track in the PTC these real world PTC systems is the centralized safety test bed has its own power supply that has been routed monitoring system that can remotely control trains in order through a relay mechanism to allow the safety system to shut to prevent accidents. off that block’s power supply to prevent an impending The five primary software components of the system collision. Figure 6 below shows the custom-designed wiring (routing, safety, display, layout and manual trablet control) consolidation circuit board (purple) that integrates multiple are interfaced to each other and the track via a SQL server off-the-shelf Digitrax controller boards (green) with the wiich maintains a set of SQL tables that reflect the track arrays of relays used by the safety system to shut off power status (as read from the track) and track commands (as to blocks of track. Thus, even with a defect in the issued by the various control components) scheduling software failure, the trains should never collide. The trains and track switches are controlled by and Short circuit protection is offered by fuses located in series observered by a set of Digtrax LocoNet hardware which with each relay. silultantously monitors locomotive location while issuing commands to the locomotives (speed, direction) and switches (through, bypass). The component that reaches across the SQL database and LocoNet hardware is called the SQL-LocoNet bridge. As shown in Figure 4 below, our PTC test bed has been recently re-engineered to increase the total number of individually-controllable blocks of track to over 100 blocks. The track layout consists of two different train systems, a main line track consisting of the outside ovals and rail yards and the interurban system that consists of the figure-eight loops connected by track that crosses the main line track at four crossover points. The added complexity of these crossovers present points of interference that must be accounted for in both the scheduling system and the safety- monitoring systems. An Arduino-based block occupancy detection and safety system has recently been integrated into the PTC test bed. Redundant occupancy detection is employed to prevent collisions in case of hardware failures. As with real-world Figure 5 – Photo of Arduino-based safety monitoring system trains, current sensing is employed to detect the presence of a locomotive within a specific block of track [11]. As previously reported, fine grain localization will be provided wirelessly by inertial measurement unit/tie counting Int'l Conf. Frontiers in Education: CS and CE | FECS'15 | 153

to have each of the teams performa an integration activity with the code that was developed after each sprint. Because the integration was managed by the SQL-LocoNet bridge, an asynchronoously updated database, students could mix and match old and new (tried true or developmental) components as they wished.

5 Observations and Discussion Although the systems integration plan seemed reasonable at the time, a number of issues arose that led to a somewhat different execution. Agile processes call for replacement of formal documentation by personal communication in the form of daily meetings and bi-weekly retrospective/prospective planning meetings. The students in the class are somewhat Figure 6 – Photo of safety system relay arrays. different than your typical undergraduates in that most of them have full time jobs in the local industrial and research community that is supported by the US Army's Redstone 4 Pilot Systems of Systems Integration Arsenal and NASA's Marshall Space Flight Center. Typically the only one-on-one physical meetings occurred Study after class which ended at 7pm two times a week. Generally the meetings that did occur were too short 4.1 Setup and too dense (30 students attended class regularly of the 32 As discussed earlier, the software development task enrolled). As a result the absence of formal documentation was divided up into five parts with an SQL database acting and the absence of regular inter and intra team meetings as the interface mechanism. By making the interface resulted in numerous teams moving forward with between the components a database, it was possible for each development before fully developed ICDs describing the team to develop and test their code as it evolved to ensure it interface were created. Conomintantly, students developed met the interface description of the database tables. The unit test methodologies for their personal incarnation of the large enrollment in the course (30+ students) resulted in a data base. Although several class sessions were dedicated to total of ten development teams of 3-4 students each. To the emerging bifurcation of the database standards, increase the likelihood that a fully functional, five part differences remained until the very end of class when system would be developed, two teams, an A team and a B integration testing was performed as a formal requirement. team, were assigned to develop competing implementations Another issue that arose was the continuous evolution for each of the five parts. of the target hardware. An early version of the hardware A key requirement for the fully integrated system was was available at the beginning of the term for student use. that all parts delivered must be interoperable with each other However, the final complete layout with all its whistles and so that all teams would have to interact with each other to bells was not available until the second half of the term. The develop appropriate component interfaces. For example, the students failed to appreciate the fact that such issues arise train scheduler developed by the A team must be frequently in the development of real world systems. interchangeable with the train scheduler developed by the B During the study, the university was also closed team in the fully-integrated system. Moreover, any part 1 unexpectedly due to several snow days. The university was component must be compatible with all parts 2 – 5. closed for up to two weeks equivalent workdays which was unprecedented in the previous 25 years history of the 4.2 Systems Integration Plan authors. Students used these periods of reduced communication to forge ahead on their own vision of the The intention at the onset of the project was to use interfaces, which resulted in more aggravation and breakage continuous integration which is a primary property of both of code. agile integration and traditional integration processes. In Finally, some methodologies used by students, such as continuous integration, components and builds of copy and paste of large code segments rather than writing a components are integrated as they become available. parameterized function or developing table driven code, Sketches of a component's capabilities are to be serially resulted in at least one team producing 25,000 source lines replaced by their ever refined version as time goes on. If of code. While the code was highly repetitive, each defect some component lags the others in developmental maturity, detected required the repetitive repair of each block of legacy components could continue to be used. The goal was copied code for each defect detected, which likely increased 154 Int'l Conf. Frontiers in Education: CS and CE | FECS'15 |

the probability that one of the students might have missed must be integrated on a continuous basis to enforce applying the repair in one or more places. acceptance of interface specifications rather than a wild west approach to interface design (my way or else). 6 Conclusions 7 Acknowledgements The Agile course upon which this paper is based is undergoing evolution as we speak and the success of the We would like to thank Dr. George Petznick, our endeavor has only been demonstrated a few days ago. Of the resident train expert, for his assistance in the design and 10 team development projects, 9 were completed all the way construction of the Positive Train Control test bed. through systems integration. The goal to have students develop a large (each component ran thousands of source 8 References lines of code) multicomponent project which involved component development and system integration based on [1] Stephen R. Schach, Object-Oriented & Classical agile processes was much harder than expected. Even Software Engineering, 7th edition, McGraw-Hill, 2007 though the course is ongoing, several important lessons have [2] RTCA DO-178C, “Software Considerations in been learned for future enhancements to the course. Airborne Systems and Equipment Certification”, December

13, 2011. Lesson Learned: Face-to-face communication must occur Students’ rapid and eager acceptance of low- [3] S.H. Kan, S.D. Dull, D.N. Amundson, R.J. Lindner, documentation processes was not coupled with the need for and R.J. Hedger, “AS/400 Software Quality Management,” much greater verbal and meeting mediated communication. IBM Systems Journal, vol 33, no. 1, 1994. Agile replaces detailed specification and design [4] B.W. Boehm, “A Spiral Model of Software documentation with face-to-face communication. The Development and Enhancement,” IEEE Computer, vol. 21, integration plan hinged on these face-to-face May 1988. communications to resolve component interface issues early on. Since some teams failed to conduct these face-to-face [5] StatSVN utility. http://www.statsvn.org/ discussions and other teams failed to abide by the decisions [6] David Kelly and Mary Sadowski, “Peer Evaluation made during these discussions, interface issues continued to within a Team Design Project”, ASEE Conference arise even in the last sprints. For interfaces between teams Proceedings the old fashioned signed off ICDs would probably have [7] David J. Coe, Joshua S. Hogue, and Jeffrey H. Kulick, been a better implementation. For processes that require "Software Safety Engineering Education," 2011 significant communication within and between team International Conference on Software Engineering Research members communication vehicles such as Skype and and Practice (SERP'11), WORLDCOMP 2011, July 18-21, Google drive should be set up at the very earliest stage. 2011, Las Vegas, NV. Moreover, points must be assigned to these communications as incentive for students to engage in them. [8] Travis Cleveland, David J. Coe, and Jeffrey H. Kulick, "Video Processing for Motion Tracking of Safety Critical Lesson Learned: All code is owned by the team Systems," 2013 International Conference on Software Students who have not worked in an environment Engineering Research and Practice (SERP'13), where breakage is a measure of the success of the project WORLDCOMP 2013, July 22-25, 2013, Las Vegas, NV. rather than the loss of an individual’s personal code [9] Scott Schiavone, Sjohn Chambers, Sunny Patel, Lee contribution to the project. Agile is predicated on far more Ann Hanback, David J. Coe, Jason Winningham, B. Earl rework and scrapping of codes that are developed earlier in Wells, George Petznick, and Jeffrey H. Kulick, "UAH the process. Students feel that each line of source code is OnTrack: Precision Navigation System for Research on The fully formed work product are loathe to scrap or rework Software Safety Issues of Positive Train Control," 2014 them. Agile processes gain their ability to identify and International Conference on Software Engineering Research rectify early errors only through much larger breakage in the and Practice (SERP’14), WORLDCOMP 2014, July 21-24, work product than processes with great planning and 2014, Las Vegas, NV. breakage should be looked upon as progress rather than lost [10] David J. Coe and Jeffrey H. Kulick, "Positive Train children. Control: Concepts, Implementations, and Challenges," 2014 International Conference on Software Engineering Research Lesson Learned: Practice continuous integration within and Practice (SERP’14), WORLDCOMP 2014, July 21-24, and across components. 2014, Las Vegas, NV. Continuous integration is required for success of agile [11] American Railway Association, The Invention of the based systems of system integration. Not only must each Track Circuit, New York, 1922. team develop working, testable product each sprint, these