Flight Standards Service

Alaskan Region, Flight Standards Division

Report to the Manager, AAL-200

System Safety Evaluation

Capstone Phase 1 Implementation

Yukon Kuskokwim Delta, Western

System Safety and Analysis Branch, AAL-240

March 5, 2003

Page 1 of 6 Executive Summary

Purpose. To evaluate Alaskan Region, Flight Standards Division effectiveness and participation in the Capstone Phase I implementation and operational approval process.

Background. Phase I of the Capstone project has been operational in Western Alaska since 1998. Twenty-seven FAR Part 135 operators have been using Capstone throughout the operating area with equipment installed in approximately 180 individual aircraft. In addition there are 15 Capstone equipped aircraft operating under Part 91. Phase I installations are primarily for VFR operations.

An initial purpose of Capstone was to reduce aircraft accidents and fatalities due to controlled flight into terrain (CFIT), and a review of the implementation process by Mitre® Corporation shows an overall safety improvement in the Capstone Phase I operations area. Accident statistics, however, do not show a reduction in CFIT accidents and the safety improvement in Western Alaska could not be directly attributed to Capstone. Data also suggested that not all pilots have been trained effectively in the proper application of Capstone and may not be using the equipment as designed.

Capstone Phase II involves the installation of additional advanced avionics and includes plans for IFR operations in Southeast Alaska. Phase II is scheduled to be implemented during the first quarter of calendar year 2003. Equipment installation is currently underway and training program modules are being evaluated for deployment to operators.

The Flight Standards Division Manager requested a systems evaluation of Capstone Phase I to determine the validity of reports that the program is not being used effectively. Lack of pilot training and non-standard use of equipment indicates that Federal Aviation Administration (FAA) Flight Standards responsibilities may not have been met during the training program and operational approval process.

Conclusions. Using questionnaires to survey FAA and operator personnel involved in Capstone Phase I, the System Safety and Analysis Branch determined that many operators did not develop comprehensive policy and procedure for the use of Capstone equipment. The result was non-standard use of the equipment by company pilots. The evaluation also indicated that operator training programs were not approved in accordance with FAA directives.

Recommendations. The following actions are recommended based on evaluation findings. 1. Establish national guidance for implementation and operational use of Capstone. Current guidance on technology related to the Capstone project should be reviewed and updated. 2. Establish effective liaison between Capstone Program Office and Flight Standards Division, AAL-200, which includes written procedure regarding that interface. 3. Standardize the approval process for use of the University of Alaska Anchorage Capstone training module. 4. Ensure certificate holders develop policy and procedure for use of Capstone equipment. 5. Provide mandatory Capstone program training for inspectors who are reviewing operator programs for acceptance or approval as appropriate. 6. Develop standards for using Capstone equipment during check flights. 7. Provide Capstone equipment operational technical training for inspectors who will be conducting check flights. This may require use of the regional King Air or rental aircraft properly equipped with Phase I or Phase II equipment as appropriate. 8. Ensure that the PTRS and SDR databases are used for Capstone related activities in accordance with national guidance. 9. Complete the TapRooT® analysis of causal factors identified through the SnapCharT® and Root Cause Tree® process. The process includes documenting and tracking proposed corrective actions for each causal factor as well as identifying the office and/or individual responsible for that corrective action.

Page 2 of 6 Purpose and Scope. Evaluate Alaskan Region, Flight Standards Division participation in the Capstone Phase I implementation and operational approval process to determine if Flight Standards responsibilities were met. Data gathered during the evaluation will be used to enhance phase II implementation.

Background. The Capstone project was initiated in 1997 by a congressional mandate to introduce advanced electronics systems to improve , particularly in the FAR Part 135 environment. A primary input to this effort was a study of operations in Alaska (Aviation in Alaska, NTSB/SS-95/03) that recommended using global positioning and satellite-based communications systems to provide an infrastructure that better meets Alaska’s air transportation system needs. An offshoot of Vice President Al Gore’s Flight 2000 program, Capstone was designed to reduce the fatal accident rate due to navigational errors and mid-air collisions. Under Phase I of Capstone, the Yukon Kuskokwim Delta in Western Alaska was selected to test the first generation of avionics and ground based electronics to improve safety and system efficiency.

The Delta is an area of extensive FAR Part 135 commuter operations originating from a central hub located at Bethel, Alaska. The operating environment is hostile most of the year and includes severe winter weather with low temperatures, icing, snow, high winds, and widespread whiteout conditions. Most operations are VFR using small, single engine aircraft with a high incidence of CFIT accidents throughout the area. The introduction of new situational awareness tools such as terrain and mid air collision warnings for VFR operations is expected to have the greatest impact in reducing accidents and improving safety.

Following deployment of Phase I in Western Alaska, a second wave of Capstone, Phase II, is being implemented in Southeast Alaska, where there is an equally hostile environment, but for different reasons. This area is comprised generally of coastal mountains with numerous valleys and fjords. Like Western Alaska, local communities are served primarily by air using small, single engine aircraft also with a high incidence of CFIT accidents due to low visibility, rain, and fog. Phase II uses more sophisticated technologies and IFR en route and approach operations.

After three years of Phase I operations, the Capstone Program Office (CPO) commissioned the Mitre® Corporation to collect data, review project accomplishments, and determine if primary objectives were met. In August 2002, the Mitre® Corporation published an interim report summarizing preliminary findings that indicated lack of pilot training and non-standard use of the Capstone equipment by pilots. With Phase II implementation pending in Southeast Alaska, it was decided that a complete systems evaluation of Capstone was warranted to determine if Flight Standards in the Alaskan Region was fulfilling its responsibilities, which include training program surveillance as well as operational approvals. Evaluation findings would first be applied to enhance Phase II and then be employed retroactively where appropriate in Phase I.

An evaluation team was formed with specialists from the System Safety and Analysis Branch (AAL-240) as the nucleus. The team included an Anchorage Flight Standards District Office principal operations inspector experienced in Capstone Phase I, along with operations and airworthiness inspectors from the Juneau FSDO, which will be the field office responsible for operational approvals for Phase II. Additional support was available from field supervisors and managers during the actual evaluation process.

Methodology. In the past several years, the FAA initiated a program to ensure that certificate holders apply the principles of risk management in their operations. This follows the concepts of system safety, which uses managerial techniques in a systematic manner to identify and control hazards. System safety has been introduced into FAR Part 121 operations through Air Transportation Oversight System (ATOS) and Certification Standardization and Evaluation Team (CSET) programs. The Alaskan Region is active in introducing system safety to local FAR Part 135

Page 3 of 6 operations and conducting a systems evaluation on Capstone provides opportunity to apply these principles to that program.

The first step was to identify the purpose of the evaluation in terms of functional requirements and existing problems. Data in the Mitre® report indicated a lack of effective pilot training and non-standard use of Capstone equipment. The evaluation team believed that if these two conditions actually existed, they reflected potential failure in program oversight since Flight Standards is responsible for approving training programs and operating authorizations. A systems evaluation would determine if these responsibilities were met during Capstone implementation.

The next step was to identify local stakeholders that affect Flight Standards with respect to Capstone Phase I in Alaska. Applying the principles of system safety, the team noted that the interfaces attribute points directly to where Flight Standards played an active role in the process by granting approval for certificate holders to use the equipment. The team determined that there were four key stakeholder groups located in Alaska whose direct exchange of information affected operational approval. The four stakeholders identified were the Capstone Program Office who developed the program, certificate holder management who exercised operational control, company pilots who used the equipment in flight, and Flight Standards inspectors who conducted field surveillance and granted approval in accordance with national policy. The team decided to conduct direct interviews with representatives from each stakeholder group to determine the scope of the interface and the effectiveness of the approval process. For the last step, the team used standard investigative techniques to design survey questionnaires and gather data.

At the time of the evaluation, there were 27 certificate holders using Capstone equipped aircraft. By reviewing the operator size, scope and accident history, the system evaluation team selected nine operators to be surveyed. In December 2002, team members interviewed representatives from the four stakeholder groups. Answers to questions were returned to an AAL-240 representative in the same form and manner as collected to assure accuracy in interpreting responses. The data was then sanitized, consolidated, and summarized before specific issues were identified and findings formulated. (Appendix 1, Detailed Summary of Surveys)

The following personnel were interviewed and provided data for the evaluation. A member of the Capstone Program Office staff provided information regarding implementation procedure. At least two of the three principal inspectors assigned to each certificate holder were interviewed, with some principals assigned to multiple operators. At least one of the required management personnel for each certificate holder was interviewed, either the Director of Operations or Chief Pilot. The Chief Operating Officer for one certificate holder was interviewed in lieu of the Director of Operations, who was not available. Lastly, twenty-one pilots were interviewed, with at least two pilots from each operator. One certificate holder was an exception and no company pilots were interviewed due to lack of availability during the evaluation period. A chance encounter, however, allowed one member of the survey team to interview pilots from an operator not listed on the original plan and that information was included.

To review and analyze system weaknesses, the AAL-240 branch employed a nationally recognized root cause analysis system developed by System Improvements, Inc., of Knoxville, Tennessee. This software program, called TapRooT®, enabled an in-depth, step-by-step review of the process and facilitated identifying causal factors for negative outcomes. The software then assisted the team in determining root cause and documenting corrective actions.

Analysis. Using data collected from the surveys, which supported the interim Mitre® report, AAL-240 developed a problem statement to initiate the SnapCharT® analysis.

“Pilots/operators are not fully using/implementing the Capstone program as intended”

Page 4 of 6 The process identified key actions and events in Capstone implementation as well as conditions that affected or influenced those events. Each event and condition was then analyzed individually for its impact on the problem statement. If an event or condition was determined to directly contribute to the problem, it was identified as a “causal factor”. Using the TapRooT® “SnapCharT®” software, this process was graphically charted in a logical progression toward the problem identified by the problem statement. Appendix 2 is the SnapCharT® graphic presentation of that evaluation.

After developing the graphic flow diagram, each identified causal factor was analyzed using a second TapRooT® software system, the “Root Cause Tree ®”. Like the SnapCharT®, the Root Cause Tree® is a structured process to determine and document where the “root cause” of each causal factor originated. Appendix 3 is a summary of the Root Cause Tree® report.

Concerns. Although the Capstone project and equipment received an overall favorable and enthusiastic response from operator management and pilots, survey results indicated that pilots were not fully using the Capstone equipment as designed. The following system weaknesses were identified.

1. National guidance was incomplete, confusing and conflicted with information published by the Capstone Program Office (Appendix 2, SnapCharT® Causal Factor CF1). a. Capstone Program Office information was not published through headquarters as required. b. Capstone Program Office information was sent directly to inspectors without going through the AAL-200 chain of command. c. PTRS data indicated that inspectors did not follow either the formal and informal guidance. d. Inspectors generally were unable to identify FAA or Flight Standards policy regarding the Capstone project, program implementation, or use of the equipment.

2. Avionics and maintenance inspectors received limited written policy and guidance on Capstone installations (Appendix 2, SnapCharT® Causal Factor CF2). a. Most operators’ aircraft equipment lists have not been modified to reflect the newly installed Capstone equipment. b. Most avionics and maintenance inspectors did not know the equipment had been initially installed. c. Airworthiness inspectors stated they had little if any involvement or responsibility for Capstone. d. Most airworthiness inspectors did not receive information on the application of the Capstone STC and many were not aware of the STC or program participation until the operator submitted paperwork for approval after the equipment was installed. e. MELs have not been modified to reflect Capstone equipment. f. Few inspectors were aware of specific Capstone procedures for reviewing manuals, training programs, operations specifications, and minimum equipments lists.

3. FAA approval for operator training programs was not accomplished in accordance with Order 8400.10 on most Capstone operators (Appendix 2, SnapCharT® Causal Factor CF3). a. Surveys indicated that many operations inspectors believed the Capstone office was performing training approval functions or that the training module was pre-approved when delivered to the operator by UAA. b. Interviews indicated that some training programs distributed to operators by UAA received initial approval without further review by the principal operations inspector. c. Many operations inspectors did not attend the training offered by UAA and Capstone. The training was not mandatory, attendance records were not kept, and inspectors that did attend did not initiate PTRS records for the training. d. Few inspectors were aware of specific Capstone procedures for reviewing manuals, training programs, operations specifications, and minimum equipments lists.

Page 5 of 6

4. The training module was not delivered to every pilot that used Capstone (Appendix 2, SnapCharT® Causal Factor CF4). The Capstone Program Office stated that the UAA training module was developed to comply with Order 8400.10, and would be delivered to company pilots through a train-the-trainer program. a. Most pilots stated that they received initial training but there was no company policy requiring them to use the equipment as trained. In most cases there has been no recurrent training. b. Most operators did not modify the training module to reflect company policy or procedures. c. Most operators did not audit/evaluate pilots for their ability to use Capstone. d. Operations inspectors have not required pilots to use Capstone during FAA check flights.

5. Most operators did not have written policy or procedures for using Capstone equipment (Appendix 2, SnapCharT® Causal Factor CF5). a. Most company management stated that policies and procedures were part of the training program and not in the manual system. b. Most company pilots were unable to identify company policy other than to turn it on when flying. Pilots stated in general they have used the equipment as they wanted to. c. There were few PTRS records found that indicated review of company manuals for policy or procedure related to Capstone. d. An analysis of a Capstone-equipped CE 207 crash in April 2001 reflected no company policy for Capstone and indicated improper use of the equipment by the pilot.

6. The SnapCharT® shows two equipment causal factors related to equipment development that were identified during prior investigation of a Capstone-equipped Cessna 207 crash in April 2001 (Appendix 2, SnapCharT® Causal Factors CF6 and CF7). These two factors, equipment displays and controls, were documented in the SnapCharT® because they related directly to the pilot’s decision-making process during that flight. Further analysis of these items was beyond the scope of this evaluation.

Recommendations. The following actions are recommended based on evaluation findings.

1. Establish national guidance for implementation and operational use of Capstone. Current guidance on technology related to the Capstone project should be reviewed and updated. 2. Establish effective liaison between Capstone Program Office and Flight Standards Division, AAL-200, which includes written procedure regarding that interface. 3. Standardize the approval process for use of the UAA Capstone training module. 4. Ensure certificate holders develop policy and procedure for use of Capstone equipment. 5. Provide mandatory Capstone program training for inspectors who are reviewing operator programs for acceptance or approval as appropriate. 6. Develop standards for using Capstone equipment during check flights. 7. Provide Capstone equipment operational technical training for inspectors who will be conducting check flights. This may require use of the regional King Air or rental aircraft properly equipped with Phase I or Phase II equipment as appropriate. 8. Ensure that the PTRS and SDR databases are used for Capstone related activities in accordance with national guidance. 9. Complete the TapRooT® analysis of causal factors identified through the SnapCharT® and Root Cause Tree® process. The process includes documenting and tracking proposed corrective actions for each causal factor as well as identifying the office and/or individual responsible for that corrective action.

Appendix 1, Detailed Summary of Surveys Appendix 2, SnapCharT®

Page 6 of 6