Prediction of Foreign Object Debris/Damage Fod Type Based In

University of Texas at El Paso DigitalCommons@UTEP

Open Access Theses & Dissertations

2013-01-01 Prediction Of Foreign Object Debris/damage Fod Type Based In Human Factors For Aeronautics Using Logistic Regression Model David Ricardo Romo University of Texas at El Paso, [email protected]

Follow this and additional works at: https://digitalcommons.utep.edu/open_etd Part of the Aerospace Engineering Commons, and the Industrial Engineering Commons

Recommended Citation Romo, David Ricardo, "Prediction Of Foreign Object Debris/damage Fod Type Based In Human Factors For Aeronautics Using Logistic Regression Model" (2013). Open Access Theses & Dissertations. 1721. https://digitalcommons.utep.edu/open_etd/1721

This is brought to you for free and open access by DigitalCommons@UTEP. It has been accepted for inclusion in Open Access Theses & Dissertations by an authorized administrator of DigitalCommons@UTEP. For more information, please contact [email protected]. PREDICTION OF FOREIGN OBJECT DEBRIS/DAMAGE TYPE BASED IN HUMAN FACTORS FOR AERONAUTICS USING LOGISTIC REGRESSION MODEL

DAVID RICARDO ROMO Department of Industrial Manufacturing & Systems Engineering

APPROVED:

Bill (Tzu-Liang) Tseng, Ph.D., Chair

Jaime Sanchez, Ph.D.

Eric Smith, Ph.D.

Benjamin C. Flores, Ph.D. Dean of the Graduate School

by David Ricardo Romo

2013

To my Amazing Family

Beloved Girlfriend. PREDICTION OF FOREIGN OBJECT DEBRIS/DAMAGE TYPE BASED IN HUMAN FACTORS FOR AERONAUTICS USING LOGISTIC REGRESSION MODEL

DAVID RICARDO ROMO, Bachelor of Science

THESIS

Presented to the Faculty of the Graduate School of

The University of Texas at El Paso in Partial Fulfillment of the Requirements for the Degree of

MASTER OF SCIENCE

Department of Industrial Manufacturing & Systems Engineering

THE UNIVERSITY OF TEXAS AT EL PASO December 2013

Acknowledgements

First, I would like to thank God for the gift of life and for the entire blessing I have received in my life. I want to thank Dr. Bill Tseng for the opportunity of being part of this research and many other projects through my higher education. In addition, I would like to thank Dr. Jaime Sanchez for all his support and guidance; for sharing his dedication, knowledge, and passion for his work that highly motivated me to always give my best effort in this presented research. Moreover, I would like to thank

Dr. Alexander Eksir, and Mr. Guadamuz for considering me to develop this thesis.

Second, I want to thank all the student members of the Intelligent Systems Engineering Lab for their support and advice regarding this study. Besides, I would like to thank their motivation, jokes, and fraternity that always helped me to reduce the stress. It has been a pleasure to be part of this family that after the time and experiences we have lived together, I consider all of them brothers and sisters rather than workmates.

In addition, I would also like to thank my family, my parents Francisco and Alicia for their unconditional support, inspiration and motivation through my life. Thanks to my siblings Francisco,

Luis, and Alicia for their support and specially for being my role models since my childhood. Also, I would like to thank my closest friends for always cheering me up, for all their support, and all their words full of motivation.

Specially, I would like to thank my girlfriend Lauren. Thank you for not only being through the development of this thesis, but for being here during my entire college education. Thank you for all your support, motivation, and inspiration during all these years. Also, thank you for all the jokes and effort to cheer me up in the toughest moments. My life would not be the same without you.

Finally, I would like to thank all the people that in some way helped me through all these years of education, and especially during the development of this thesis.

v Abstract

Foreign Object Debris/Damage (FOD) has been an issue for military and commercial aircraft manufacturers since the early ages of aviation and aerospace. Currently, aerospace is growing rapidly and the chances of FOD presence are growing as well. One of the principal causes in manufacturing is the human error. The cost associated with human error in commercial and military aircrafts is approximately accountable for 4 billion dollars per year. This problem is currently addressed with prevention programs, elimination techniques, and designation of FOD areas, controlled access, restrictions of personal items entering designated areas, tool accountability, and the use of technology such as Radio Frequency Identification (RFID) tags, etc. All of the efforts mentioned before, have not show a significant occurrence reduction in terms of manufacturing processes. On the contrary, a repetitive path of occurrence is present, and the cost associated has not declined in a significant manner.

In order to address the problem, this thesis proposes a new approach using statistical analysis.

The effort of this thesis is to create a predictive model using historical categorical data from an aircraft manufacturer only focusing in human error causes. The use of contingency tables, natural logarithm of the odds and probability transformation is used in order to provide the predicted probabilities of each aircraft.

A case of study is shown in this thesis in order to show the applied methodology. As a result, this approach is able to predict the possible outcomes of FOD by the workstation/area needed, and monthly predictions per workstation. This thesis is intended to be the starting point of statistical data analysis regarding FOD in human factors. The purpose of this thesis is to identify the areas where human error is the primary cause of FOD occurrence in order to design and implement accurate solutions. The advantages of the proposed methodology can go from the reduction of cost production, quality issues, repair cost, and assembly process time. Finally, a more reliable process is achieved, and the proposed methodology may be used in other aircrafts.

vi Table of Contents

Acknowledgements...... v

Abstract ...... vi

Table of Contents...... vii

List of Tables ...... x

List of Figures ...... xi

Chapter 1: Introduction ...... 1 1.1 Objective ...... 1 1.2 Contribution ...... 2 1.3 Motivation ...... 2

Chapter 2: Problem Statement ...... 3

Chapter 3: Literature Review ...... 4 3.1 FOD Definition ...... 4 3.2 FOD Resources ...... 4 3.3 FOD Cases ...... 5 3.4 FOD Categories ...... 6 3.5 FOD Area Designation ...... 6 3.6 Human Factors in FOD...... 7 3.6.1 Dirty Dozen ...... 8 3.6.2 Worker Nature ...... 10 3.7 FOD Focus Areas ...... 11 3.7.1 FOD Detection ...... 11 3.7.2 FOD Prevention ...... 12 3.7.3 Training & Promotion ...... 15 3.8 Foreign Object Elimination ...... 16 3.9 Predictive Models ...... 17 3.10 Natural Logarithm of the Odds (Log Odds) ...... 18 3.11 Logistic Regression ...... 19 3.12 Validation Criteria ...... 24

vii Table of Contents (continued)

Chapter 4: Methodology ...... 26 4.1 Materials ...... 27 4.1.1 Software ...... 27 4.1.2 Quality Assurance Reports ...... 27 4.2 Methods ...... 27 4.3 Data Preparation ...... 28 4.4 Data Classification ...... 28 4.5 Identification of Factors...... 28 4.6 Contingency Tables ...... 29 4.7 Model Construction ...... 30 4.8 Model Validation ...... 31 4.9 Natural Logarithm of the Odds ...... 33 4.10 Probability Transformation ...... 35 Chapter 5: Case Study ...... 36 5.1 Data...... 36 5.1.1 Data Collection ...... 36 5.1.2 Data Preparation ...... 36 5.1.3 Data Preparation Summary ...... 38 5.2 Advanced Fighter ...... 38 5.2.1 Contingency Tables ...... 38 5.2.2 Model Construction ...... 40 5.2.3 Model Validation & Selection ...... 42 5.2.4 Natural Logarithm of the Odds ...... 43 5.2.5 Probability Transformation ...... 45 5.3 Large Cargo Aircraft ...... 47 5.3.1 Contingency Tables ...... 47 5.3.2 Model Construction ...... 48 5.3.3 Model Validation & Selection ...... 48 5.3.4 Natural Logarithm of the Odds ...... 49 5.3.5 Probability Transormation...... 51 Chapter 6: Conclusions and Recommendations ...... 53 6.1 Conclusions ...... 53

viii Table of Contents (continued)

6.2 Recommendation ...... 53 References ...... 55

Vita……………...... 89

ix List of Tables

Table 3.1: FOD Categories...... 6

Table 3.2: FOD Methods ...... 14

Table 4.1: Example of Pearson P-value ...... 30

Table 4.2: Example of Whole Model Test ...... 32

Table 4.3: Example of Generalized R-square and Misclassification Rate. JMP Portion ...... 33

Table 4.4: Likelihood Ratio Test ...... 33

Table 5.1: Variants per Aircraft ...... 37

Table 5.2: Data Preparation Summary ...... 38

Table 5.3: Advanced Fighter Month by FOD Type ...... 39

Table 5.4: Advanced Fighter Workstation by FOD Type...... 39

Table 5.5: Advanced Fighter Variant by FOD Type ...... 39

Table 5.6: Advanced Aircraft Significant Variables ...... 40

Table 5.7: Advanced Fighter. Output from Models ...... 42

Table 5.8: Advanced Fighter Input Variables and Estimates ...... 44

Table 5.9: Large Cargo Aircraft Month by FOD Type...... 47

Table 5.10: Large Cargo Aircraft Workstation by FOD Type ...... 47

Table 5.11: Large Cargo Aircraft Significant Variables ...... 48

Table 5.12: Large Cargo Aircraft Output from Models...... 49

Table 5.13: Large Cargo Aircraft Input Variables and Estimates ...... 51

List of Figures

Figure 3.1: Concorde Taking Off With a Malfunction Caused by Titanium Debris ...... 5

Figure 3.2: Human Factor Accident Scenario ...... 9

Figure 3.3: Radar Detection Technology ...... 12

Figure 4.1: Methodology Flow Diagram ...... 26

Figure 4.2: Example of Mosaic Plot ...... 29

Figure 4.3: Example of JMP Model Specification Window ...... 31

Figure 5.1: Actual JMP Model Specification Window...... 41

Figure 5.2: Advanced Fighter Monthly Predicted Probabilities for Final Assembly 1 ...... 46

Figure 5.3: Large Cargo Aircraft Monthly Predicted Probabilities for Workstation 2 ...... 52

Chapter 1: Introduction

Foreign Object Damage/Debris (FOD) has being the most important issue for military and commercial aircrafts. It costs to commercial airlines an average of 4 billion dollars a year. This includes damage to the aircraft, such as blade damage; tire damage and reposition, body damage, loss of aircraft, flight delays/cancelation, and loss of customers. In summary, the consequences the current problem is leading to production delays, an increase in production cost, and a decrease in the quality o the product.

FOD causes not only damage to aircrafts but also the loss of human lives. FOD is most commonly found during manufacturing processes and runways. Today, there exist a decent number of technologies specially designed for FOD detection in runways and taxiways. On the other hand, there is no evidence found in prevention or detection technologies regarding FOD during manufacturing and assembly processes of the aircraft. Human error has being identified as one of the primary causes of presence of FOD in manufacturing. There exist different policies, trainings, guidelines adopted by all different aircraft manufacturers across the U.S. but nothing really effective in foreign object elimination inside the manufacturing processes.

1.1. Objective

At present, there is no evidence of a statistical approach that has been found to address the FOD issue in a Human Error perspective. This study seeks to develop a new approach to address the problem.

This will cover the analysis of historical data and the application of the methodology to be discussed, and the benefits for aerospace industry. The proposed approach should enable the aircraft manufacturer to predict the FOD type for given conditions. A more specific scientific objective and contribution of this research are to develop a predictive model based in historical discrete data to estimate the

1 probabilities of FOD type and occurrence of it. Although, the proposed methodology may be applicable to other types of aircrafts and likewise for other aircraft manufacturers.

1.2. Contributions

This thesis covers the data preparation, selection of significant factors, and the chosen methodology to conduct the research. Moreover, this thesis also includes the application of the methodology in a real world scenario based in real manufacturing data. Additionally, this thesis can serve as a professional guidance for the aerospace industry in order to address the FOD problem.

Benefits of the proposed methodology application can be a significant reduction of FOD occurrence, an accurate identification of workstation/areas of more recurrent FOD, and a significant cost of defect and production reduction, and as a consequence, an increase of profit is feasible.

1.3. Motivation

Human error has being the primary cause in FOD incidents where not only aircraft damage has being produced, but the loss of human lives during flight or combat missions. There is no evidence found of a similar approach using statistics to address the current issue. The primary motivation of this research is the large consequences that human error has, and the monetary cost associated with the problem as well. I would like to address the actual target that has been a large problem during the past years. It is important to clarify that aircrafts are becoming more sophisticated and complex in the actual days. As a consequence, manufacturing processes have become more multifaceted for the personnel who have a close interaction with the aircraft, leaving the open gap for an error to occur. Having said so, I would like to address this huge problem using statistics to predict FOD occurrence. Additionally, this thesis would serve as a start point of the usage of statistics regarding human error in FOD prediction and elimination. Finally, the contributions in the study could be further analyzed and future contributions to the methodology may be added.

Chapter 2: Problem Statement

Aircraft manufacturers have a multi-billion FOD recurrence. Besides, as the aircrafts are becoming more sophisticated and complex, the cost of FOD detection and removal is rapidly increasing.

Aircrafts Manufacturers are concern in two of the most produced aircrafts with the highest FOD recurrence and cost among the variety of aircrafts. It is requested by the company that this study is conducted for the “Advanced Fighter” and “Large Cargo Aircraft”.

Moreover, since several FOD prevention and elimination programs have being implemented with no significant improvement, a new approach is needed. This research is focusing in the use of statistics and predictive models to significantly improve the efforts in prevention and elimination. The study is conducted by the use of estimates calculated by the natural logarithm of the odds and the transformation of these estimates to actual probabilities.

The main objective is to predict the FOD type focusing only in human factor in order to eliminate the human error leading to defect recurrence. As discussed in following chapters, human error has being present in the aircraft manufacturing business since its creation and it cannot be completely eliminated because of human nature. However, it is expected to reduce significantly the cost and frequency associated to FOD occurrence caused by human error.

Chapter 3: Literature Review

3.1. FOD Definition

FOD is defined as a substance, debris or article alien to a vehicle or system, which would potentially cause damage. Foreign Object Debris causes Foreign Object Damage (Feeler, 2002). Also it is defined as any damage attributed to a foreign object that can be expressed in physical or economic terms, which may or may not reduce the product’s safety and/or performance characteristics. Foreign

Object Debris (FOD) is defined as a substance, debris, loose hardware, or article alien to a vehicle or system, which could potentially cause damage (NASA, 2011). A Critical Foreign Object (FO) is defined as foreign objects in areas where migration is possible through tolling holes, intakes, etc., which are probable to cause system or component damage. In summary, Foreign Object Damage/ Debris and

Foreign Object are referring to any object(s) or substance(s) that does not belong or to the vehicle.

3.2. FOD Resources

FOD has been identified to come from many resources. Foreign objects are most commonly found in the open environment but many resources can cause them. The main three focus causes of foreign objects are airport infrastructure, aircraft operations, and personal belongies.

First, deterioration, maintenance and construction are key contributors to FOD. For example, pieces of concrete from pavement or fatigue corners, and building materials can easily be blown by the wind or carried by any vehicle from construction areas to airplane maneuver areas. These small objects can be easily suctioned by the intakes of the aircraft and cause potential damage to the vehicle and passengers.

Second, Refueling, catering, cabin cleaning, and baggage and cargo handling can produce broken materials. Also, maintenance aircraft operations play an important role since the maintenance crew utilizes small tools in order to perform their tasks. These small objects can easily fall from aircraft and remain in the taxiway leading to potential aircraft damage. 4

Finally, personal belongies emphasizes the personal items such as watches, pens, coins, etc.

There exists evidence of unintentional presence of personal items in runways and inside aircrafts. They are one of the most common foreign objects found inside the aircraft body. This is a key factor in the

FOD environment.

3.3. FOD Cases

There has been evidence of accidents caused by FOD. Some of the most popular FOD incidents recorded are the bird strikes, volcanic ashes, and the Concorde accident (Procaccio, 2008).On 24 June

1982, Flight 9 of British Airways flew into a volcanic ash cloud over the Indian Ocean. The Boeing 747-

200B suffered engine surges in all four engines until they all failed. Aircraft dived down until it exited the ash cloud allowing the airborne ash to clear engines, which were then restarted. The cockpit windshield was badly pitted by the ash particles but the aircraft landed safely.

On January 15, 2009, Flight 1549 of US Airways flew into a flock of Canada geese and suffered a double engine failure. The pilot ditched the aircraft in the Hudson River saving all the passengers lives. The crash of a Concorde, Air France Flight 4590, near Paris on July 25, 2000 was caused by a piece of titanium debris on the runway (FOD). As a result, all 100 passengers and nine crew on board the flight, as well as four people on the ground, were killed.

Figure3.1: Concorde Taking Off With a Malfunction Caused by Titanium Debris. 5

3.4. FOD Categories

In a manufacturing perspective, FOD has being identified as a very frequent defect. In order to address the problem, BOEING has identified and categorized the FOD issues into five different areas.

For the objective of this research, this FOD categorization shall be followed for the desired output.

Table 3.1 shows the five categories of FOD and the description of each one.

Table 3.1: FOD Categories.

FOD Type Examples

Manufacturing Debris Metal shavings, loose sealant, rivet tails. Panstock Washers, rivets, bolts, screws, pins.

Consumables Q-tips, caps, bags, tapes, rags, cleaners, string ties. Tools/Shop Aids Wrenches, sockets, screwdrivers, clamps. Trash Plastic wrap, paper, used tape, cardboard.

3.5.FOD Area Designation

Any area where flight hardware is in place and exposure to foreign objects would potentially cause a system or product failure due to deterioration, malfunction or damage. FOD sensitive areas are required to be designated based o the risk associated with a Foreign Object (FO) and the activities performed in the specific area. The risk shall take into account both the consequences and probability a foreign object will not be found and controlled.

Important leaders such as NASA, Boeing, and Lockheed Martin follow the previous model of area designation. They share the following division and definition for each one of them. The following

Area designation is taken from NASA, 2011 FOD Prevention Program. Area designation consists in four

6 different main areas. These areas are Non-FOD sensitive, FOD awareness, FOD control, and FOD critical areas.

First, Non-FOD sensitive is defined as n area where the risk associated with a foreign object is negligible and no FOD control measures are needed. These areas do not need special indications or restrictions in order to access them. They do not hold potential foreign objects neither they are related to the aircraft area.

Second, FOD awareness can be described, as an area where the risk associated with a foreign object resulting in hardware damage/contamination is low. A FOD prevention area where manufacturing/modification processes remains open without any potential FOD entrapment. This includes but is not limited to components or assemblies undergoing manufacturing or modification without any closeout activities on the product.

Third, FOD control is defined, as an area where the risk associated with a foreign object resulting in hardware damage/contamination is medium. Area where assembly or modification processes occurs.

This includes but is not limited to components or assemblies undergoing manufacturing or modification in the process of becoming a completed aircraft.

Finally, FOD critical is identified, as an area where the risk associated with a foreign object resulting in hardware damage/contamination is high. A FOD prevention area where assembly, modification and flight/ground operations require the highest level of preventative measures. The elimination of FOD contamination, entrapment, mitigation or damage is most critical to safeguard the product.

3.6. Human Factors in FOD

Human factors are the science whose goals are to identify, describe, predict, and control the performance of a human operator in a determined system. It is concerned with optimizing the

7 relationship between people and their activities by the systematic application of human sciences, integrated within the framework of engineering.

Human factors issues may be broken down into four main categories, which can be characterized in the SHEL model of Software, Hardware, Equipment, and Liveware. Each of these categories is directly affected by human interaction, which is the most flexible and adaptable part of the aviation system. (Procaccio, 2008)

The Boeing Corporation was the pioneer in addressing the Human Factors in the decade of

1960’s. They utilized their experts and specialists in the field, where the majority was pilots and mechanics, and introduced the Maintenance Error Decision Aid (Graeber, 1999). U.S. airlines rapidly followed the model, Northwest and Continental developed a human factors training program called

Crew Coordination Concepts (Johnson, 1997). Transport Canada developed the Human Performance in

Maintenance Program based on the Continental’s program. The focus of the program was a do-it- yourself, a training methodology for maintenance personnel, to improve awareness o the effects of human limitations in the aviation maintenance.

3.6.1. Dirty Dozen

One of the most important products of the Human Performance in Maintenance Program was the

Dirty Dozen. Transport Canada researchers identified the following factors as the causes of a person making an error in judgment, resulting in a flawed FOD process.

One or more of the following usually causes errors:

1) Lack of Communication, 2) Complacency, 3) Lack of Knowledge, 4) Distractions, 5) Lack of Teamwork, 6) Fatigue,

7) Lack of Resources, 8) Pressure, 9) Lack of assertiveness, 10) Stress, 11) Lack of Awareness, 12) Norms.

The presence of these factors can be found and attacked in any stage of the aircraft; manufacturing, flight test, aircraft and ramp maintenance. Besides, since its simplicity, the Dirty Dozen has become a popular training tool in aviation maintenance and safety seminars. (Cunningham, 2007)

Every incident is a cumulative result of more than one event. The holes in Figure 3.2 represent random events, as the slices rotate the holes line up providing a trajectory of accident opportunity. At this point, an accident is eminent. Therefore, it is paramount that when a FOD incident occurs, every possible step is taken to identify and correct the latent conditions, active failures and root causes, in order to prevent future events from happening.

Figure 3.2: Human Factor Accident Scenario.

The International Civil Aviation Organization (ICAO) defines human error as “the failure of planed actions to achieve their desired goal.” Human error is unintentional in the majority of the

9 reported FOD incidents. Assuming human error is unintentional, the situation can be categorized in two actions; execution failure and mistakes.

Mistakes are the operations that a technician performs as planned but the process is not adequate to achieve the desired result. On the other hand, execution failures are error that may result of actions that do not go as planned. An example would be an unintentional introduction of a foreign object (i.e. tool) to an aircraft during any manufacturing or maintenance process.

Failures can be further divided into active or latent. Active failures are the result of unsafe acts committed by individuals at the human system interface, and those actions might have immediate consequences. Latent failures are errors that only become evident when they combine with a trigger factor to overcome the system’s defenses. In most cases, latent failures are attributed to local factors that are present in the workplace and organizational factors.

3.6.2. Worker nature

The first step to apply human factors in the Industry is to understand the work force. It is important for an accurate analysis and application of human factors to define the characteristics of the technicians. Experts have defined the aviation technician characteristics as follows:

1) Dependable, 2) Loyal, 3) Possess Integrity, 4) Modest, 5) Follow written Instructions, 6) Independent, 7) Possess Reasoning Capabilities.

Taking these characteristics into account would help to the proper and accurate design of manufacturing processes, detection and maintenance programs. Better processes and programs are most likely to success in reduction of FOD incidents (Lock, 2011).

3.7. FOD Focus Areas

The following sections will cover the most relevant efforts found to address the actual problem.

The mentioned areas plays an extremely important role about the promptly address of defect recurrence.

They go in detail explaining and addressing the important aspects of FOD detection and prevention, and training and promotion as well.

3.7.1. FOD Detection

Adequate control of FOD at any facility begins with a regularly scheduled inspection and detection program. Close examination it is necessary to assess the condition or to discovery any irregularities. In this case, it would be in the form of actively conducing inspections to monitor and mitigate any threads. When an inspection is conducted, it is important to inspect areas where similar defects have been reported before. Foreign Objects may have relocated since it was reported. Structured inspections should be adopted by any facility, which deals with aircrafts such as airports and manufacturers. Detection is the bottom line in the elimination process. Detection in most manufacturing facilities, airports, and in the military, is done by humans. It consists of three different methods: manual, supplemental, and automated. Manual technique is commonly found as a human operator in a vehicle performing FOD search. Supplemental refers to a human-operated camera with the usage of mobile sensors making the foreign object easier to detect. These previous methods leave the gap open for one or more factors to be present and cause an incident.

Currently, automated systems have being used in order to have more accurate inspections for defect detections, and also to minimize the chance of human error. Some of the technology used is

Radar, electro-optical and Hybrid.

Radar technology utilizes millimeter-wave radar. It uses extremely high frequency in the range of 30 to 300 GHz. There are both fixed and mobile systems that incorporate millimeter-wave radar. The

11 mobile device can scan an approximate of 80 degrees in front o the vehicle. The antenna can scan up to

30 scans per minute and provides a detection distance of 650 ft.

Figure 3.3: Radar Detection Technology

Electro-optical technology utilizes electro-optical sensors. This technology features self- calibrating cameras, automated scene analysis, and configurable scan resolution for different object sizes. The remotely placed electro-optical sensors provide continuous surveillance of the runway surface

24 hours a day and seven days a week. Objects can be detected at night without supplemental illumination. Hybrid technologies use a combination of radar and electro-optical data as the primary means to detect FOD (Prather, 2011). This technology has show improvements in the detection of FOD and the majority of the airports in the world currently use it.

3.7.2. FOD Prevention

Several companies and airports have FOD prevention programs. These programs reduce the incurred from lost tools, damage equipment and the impact on schedule. The primary goals of the prevention programs are to provide a standardize approach, maintain awareness, prevention, and compliance, and to ensure operational processing areas are safe, clean, and free of foreign object debris.

Most FOD can be attributed to poor housekeeping, facilities deterioration, improper maintenance or careless assembly, not keeping full account of hardware, tools and materials, and inadequate operational practice. An effective prevention program identifies potential problems, corrects negative factors, provide awareness, effective employee training, and uses industry lesson learned for continued improvement. Organizational planning and function must include provisions for the prevention of foreign object in sensitive areas.

Prevention practices in the manufacturing environment can be simple procedures as removing personal items such as watches rings, jewelry or chains, and wearing tighter clothing that could not be drawn into engine intakes. On the other hand, the military outlines a rule or wearing button-less and pocket-less coveralls when physical entry is needed to inspect engine or exhaust areas. FOD-walks, sweeps are performed at the areas where FOD is a potential issue. The maintenance technicians perform the FOD-walks. They take place as much as twice a day, to as little as four times a year depending on the company policy. As a direct result of these programs many tenants have invoked more stringent housekeeping measures, this reducing the chance of a FOD related incident.

This research reviews a number of FOD prevention programs supplied by the aviation industry leaders (i.e. Boeing, NASA, Lockheed Martin Aeronautics). The program elements are combined to provide a list of available practices of FOD methods. Table 3.2 summarizes the FOD strategies and methods with the associated components identified through the research. FOD strategies have being adopted by aircraft manufacturers to eliminate FOD recurrence.

Table 3.2: FOD Methods. (Johnson, Kraus, Mason, Watson, 2001)

FOD Management Methods Components Preventive Practices  FOD walks  Runway sweepers  Tool Control Training Courses  Program and procedures/policies  Classroom  Definitions and causes  Computer-based Training (CBT)  Safe workmanship practices:  On-The-Job Training (OJT) Visibility Charts, Trend analysis, report cards, performance reviews, customer comments, and reviews.  Cleaning and Inspection of parts  Individual responsibilities  Hand-on training from supervisors on preventive practices  Learning Resource Centers  Types of FOD  FOD Labs- Visually presents FOD damage, mishap videos and training. (Refer to CBT, Course Dev., Learning Resource centers) FOD Committees, Specified Personnel  Define FOD terms, identify major contributors, identify HOW, list key reference documents for FOD, explain how inspections are conducted  Building and maintaining FOD training/ awareness programs Housekeeping Guidelines/Rules  Proper storage, shipment, and handling of components  Accountability of control of tools and hardware  “Clean as you go” Consulting  Necessary revisions recommended  Ensure contractors are complying with set regulations and standards FOD Awareness Job Aids  Visual job aids, management aids, tool shadowboxes, stickers, high visibility posters  Awards Programs 14

3.7.3. Training & Promotion

The purpose and primary objective of training programs is to increase the awareness of the employees as to the causes and effects of FOD, emphasize and train good work habits through structured and standardized work disciplines, and promote active involvement through awareness programs.

Prevention programs are associated with design, development, manufacturing, assembly, testing and repair, refurbishment, modifications, maintenance and general operations. In most leader companies, prevention programs are addressed and performed through initial training and recurrent training to maintain and refresh the awareness of the employees. Training is fundamentally critical in order to the impacts of human factor issues related to FOD be minimized (Randolph, 2004).

The first step in promoting a prevention program/training is to make certain that all personnel with direct contact to the aircraft, including ramp and gate areas, receive proper initial training. It is also important to create a positive culture where a safe and FOD-free work environment is the number one priority. Foreign object prevention cannot be addressed in one session. It must be continuous and multifaceted program that does not become monotonous. The program should become the key chain and forms a culture for everyone who works closely with the aircraft. In order to create a positive culture regarding this issue, it is necessary for the management to be fully involved and committed.

Once initial training has been conducted, it is important to promote a management program.

Promotion can occur in a variety of ways, but is best accomplished by relying on multiple methods.

Commitment of management with a hand attached to employees is extremely important to create conscious of the magnitude of the FOD problem. Visibility plays a key factor with the designation of areas to let any person know that he/she is entering a controlled area at any level. Awareness is completely necessary in order to detect FOD. These three previous factors are the most effective combination that minimizes incidents. 15

3.8. Foreign Object Elimination

Nowadays exist a number of programs developed by leader companies. One of the most important guidelines for Foreign Object Elimination (FOE) was created by the National Center for

Aerospace & Transportation Technologies (NCATT). Inside the guidelines, the program addresses twelve industry identified knowledge areas, functions and activities specifically designed to prevent the introduction of a foreign object to an aircraft or space products. These standards are derived from NAS

412 – Foreign Object Damage / Foreign Object Debris (FOD) Prevention.

The twelve mentioned standards are as Follows:

1) Basic Terms & Definitions. – Definition of Foreign Object Damage/Debris, meaning of

methods used to prevent the presence of FOD, and explain the consequences of FOD

incidents.

2) Housekeeping. – Identify the relationship of basic facts and state general principles

relevant to the aviation and aerospace industry methodology on housekeeping.

3) Tool Accountability. – Explains a variety of methods used for tool control and

accountability, and find the most adequate method for every process.

4) Hardware accountability. – Explains and describes methods and principles for controlling

hardware.

5) Lost Items. – Emphasize the importance of reporting lost or found items as soon as

possible and explain the importance and justification of why technicians are encouraged

and responsible to report missing items.

6) Physical Entry & Personnel Control. – Emphasize the requirements for entry into and

working within designated FOD areas increase as the final product progress through the

manufacturing/assembly facility to the customer delivery.

7) Reporting & Investigate. – Identify facts about reporting FOD incidents and investigation

requirements.

8) Material Handling. – Explain material handling techniques such as storage and

transportation.

9) Parts Protection. – Explains definition and use of FOD barrier, and sensitive products.

10) Hazardous Material. – Explains the importance of careful control, material handling, and

disposal o hazardous material can prevent FOD incidents.

11) Wildlife/Environment. – Identify and create conscious of FOD danger hidden in the

environment, such as a small bird can be suctioned into the engine causing damages to

the aircraft or even cost to the human life.

12) FOD Effects. – Identify and describes the impact of FOD incidents to the aviation and

aerospace industry across the country.

As mentioned before, Foreign Object Elimination (FOE) basically depends on a series of guidelines for personnel, which interacts directly with the aircraft. These techniques or methods in combination with new arising technologies (mentioned in previous sections) can minimize the degree of impact of a given incident. However, these existing practices are not enough for a successful elimination of foreign objects. This is because the magnitude of the problem is extremely high. In the next chapter, a new approach has been identified in order to address the problem and achieves the goal of this research; eliminate FOD, (Procaccio, 2008).

3.9. Predictive Models

Predictive models have being used inside the industry as an effective method. Most models

(studies) have several explanatory variables, and they may be continuous or categorical as well. The goal of the models is to describe their effects on response variables. There exist three main models used for prediction using categorical data. Binomial Generalized Linear Model, Probit, and Logistic 17

Regression. For the purpose of this research, the best model that fits the data is the Logistic Regression.

This is because Binomial it is focusing only in binary output instead of multiple outputs. Moreover,

Probit is the base of the Logistic Regression and it is also intended and most used for binary data.

Finally, Logistic Regression has being the most successful model when dealing with categorical data and having a multinomial response.

3.10. Natural Logarithm of the Odds (Log Odds)

Log odds are also called logistic regression coefficients, or parameter estimates in logistic regression output either ordinal or multinomial regression. Odds ratios and logits are important basic terms in logistic regression. A logit is a parameter estimate in the logistic regression output. Logistic regression has a logit link function, meaning that logistic regression calculates changes in the log odds of the dependent, not changes in the dependent itself. Parameter estimates ( are logits of explanatory variables used in the logistic regression equation to estimate the log odds that the dependent equals to the highest/last value (used in multinomial regression). The value of the logit is the value of the change in the log odds o the dependent variable per unit change in the predictor variable, positive or negative. The parameter estimates table for multinomial logistic regression will contain factor or covariate parameter for each category of the dependent variable except the last category. If the predictor is a covariate, there will be a single set of parameters for each value of the categorical dependent except the reference category. If the predictor is a factor, there will be one parameter row for each of that predictor’s category except the reference category. If a logit ( is significant and positive, then that parameter increases the odds of the given response (category) of the dependent (response) variable compared to the reference category response. If negative, that parameter decrease to odds of that response compared to the reference category response (Andersen, 1980).

The logit of a number p between 0 and 1 is given by the following formula:

( ( ) ( ( (1)

Where p is a probability, then p/(1-p) is the corresponding odds. The logit of the probability is the logarithm of the odds. The base of the logarithm function used for the explanation of the formula but the natural logarithm with base e is the most often used. Logistic Regression is the inverse of the Log

Odds showing the link between the log-odds and the Logistic Regression.

3.11. Logistic Regression

Logistic Regression is the most important model for categorical response data. It has been increasingly used in a very wide variety of applications. It is part of the statistical models named generalized linear models (GLM). This type of models includes linear regressions, analysis of variance

(ANOVA), and multivariable statistics, and log linear regressions (Agresti, 2002).

Logistic regression is most commonly used because it is possible to predict discrete outcome from a set of variables. That may be discrete, dichotomous, continuous, or even a combination of any of these. On the other hand, cases where the independent variables are categorical or a mix of categorical and continuous data, logistic regression is always preferred (Hilbe, 2009).

Binomial logistic regression is a form of regression, which is used when the dependent is a dichotomy and the independents are of any type. On the other hand, Multinomial logistic regression exists to handle the case of dependents with more classes than two. Logistic Regression applies maximum likelihood estimation after transforming the dependent into a logit variable (natural logarithm of the odds of the dependent occurring or not). With that being said, logistic regression estimates the probability of a certain event to occur. Logistic regression does not assume linearity of relationship between the dependent and independent variables. It does not require variables that are normally distributed. In general, logistic regression has less stringent requirements. However, it requires that the

19 observations are independent and linearly related to the logit of the dependent. Logistic regression success can be accurately addressed by the correct classification of the ordinal dependent. Also, goodness of fit tests such as model chi-square, are variable indicators of model appropriateness as is the

Wald Statistic to test the significance of individual independent variables.

In most cases, in logistic regression, the dependent variable is dichotomous; meaning hat the dependent variable can take the value 1 with a probability of success , or the value 0 with a probability of failure 1- (Harvill, 2009). These variables are called Bernoulli or binary variables. Although, applications of logistic regression model have also been extended to cases where the dependent variable is of more than two cases; this, as previously mentioned, is known as multinomial regression.

In logistic regression, the predictor or independent variables can take any form. In other words, logistic regression makes no assumption about the distribution of the predictor variables. Once again, they do not need to be normally distributed or linearly related, or even have equal variance within groups of data (Christensen, 1997). Instead of having linear function for the relationship between the predictor and response variables, the logistic regression functions is used as a transformation of :

( (2) ( ∑

Where:

= The constant of the equation.

= The coefficient of the predictor variables.

= Predictor variable.

There exist two main uses of logistic regression. Since logistic regression calculates the probability of success over the probability of failure, the results of the analysis are in the form of an odds ratio, so it is possible to do accurate prediction of groups. For example, logistic regression is often used to determine group categories. This is most used in the prediction of group memberships or determining

20 insurance policies. Logistic regression also provides knowledge of the relationships and strengths among the variables (Balakrishnan, 1991).

The main objective of logistic regression is to accurately predict the category or type of outcome for individual cases using the closest model. In order to do so, a model is created including all the predictor variables that are found to be useful in predicting the response variable. Several options are available during the model creation stage. It is possible to enter the variables into the model in the specified order by the analyst or logistic regression can test the fit of the model after entering or removing each coefficient; this is called stepwise regression.

Stepwise regression is more used for explanation purposes of research but it is not recommended for theory testing (testing of apriori hypotheses of the relationship between variables). Exploratory testing makes no-apriori assumptions of the relationship among the variables since the goal is to discover the existent or possible relationships. On the other hand, backward stepwise regression seems to be the best method of exploratory analyses, where it begins with a full model and variables are eliminated from the model in an iterative basis. The fit of the model is tested after the elimination of the variables that have found to be not significant to the model to ensure the adequacy of it. Where no more variables can be eliminated fro the model, the analysis is officially completed.

Logistic Regression works with two types of design variables. They could be nominal or ordinal factors. Logistic regression is able to consider factors inside the model either nominal or ordinal data but the output should be nominal. It is also possible the usage of interactions between the desired factors. It is important to clarify that only possible interaction between factors is considered for the accuracy of the model.

Multinomial Logistic Regression is extensions of binary logistic regression that allow the simultaneous comparison of more than one category. Which means, the log odds of three or more contrasts are estimated simultaneously. 21

Contingency Tables

Contingency tables areas used to display the relationship between categorical variables. The joint distribution between categorical variables determines the relationship among them. They consist in a rectangular table having I rows for categories and J columns for categories. The cells of the table represent the IJ possible outcomes. (Agresti, 2002). They are called contingency tables since the cells contain frequency counts of outcomes for a sample. Contingency tables are constructed by listing all levels of one variable as row in a table, and the levels of the other variables as columns. Then finding the joint or cell frequency for each cell. The cell frequencies are then summed across both rows and columns. The sums are placed in the margins, the values of which are called marginal frequencies

(Agresti, 1996). These expected frequencies are computed from the totals as follows.

(3)

Where:

= Is the expected frequency for cell IJ.

= Is the total for the ith row.

= Is the total for the jth column.

T= Total number of observations.

The outcomes are summarized in the contingency tables. Four main areas construct the main body of the table. Count is the number of observations in each cell. Total percentage is the percentage of the total number of observations represented by the cell count. Column percentage is the percentage of the observations in the column represented by the cell frequency. Row percentage is the percentage of observations in the row represented by the cell frequency. Column and Row percentages values sum to

100% for each column and row (Agresti, 2002).

Structural Zeros

As long as the probability of falling into row category I, and the probability of falling into column category J are both non-zero, the expected probability of falling into cell IJ is also non-zero under the usual contingency table model of independence. If the total sample size is very small, or if there are many cells in the table, hen it may happen that no observations are recorded for a particular IJ cell. These zero values in a contingency table are sampling zeroes. However, the actual process that creates the observations may produce cells in the contingency table in which observation can never occur. The zero values that must occur in these cells are structural zeroes. A contingency table containing a large count of structural zeroes is an incomplete contingency table (Bishop, 1975).

Chi-Square Test

The use of statistical test of chi-square, looking at Pearson p-value, helps to statistically show a variable to be significant to the model (Howell, 2010). The chi-square test verifies the null or alternative hypothesis. The null hypothesis (Ho) states that row and column factors are independent against the response variable. On the other hand, the alternative hypothesis ( ) states that row and column factors are not independent against the response variable. In order to reject the null hypothesis the Pearson p- value shall be equal or less than .05. a factor is not independent to the response variable when the null hypothesis is rejected making the factor considerable to the model.

The chi-square test involves using the chi-square distribution to approximate the underlying exact distribution. The best approximation will be most likely in the sampling scheme (multinomial).

The approximation becomes better as the expected cell frequencies grow larger, and may be inappropriate for contingency tables with very small-expected cell frequencies. For contingency tables with expected cell frequencies less tan 3, the chi-square approximation may not be reliable. A standard rule of thumb is to avoid using the chi-square test for contingency tables with expected cell frequencies

23 less than 1, or when more than the 20% of the contingency table cells have expected cell frequencies less than 3 (Agresti, 2002).

3.12.Validation Criteria

The following topics are included in this thesis to display the key attributes in order to validate and accept a model. These criteria shall be satisfied in each aspect in order for a model to be considered as a potential solution to the specific problem. The output of each validation criteria will determine the goodness of prediction of the model. This criterion consists of the Test of Goodness of Fit, Generalized

R-square, and Misclassification Rate.

First, The test of goodness of fit, Chi-square (Hosmer-Lemeshow), divides subjects into decimals based on predicted probabilities. Then, it performs a chi-square from observed and expected frequencies.

Next, a probability value is computed from the chi-square distribution to test the fit of the logistic model. If the goodness of fit test statistic is greater than .05 ( ), we fail to reject the null hypothesis where there is no difference between observed and model-predicted values. In order to have a significant model, a value of <. 05 is required in the test. As the sample size gets larger, the test of goodness fit can find smaller and smaller differences between predicted and observed values to be significant.

Second, There is no widely accepted direct analog R-square. This is because an R-square measure seeks to make a statement about the percent of variance explained, but the variance of a categorical dependent variable depends on the frequency distribution of the variable. For dependent variables, variance is at a maximum for a 50-50 split and the more lopsided the split, the lower de variance. This means that R-squared measures for logistic regressions with differing marginal distributions of their respective dependent variables cannot be compared directly. Furthermore, a number of logistic R-squared measures have been proposed. These measures are not goodness-of-fit tests but rather attempt to measure strength of association.

R-square can be used in binary logistic regression but not in multinomial logistic regression. In order to obtain R-square, save the predicted values from logistic regression and run a bivariate regression on the observed dependent values. Logistic regression can yield deceptively high R-square values when you have many variables relative to the number of cases, keeping in consideration that the number of variables included k-1 variables for every categorical independent variable having k categories.

Pseudo R-square is an Aldrich and Nelson’s coefficient, which serves as an analog to the squared contingency coefficient, with an interpretation like R-square. Its maximum is les than 1. It may be used in multinomial logistic regression (Gelman, 2000).

Generalized R-square (Cox and Snell) is a generalization of the R-square measure that simplifies to the regular R-square for continuous normal responses. It is scale o have a maximum of 1. The value is

1 for a perfect model, and 0 for a model no better than a constant model. In summary, generalized R- square values closer to 1 indicate a better fit. It is important to note that in terms of binomial response and multinomial response, it is significantly hard to get a value close to one. This is because, when using binomial logistic or multinomial logistic, if the category is not correctly predicted there is no percentage of approximation to the correct output as in linear regression which uses continuous data. However, literature review states that a Generalized R-square between .4 and .6 are acceptable values when dealing with nominal data.

Finally, Another indicator of a good fit for the logistic regression model is the misclassification rate. This indicator is intended to be used for categorical data, which is present in binomial logistic, and multinomial logistic. Since the output of the model is binary or multinomial, misclassification rate is the rate for which the response category with the highest fitted probability is not the observed category. The value is intended to be 0 for a perfect model, and 1 for a completely wrong model. This indicator is very important when dealing with nominal data. It is a reliable indicator of a good fit of the model. 25

Chapter 4: Methodology

The focus of the current chapter is to address the top-level FOD questions regarding human error associated with foreign object occurrence. Specially, the identification of the relevant input variables; the selection of the best model, and the identification if an interaction is needed. As each step is developed, the more detailed questions regarding each step will be addressed. The result of this chapter will yield a potential prediction model that responds to the necessity of elimination of human error involved in FOD. Finally, a case of study is presented further in this thesis in order to illustrate the application of the chosen methodology.

Figure 4.1: Methodology Flow Diagram.

4.1. Materials

In order to conduct the study, it is necessary to obtain the correct materials. These materials include the raw data and the software used to analyze it. The following subsections describe in details the type o data to be analyzed, and the software specifications used in this thesis.

4.1.1. Software

In order to perform analysis and develop an adequate predictive model, statistical software is needed. JMP 11 trial version has being selected for this study because of its simplicity and its user- friendly features. JMP is able to import databases from the most used software or data storage and analysis such as Microsoft Excel. JMP software is focused on exploratory data analysis and visualization. It is intended to investigate data to learn something unexpected, as opposed to confirming a hypothesis. JMP links statistical data to graphic representation. Moreover, it has the capabilities to not only perform logistic regression, but other types of regression (predictions) such as linear regression and so others. This software would be used as a statistical tool for the determination of significant factors, the development, construction, and evaluation of the model. Finally, this software would be used to perform final results.

4.1.2. Quality Assurance Reports

Quality Assurance Reports (QARs) are the primary data to be analyzed. Quality engineers of the aircraft manufacturer that provided the data write these reports. They contain all the information regarding the defect recorded. This involves the location, type, cost, and frequency o defect. In order to conduct the analyses, this research only focuses on defects classified as FOD.

4.2. Methods

The following sections covers and describe all the steps involved in order to apply the logistic regression approach. They go in detail in the step-by-step explanation of all the conducted procedures

27 and the interpretation of the results. These sections go from data preparation throughout calculation o the log odds, and the transformation of the estimates to actual probabilities.

4.3. Data Preparation

The first step is to properly clean the received data. It is well known that manufacturers of any type deal with large amount of data. Aircraft manufacturers are not exempt. They not only deal with large amount databases, but with a wide variety of databases such as, production, personnel, quality databases, etc. this thesis is focusing in quality databases. This is because FOD is treated as a defect inside manufacturing. Quality databases are a recompilation of quality reports. For the accuracy of the study, it is important to remove all the data that is not significant to the objective of the thesis. It is important to carefully analyze the data that has being received. Completely accurate understand of data is the most important part when performing data cleaning. Failure to do so may result in important data ignorance or non-important data consideration. The purpose of data preparation is to eliminate incomplete records that are not useful for this study.

4.4. Data classification

The data received for this research is in the majority nominal data. Meaning that the majority of the data has a binary or multivariate outcome (i.e. month, year, workstation). However, a few presences of continuous data are present. Continuous data is constituted by integer values such as defect frequency and total cost. It is extremely important to group data (if possible) in terms of simplicity and easy identification of data. Also, this practice is helpful when statistical analyses are conducted. Moreover,

4.5. Identification of Factors

Once the data preparation process is done it is time to start objectively selecting the factors that can be used for the selected model. In order to do so, the best practice is to identify the correlation between factors in raw data. This could be a repetition of an error given other circumstances. Some patterns may exist that may lead the circumstances to a non-desirable scenario. Once the factors are pre- 28 selected, it is now time for a statistical justification, and to discover if they are truly significant to de model. It is highly recommend to never under estimate any factor. Even though the smallest correlation is identified, it is recommended to verify by the use of any statistical method such as contingency tables.

4.6. Contingency Tables

Contingency tables, as previously discussed, are used to verify independence of the factor to the predictive model. These contingency tables are performed by JMP software to analyze nominal data.

JMP uses a graphical representation of the contingency tables called Mosaic Plot. Figure 4.2 shows an example of the Mosaic Plot.

Figure 4.2: Example of JMP Mosaic Plot

The mosaic plot represents the relationship between 2 response variables. It shows in the x-axis the frequency of occurrence of the variable in discussion. In the y-axis, it represents the percentage of the total data for category. This mosaic plots helps to visually identify the most frequent variables with their associated percentages.

The next output from JMP is the contingency table with values. As explained previously, it shows the count, and total percentage for each variable, the percentage value with respect of columns and rows. As explained in chapter two, the analysis becomes more accurate when large data is used. In case of having small data or the connection between the variables does not exist, there is a high probability of having structural zeroes. In case of having a large amount of low values or structural zeroes, a deeper data cleaning is recommended. By doing so, chances of facing these special cases are reduced or eliminated.

Moreover, JMP shows the last output. It is the test of statistics, which are the rejection/acceptance criteria for the analyzed variable. As described in chapter two, the test of statistics is based on the chi-square p-value (Pearson). The basic rule of thumb is if the Pearson p-value is less or equal to .05, the analyzed variable is significant to the model. In other words, the variable shall be included in the model.

Table 4.1: Example of Pearson P-value.

ChiSquare Prob>ChiSq Likelihood Ratio 81.562 0.0005* Pearson 80.790 0.0006*

4.7.Model Construction

Now that the factors are statistically selected, it is time to construct the model. JMP offers a variety of regression analyses. JMP automatically recognizes the type of data that is present. Based on that, it automatically suggests the best model possible but it is not mandatory. The model can be modified or changed as needed. For the nature of the data, Logistic Regression Model is selected.

The first step when constructing the model is the selection of the response variable (output).

Then, the key of the model is to introduce one by one the previously selected input variables. It is important to clarify that it is possible to JMP to add the interaction of factors to the model, as well as the

30 factor to a given exponent such as quadratic interaction. It is important to state that interactions are only a good contribution to the model when they are possible. The addition of non-real interactions to the model may or may not improve the model but the model will lose the accuracy and the model will be obsolete.

Another important aspect of the model is that it is also possible to include a variable to be the weight of the formula. Moreover, the software has the option to include a variable to be the frequency variable. All the modification done in the selection of factors is highly affecting the desired model.

Figure 4.3: Example of JMP Model Specification Window

4.8.Model Validation

The following sections are intended to show the key attributes in order to validate and accept a model. These criteria shall be satisfied in each aspect in order for a model to be considered as a potential solution to the specific problem. The output of each validation criteria will determine the goodness of prediction of the model.

Whole Model Test

Once the statistically selected factors, and the response variable are selected as well, the next step is to run and check the output. The first part of the model that needs to be checked is the p-vale from the chi-square analysis (see table 4.2). This p-value is the first indicator for the significance of the model. As discussed in chapter two, the p-value shall be less or equal than in order for the model to be significant. Meaning that the model, based on the selected factors, may be used for its main prediction objective.

Table 4.2: Example of whole model test.

Model -LogLikelihood DF ChiSquare Prob>ChiSq Difference 97.89877 28 195.7975 <.0001* Full 195.21973 Reduced 293.11850

Generalized R-square & Misclassification Rate

Once this validation value is satisfied, the next step is to check the Generalized R-square. This value goes from 0 to 1, meaning 1 for a perfect model and 0 for a defective model. This value is most used in order to compare model in order to find the best model. Having said so, Generalized R-square can be used as a “stop” measure. A best model is determined when there is no other model that can improve your R-square.

Another validation criteria, present in the same table as the Generalized R-square, is the misclassification rate. This criteria, as its name says, is in some terms the probability of error in the classification of the response variable. For the focus of this study, this is most used for multinomial response variables. See table 4.3 below for generalized R-square and misclassification rate.

Table 4.3 Example of generalized R-square and misclassification rate. JMP portion.

Measure Training Entropy RSquare 0.3340 Generalized RSquare 0.5369 Mean -Log p 0.5862 RMSE 0.4226 Mean Abs Dev 0.3186 Misclassification Rate 0.2012 N 333

Likelihood Ratio Test

Once the previous validation criteria are satisfied, the next step is to check the Likelihood Ratio

Test (see Table 4.4). This is used to check the significance of the variables with respect the response variable. It relies in the same principle as the contingency tables and the whole model test. The p-value is calculated by the use of the chi-square. This test confirms the findings in the contingency tables. It confirms the significance of the input variables using the p-value rule of thumb as decision criteria. It is important to make a note regarding this analysis. There is some cases where the input variables are significant but with a warning of “Lost of Degrees of Freedom”. This, in most cases, is originated by the addition of interactions between input variables that are not feasible. If the input variables are significant and no warnings are present, the model is well accepted in the likelihood ratio test criteria.

Table 4.4: Likelihood Ratio Test

Source Nparm DF ChiSquare Prob>ChiSq COUNTRY 6 6 14790.1956 <.0001* MONTH 22 22 11752.291 <.0001*

4.9.Natural Logarithm of the Odds

Once the best model is selected, without any violation to the validation criteria, the next step is to start using the output from the model to construct the prediction probabilities. These values are calculated based in the last factor of each input variable. Log Odds are calculated for all factors in the

33 response variable with the exception of the strongest factor. As an example, suppose the response variable is conformed by k number of categories. Log odds are calculated for k-1 categories. This is because log odds are shown as a comparison or ratio of the k categories over the strongest k category in the response variable. That is the primary reason for the k-1 circumstances. As a consequence, there is no evidence o Log Odds result for the dominant or strongest factor inside the response variable.

Since Log Odds are calculated for k-1 categories inside the response variable, it is naturally to expect k-1 number of Log Odds calculations. This means, a k-1 number of formulas are performed in order to calculate the values based in the strongest category.

Log Odds are calculated using the following formula:

∑ ∑ (

Where:

= Log Odds for each category inside the response variable.

= The intercept coefficient calculated value.

= First input variable estimate coefficient calculated value.

= Selected first input variable. [0,1]

= Second input variable estimate coefficient calculated value.

= Selected second input variable. [0,1]

= Interaction input variable estimate coefficient calculated value.

= Selected Interaction input variable. [0,1]

It is important to note that the factors of every input variable are mutually exclusive. Denoting that no more than one factor inside the input variable can be selected following a binary selection. Once

34 a category is selected in each input variable its value would be 1, and all other factors become 0. This rule applies for all input variables and the possible interactions between them.

The calculated value for the Log Odds base is always 1. This is because the Log Odds are calculated with respect the strongest category. When comparing the strongest category with itself, the division is interpreted as based category/based category. Which no matter what the estimated values is, it would be

4.10. Probability Transformation

Once the log odds are calculated for each k-1 input variable, the next step is to transforms those values to real probabilities values. The reason for doing this step is to deliver the desired output.

Moreover, these values are way easier to understand since they go from 0 to 100%. These probabilities are calculated based in previously selected factors. The intention if this is to create possible or desired scenarios and predict the future outcomes. As well expected, the sum of probabilities of all the factors in the response variable must add 100%. Failure to satisfy this condition is an indicator that the probabilities are not being correctly transformed, or the log odds are not correctly calculated.

The probabilities are calculated as follows:

( (5) ( ∑

The nominator represents the Log Odds in an exponential interpretation. Where the nominator can be interpreted as to the log odds, and the denominator as 1 plus to the log odds. This formula can be interpreted as the chances to succeed over the chances of failure.

The probability for the based factor inside the response variable is calculated as follows:

(6) ( ∑

This can be interpreted as the probability of the base to occur given specific circumstanced is calculated as 1 over 1 plus to the Log Odds. 35

Chapter 5: Case of Study

5.1. Data

This section discusses all the process of data cleaning. Sections 5.2 and 5.3 show the applied methodology for each aircraft in their specific scenario. This is because they deal with their respective data that not all of them are common among the mentioned aircrafts. However, there exist a minimum of shared information among these aircrafts mentioned in the following subsections.

5.1.1. Data Collection

Data collection is the first step in order to start any type of analysis. It is very important to obtain the representative data. For this study, data has being proportioned by the aircraft manufacturer company. The received data is conformed by Quality Reports. These quality reports are records defects, which includes where the defect has being found, the aircraft involved, the actions needed, the result of the action, and any further investigation.

5.1.2. Data Preparation

Data received from an aircraft manufacturer consists in a master database. This massive database is contains several characteristics making this database unique. It consists in a large quantity of records where the most relevant factors were taking in consideration.

The data considered for this research is involved with two aircrafts, the Advanced Fighter and the Large Cargo Aircraft. Moreover, each aircraft has their corresponding variants. Table 5.1 shows the variants per Aircraft used for this study.

Table 5.1: Variants per Aircraft.

Aircraft Variants

Advanced Fighter  X  Y  Z Large Cargo Aircraft  I  II

More information such as workstation for each aircraft is important for the study. There exists a several number of workstations for each aircraft in the used database. Further analyses are needed in order to determine the final number of workstations per aircraft. These analyses are covered in later sections. Another relevant information found and used in terms of data cleaning are the actual defect, the frequency, and the cause of defect. This information serves as a filter for the final selected data for both aircrafts. Only causes and defects defined by the aircraft manufacturer as human factor would be considered for this study.

Grouping the data is important in order to get a better interpretation of it. Following the Boeing identification of FOD, the categories described by Boeing are going to be used as the final output.

Which means, the response variable will be the FOD type that at the same time is divided into five different categories defining the model as multinomial. The five FOD categories identified by Boeing and used for this thesis are as follows:

1) Trash, 2) Tools/Shop Aids, 3) Panstock, 4) Manufacturing Debris, 5) Consumables.

Finally, the received data shows dates of defect, and cost. These are important features among data that may be used for future analyses.

5.1.3. Data Preparation Summary

A large amount of information was removed from the database because of its lack of relevance to the proposed methodology. On the other hand, only important, well recorded, and complete data can be used in order to apply the proposed statistical methodology to both Advanced Fighter and Large Cargo

Aircraft. Table 5.2 shows the quantity of data in good conditions for the application of the methodology by each aircraft to be analyzed.

Table 5.2: Data Preparation Summary.

Aircraft Useful Records Advanced Fighter 937 Large Cargo Aircraft 333

5.2.Advanced Fighter

The following sections are intended to discuss the process followed for the Advanced Fighter

Aircraft. These sections cover the entire analyses done to this aircraft in order to find the best predictive model possible and the results of it.

5.2.1. Contingency Tables

The first step is to identify the possible input variables. Since the response variable of the predictive model is the FOD type, it is possible to have as a potential input variable the month, workstation, variant, for the Advanced Fighter. However, an obstacle is present after performing the contingency table analysis. When performing the analysis for the Advance Fighter between workstations against the response variable (FOD type). A large amount of zeros automatically turned on the JMP alarms giving a warning. This alarm warned that more that the 20% of the IJ cells have expected values less than 3. This condition is under no circumstances acceptable since the accuracy of the model would 38 be questionable. In order to fix the possible input variable, a workstation grouping is necessary for this aircraft. This technique is highly recommended since it does not delete useful data, it only rearranges the data in a way where the quantity of data increases, and at the same time the categories of the input variable reduces significantly. Final usable workstations for the Advanced Fighter are: Forward, Wing 1,

Wing 2, Wing 6, Wing 7, Wing 8, Wing 9, Wing 10, Wing 11, Final Assembly 1, Final Assembly 3, and

Mate.

After the data has being prepared based in the final workstations and final categories in the response variable, the contingency tables are performed again. Fortunately, the warnings and zeros have disappeared from the contingency tables. The following tables are from different possible input variables that were measured against the response variable. They are only showing the decision criteria based on the Pearson p-value from the chi-square test (Refer to appendix A for complete contingency tables).

Table 5.3: Advanced Fighter Month by FOD Type.

N ChiSquare Prob>ChiSq Likelihood Ratio 81.562 0.0005* Pearson 80.790 0.0006*

Table 5.4: Advanced Fighter Workstation by FOD Type.

Test ChiSquare Prob>ChiSq Likelihood Ratio 264.280 <.0001* Pearson 252.116 <.0001*

Table 5.5: Advanced Fighter Variant by FOD Type.

Test ChiSquare Prob>ChiSq Likelihood Ratio 12.442 0.1325 Pearson 12.584 0.1270

Test performed, as part of the contingency table analyses, follows the simple rule of thumb for critical criteria. If the p-value is equal or less than .05, there is evidence of a non-independence behavior between the possible input variable with the response variable. For the Advanced Fighter, the variables

39 taken in consideration to the model are Month and workstation. Variant failed the test by the presence of a Pearson p-value greater than .05. This variable is not affecting the response variable, so it is not going to be included for the construction of the model. However, this assumption cannot be completely accepted. Contingency tables show the dependency between two factors, they are not completely deciding the factors to be included for the model construction. This means that a possible input variable is not significant during the contingency table analysis but it may or may not be significant when running the model. If it is significant, it means that it must be included in the model.

The significant variables are show in table 5.6.

Table 5.6: Advanced Aircraft Significant Variables.

Aircraft Significant Variables Not Significant Variables Advanced Fighter  Month  Variant  Workstation

5.2.2. Model Construction

Once the response variable and significant variables are selected is time to identify the best model possible for the Advanced Fighter. Nominal Logistic Model was run using JMP. Figure 5.1 shows the actual JMP specification window.

Moreover, it is important to select, in the personality menu, the nominal logistic model. The selection of the correct model is crucial to the study. If other model is selected all the analyses done are irrelevant since other model will not predict nominal data.

As showed in Figure 5.1, JMP has the feature to include a weight and frequency variable for the predictive model as well as the possibility of interactions between the input variables. These interactions can be a first level interaction, which funds a simple interaction between two or more different variables is possible. Also, quadratic interactions are possible. This means that an interaction of only one factor is

40 possible. It is important to remember that only real interactions are expected for the model. If an interaction improves the model but it is not real, the model becomes unstable.

Figure 5.1: Actual JMP Model Specification Window.

According to the analysis previously done using contingency tables, month and workstation are significant variables for the Advance Fighter Aircraft. A first level interaction between these variables is possible since it is true to say that a workstation is performing in a given month. However, a quadratic or higher interaction is not possible among the input variables. This is because it is impossible have a specific workstation twice, and it is impossible to be in two months at the same time as well.

Another important output to the creation of the model is the selection of weight and frequency variables. Since money is the primary impact in FOD related issues, it can be selected as the weight variable. On the other hand, master database shows the defect count per quality record. The main objective is to reduce or even eliminate FOD occurrence, which means the reduction of cost and defect counts related to FOD. Having this clarified, defect can serve as the frequency variable. In summary, the

41 inclusion of weight and frequency variables associated with the input variables and their interactions among them can significantly improve the model response.

5.2.3. Model Validation & Selection

Now that the input, weight, frequency, and response variables are identified, the next step is to start modeling and follow the validation criteria in order to select the best FOD predictive model.

Several models were constructed, ran, and evaluated but in most cases the validation criteria were not successfully achieved. Table shows the best models performed by this study. Also, it is explaining the characteristics of each model. This means if an interaction, weight, or frequency was included in the model. Moreover, it shows a summary of the validation criteria values such as Whole Model Test

(WMT), Generalized R-square value and the Misclassification Rate.

Table 5.7: Advanced Fighter. Output from Models

ADVANCED FIGHTER Model Input Variables Interaction Weight Freq WMT G. R-square MR Month I    <.0001 0.5058 0.4468 Workstation Month II    <.0001 0.5143 0.4348 Workstation Month III    <.0001 0.7798 0.3335 Workstation Month IV    <.0001 0.8006 0.3077 Workstation

By interpreting the results above, it is easy to note that the best model for the Advanced Fighter is model “IV”. It contains the input variables month and workstation and its interactions. It also contains the weight and frequency factors, which are total cost and defect count. This is the best model since it has the larger Generalized R-square, the lowest misclassification rate.

5.2.4. Natural Logarithm of Odds

Once the logistic regression models are selected and validated, the Log Odds can now be calculated. The log odds formula is now going to be slightly modified from the original. This is because the factors selected are going to compose the Log Odds formula.

As discussed in previous chapters, the calculation of the Log Odds is based in the strongest category in the response variable FOD Type. For the Advanced Aircraft, the strongest category inside

FOD type is the category of Trash. This category is taken as the base category, stating that the Log Odds are calculated based in the Category of Trash. Since a total of five categories of the response variable are present, a total of 4 Log Odds formulas are develop for the remaining categories. The calculated Log

Odds are shown as Consumables/Trash, Manufacturing Debris/Trash, Panstock/Trash, and Tools/Trash.

This satisfies the k-1 of the Log Odds. The base category is calculated separately.

∑ ∑ (

Where:

= Log Odds for each category inside the response variable.

= The intercept coefficient calculated value.

= Month estimate coefficient calculated value.

= Selected Month. [0,1]

= Workstation estimate coefficient calculated value.

= Selected second input variable. [0,1]

= Workstation-Month interaction estimate coefficient calculated value.

= Selected Workstation-Month Interaction. [0,1]

The formula is a double summation of the estimates of the selected values for both predicting variables. As discussed before, the predicting has categories that are mutually exclusive. When an input is selected it becomes a number 1 and the remained becomes zero. Then the multiplication of the estimate times the binary response of the selected factor [0,1]. This principle is the base of the formula and it applies for the Month, Workstation, and the Month-Workstation interaction. The first summation is referring to the month factors. They go from January to December, so the month goes from January to

December. The second summation is referring to the 12 available workstations that can be chosen for the Advanced Fighter. Finally, the Month-Workstation interaction can only be possible between the previously selected categories.

Having the following selected month, workstation, and estimate values (calculated in the predictive model) in Table 5.8 the Log Odds and Probability Transformation areas calculated as follows.

Table 5.8: Advanced Fighter Input Variables and Estimates.

∑ ∑ ( ( ( ( ( (

( ∑ ∑ ( ( ( ( ( (

∑ ∑ ( ( ( ( ( (

( ∑ ∑ ( ( ( ( ( (

5.2.5. Probability Transformation

Now that the Log Odds are calculated for the all categories in the response variable FOD Type, the next step is to transforms those values to actual probability values. For the purpose of this research, probability values are shown as decimals going from 0 to 1. Taking previous equation as a base, a small modification in the factors is done in order to fit the new input variables. The probability transformation for the k-1 prediction categories for both aircrafts is as follows.

∑ ∑

(8) ∑ ∑

∑

Where the nominator is interpreted as to the Log Odds over 1 plus to the Log Odds. The probability transformation for the base factor in the response variable is calculated as follows.

(9)

∑

The probability formula can be interpreted as 1 over 1 plus to the Log Odds. It is important to clarify that both aircrafts have the same input variables, so the formulas do not change with the exemption of workstations amount.

By using the Log Odds results in section 5.2.4. The probability transformation results are as follows: (See Appendix E for final projected probabilities per workstation per month)

(

( ( ( (

(

( ( ( (

(

( ( ( (

(

( ( ( (

Finally, the calculations above are the ones followed in order to perform the projections for each workstation. Figure 5.2 shows the projected predictions for the calculated workstation in the Advanced Fighter Final Assembly 1.

Final Assembly 1 1.00 0.90 0.80 0.70 0.60 0.50 Probability 0.40 0.30 0.20 0.10 - JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.04 0.49 0.12 0.72 0.12 0.25 0.41 0.87 0.28 0.00 0.61 0.91 Tools 0.00 0.03 0.00 0.00 0.88 0.25 0.19 0.00 0.44 0.05 0.04 0.01 Panstock 0.93 0.22 0.77 0.04 0.00 0.25 0.00 0.03 0.18 0.72 0.14 0.08 Manufacturing Debris 0.04 0.00 0.12 0.17 0.00 0.00 0.13 0.03 0.10 0.03 0.04 0.00 Consumables 0.00 0.26 0.00 0.07 0.00 0.25 0.27 0.06 0.00 0.20 0.17 0.00

Figure 5.2: Advanced Fighter Probabilities for Final assembly 1 by Month. 46

5.3. Large Cargo Aircraft.

The following sections are intended to discuss the process followed for the Large Cargo Aircraft.

These sections cover the entire analyses done to this aircraft in order to find the best predictive model possible and the results of it.

5.3.1. Contingency Tables

As found in the Advanced Fighter, the Large Cargo Aircraft has the same problem when comparing the response variable against the workstation variable. A large amount of zeros automatically turned on the JMP alarms giving a warning. This alarm warned that more that the 20% of the IJ cells have expected values less than 3. This condition is under no circumstances acceptable since the accuracy of the model would be questionable. In order to fix the possible input variable, a grouping it is necessary for this aircraft. The final workstations for the Large Cargo Aircraft are workstations 1, 2, 3, and 4.

After the data has being modify based in the final workstations and final factors in the response variable, the contingency tables are performed again. Fortunately, the warnings and zeros have disappeared from the contingency tables. The following tables are from different possible input variables that were measured against the response variable. The following tables 5.9 and 5.10 show the results for the Large Cargo Aircraft (Refer to appendix B for complete contingency tables).

Table 5.9: Large Cargo Aircraft Month by FOD Type

Test ChiSquare Prob>ChiSq Likelihood Ratio 9467.080 <.0001* Pearson 11883.50 <.0001*

Table 5.10: Large Cargo Aircraft Workstation by FOD Type.

Test ChiSquare Prob>ChiSq Likelihood Ratio 12504.98 <.0001* Pearson 11422.60 <.0001*

Since the p-value is less than .05 for both month and workstation variables, there is evidence of a strong impact of the workstation and month in the response variable. There is evidence of a strong correlation between the possible input variable with the response variable. For the Large Cargo Aircraft, the variables taken in consideration to the model are Month and workstation. Variant has not been selected for this study since all clean data relies in only one variant of the Large Cargo Aircraft.

The significant variables are show in Table 5.11.

Table 5.11: Large Cargo Aircraft Significant Variables.

Aircraft Significant Variables Not Significant Variables Large Cargo Aircraft  Month  Variant  Workstation

5.3.2. Model Construction

According to the analysis previously done using contingency tables, month and workstation are significant variables for the Large Cargo Aircraft. A first level interaction between these variables is possible is possible but not a quadratic interaction as well.

Another important output to the creation of the model is the selection of weight and frequency variables. As discussed in the Advance Fighter, the inclusion of weight and frequency variables associated with the input variables and their interactions among them can significantly improve the model response.

5.3.3. Model Validation & Selection

Several models were constructed, ran, and evaluated but in most cases the validation criteria were not successfully achieved. Table 5.12 shows the best models performed by this study. Also, it is explaining the characteristics of each model. This means if an interaction, weight, or frequency was included in the model. Moreover, it shows a summary of the validation criteria values such as Whole

Model Test (WMT), Generalized R-square value and the Misclassification Rate. 48

Table 5.12: Large Cargo Aircraft Output from Models.

LARGE CARGO AIRCRAFT Model Input Variables Interaction Weight Freq WMT G. R-square MR Month I    <.0001 0.5369 0.2012 Workstation Month II    <.0001 0.6827 0.2196 Workstation Month III    <.0001 0.8483 0.1626 Workstation Month IV    <.0001 0.7572 0.2192 Workstation

For the Large Cargo aircraft, the best model is model “III”. This model contains the input variables month and workstation, the interaction of them, and the weighted factor total cost. This is the best model since it has the largest Generalized R-square value, and the lowest misclassification rate.

5.3.4. Log Odds

As discussed in previous sections, the calculation of the Log Odds is based in the strongest factor in the response variable FOD Type. For the Advanced Aircraft, the strongest factor inside FOD type is the category of Trash. This category is taken as the base factor, meaning that the Log Odds are calculated based in the Category of Trash. For the Large Cargo Aircraft, only three categories are possible in the response variable FOD Type. The strongest category is Trash, which is taken as the based for the calculation of the Log Odds. The calculated Log Odds are shown as Manufacturing Debris/Trash, and Tools/Trash. This satisfies the k-1 of the Log Odds. The base factor is calculated separately.

Advanced Aircraft developed formula is as follows:

∑ ∑ (

Where:

= Log Odds for each factor inside the response variable.

= The intercept coefficient calculated value.

= Month estimate coefficient calculated value.

= Selected Month. [0,1]

= Workstation estimate coefficient calculated value.

= Selected second input variable. [0,1]

= Workstation-Month interaction estimate coefficient calculated value.

= Selected Workstation-Month Interaction. [0,1]

The formula is a double summation of the estimates of the selected values for both input variables. As discussed before, the input variables are mutually exclusive. When an input is selected it becomes a number 1 and the remained becomes zero. Then the multiplication of the estimate times the binary response of the selected factor [0,1]. This principle is the base of the formula and it applies for the Month, Workstation, and the Month-Workstation interaction. The first summation is referring to the month factors. They go from January to December. The second summation is referring to the 4 available workstations that can be chosen for the Large Cargo Aircraft. Finally, the Month-Workstation interaction can only be possible between the previously selected factors.

Having the following selected month, workstation, and estimate values (calculated in the predictive model) in Table 5.13 the Log Odds and Probability Transformation areas calculated as follows. See Appendix F for final projected probabilities per workstation per month.

Table 5.13: Large Cargo Aircraft Input Variables and Estimates.

Log Odds:

∑ ∑ ( ( ( ( ( (

5.3.5. Probability Transformation

Probability values are shown as decimals going from 0 to 1. Taking previously explained equation as a base; a small modification in the factors is done in order to fit the new input variables. The probability transformation for the k-1 prediction categories for both aircrafts is as follows.

(11)

∑

Where the nominator is interpreted as to the Log Odds over 1 plus to the Log Odds. The probability transformation for the base factor in the response variable is calculated as follows.

(12)

∑

By using the Log Odds results in section 5.3.4. The probability transformation results are as follows:

(See Appendix F for final projected probabilities per workstation per month)

(

( (

(

( (

The calculations above are the ones followed in order to perform the projections for each workstation. Table 5.3 shows the projected predictions in the Large Cargo Aircraft workstation

WORKSTATION 2 1.00 0.90 0.80 0.70 0.60 Probability 0.50 0.40 0.30 0.20 0.10 - JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.39 0.00 0.00 0.00 0.00 Tools 1.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.43 0.00 0.00 Manufacturing Debris 0.00 1.00 1.00 1.00 1.00 0.00 1.00 0.61 1.00 0.57 1.00 1.00

Figure 5.3: Large Cargo Aircraft Predicted Probabilities for Workstation 2 by Month.

Chapter 6: Conclusions and Recommendation

6.1. Conclusions

The main and novel achievement of the study presented in this thesis is the FOD prediction per month per workstation for two different aircraft, the Advance Fighter and the Large Cargo Aircraft using

Logistic Regression Models. To the author’s knowledge, there is no previous published evidence of the use of statistical models to predict Foreign Object Damage/Debris cases. In addition, this prediction has being conducted not only for a given month and workstation, but it is able to predict the possible type of defect making this effort more significant in terms of research. The methodology used proposes the use of statistical methods and techniques to identify the main cause factors of the recurrent quality problem as a new problem solving approach in the aviation industry. Moreover, Logistic Regression statistically verifies the significance of the models tested. The comparison among the validation criteria will show the best model possible that it’s the quality data that has being used for the analysis. The results presented in this thesis are the projection for each workstation per month for both aircrafts. These projected results can serve as a starting point in the identification of the areas with more FOD occurrence observed and identify the type of defect. The fact that the discussed methodology is a non- expensive effort, and the FOD occurrence predictions are feasible give a very significant motivation and interest to the industry to take in consideration the proposed methodology. The advantages of the proposed methodology can go from the reduction of cost production, quality issues, repair cost, and assembly process time. Finally, a more reliable process is achieved, and the proposed methodology may be used in other aircrafts.

6.2. Recommendations

The proposed methodology covers the details about finding relevant factors, choosing weight, frequencies and interactions in order to construct the model. Moreover, the used statistical software shows the main differences between the models and their predicting capabilities. It is possible to select 53 the based model based in previously described validation criteria. Once the best available model is selected and the results are calculated and predictions are performed per area it is time to go to the next step. The proposed models can be improved by finding more reliable data for the analysis. If more complete an accurate data can be found, it means that more data can be used. If more data is available, the model can be highly improved in terms of predictions. The model will have more historical data meaning more records of previous cases.

Furthermore, with the predicted FOD projections, the next step is to focus in the Foreign Object

Elimination. By using the projections presented in this thesis, Industry can focus in in developing or finding methods or techniques to eradicate the FOD type present in each workstation. Human error can be minimized by the use of existing technologies for eradicating FOD in the exterior. They can be applied in the manufacturing side in order to minimize the impact oh human error. Technologies from the medical surgery field can be implemented to detect and eliminate Foreign Object from a body, in this case, an aircraft. Also, other technologies from different areas can be implemented in workstation to eliminate the human error of the scene.

References

Agresti, Alan. (1996.) An Introduction to categorical Data Analysis. Florida: University of Florida, Gainesville, FL.

Agresti, Alan. (2002.) Categorical Data Analysis. 2nd Ed. Florida: University of Florida.

Andersen, Erling B. (1980.) Discrete Statistical Models with Social Science Applications. North Holland

Balakrishnan, N. (1991.) Handbook of the Logistic Distribution. Marcel Dekker, Inc.

Bishop, Y. M. M.; Fienberg, S. E.; Holland, P. W. (1975.) Discrete Multivariate Analysis: Theory and Practice. MIT Press.

Christensen, Ronald (1997.) Log-linear models and logistic regression. Springer Texts in Statistics (Second ed.). New York.

Cunningham, Ron. August (2007.) Human Factors in FOD Prevention. 27th Nation Aerospace FOD Prevention Conference. Feeler, Robert A. (2002.) Improvement Understanding of Human Factors Could Reduce Foreign Object Damage. Flight Safety Foundation. Aviation Mechanics Bulletin. Vol. 50 No. 4

Gelman, Goegebeur, Tuerlinckx, Van Mechelen. (2000.) Diagnostic Checks for Discrete Data Regression Models Using Posterior Predictive Simulations.

Harvill, Jane L. (2011.) Introduction to Logistic Regression for Use in ST 6211 Statistical Consulting.

Hilbe, Joseph M. (2009.) Logistic Regression Models. Chapman & Hall/CRC Press.

Howell, David C. (2010.) Statistical Methods for Psychology, 7th ed. Belmont, CA; Thomson Wadsworth.

Johnson, Kraus, Mason,Watson. (2001.) Reducing Foreign Object Damage Through Improved Human Perormance: Best Practices. USA: Galaxy Scientific Corporation.

National Aeronautics and Space Administration. (2011.) Foreign Object Damage (FOD) Prevention Program. USA: Langley Research Center.

Prather, Daniel. (2011.) Current Airport Inspection Practices Regarding FOD (Foreign Object Debris/Damage). Washington D.C. Procaccio, Felice. (2008.) Effectiveness of FOD Control Measures. Embry-Riddle Aeronautical University Worlwide Campus.

Randolph, Sherry. (2004.) FOD Prevention Training. United Space Alliance. USA.

APPENDIX A. Advanced Fighter Contingency Tables

Figure A.1: FOD type vs. Month Mosaic Plot

TableA.1: FOD Type vs. Month Contingency Table.

Count Consumables Manufacturing Panstock Tools Trash Total Total % Debris Col % Row % 01 4 12 34 5 20 75 0.43 1.28 3.63 0.53 2.13 8.00 3.64 10.26 10.63 3.45 8.16 5.33 16.00 45.33 6.67 26.67 02 7 12 19 8 19 65 0.75 1.28 2.03 0.85 2.03 6.94 6.36 10.26 5.94 5.52 7.76 10.77 18.46 29.23 12.31 29.23 03 18 13 28 16 13 88 1.92 1.39 2.99 1.71 1.39 9.39 16.36 11.11 8.75 11.03 5.31 20.45 14.77 31.82 18.18 14.77 04 12 10 34 9 27 92 1.28 1.07 3.63 0.96 2.88 9.82 10.91 8.55 10.63 6.21 11.02 13.04 10.87 36.96 9.78 29.35

05 4 7 16 13 15 55 0.43 0.75 1.71 1.39 1.60 5.87 3.64 5.98 5.00 8.97 6.12 7.27 12.73 29.09 23.64 27.27 06 4 3 23 9 21 60 0.43 0.32 2.45 0.96 2.24 6.40 3.64 2.56 7.19 6.21 8.57 6.67 5.00 38.33 15.00 35.00 07 15 7 23 11 24 80 1.60 0.75 2.45 1.17 2.56 8.54 13.64 5.98 7.19 7.59 9.80 18.75 8.75 28.75 13.75 30.00 08 7 13 33 17 26 96 0.75 1.39 3.52 1.81 2.77 10.25 6.36 11.11 10.31 11.72 10.61 7.29 13.54 34.38 17.71 27.08 09 6 7 17 20 21 71 0.64 0.75 1.81 2.13 2.24 7.58 5.45 5.98 5.31 13.79 8.57 8.45 9.86 23.94 28.17 29.58 10 10 19 28 11 8 76 1.07 2.03 2.99 1.17 0.85 8.11 9.09 16.24 8.75 7.59 3.27 13.16 25.00 36.84 14.47 10.53 11 16 7 32 15 23 93 1.71 0.75 3.42 1.60 2.45 9.93 14.55 5.98 10.00 10.34 9.39 17.20 7.53 34.41 16.13 24.73 12 7 7 33 11 28 86 0.75 0.75 3.52 1.17 2.99 9.18 6.36 5.98 10.31 7.59 11.43 8.14 8.14 38.37 12.79 32.56 Total 110 117 320 145 245 937 11.74 12.49 34.15 15.47 26.15

Table A.2: FOD Type vs. Workstation Pearson P-value

n ChiSquare Prob>ChiSq Likelihood Ratio 81.562 0.0005* Pearson 80.790 0.0006*

Figure A.2: FOD Type vs. Workstation Mosaic Plot

Table A.3: FOD Type vs. Workstation Contingency Table

Count Consumables Manufacturing Panstock Tools Trash Total Total % Debris Col % Row % FINAL ASSY 1 22 16 35 18 63 154 2.35 1.71 3.74 1.92 6.72 16.44 20.00 13.68 10.94 12.41 25.71 14.29 10.39 22.73 11.69 40.91 FINAL ASSY 3 13 11 9 21 16 70 1.39 1.17 0.96 2.24 1.71 7.47 11.82 9.40 2.81 14.48 6.53 18.57 15.71 12.86 30.00 22.86 FORWARD 1 16 12 2 1 32 0.11 1.71 1.28 0.21 0.11 3.42 0.91 13.68 3.75 1.38 0.41 3.13 50.00 37.50 6.25 3.13 MATE 18 26 45 23 18 130 1.92 2.77 4.80 2.45 1.92 13.87 16.36 22.22 14.06 15.86 7.35

13.85 20.00 34.62 17.69 13.85 WING 1 4 1 2 5 3 15 0.43 0.11 0.21 0.53 0.32 1.60 3.64 0.85 0.63 3.45 1.22 26.67 6.67 13.33 33.33 20.00 WING 10 12 7 11 9 17 56 1.28 0.75 1.17 0.96 1.81 5.98 10.91 5.98 3.44 6.21 6.94 21.43 12.50 19.64 16.07 30.36 WING 11 25 15 50 22 43 155 2.67 1.60 5.34 2.35 4.59 16.54 22.73 12.82 15.63 15.17 17.55 16.13 9.68 32.26 14.19 27.74 WING 2 10 7 21 21 2 61 1.07 0.75 2.24 2.24 0.21 6.51 9.09 5.98 6.56 14.48 0.82 16.39 11.48 34.43 34.43 3.28 WING 6 0 9 18 3 1 31 0.00 0.96 1.92 0.32 0.11 3.31 0.00 7.69 5.63 2.07 0.41 0.00 29.03 58.06 9.68 3.23 WING 7 3 5 38 11 41 98 0.32 0.53 4.06 1.17 4.38 10.46 2.73 4.27 11.88 7.59 16.73 3.06 5.10 38.78 11.22 41.84 WING 8 1 4 40 4 25 74 0.11 0.43 4.27 0.43 2.67 7.90 0.91 3.42 12.50 2.76 10.20 1.35 5.41 54.05 5.41 33.78 WING 9 1 0 39 6 15 61 0.11 0.00 4.16 0.64 1.60 6.51 0.91 0.00 12.19 4.14 6.12 1.64 0.00 63.93 9.84 24.59 Total 110 117 320 145 245 937 11.74 12.49 34.15 15.47 26.15

Table A.4: FOD Type vs. Workstation Pearson P-value

Test ChiSquare Prob>ChiSq Likelihood Ratio 264.280 <.0001* Pearson 252.116 <.0001*

Figure A.3: FOD Type vs. Variant Mosaic Plot

Table A.5: FOD Type vs. Workstation Pearson P-value

Count Consumables Manufacturing Panstock Tools Trash Total Total % Debris Col % Row % X 45 56 180 71 132 484 4.80 5.98 19.21 7.58 14.09 51.65 40.91 47.86 56.25 48.97 53.88 9.30 11.57 37.19 14.67 27.27 Y 51 43 111 60 84 349 5.44 4.59 11.85 6.40 8.96 37.25 46.36 36.75 34.69 41.38 34.29 14.61 12.32 31.81 17.19 24.07 Z 14 18 29 14 29 104 1.49 1.92 3.09 1.49 3.09 11.10 12.73 15.38 9.06 9.66 11.84 13.46 17.31 27.88 13.46 27.88 Total 110 117 320 145 245 937 11.74 12.49 34.15 15.47 26.15

Table A.6: FOD Type vs. Workstation Pearson P-value

Test ChiSquare Prob>ChiSq Likelihood Ratio 12.442 0.1325 Pearson 12.584 0.1270

APPENDIX B Large Cargo Aircraft Contingency Tables

Figure B.1: FOD Type vs. Month Mosaic Plot.

Table B.1: FOD Type vs. Month Contingency Table.

Count Manufacturing Tools Trash Total Total % Debris Col % Row % 01 212 30 160 402 0.95 0.13 0.72 1.80 1.93 1.12 1.86 52.74 7.46 39.80 02 424 62 502 988 1.90 0.28 2.25 4.43 3.86 2.30 5.82 42.91 6.28 50.81 03 333 142 559 1034 1.49 0.64 2.51 4.64 3.03 5.28 6.48 32.21 13.73 54.06 04 90 13 219 322 0.40 0.06 0.98 1.44 0.82 0.48 2.54 27.95 4.04 68.01

05 88 26 125 239 0.39 0.12 0.56 1.07 0.80 0.97 1.45 36.82 10.88 52.30 06 481 21 784 1286 2.16 0.09 3.52 5.77 4.38 0.78 9.09 37.40 1.63 60.96 07 3917 108 1373 5398 17.57 0.48 6.16 24.21 35.66 4.01 15.92 72.56 2.00 25.44 08 2508 62 2192 4762 11.25 0.28 9.83 21.36 22.83 2.30 25.42 52.67 1.30 46.03 09 1806 27 477 2310 8.10 0.12 2.14 10.36 16.44 1.00 5.53 78.18 1.17 20.65 10 1025 1041 1779 3845 4.60 4.67 7.98 17.25 9.33 38.70 20.63 26.66 27.07 46.27 11 100 13 309 422 0.45 0.06 1.39 1.89 0.91 0.48 3.58 23.70 3.08 73.22 12 0 1145 143 1288 0.00 5.14 0.64 5.78 0.00 42.57 1.66 0.00 88.90 11.10 Total 10984 2690 8622 22296 49.26 12.06 38.67

Table B.2: FOD Type vs. Workstation Pearson P-value.

Test ChiSquare Prob>ChiSq Likelihood Ratio 9467.080 <.0001* Pearson 11883.50 <.0001*

Figure B.2: FOD Type vs. Workstation Mosaic Plot.

Table B.3: FOD Type vs. Workstation Contingency Table.

Count Manufacturing Tools Trash Total Total % Debris Col % Row % WORKSTATION 1 130 131 2528 2789 0.58 0.59 11.34 12.51 1.18 4.87 29.32 4.66 4.70 90.64 WORKSTATION 2 9038 1908 1525 12471 40.54 8.56 6.84 55.93 82.28 70.93 17.69 72.47 15.30 12.23 WORKSTATION 3 1481 442 780 2703 6.64 1.98 3.50 12.12 13.48 16.43 9.05 54.79 16.35 28.86 WORKSTATION 4 335 209 3789 4333 1.50 0.94 16.99 19.43 3.05 7.77 43.95 7.73 4.82 87.45 Total 10984 2690 8622 22296 49.26 12.06 38.67

Table B.3: FOD Type vs. Workstation Pearson P-value.

ChiSquare ChiSquare Prob>ChiSq Likelihood Ratio 12504.98 <.0001* Pearson 11422.60 <.0001*

APENDIX C Advanced Fighter Model

Table C.1: Advanced Fighter Best Model

ADVANCED FIGHTER Term Estimate Std Error ChiSquare Prob>ChiSq Intercept -2.1428359 98.703786 0 0.9827 MONTH[01] -0.9916869 0.0789367 157.83 <.0001 MONTH[02] -2.4501764 0.0492933 2470.7 <.0001 MONTH[03] 2.38484626 0.0589832 1634.8 <.0001 MONTH[04] -0.2115751 0.0466722 20.55 <.0001 MONTH[05] -0.0313249 0.0488219 0.41 0.5211 MONTH[06] -0.4049892 0.0749251 29.22 <.0001 MONTH[07] 1.62230226 0.0404293 1610.2 <.0001 MONTH[08] -1.394547 0.0493016 800.1 <.0001 MONTH[09] -0.282635 0.05802 23.73 <.0001 MONTH[10] 1.89760051 0.0618875 940.16 <.0001 MONTH[11] 0.69934866 0.0375409 347.04 <.0001 WORKSTATION[FINAL ASSY 1] 0.70554119 98.70379 0 0.9943 WORKSTATION[FINAL ASSY 3] 1.97189061 98.703793 0 0.9841 WORKSTATION[FORWARD] -0.3481444 98.703916 0 0.9972 WORKSTATION[MATE] 2.20286379 98.703794 0 0.9822 WORKSTATION[WING 1] 2.62950352 98.70382 0 0.9787 WORKSTATION[WING 10] 1.88695402 98.703802 0 0.9847 WORKSTATION[WING 11] 1.19253331 98.703788 0 0.9904 WORKSTATION[WING 2] 5.76073062 98.703921 0 0.9535 WORKSTATION[WING 6] -14.105532 1085.7416 0 0.9896 WORKSTATION[WING 7] -0.7094414 98.703798 0 0.9943 WORKSTATION[WING 8] 0.14168054 98.703806 0 0.9989 Intercept -1.6606565 171.39474 0 0.9923 MONTH[01] 1.03025611 0.0400059 663.2 <.0001 MONTH[02] -2.2716188 0.0343291 4378.7 <.0001 MONTH[03] 1.14976878 0.0617432 346.77 <.0001 MONTH[04] 0.14524522 0.0406695 12.75 0.0004 MONTH[05] 0.32778951 0.0401241 66.74 <.0001 MONTH[06] 0.93031314 0.046529 399.77 <.0001 MONTH[07] -1.3235091 0.0696662 360.92 <.0001 MONTH[08] -0.6064563 0.0329837 338.06 <.0001 MONTH[09] -0.5723444 0.0548186 109.01 <.0001

MONTH[10] 2.02821508 0.0566679 1281 <.0001 MONTH[11] -0.4154889 0.0459608 81.72 <.0001 WORKSTATION[FINAL ASSY 1] -0.8022533 171.39475 0 0.9963 WORKSTATION[FINAL ASSY 3] 1.23829687 171.39475 0 0.9942 WORKSTATION[FORWARD] 3.15565252 171.39476 0 0.9853 WORKSTATION[MATE] 2.3695066 171.39475 0 0.989 WORKSTATION[WING 1] -0.6093427 171.3949 0 0.9972 WORKSTATION[WING 10] 2.41948863 171.39475 0 0.9887 WORKSTATION[WING 11] 1.56732664 171.39475 0 0.9927 WORKSTATION[WING 2] 5.23380418 171.39482 0 0.9756 WORKSTATION[WING 6] 7.47878814 171.39488 0 0.9652 WORKSTATION[WING 7] -1.4731135 171.39476 0 0.9931 WORKSTATION[WING 8] -0.2973733 171.39475 0 0.9986 Intercept 1.20701289 0.0293068 1696.2 <.0001 MONTH[01] 1.59451303 0.0300104 2823 <.0001 MONTH[02] -2.5515733 0.0315941 6522.3 <.0001 MONTH[03] 1.60341904 0.0545631 863.57 <.0001 MONTH[04] -0.0479065 0.0297686 2.59 0.1076 MONTH[05] -0.8911557 0.0340311 685.73 <.0001 MONTH[06] -0.4335585 0.0444193 95.27 <.0001 MONTH[07] 0.39547266 0.0351255 126.76 <.0001 MONTH[08] -0.0620072 0.0218816 8.03 0.0046 MONTH[09] -0.5486216 0.0443085 153.31 <.0001 MONTH[10] 1.19429102 0.0534972 498.38 <.0001 MONTH[11] 0.22292107 0.0303927 53.8 <.0001 WORKSTATION[FINAL ASSY 1] -2.1193665 0.0348241 3703.9 <.0001 WORKSTATION[FINAL ASSY 3] -1.1953422 0.0474033 635.87 <.0001 WORKSTATION[FORWARD] -0.0214975 0.0735779 0.09 0.7702 WORKSTATION[MATE] 0.66831077 0.0435205 235.81 <.0001 WORKSTATION[WING 1] -1.520963 0.1100308 191.08 <.0001 WORKSTATION[WING 10] -0.9923765 0.059396 279.15 <.0001 WORKSTATION[WING 11] -0.9200746 0.0330809 773.55 <.0001 WORKSTATION[WING 2] 2.78459545 0.1650644 284.59 <.0001 WORKSTATION[WING 6] 5.13062933 0.2185993 550.86 <.0001 WORKSTATION[WING 7] -1.5101214 0.0374631 1624.9 <.0001 WORKSTATION[WING 8] -0.3496016 0.0405979 74.15 <.0001 Intercept 0.13035642 0.0309798 17.71 <.0001 MONTH[01] 0.31951345 0.0540729 34.92 <.0001 MONTH[02] -3.357947 0.0571721 3449.7 <.0001 MONTH[03] 0.80316418 0.0658597 148.72 <.0001 MONTH[04] -0.8906662 0.0543822 268.24 <.0001

MONTH[05] -1.2301364 0.0628762 382.77 <.0001 MONTH[06] -0.1270868 0.0621961 4.18 0.041 MONTH[07] 1.14854107 0.0416129 761.79 <.0001 MONTH[08] 0.0200373 0.0327533 0.37 0.5407 MONTH[09] 0.96493392 0.0454202 451.33 <.0001 MONTH[10] 2.61467308 0.0567044 2126.2 <.0001 MONTH[11] 0.68796566 0.0368191 349.13 <.0001 WORKSTATION[FINAL ASSY 1] -2.9060847 0.0474383 3752.8 <.0001 WORKSTATION[FINAL ASSY 3] 0.4803543 0.0474917 102.3 <.0001 WORKSTATION[FORWARD] 0.04726044 0.0784275 0.36 0.5468 WORKSTATION[MATE] -0.3090486 0.0519366 35.41 <.0001 WORKSTATION[WING 1] -0.4377112 0.1072615 16.65 <.0001 WORKSTATION[WING 10] 0.14966625 0.0595628 6.31 0.012 WORKSTATION[WING 11] -1.1208386 0.0380319 868.54 <.0001 WORKSTATION[WING 2] 3.67170884 0.1670003 483.39 <.0001 WORKSTATION[WING 6] 4.9328389 0.2202002 501.83 <.0001 WORKSTATION[WING 7] -2.2278062 0.0522221 1819.9 <.0001 WORKSTATION[WING 8] -1.8095762 0.0594684 925.94 <.0001

APENDIX D Large Cargo Aircraft Model

Table D.1: Large Cargo Aircraft Best Model.

LARGE CARGO AIRCRAFT Term Estimate Std Error ChiSquare Prob>ChiSq Intercept 13.187444 14877.069 0 0.9993 WORKSTATION 1 -31.504338 14925.833 0 0.9983 WORKSTATION 2 66.1964558 48005.062 0 0.9989 WORKSTATION 3 -12.145757 19731.754 0 0.9995 MONTH[01] -7.3698287 23447.389 0 0.9997 MONTH[02] 74.6205863 98440.775 0 0.9994 MONTH[03] 1.66636357 8975.1753 0 0.9999 MONTH[04] 23.1096137 19128.187 0 0.999 MONTH[05] 23.1095928 28175.622 0 0.9993 MONTH[06] 2.06225923 8975.1753 0 0.9998 MONTH[07] -8.9620436 14894.421 0 0.9995 MONTH[08] -24.943566 21088.083 0 0.9991 MONTH[09] -18.099047 21400.718 0 0.9993 MONTH[10] -14.737921 15735.947 0 0.9993 MONTH[11] -25.193016 19569.235 0 0.999 WORKSTATION 1*MONTH[01] 24.3004285 23478.345 0 0.9992 WORKSTATION 1*MONTH[02] -80.130457 99261.67 0 0.9994 WORKSTATION 1*MONTH[03] -7.3785558 12316.129 0 0.9995 WORKSTATION 1*MONTH[04] -28.678075 22181.913 0 0.999 WORKSTATION 1*MONTH[05] -27.898802 31202.753 0 0.9993 WORKSTATION 1*MONTH[06] -7.8966738 11625.43 0 0.9995 WORKSTATION 1*MONTH[07] 23.4304283 14943.129 0 0.9987 WORKSTATION 1*MONTH[08] 43.4740342 21010.84 0 0.9983 WORKSTATION 1*MONTH[09] 13.086983 22361.84 0 0.9995 WORKSTATION 1*MONTH[10] 8.90350205 19321.858 0 0.9996 WORKSTATION 1*MONTH[11] 19.3586086 25361.897 0 0.9994 WORKSTATION 2*MONTH[01] -72.014316 72089.792 0 0.9992 WORKSTATION 2*MONTH[02] 222.524143 292902.52 0 0.9994 WORKSTATION 2*MONTH[03] 1.65544398 20178.159 0 0.9999 WORKSTATION 2*MONTH[04] 66.5800735 47221.45 0 0.9989 WORKSTATION 2*MONTH[05] 41.9467306 58043.246 0 0.9994 WORKSTATION 2*MONTH[06] 2.68902793 19764.155 0 0.9999 WORKSTATION 2*MONTH[07] -46.270548 48053.464 0 0.9992 WORKSTATION 2*MONTH[08] -53.985194 41771.171 0 0.999

WORKSTATION 2*MONTH[09] -37.133544 41986.721 0 0.9993 WORKSTATION 2*MONTH[10] -41.831439 48493.379 0 0.9993 WORKSTATION 2*MONTH[11] -30.039469 43054.776 0 0.9994 WORKSTATION 3*MONTH[01] 30.4798881 39407.652 0 0.9994 WORKSTATION 3*MONTH[02] -75.219629 95859.456 0 0.9994 WORKSTATION 3*MONTH[03] 0 0 . . WORKSTATION 3*MONTH[04] 0 0 . . WORKSTATION 3*MONTH[05] 0 0 . . WORKSTATION 3*MONTH[06] 0 0 . . WORKSTATION 3*MONTH[07] 6.82174471 19744.841 0 0.9997 WORKSTATION 3*MONTH[08] 0 0 . . WORKSTATION 3*MONTH[09] 0 0 . . WORKSTATION 3*MONTH[10] 13.6962653 23413.291 0 0.9995 WORKSTATION 3*MONTH[11] 0 0 . . Intercept 6.9826703 15354.578 0 0.9996 WORKSTATION 1 -19.964383 15340.121 0 0.999 WORKSTATION 2 51.3397735 49166.109 0 0.9992 WORKSTATION 3 -11.437935 19978.693 0 0.9995 MONTH[01] -7.273789 21608.885 0 0.9997 MONTH[02] 77.1210891 98808.865 0 0.9994 MONTH[03] 5.76745095 9467.4277 0 0.9995 MONTH[04] 4.45524985 25696.234 0 0.9999 MONTH[05] 4.45522221 38939.224 0 0.9999 MONTH[06] 4.45526456 9467.4277 0 0.9996 MONTH[07] -14.075567 15518.19 0 0.9993 MONTH[08] -26.554781 21660.957 0 0.999 MONTH[09] -19.17495 21711.285 0 0.9993 MONTH[10] -2.4959386 16109.746 0 0.9999 MONTH[11] -19.696658 19783.82 0 0.9992 WORKSTATION 1*MONTH[01] -3.2119502 24055.706 0 0.9999 WORKSTATION 1*MONTH[02] -66.510954 98930.31 0 0.9995 WORKSTATION 1*MONTH[03] 3.68545373 10163.036 0 0.9997 WORKSTATION 1*MONTH[04] 5.90962914 25960.577 0 0.9998 WORKSTATION 1*MONTH[05] 7.77725433 39114.165 0 0.9998 WORKSTATION 1*MONTH[06] -15.62486 11929.571 0 0.999 WORKSTATION 1*MONTH[07] 24.3073828 15503.886 0 0.9987 WORKSTATION 1*MONTH[08] 16.9679422 22022.445 0 0.9994 WORKSTATION 1*MONTH[09] 31.0336599 21592.51 0 0.9989 WORKSTATION 1*MONTH[10] -8.6736981 19579.946 0 0.9996 WORKSTATION 1*MONTH[11] 8.5271792 25482.08 0 0.9997 WORKSTATION 2*MONTH[01] -26.897567 64844.379 0 0.9997

WORKSTATION 2*MONTH[02] 226.422905 293295.45 0 0.9994 WORKSTATION 2*MONTH[03] -9.9642189 19631.554 0 0.9996 WORKSTATION 2*MONTH[04] 9.74213423 57396.693 0 0.9999 WORKSTATION 2*MONTH[05] 6.55305123 78675.153 0 0.9999 WORKSTATION 2*MONTH[06] 30.8713101 24238.238 0 0.999 WORKSTATION 2*MONTH[07] -44.246878 49274.19 0 0.9993 WORKSTATION 2*MONTH[08] -54.236456 42954.749 0 0.999 WORKSTATION 2*MONTH[09] -39.147494 43078.972 0 0.9993 WORKSTATION 2*MONTH[10] -33.282665 49462.48 0 0.9995 WORKSTATION 2*MONTH[11] -38.625577 42413.073 0 0.9993 WORKSTATION 3*MONTH[01] 11.7298615 47938.324 0 0.9998 WORKSTATION 3*MONTH[02] -95.139511 96017.414 0 0.9992 WORKSTATION 3*MONTH[03] 0 0 . . WORKSTATION 3*MONTH[04] 0 0 . . WORKSTATION 3*MONTH[05] 0 0 . . WORKSTATION 3*MONTH[06] 0 0 . . WORKSTATION 3*MONTH[07] 16.7175658 20104.697 0 0.9993 WORKSTATION 3*MONTH[08] 0 0 . . WORKSTATION 3*MONTH[09] 0 0 . . WORKSTATION 3*MONTH[10] 31.1025404 22115.552 0 0.9989 WORKSTATION 3*MONTH[11] 0 0 . .

APENDIX E Advanced Fighter Results

Final Assembly 1 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.04 0.49 0.12 0.72 0.12 0.25 0.41 0.87 0.28 0.00 0.61 0.91 Tools 0.00 0.03 0.00 0.00 0.88 0.25 0.19 0.00 0.44 0.05 0.04 0.01 Panstock 0.93 0.22 0.77 0.04 0.00 0.25 0.00 0.03 0.18 0.72 0.14 0.08 Manufacturing Debris 0.04 0.00 0.12 0.17 0.00 0.00 0.13 0.03 0.10 0.03 0.04 0.00 Consumables 0.00 0.26 0.00 0.07 0.00 0.25 0.27 0.06 0.00 0.20 0.17 0.00

Figure E.1: Predicted Probabilities for Final Assembly 1

Final Assembly 3 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.17 0.99 0.01 0.39 0.07 0.00 0.00 0.00 0.12 0.00 0.02 0.01 Tools 0.78 0.01 0.10 0.00 0.00 1.00 0.00 1.00 0.06 0.78 0.70 0.97 Panstock 0.00 0.00 0.33 0.00 0.04 0.00 0.33 0.00 0.30 0.00 0.12 0.00 Manufacturing Debris 0.06 0.01 0.28 0.00 0.07 0.00 0.00 0.00 0.52 0.22 0.00 0.01 Consumables 0.00 0.00 0.28 0.61 0.82 0.00 0.67 0.00 0.00 0.00 0.16 0.00

Figure E.2: Predicted Probabilities for Assembly 3

FORWARD 1.00

0.90

0.80

0.70

Probability 0.60

0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 1.00 0.00 0.00 1.00 1.00 0.00 0.00 0.00 Tools 0.00 0.00 0.00 0.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 Panstock 0.62 0.35 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.68 0.46 0.36 Manufacturing Debris 0.38 0.37 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.32 0.54 0.64 Consumables 0.00 0.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Figure E.3: Predicted Probabilities for FORWARD

MATE 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.23 0.00 0.06 0.00 0.07 0.24 0.12 0.12 0.01 0.00 0.00 Tools 0.00 0.02 0.05 0.17 0.25 0.00 0.11 0.45 0.25 0.15 0.00 0.00 Panstock 0.43 0.16 0.46 0.72 0.49 0.80 0.36 0.22 0.30 0.39 0.57 1.00 Manufacturing Debris 0.57 0.57 0.42 0.00 0.26 0.00 0.10 0.09 0.00 0.30 0.33 0.00 Consumables 0.00 0.02 0.07 0.06 0.00 0.13 0.19 0.12 0.33 0.15 0.11 0.00

Figure E.4: Predicted Probabilities for MATE

WING 1 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.13 0.81 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Tools 1.00 1.00 - 0.00 0.19 1.00 0.00 1.00 1.00 0.00 0.41 0.00 Panstock - 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.59 1.00 Manufacturing Debris - 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Consumables - 0.00 1.00 0.87 0.00 0.00 1.00 0.00 0.00 1.00 0.00 0.00

Figure E.5: Predicted Probabilities for WING 1

WING 2 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.00 Tools 0.00 0.00 0.17 0.21 1.00 0.00 0.50 0.63 0.24 1.00 0.14 0.00 Panstock 0.00 0.19 0.09 0.24 0.00 0.00 0.25 0.37 0.67 0.00 0.71 0.00 Manufacturing Debris 0.00 0.00 0.13 0.37 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 Consumables 1.00 0.81 0.60 0.17 0.00 0.00 0.12 0.00 0.08 0.00 0.14 1.00

Figure E.6: Predicted Probabilities for WING 2

WING 6 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.00 0.00 Tools 0.00 0.03 0.00 0.10 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 Panstock 0.53 0.84 1.00 0.26 0.78 0.65 0.00 1.00 0.00 0.06 1.00 1.00 Manufacturing Debris 0.47 0.13 0.00 0.64 0.22 0.24 0.00 0.00 1.00 0.03 0.00 0.00 Consumables 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00

Figure E.7: Predicted Probabilities for WING 6

WING 7 1.00

0.90

0.80

0.70

Probability 0.60

0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.49 0.80 0.18 0.17 0.87 0.54 0.27 0.70 0.57 0.09 0.29 0.53 Tools 0.01 0.10 0.14 0.06 0.00 0.00 0.00 0.00 0.22 0.22 0.08 0.00 Panstock 0.32 0.10 0.21 0.75 0.13 0.46 0.70 0.30 0.22 0.69 0.63 0.47 Manufacturing Debris 0.13 0.00 0.02 0.01 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 Consumables 0.03 0.00 0.45 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Figure E.8: Predicted Probabilities for WING 7

WING 8 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 1.00 0.00 0.24 0.40 0.61 0.17 0.39 0.00 0.15 0.00 0.00 Tools 0.00 0.00 0.00 0.00 0.00 0.15 0.00 0.24 0.00 0.00 0.00 0.00 Panstock 1.00 0.00 0.00 0.64 0.60 0.24 0.83 0.37 1.00 0.47 0.94 0.91 Manufacturing Debris 0.00 0.00 1.00 0.12 0.00 0.00 0.00 0.00 0.00 0.19 0.06 0.09 Consumables 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00

Figure E.9: Predicted Probabilities for WING 8

WING 9 1.00

0.90

0.80

0.70

Probability 0.60 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 1.00 0.08 0.02 0.03 0.00 0.09 0.00 0.00 Tools 0.00 0.00 0.00 0.00 0.00 0.92 0.00 0.97 0.02 0.00 0.00 0.00 Panstock 1.00 0.00 0.00 0.00 0.00 0.00 0.30 0.00 0.98 0.00 1.00 0.86 Manufacturing Debris 0.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Consumables 0.00 1.00 0.00 0.00 0.00 0.00 0.68 0.00 0.00 0.91 0.00 0.14

Figure E.10: Predicted Probabilities for WING 9

WING 10 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.53 0.10 0.25 0.20 0.34 0.29 0.00 0.00 0.32 0.05 0.00 Tools - 0.30 0.06 0.06 0.41 0.00 0.00 0.61 0.00 0.00 0.05 0.00 Panstock 0.00 0.00 0.61 0.06 0.39 0.00 0.49 0.00 1.00 0.00 0.05 0.00 Manufacturing Debris 0.00 0.00 0.00 0.63 0.00 0.00 0.00 0.39 0.00 0.59 0.00 0.00 Consumables 1.00 0.18 0.22 0.00 0.00 0.66 0.22 0.00 0.00 0.09 0.84 1.00

Figure E.11: Predicted Probabilities for WING 10

WING 11 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.48 0.83 0.08 0.47 0.19 0.00 0.07 0.05 0.33 0.04 0.28 0.16 Tools 0.09 0.02 0.15 0.00 0.02 0.13 0.30 0.00 0.50 0.01 0.05 0.01 Panstock 0.37 0.06 0.44 0.39 0.08 0.03 0.20 0.77 0.02 0.17 0.19 0.82 Manufacturing Debris 0.00 0.08 0.00 0.00 0.71 0.85 0.02 0.17 0.12 0.73 0.00 0.00 Consumables 0.06 0.00 0.32 0.14 0.00 0.00 0.41 0.00 0.03 0.06 0.47 0.01

Figure E.12: Predicted Probabilities for WING 11

APENDIX F Large Cargo Aircraft Results

WORKSTATION 1 1.00

0.90

0.80

0.70

0.60 Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.80 0.91 0.97 0.93 0.68 1.00 0.92 0.45 0.75 1.00 1.00 1.00 Tools 0.00 0.09 0.03 0.07 0.32 0.00 0.06 0.00 0.25 0.00 0.00 0.00 Manufacturing Debris 0.20 0.00 0.00 0.00 0.00 0.00 0.02 0.55 0.00 0.00 0.00 0.00

Figure F.1: Predicted Probabilities for WORKSTATION 1

WORKSTATION 2 1.00

0.90

0.80

0.70

0.60

Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.39 0.00 0.00 0.00 0.00 Tools 1.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.43 0.00 0.00 Manufacturing Debris 0.00 1.00 1.00 1.00 1.00 0.00 1.00 0.61 1.00 0.57 1.00 1.00

Figure F.2: Predicted Probabilities for WORKSTATION 2

WORKSTATION 3 1.00

0.90

0.80

0.70

0.60 Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.39 0.05 0.00 0.00 0.04 0.67 1.00 1.00 0.00 1.00 0.26 Tools 0.00 0.00 0.19 0.00 0.00 0.04 0.11 0.00 0.00 1.00 0.00 0.00 Manufacturing Debris 1.00 0.61 0.76 1.00 1.00 0.92 0.22 0.00 0.00 0.00 0.00 0.74

Figure F.3: Predicted Probabilities for WORKSTATION 3

WORKSTATION 4 1.00

0.90

0.80

0.70

0.60 Probability 0.50

0.40

0.30

0.20

0.10

- JAN FEB MAR APR MAY JUNE JULY AUG SEP OCT NOV DEC Trash 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.00 0.99 0.01 1.00 0.00 Tools 0.00 0.02 0.11 0.00 0.00 0.02 0.00 0.00 0.00 0.99 0.00 0.00 Manufacturing Debris 0.99 0.98 0.89 1.00 1.00 0.98 0.99 0.00 0.01 0.00 0.00 1.00

Figure F.4: Predicted Probabilities for WORKSTATION 4

Vita

David Ricardo Romo was born in El Paso Texas on March 6th, 1990. He started college education on August 2007 at El Paso Community College, and transferred to the University of Texas at

El Paso on January 2010. He graduated with a Bachelor of Science in Industrial Engineering in May 2012. It was on his last semester of undergraduate studies that he begun to work in the Research Institute for Manufacturing and Engineering Systems under the guidance of Dr. Tseng.

In August 2012, he enrolled in the Industrial Engineering graduate program of The University of Texas at El Paso. He continued working with the Research Institute for Manufacturing and Engineering Systems research group for Dr. Tseng. He conducted research in quality control, and had the opportunity to work in projects for Lockheed Martin through the university. He had the opportunity to publish a paper in the Institute of Industrial Engineers, and had the privilege to attend the Institute of Industrial Engineer Annual Conference & Expo 2013 in San Juan, Puerto Rico. In addition, David was selected to collaborate in an Internship with Lockheed Martin in Fort

Worth, Texas from June to August 2013 working as a Quality Engineer. In this position, he was able to apply his knowledge and education in quality control and statistical process control to improve the quality of the product.

Permanent address: 8725 Orion Pl. El Paso, TX 79904

This thesis was typed by David Ricardo Romo.