Tim Mccoy: Mcnair Scholar

“Misrepresenting Quality Data through Incorrect Statistical Applications– A Statistical Quality Control (SQC) Case Study” Your Picture Tim McCoy: McNair Scholar Here

Dr. Richard.Yearout: Mentor

Department of Mathematics

Abstract

Statistical Process Control (SPC) is an essential element for quality assurance. A major SPC tool is the control chart. The investigator examined a western North Carolina manufacturer who was using incorrect control chart methodology. Client specifications required strict adherence to specified tolerances. Initial observations uncovered malpractice that appeared to be the result of ignorance and economic constraints. It was noted that systems used to measure quality performance had misapplied accepted statistical applications. Recognizing the risk of misapplication of SPC, the investigator examined the tradeoff between continuing the present practice versus adhering to statistically sound 6control. The principle error was constructing control charts without statistically calculated control limits, which makes the current state of statistical stability not only unknown, but also noninferable. Thus, validation of process capability cannot be assured. After a full discussion explaining why “specification” charts with sample averaging are inappropriate for process monitoring, proposed solutions with appropriate examples are recommended for quality and industrial engineers.

Introduction

Since international trade has become an essential component of manufacturing strategy, quality has surfaced as one of production's main foci. In today’s international environment, Statistical Process Control (SPC) is a fundamental element of any organization’s quality master plan. One of the main tools of SPC is the control chart. Originally introduced by Dr. Walter Shewhart in the 1920’s, the control chart has revolutionized the way companies in both manufacturing and service industries monitor quality7. This study examines a specific case where a western North Carolina (wNC) manufacturer was utilizing incorrect control chart methodology. Client product specifications require all vendors, including this wNC manufacturer, to implement the Quality System 9000 (QS-9000), which is the automotive industry’s supplement to the International Standards Organization series 9000 (ISO-9000). In other words, QS-9000 certified suppliers are more than compliant with ISO-9000 guidelines. Initially this organization’s malpractices appeared to be the results of the manufacturer’s ignorance and economic rationale for training and sampling costs. Correctly applying statistical control charts has been shown in saving resources by reducing scrap and re-work, not to mention saving potential external cost from product failure and contract loss. The critical error in this case study is the manufacturer using control charts without statistically calculated control limits. This makes the current state of statistical stability not only unknown, but non-inferable. Thus validation of process capability cannot be assured. “The accepted practice in the automotive industry is to calculate capability only after a process has been demonstrated to be in a state of statistical control.”1 In addition to the process capability issue, it appeared that the several internal and external systems used to measure the organization’s quality performance were distorting the applications of accepted statistical practices. Ambiguous requirements within the company’s quality system allowed the organization to incorrectly implement and guarantee obligatory quality levels. The specific customer requirements in question will be analyzed according to QS-9000 criteria to determine the degree of misinterpretation from the data derived from quality inspections. First, an introduction to control charts and their various types is needed. There is a dichotomy among type of control charts, being variable charts and attribute charts. Variable charts are used for numeric data; characteristics that can be measured. Attribute charts are used for categorical data; pass/fail, ok/nok characteristics. The formulation of the calculated control limits differ for either type and become further differentiated depending on the situation. Variable control charts were of interest in this investigation. The basic idea of variable charts is to compare process characteristics such as the mean (in the x-bar chart) and the spread (in the R chart) to its natural variability in the format of a visual display. The idea is to produce a signal that indicates when to take action on the system and when to leave the process alone. By taking samples of subgroups and plotting their means (or range) over time, the charts can use statistical theory to signal when an event with low probability happens, such as when a mean is outside 3from the average mean. It should be noted this is exactly where the calculated control limits are placed on an x-bar chart so that when a point crosses over a control limit a signal is made that the process has shifted. Refer to the QS-9000 form in Figure 5 for examples of an x-bar and R chart and methodology of data collection. Of critical importance for this investigation is the nature of specific product being made. Type II error (consumer's risk), as it applies to the specific industry cannot be overlooked. In strict statistical terms, Type II error is the probability of failing to reject a null hypothesis, when in fact the null hypothesis was false6. In practical terms, it is “keeping and shipping a bad part” (as opposed to Type I error which is “throwing away a good part”). Assumed probabilities do not match up with actual practiced probabilities of Type II errors for this situation. How exactly was the control chart being misused here? What are the implications of such a malpractice? After analysis and discussion, the proposed solutions are given to consider more appropriate control chart methods such as the standard x-bar and R charts, and alternative control charts for individual measurements.

Observations of Incorrect Charting Practices

In midst of daily operations, the investigator monitored charts in the facility. While recording limit changes and chart attributes, the investigator noticed that the variable charts had no calculations of control limits that are used in the typical control chart, depicted in Figure 13. The charts were being used with specification limits, thus nullifying the investigative effect on the variation in the process, the central goal of the control chart. Also, using sample averaging and plotting averages against specification limits for characteristics is hazardous.

Figure 1: Control Chart vs. Observed Specification Chart Situation

Control charts are formulated such that “too much” or “special cause” variability in the process causes an out-of control situation. Using charting based on limits with specifications does not allow similar answers to be justifiably formed from an out-of-specification situation. True, both are undesired and while specifications are what the customer requires in the end, warning flags are raised in cases of non-control- chart monitoring. One of which is the validity of Process Capability studies, where assumptions of “in control” are required. By using specification (spec) charts instead of control charts, this cannot be assumed. Thus validity of requalifications and other various Q.A. efforts are in question. Monitoring charts that raise warning signals only when near or cross over specifications is dangerous because: 1) Charts with specifications as limits and sample averaging allow out-of-specification parts to be made while the charts do not reflect this, 2) there may exist an out of control situation as far as too much variation within the process, regardless of closeness to specifications, 3) and assumptions required for other SPC tools like capability studies may be violated.

Discussion of Process Capability

Process capability is defined to be the capability of the process to meet design specifications4. Process capability can only be determined and hold validity once a process has demonstrated a state of statistical control, via correct control chart methods with calculated control limits. The two most common measures of process capability are the Process Capability Ratio, Cp, and the Process Capability Index, or 5 Cpk . Their equations are shown here:

Cp = (USL – LSL)/ 6 (1) Cpk = Minimum of: [ ( x - LSL)/3, (USL - x )/3] (2)

The population (process) standard deviation is , the Upper Specification Limit is USL, the Lower Specification Limit is LSL, and the average of the subgroup means is x-double-bar, which is the centerline of the x-bar control chart. Note that the process standard deviation is rarely known and is replaced by estimates. A commonly used estimate is R-bar/d2 for the process standard deviation (although other estimates exist) 2. The R-bar and x-doublebar are taken from the control charts where the process has demonstrated statistical control and thus can be used for process capability analysis. This is the standard procedure for performing process capability studies. In the present case, the state of statistical control is unknown with use of specification charts. Thus, the validity of using R-bar/d2 as an estimate of is unfounded. In cases where the process is not in statistical control, the Rbar/ d2 estimate will not reflect well due to the presence of special variation. This is another crucial misapplication on behalf of the manufacturer in question, whose capability performance is a major part of the customer’s requirements.

Motivation for Chart Revision

At the center for this motivation for revision is that specification limits do not work well with sample averaging, because out-of-specification parts can be produced without being caught by the monitor if the average of the sample “averages” to be in specification. It is the individual piece that needs to be upheld to specifications, not the average of a sample of pieces. The reason the average is used on control charts with statistically calculated limits (UCL’s and LCL’s and not USL’s and LSL’s) is because calculating the average and plotting it against the calculated control limits allow for unbiased monitoring of variation due to common causes. Once a sample average crosses the calculated control limit then assignable cause should be sought. This knowledge is based upon sound statistical theory like the Central Limit Theorem. However, in the case of using specifications as limits for charts, out-of specification parts can be made and the chart looks in specification! “If specification limits are drawn on the average(s) chart, there is a natural tendency to compare the subgroup averages, plotted on the chart, with the specification limits. This sometimes leads to the false conclusion that whenever an average is within specification limits, all the product is within specifications.”3 Consider Figure 23 where the situation depicts individual pieces being out-of-specification and the average plots inside specifications on the chart. Figure 2: Plots of Individual measurements vs. Averages of Five to Specification Limits

In Figure 2 there exist sample points that are in specification and out-of-specification on the low side at the same time, which allows the average to be plotted in specification. Another potential case exists when a group of points is outside a specification limit on the high side and out of specification on the low side at the same time, and the average plots in. The monitor is unaware of the individual parts being out of specification if they are only monitoring the average, especially if the chart does not explicitly disclose engineering specifications on the chart. Impacts of Misleading QS-9000 Requirements

How could a situation like this evolve under the strict quality system of QS-9000? Reviewing the requirements placed on suppliers by QS-9000 may reveal the rationale. Shown in Figure 3 are the QS-9000 requirements that suppliers are expected to follow regarding process monitoring1.

Figure 3: QS-9000 Guidelines, 3rd Edition

In no part do the requirements specifically state that control charts with calculated control limits must be employed. Instead, the requirements leave gray areas of interpretation open, letting the supplier themselves determine appropriate statistical techniques. QS-9000 does remark, “The supplier shall identify the need for statistical techniques required for establishing, controlling and verifying process capability and product characteristics.” Regarding appropriate statistical techniques, the manufacturer has not implemented control charts but rather specification charts, which cannot verify process capability. Also in the SPC book of QS-9000, no model is given where specification limits are being used 2. From the introduction and throughout, control charts are being correctly demonstrated with calculated control limits. Thus, it can be deduced that QS-9000 correctly dictates the use of control charts with calculated control limits and not the observed situation here. But one may see how these requirements can be misinterpreted to lead to the observed situation. Therefore, the impacts of these ambiguous requirements may lead suppliers required to follow QS-9000 guidelines to incorrectly implement and monitor with improper SPC techniques and practices.

Scope of Misapplication

The charting practices that were discovered did not conform to QS-9000 requirements and did not allow the state of statistical control to be known. The manufacturer was using charts with specification limits on every numeric characteristic being monitored (not with attribute or pass/fail characteristics). Since a malpractice in the initial formulation of the control chart was detected, data were collected on the number of control charts that could be included in this misapplication. Seen in Table 1 is the quantity of control charts used at the facility, noting this was happening to all charts that used samples of three and five individuals. Variable charts with one and two individuals were also not constructed with calculated control limits but were not quite as bad, since they did not use sample averaging (impractical for samples of one and not employed for two).

Table 1: Frequency of occurrence of specification charts with sample averaging Frequency type number of charts % with averaging 5 times per shift 67 46.5% 3 times per shift 2 1.4% 2 times per shift 6 0.0% 1 time per shift 69 0.0% Total 144 47.9%

The total number of characteristics under this scope is one hundred and forty-four (144). This misapplication of the control charts occurred in situations where a sample average was computed on monitoring characteristics. This is alarming because the supplier in question was using these kinds of charts for their most important characteristics. Less important variables were being monitored using attribute charts with pass/fail monitoring. Thus, this misapplication is being applied to the most critical characteristics, since they require more than simple pass/fail monitoring. They require variable chart monitoring, and commonly utilize destructive testing and manual off-line tests that are more costly. Thus, if the type of monitoring is misleading insofar that the theoretical statistical foundations are not being applied properly; this monitoring is an economic waste. This misapplication is occurring on all characteristics in the facility that require samples of three and five. Thus, the manufacturer has deemed these characteristics important enough to spend extra cost to ensure detection of special variation. However, this is nullified in that the charts will allow for potential false or no alarms to be sounded when possible special variation is present. This is because the control chart has been incorrectly implemented into the quality system.

Application of Correct Practices

The correct control chart formulation with calculated control limits was illustrated in Figure 1. Notice the calculated control limits that are estimates of +/- 3. These limits follow correct statistical theory by use of the Central Limit Theorem. The Central Limit Theorem discloses that a sample average follows Normal distribution probabilities (despite whatever distribution its individuals follow) once a large enough sample size has been achieved 6. Thus, the chance that a sample average exceeds 3 standard deviations from the mean of sample averages is approximately 27 out of 10,000 (.0027). This probability represents the chance that a single point on the x-bar chart goes outside a control limit. This is further emphasized by QS-9000 in their manuals designed for QS- 9000 compliant suppliers. On their following control chart form in Figure 4 2, the analyst is given a complete and fundamentally sound template for producing both x-bar and R control charts for numeric characteristics. It is this template that was misapplied in the manufacturer’s quality system. QS-9000 manuals also divulge other various control charts for numeric variables such as x-bar and s charts, median charts, and control charts for individuals, but none with specification limits. Directly on their x-bar and R chart form QS-9000 states process capability (which analyzes performance to specifications) can only be determined when the process is in (statistical) control. QS-9000’s adoption of a sound control chart is presented in Figure 4: Figure 4: QS-9000 form for x-bar and R control charts

Notice the QS-9000 form does include calculations of control limits for both x-bar chart and R chart. It further discloses “rules for suspicion” such as “runs of 7” derived from probability theory which are only valid in the use of control charts with calculated control limits, since these rules have been formulated according to probability laws. These and other rules have no basis for corrective action on charts without calculated control limits, which is the situation under scope here. The form assists in the proper plotting and analysis by listing all prudent information such as applicable calculations and constants for particular subgroup sizes, the rules for suspicion besides out of control points, and other information like engineering specifications and sample size and frequency. By putting both x-bar and the R charts together into one form, QS-9000 had made it easy for the two charts to be pair wise monitored, as they should be.

Discussion

It can be deduced that using the right rules with the wrong analysis will yield inconclusive results. This is the essence of the situation here. Engineering specifications cannot replace control limits on control charts and result in the same benefits. The use of Shewhart control charts results in a different kind of analysis than plotting subgroup averages against specifications. The Type II error cannot be inferred in the latter case, which is unacceptable risk since preventing Type II error is a high priority considering the particular industry. Performing to specifications and being in a state of statistical control are two different goals manufacturers strive for. Customers require specifications to be met; yet manufacturers can predict better on their process when in a state of statistical control. When quality analysis of data misleads the manufacturer into a false confidence of satisfactory quality levels, the customer’s assurance of quality is nullified. In summary, the charting method with specification limits and sample averaging was happening on important numeric variables. SPC tools were being misapplied which lead to not only the QS-9000 guidelines being misinterpreted but also to process capability being nullified. For this specific case, inference on Type I and II errors have been misdirected and the standard “runs rules” for corrective action are invalid since control chart limits formulated with probability laws are not being applied. It has been shown where use of specification limits with sample averaging is bad for producing product to blueprint tolerances and given evidence that the correct standard is adherence to correct control chart methodologies. Control charts are only beneficial when properly applied with understanding and precision. Only then can the benefits described by Shewhart and all subsequent authors be enjoyed.

Conclusions and Recommendations

When sound control chart methods are not used, the state of statistical control is unknown. Thus, the manufacturer nullified any validation of obligatory process capability levels. Also, the manufacturer has allowed a non-inferable level of Type II error to exist. This is a most dangerous practice, since the importance of Type II error, or consumer’s risk, cannot be underemphasized due to the nature of the industry. For the immediate short term it is recommended that all specification charts with sample averaging be switched to charts for individuals to avoid making a higher percentage of potential Type II errors. This is an immediate temporary fix, with the recommended course of action being to implement x-bar and R control charts for all numeric characteristics and determine the economic and quality level effects on scrap and rework. Also, it is recommended that training for operators and management be provided on correct SPC procedures to ensure no repeat of this situation. Another macro/long-term recommendation would be revision of the QS-9000 guidelines to resolve ambiguity in interpretation and application of control chart methods that could be dangerously misleading. Proper training and careful attention to correct procedures can then lead to all the fruitful benefits of sound SPC practices.

References

[1] Automotive Industry Action Group (August 1994). Quality System Requirements – QS-9000 Third Edition, AIAG, MI. [2] Automotive Industry Action Group (1991). QS-9000 Statistical Process Control – SPC, AIAG, MI. [3] Grant, E.L. and Leavenworth, R.S. (1996). Statistical Quality Control, 7th Edition, McGraw-Hill Companies, Inc., New York. [4] Krajewski, L.J. and Ritzman, L.P. (1999). Operations Management Strategy and Analysis 5th edition, Addison-Wesley, Reading. [5] Lyda, M., Blackwelder, C., Yearout, R., Nelms, L. (2000). Process Capability Ratio and Process Capability Index: When is Each Appropriate? (Two Case Studies to Illustrate Appropriate Use, International Journal of Industrial Engineering Theory, Application, and Practice, 2000, pp. 341- 347. [6] McCabe, G.P. and Moore, D.S. (2000). Introduction to the Practice of Statistics, W.H. Freeman and Company, New York. [7] Shewhart, W.A. (1931). Economic Control of Quality of Manufactured Product, D. Van Nostrand Company, Inc., Princeton, N.J. Republished by American Society for Quality Control, Milwaukee, Wis., 1980.