Six Sigma Metric Analysis for Analytical Testing Processes
Total Page:16
File Type:pdf, Size:1020Kb
DIAGNOSTICS Six Sigma Metric Analysis for Analytical Testing Processes Sten Westgard, M.S., Westgard QC DIAGNOSTICS Introduction Adopting Six Sigma as the Goal for Laboratory Testing Laboratories seek objective assessment and comparison of analytical methods and instrumentation performance. Six Sigma is a widely-accepted quality management Unfortunately, there are few ways to compare systems system, perhaps best known outside of healthcare as the 1 on a level playing field to make an “apples to apples” product of innovation at General Electric and Motorola. comparison. Current methods of assessment can be Six Sigma is also well known for the colorful titles of its arbitrary, relying on unclear “state of the art” assessments, practitioners – green belt (part-time Six Sigma worker), or focusing more on easily tangible efficiency metrics, black belt (full-time Six Sigma worker), master black such as speed, cost, or ease of use. Analytical goals and belt (consultant to black belts), and champion (executive requirements for the quality delivered by a test are often proponent of Six Sigma efforts). Six Sigma has been overlooked during the decision-making process leading adopted by both manufacturing and service industries, as to the purchase of instrumentation. Rapidly changing well as healthcare institutions, from, hospitals to reference regulatory schemes increase the confusion over acceptable laboratories. standards for instrument and method quality. Six Sigma is a metric that quantifies the performance of A technique to objectively and quantitatively assess the processes as a rate of Defects-Per-Million Opportunities, performance of methods, instruments, and laboratories (DPM, or DPMO). Six Sigma programs also encompass is laid out in this paper. The technique consists of three robust techniques such as Define-Measure-Analyze- components: (1) the Six Sigma metric, a widely-accepted Improve-Control (DMAIC) and Root Cause analysis to find measure of quality management, process improvement, and eliminate defects and variation within a process. and universal benchmarking; (2) quality requirements in the form of specific quantitative goals for analytical tests; The goal of Six Sigma, in its simplest distillation, is to and (3) performance data from method validation and eliminate or reduce all variation in a process. For example, verification studies or routine laboratory data. variation in a process leads to wasted effort and resources on retesting and workarounds. Reducing defects reduces One way to understand how Sigma metric analysis costs, and improves performance and profitability. A combines these three components is to picture a target process that achieves the goal of Six Sigma delivers both with an arrow (Figure 1). The shape of the target is quality and efficiency. determined by Six Sigma metrics. The size of the target is determined by the size of the quality requirement. Where The quantitative goal of Six Sigma is to create a process the arrow hits that target is determined by the method that minimizes variation until six standard deviations can performance data. fit within the tolerance limit (Figure 2). At the level of Six Sigma performance (world class quality performance), approximately three defects will occur per million opportunities. –TEa +TEa Figure 1. Bias Sigma Metric Analysis provides not only an objective assessment of analytical methods and instrumentation, but it also provides the critical design information needed for True Value operational implementation. The Sigma metric analysis CV process leads naturally to a quality control (QC) design Defects scheme using quantitative and graphic tools to determine the necessary quality control procedures for routine -6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s monitoring of methods and instruments. Figure 2: Relationship of imprecision (CV), inaccuracy (Bias) and allowable total error (TEa) in predicting defects 2 DIAGNOSTICS Q-Probe Quality Indicator % Error DPM Sigma The Six Sigma scale typically runs from zero to six, but Order accuracy 1.8% 18,000 3.60 a process can actually exceed Six Sigma, if variability is Duplicate test orders 1.52% 15,200 3.65 sufficiently low as to decrease the defect rate. In industries Wristband errors (not banded) .065 6,500 4.00 outside of healthcare, 3 Sigma is considered the minimal Therapeutic Drug Monitoring (TDM) timing errors 24.4 244,000 2.20 acceptable performance for a process. When performance falls below 3 Sigma, the process is considered to be essentially Hematology specimen acceptability 0.38 3,800 4.15 unstable and unacceptable. In contrast to other industries, Chemistry specimen acceptability 0.30 3,000 4.25 healthcare and clinical laboratories appear to be operating Surgical pathology specimen accessioning 3.4 34,000 3.30 in a 2 to 3 Sigma environment. The routine use of “2s” (i.e., 2 Cytology specimen adequacy 7.32 73,700 2.95 standard deviations or 2 SD) control limits is indicative of a Laboratory proficiency testing 0.9 9,000 3.85 Surgical pathology frozen section complacent tradition in quality control practices. Despite the 1.7 17,000 3.60 diagnostic discordance well-known problems of 2s limits – they can generate false PAP smear rescreening false negatives 2.4 24,000 3.45 rejection rates of up to 10-20%, depending on the number of controls run – many laboratories use them for all testing Reporting errors 0.0477 477 4.80 processes. The misuse of 2s limits in laboratory testing Figure 3: Sigma metrics of common laboratory processes, Nevalainen data. frequently results in erroneously-repeated controls, excessive trouble-shooting, or worse still, workarounds that artificially Revisiting the arrow and target model, Six Sigma provides widen control limits to the point that laboratories can no the shape of the target. The shape defines the goal of longer detect critical analytical errors. Six Sigma performance as the bull’s-eye, as well as the inner rings of 4 and 5 Sigma. Outside the 3 Sigma ring, Part of the power of the Six Sigma scale is its ability to provide performance is considered to have missed the target and a universal benchmark. Sigma metrics allow comparison the process is not considered to be “fit for purpose.” of different processes with each other, even comparing processes across different institutions and different industries. For example, airline safety is known to be better than Six Defining Quality Requirements Sigma with a rate of only 1.5 crashes per million departures, Knowing the shape of the target is not enough. The size while airline baggage handling in the U.S. is only 4.1 Sigma must also be described. In Six Sigma terminology, the since approximately 1% of luggage is misplaced or lost, and tolerance limits must be defined. In the clinical laboratory, U.S. airline departures perform at only 2.3 Sigma since nearly the quality required by an analytical testing process must 30% of flights are delayed, which helps to explain chronic be defined. Tolerance limits in the laboratory are best customer complaints. expressed as a total allowable error (TEa) specification. In healthcare, the Sigma performance of common processes TEa is a well-accepted concept in healthcare laboratories are less well known. When the Institute of Medicine issued as a model that combines both the imprecision and the 2 its landmark report, To Err is Human, it famously revealed inaccuracy (bias) of a method to calculate the total impact 4 that between 48,000 and 90,000 unnecessary deaths occurred on a test result. An allowable total error is the expression in U.S. hospitals every year. Examining the death rates at of how much combined imprecision and inaccuracy can the hospitals that formed the basis of the study reveals that be tolerated in the test result without negatively impacting healthcare is performing at only 3.8 Sigma. If healthcare were patient care based on interpretation of that result. achieving Six Sigma, the death rate would be only 16 to 34 deaths per year Determining the quality required by a laboratory test is not as simple as it sounds. Most laboratories do not know 3 Nevalainen’s groundbreaking work in Sigma assessment the analytical quality required by their tests. Indeed, many in the clinical lab analyzed the performance of common laboratories assume that it is not even necessary to know. laboratory processes and found that many were woefully As long as there are no direct complaints about testing inadequate Figure 3. quality, many laboratories assume that the analytical 3 DIAGNOSTICS quality they are providing is adequate. This is not the only Returning to the arrow and target model once more, crippling assumption that laboratories make. Sometimes establishing quality requirements, determines the size laboratories assume that the quality of any test is sufficient of the target. Since the use and performance of different simply because a manufacturer built the instrument and tests varies, so too does the size of the target that the made the reagents. While it’s common to assume that no arrow/process must hit. Together, Six Sigma and quality manufacturer would produce instruments and reagents that requirements provide the shape and size of the target. perform poorly, it is not good laboratory practice. Finally, Now all that remains is determining where (and if ) the laboratories frequently assume that simply following the arrow hits the target. For that, we need data on the actual manufacturer’s directions is enough to assure the quality of performance of the process . the tests they provide. Again, the fact that a manufacturer provides directions does not guarantee that the directions Measuring Six Sigma Performance in are adequate. Professional standards as well as regulatory the Laboratory requirements place the burden of selecting appropriate quality control procedures on the laboratory, specifically the Usually, Sigma performance is assessed by counting laboratory director. defects, then converting that count into a Defects Per Million Opportunities (DPM, or DPMO) rate.