forecasting
Article Cost Estimating Using a New Learning Curve Theory for Non-Constant Production Rates
Dakotah Hogan 1, John Elshaw 2,*, Clay Koschnick 2, Jonathan Ritschel 2, Adedeji Badiru 3 and Shawn Valentine 4
1 Air Force Cost Analysis Agency, Deputy Assistant Secretary for Cost and Economics, Joint Base Andrews, MD 20762, USA; [email protected] 2 Department of Systems Engineering & Management, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, USA; clay.koschnick@afit.edu (C.K.); jonathan.ritschel@afit.edu (J.R.) 3 Graduate School of Engineering and Management, Air Force Institute of Technology, Wright-Patterson AFB, OH 45433, USA; adedeji.badiru@afit.edu 4 Estimating Research & Technology Advising Branch, Cost and Economics Division, Air Force Lifecycle Management Center, Wright-Patterson AFB, OH 45433, USA; [email protected] * Correspondence: john.elshaw@afit.edu; Tel.: +1-937-255-3636 (ext. 4650)
Received: 27 August 2020; Accepted: 9 October 2020; Published: 16 October 2020
Abstract: Traditional learning curve theory assumes a constant learning rate regardless of the number of units produced. However, a collection of theoretical and empirical evidence indicates that learning rates decrease as more units are produced in some cases. These diminishing learning rates cause traditional learning curves to underestimate required resources, potentially resulting in cost overruns. A diminishing learning rate model, namely Boone’s learning curve, was recently developed to model this phenomenon. This research confirms that Boone’s learning curve systematically reduced error in modeling observed learning curves using production data from 169 Department of Defense end-items. However, high amounts of variability in error reduction precluded concluding the degree to which Boone’s learning curve reduced error on average. This research further justifies the necessity of a diminishing learning rate forecasting model and assesses a potential solution to model diminishing learning rates.
Keywords: learning curve; forecasting; production cost; cost estimating
1. Introduction The U.S. Government Accountability Office (GAO) critiqued the cost and schedule performance of the Department of Defense (DoD)’s $1.7 trillion portfolio of 86 major weapons systems in their 2018 “Weapons System Annual Assessment.” The GAO cited realistic cost estimates as a reason for the relatively low cost growth of the portfolio in comparison to earlier portfolios [1]. Congress and its oversight committees maintain a watchful eye on the DoD’s complex and expensive weapons system portfolio. Inefficient programs are scrutinized and may be terminated if inefficiencies persist. Funding of inefficient programs will also lead to the underfunding of other programs. In the public sector, these terminated and underfunded programs may result in capability gaps that negatively impact our nation’s defense. In the private sector, the inefficient use of resources often spells failure for a company. A key to the efficient use of resources is accurately estimating the resources required to produce an end-item. Learning curves are a popular method of forecasting required resources as they predict end-item costs using the item’s sequential unit number in the production line. Learning curves are especially useful when estimating the required resources for complex products. The most popular learning curve models used in the government sector are over 80 years old and may be outdated in
Forecasting 2020, 2, 429–451; doi:10.3390/forecast2040023 www.mdpi.com/journal/forecasting Forecasting 2020, 2 430 today’s technology-rich production environment. Additionally, researchers have demonstrated both theoretically and empirically that the effects of learning slow or cease over time [2–4]. A new model, named Boone’s learning curve, has been recently proposed to account for diminishing rates of learning as more units are produced [5]. The purpose of this research is to survey the need for alternative learning curve models and further examine how Boone’s learning curve performs in comparison to the traditional learning curve theories in predicting required resources. This research uses a large number of diverse production items to compare Boone’s model to the traditional theories of Wright and Crawford. While many different learning curve models exist (i.e., DeJong, Stanford B, Sigmoid, etc.), some of these others may not be as accurate in cases where the learning rate decreases over time. The next section is a review of the learning curve literature relevant to diminishing learning rates, followed by a description of our methodology and analysis to compare Boone’s learning curve to traditional models. We conclude the paper discussing managerial implications and limitations followed by recommendations for the way forward.
2. Literature Review and Background The two learning curve models cited by the GAO Cost Estimating and Assessment Guide (2009) are Wright’s cumulative average learning curve theory developed in 1936 and Crawford’s unit learning curve theory developed in 1947. Although both learning curve theories use the same general equation, the theories have contrasting variable definitions. Wright’s learning curve is shown in Equation (1):
Y = Axb (1) where Y is the cumulative average cost of the first x units, A is the theoretical cost to produce the first unit, x is the cumulative number of units produced, and b is the natural logarithm of the learning curve slope (LCS) divided by the natural logarithm of two. Note, the LCS is the complement of the percent decrease in cost as the number of units produced doubles. For example, with a learning curve slope of 80% and a first unit cost of 100 labor hours, the average cost of the first two units would be 80 labor hours, or 60 labor hours for the second unit. Regardless of the number of units produced, there is a constant decrease in labor costs with each doubling of units due to the constant learning rate. Several years following the creation of Wright’s cumulative average learning curve theory, J.R. Crawford formulated the unit learning curve theory. Crawford’s theory deviates from Wright’s by assuming that the individual unit cost (as opposed the cumulative average unit cost) decreases by a constant percentage as the number of units produced doubles. Crawford’s model is shown in Equation (2): Y = Axb (2) where Y is the individual cost of unit x, A is the theoretical cost of the first unit, x is the unit number of the unit cost being forecasted, and b is the natural logarithm of the LCS divided by the natural logarithm of two. For example, with a learning curve slope of 80% and a first unit cost of 100 labor hours, the cost of the second unit would be 80 labor hours. Note, Crawford’s unit theory is the similar to Wright’s in function form; but the difference arises in the variable interpretation lead to a different forecast. Figure1 below shows a comparison between Wright’s and Crawford’s theories using the two numerical examples provided. Cumulative average theory and unit theory will produce different predicted costs provided the same set of data despite all predicted costs being normalized to unit costs. Figure1 demonstrates this point where unit theory was used to generate data using a first unit cost of 100 and a learning curve slope of 90%. The original unit theory data was converted to cumulative averages in order to estimate cumulative average theory learning curve parameters. Forecasting 2020, 2 FOR PEER REVIEW 3
parameters were then used to predict cumulative average costs. These predicted costs were then converted to unit costs. This conversion allows for the cumulative average predictions to be directly compared to the original Unit Theory generated data. As shown in Figure 1, the cumulative average learning curve predictions first overestimate, then underestimate, and ultimately overestimate the
Forecastinggenerated2020 unit, 2 theory data for all remaining units. Together, Wright’s and Crawford’s theories form431 the basis of the traditional learning curve theory.
Figure 1. Wright’sWright’s Cumulative Average TheoryTheory vs. Crawford’s Unit Theory.
CumulativeOne assumption average of these theory traditional learning learning curve parameters. curve theories Cumulative is that they average only apply theory to estimatedprocesses athat learning may benefit curve from slope learning. of 93% andTypically, a first these unit costcosts of are 101.24. only a Thesesubset Cumulativeof total program Average costs; Theory hence parametersappropriate werecosts must then usedbe considered to predict when cumulative applying average learning costs. curve These theory predicted to yield viable costs wereparameter then convertedestimates. toIn unita complex costs. Thisprogram, conversion costs can allows be view for theed cumulativein a variety averageof ways predictionsto include recurring to be directly and comparednon-recurring to the costs, original direct Unit and Theory indirect generated costs, and data. costs As for shown various in Figure activities1, the and cumulative combinations average of learningend-items curve that can predictions be stated first in units overestimate, of hours thenor dollars. underestimate, Learning andcurve ultimately analysis focuses overestimate solely theon generatedrecurring costs unit theoryin estimating data for parameters all remaining because units. these Together, costs Wright’s are incurred and Crawford’s repeatedly theoriesfor each formunit theproduced basis of [6]. the Researchers traditional learning have also curve focused theory. solely on direct labor costs due to the theoretical underpinningsOne assumption of learning of these occurring traditional at learning the laborer curve level theories [2,3]. is thatAdditionally, they only applyresearchers to processes have thathistorically may benefit studied from end-items learning. that Typically, include theseonly the costs manufactured are only a subset or assembled of total programhardware costs; and hencesoftware appropriate elements of costs the end-item must be considered[2,3]. Lastly, when labor applyinghours in lieu learning of labor curve dollars theory are togenerally yield viable used parameterin analysis estimates.so that data In can a complexbe compared program, across costs fiscal can years be viewedwithout inthe a need variety to adjust of ways for to inflation. include recurringTherefore, andthe literature non-recurring indicates costs, using direct direct, and recurring, indirect costs, labor and costs costs in units for variousof labor activities hours. These and combinationscosts should be of considered end-items only that for can the be certain stated inelements units of that hours include or dollars. the manufacturing Learning curve or assembly analysis focusesof hardware solely and on recurringsoftware of costs an end-item. in estimating parameters because these costs are incurred repeatedly for eachAn implicit unit produced assumption [6]. Researchersin the traditional have alsolearni focusedng curve solely theories on direct is that labor knowledge costs due obtained to the theoreticalthrough learning underpinnings does not ofdepreciate. learning occurringHowever, atempirical the laborer evidence level [ 2demonstrates,3]. Additionally, that researchersknowledge havedepreciates historically in organizations studied end-items [7,8]. Argote that include [7] showed only that the manufacturedknowledge depreciation or assembled occurs hardware at both and the softwareindividual elements and theof organizational the end-item [levels.2,3]. Lastly, Many labor variations hours of in the lieu traditional of labor dollars models are make generally use of used the inconcept analysis of performance so that data candecay be (co comparedmmonly across called fiscalforgetting) years to without model thenon-constant need to adjust rates for of learning. inflation. Therefore,Forgetting theand literature its relationship indicates usingto learning direct, can recurring, take many labor costsforms in and units is of e laborssential hours. to consider These costs in shouldcontemporary be considered learning only curve for analysis. the certain elements that include the manufacturing or assembly of hardwareForgetting and software is the concept of an end-item. that an individual or organization will experience a decline in performanceAn implicit over assumption time resulting in thein non-constant traditional learning rates of le curvearning. theories Badiru is [4] that theorizes knowledge that forgetting obtained throughand resulting learning performance does not depreciate.decay is a result However, of factors empirical “including evidence lack demonstratesof training, reduced that knowledge retention depreciatesof skills, lapse in organizations in performance, [7,8]. extended Argote [ 7breaks] showed in thatpractice, knowledge and natural depreciation forgetting” occurs (p. at 287). both theAccording individual to Badiru and the [4], organizational these factors may levels. be caused Many by variations internal ofprocesses the traditional or external models factors. make Badiru use of[4] thelistsconcept three cases of performancein which forgetting decay arises. (commonly First, forgetting called forgetting) may occur to continuously model non-constant as a worker rates or of learning. Forgetting and its relationship to learning can take many forms and is essential to consider in contemporary learning curve analysis. Forgetting is the concept that an individual or organization will experience a decline in performance over time resulting in non-constant rates of learning. Badiru [4] theorizes that forgetting and resulting performance decay is a result of factors “including lack of training, reduced retention of skills, lapse in Forecasting 2020, 2 432 performance, extended breaks in practice, and natural forgetting” (p. 287). According to Badiru [4], these factors may be caused by internal processes or external factors. Badiru [4] lists three cases in which forgetting arises. First, forgetting may occur continuously as a worker or organization progresses down the learning curve due in part to natural forgetting [4]. The impact of forgetting may not wholly eclipse the impact of learning but will hamper the learning rate while performance continues to increase at a slower rate. Second, forgetting may occur at distinct and bounded intervals, such as during a scheduled production break [4] or towards the end of production as workers are transferred to other duties. Finally, forgetting may intermittently occur at random times and for stochastic intervals such as during times of employee turnover [4]. Others have expanded on the causes of forgetting and have drawn similar conclusions to Badiru [4,9–11]. This decline in performance decays the learning rate and causes longer manufacturing times and higher costs than would be forecasted using traditional learning curve theory. The concept of forgetting and its impact on non-constant rates of learning has proven relevant in contemporary learning curve research. Several forgetting models have been developed to include the learn-forget curve model (LFCM) [11], the recency model (RCM) [12], the power integration and diffusion (PID) model [13], and the Depletion-Power-Integration-Latency (DPIL) model [13] among others [10]. However, these forgetting models focus solely on the phenomenon of forgetting due to interruptions of the production process [9,10,14]. Jaber [9] states that “there has been no model developed for industrial settings that considers forgetting as a result of factors other than production breaks” (pp. 30–31) and mentions this as a potential area of future research. Although forgetting models have emerged after Jaber’s [9] article, a review of the popular forgetting models cited confirms Jaber’s statement. A related concept to the forgetting phenomenon is the plateauing phenomenon. According to Jaber [9] (2006), plateauing occurs when the learning process ceases and manufacturing enters a production steady state. This ceasing of learning results in a flattening or partial flattening of the learning curve corresponding to rates of learning at or near zero. There remains debate as to when plateauing occurs in the production process or if learning ever ceases completely [3,9,15–17]. Jaber [9] provides several explanations to describe the plateauing phenomenon that include concepts related to forgetting. Baloff [18,19] recognized that plateauing is more likely to occur when capital is used in the production process as opposed to labor. According to some researchers, plateauing can be explained by either having to process the efficiencies learned before making additional improvements along the learning curve or to forgetting altogether [20]. According to other researchers, plateauing can be caused by labor ceasing to learn or management’s unwillingness to invest in capital to foster induced learning [21]. Related to this underinvestment to foster induced learning, management’s doubt as to whether learning efficiencies related to learning can occur is cited as another hindrance to constant rates of learning [22]. Li and Rajagopalan [23] investigated these explanations and concluded that no empirical evidence supports or contradicts them while ascribing plateauing to depreciation in knowledge or forgetting. Jaber [9] concludes that “there is no tangible consensus among researchers as to what causes learning curves to plateau” and alludes that this is a topic for future research (pp. 30–39). Despite the controversy in the research surrounding forgetting and plateauing effects, empirical studies have shown learning curves to exhibit diminishing rates of learning. For instance, the plateauing phenomenon at the tail end of production was investigated by Harold Asher in a 1956 RAND study. The U.S. Air Force contracted RAND after the service noticed traditional learning curves were underestimating labor costs at the tail end of production [3]. Asher intended to study if the logarithmically transformed traditional learning curves were approximately linear. This linearity would indicate constant rates of learning throughout the production cycle. The alternative hypothesis for these learning curves was a convexity of the logarithmically-transformed traditional learning curves that would indicate diminishing rates of learning as the number of units increased [3]. An example of a learning curve with a diminishing learning rate is shown in Figure2 in logarithmic scale. The first unit Forecasting 2020, 2 433 cost is 100 with an initial learning curve slope of 80% decaying at a rate of 0.25% with each additional unit.Forecasting For 2020 example,, 2 FOR thePEER second REVIEW unit’s learning curve slope is 80.25%. 5
Figure 2. Unit Theory learning curve with a Decaying Learning Curve Slope.
Asher investigated investigated this this hypothesis hypothesis of of convex convex logarithmically logarithmically transformed transformed learning learning curves curves by byanalyzing analyzing the learning the learning curves curves of the ofvarious the variousshops within shops a withinmanufacturing a manufacturing department department producing producingaircraft. Asher aircraft. used Asher airframe used cost airframe data with cost datathe a withppropriate the appropriate amount of amount detail to of perform detail to a perform learning a learningcurve analysis curve analysison the lower on the level lower job level shops job shopswithin within the manufacturing the manufacturing department. department. He divided He divided the theeleven eleven major major kinds kinds of aircraft of aircraft manufacturing manufacturing operat operationsions into into four four shop shop groups groups each each with with a a set set of direct labor cost data [[3].3]. If non- non-constantconstant rates of learning were pr present,esent, the shop group curves would didifferffer inin their their rates rates of of learning learning and and may may themselves themselv bees convex be convex in logarithmic in logarithmic scale. This scale. would This indicate would theirindicate aggregate their aggregate learning learning curve would curve also woul bed convex also be in convex logarithmic in logarithmic scale. scale. Asher’s results showed that the learning curves of the manufacturing sh shopop group had different different learning slopes andand werewere convexconvex inin logarithmiclogarithmic scalescale [[3].3]. Asher claims the convexity within the manufacturing shopshop groupgroup learning learning curves curves is is due due to to the the disparate disparate operations operations within within the jobthe shopsjob shops and statedand stated that that each each had had their their own own unique unique learning learning curve curve [3]. [3]. He He asserts asserts that that a linear a linear approximation approximation is reasonableis reasonable for afor relatively a relatively small quantitysmall quantity of airframes of airframes produced butproduced becomes but increasingly becomes unwarrantedincreasingly forunwarranted larger quantities. for larger This quantities. is due in Th partis is becausedue in part larger because quantities larger ofquantities produced of end-itemsproduced end-items are likely toare experience likely to experience diminishing diminishing rates of learning. rates of learning. Moreover, Moreover, highly aggregated highly aggregated learning learning curves are curves also likelyare also to experiencelikely to experience diminishing diminishing rates of learning. rates of Because learning. the Because aggregated the manufacturingaggregated manufacturing cost curve is usuallycost curve the is lowest usually level the of lowest detail level on which of detail learning on which curve learning analysis iscurve performed, analysis the is manufacturingperformed, the costmanufacturing curve will have cost diminishingcurve will have rates diminishing of learning as rates cumulative of learning output as increases.cumulative These output results increases. further justifyThese results a learning further curve justify model a learning with diminishing curve model rates with of learning. diminishing rates of learning. Wright’s andand Crawford’s Crawford’s learning learning curve curve theories theories provided provided the the basis basis of the of th traditionale traditional approach approach that learningthat learning occurs occurs at a constant at a constant rate as rate the numberas the number of units producedof units produced increases. increases. Since this Since initial this discovery, initial severaldiscovery, log-linear several learninglog-linear curve learning models curve were models founded were in founded attempts in to attempts more accurately to more model accurately data frommodel manufacturing data from manufacturing processes. These processes. contemporary These co modelsntemporary diverge models from constantdiverge from rates constant of learning rates by includingof learning adjustments by including in variousadjustments forms. in The various six most forms. popular The six models most (including popular models the traditional (including model) the aretraditional shown inmodel) Figure are3 in shown logarithmic in Figure scale 3 and in logari includethmic log-log scale graphing and include lines log-log to more graphing clearly illustrate lines to themore di ffclearlyerences illustrate between models.the differences These illustrated between modelsmodels. include These the illustrated traditional models log-linear include model the or Wrighttraditional/Crawford log-linear curves, model the plateauor Wright/Crawford model [19], the curves, Stanford-B the plateau model [model24], the [19], De Jong the modelStanford-B [25], themodel S-curve [24], modelthe De [Jong21], andmodel Knecht’s [25], the upturn S-curv modele model [26 ].[21], and Knecht’s upturn model [26]. Forecasting 2020, 2 434 Forecasting 2020, 2 FOR PEER REVIEW 6
Figure 3. Comparison of Learning Curve Models (adapted from Badiru [27]). Figure 3. Comparison of Learning Curve Models (adapted from Badiru [27]). Recent studies have investigated whether the Stanford-B, De Jong, and S-Curve models more accuratelyRecent predict studies program have investigated costs in comparison whether to the the Stanford-B, traditional theories.De Jong, Moore and S-Curve [16] and models Honious more [17] accuratelystudied how predict prior experienceprogram costs in the in manufacturingcomparison to ofthe an traditional end-item alongtheories. with Moore the proportion [16] and Honious of touch [17]labor studied in the manufacturing how prior experience process ainff ectedthe manufactur the accuracying of of the an Stanford-B, end-item along De Jong, with and the S-curve proportion models of touchin comparison labor in the to the manufacturing traditional models. process The affected authors the concluded accuracy of that the these Stanford-B, models De improved Jong, and upon S- curvethe traditional models in curves comparison for only to a narrowthe traditional range of models. parameter The values. authors Their concluded research that provided these models insight improvedthat the traditional upon the learning traditional curve curves models for become only a narrow less accurate range at of the parameter tail-end of values. production Their whenresearch the providedproportion insight of human that laborthe traditional is high in learning the manufacturing curve models process. become Moreover, less accurate Honious at the [17 ]tail-end explicitly of productionreferences a when plateauing the proportion effect at the of end human of production. labor is high These in findingsthe manufacturing provide further process. justification Moreover, for Honiousinvestigating [17] explicitly non-constant references rates of a learning.plateauing effect at the end of production. These findings provide furtherThe justification Stanford-B, for De investigatin Jong, and S-Curveg non-constant univariate rates models of learning. illustrated in Figure3 alter the resulting learningThe Stanford-B, curve slope De based Jong, on and alterations S-Curve univariate to the theoretical models illustrated first unit costin Figure parameter 3 alter Athe. However,resulting learningthe learning curve curve slope slopes based of on these alterations models to are the nottheoretical directly first a function unit cost of parameter the number A. ofHowever, cumulative the learningunits produced. curve slopes The of plateau these models model andare not Knecht’s directly upturn a function model of alsothe numb illustrateder of cumulative in Figure3 eachunits produced.produce a learningThe plateau curve model whose and slope Knecht’s is directly upturn affected model by also the numberillustrated of cumulativein Figure 3 each units produce produced. a learningThe plateau curve model whose uses slope a step is function directly toaffected reduce by the the learning number rate of to cumulative 0% (i.e., the units learning produced. curve slope The plateauis 100%) model past a uses certain a step number function of cumulative to reduce the units lear produced.ning rate to In 0% contrast, (i.e., the Knecht’s learning Upturn curve slope Model is 100%)amends past the learninga certain curve number exponent of cumulative term b by unit multiplyings produced.b by In Euler’s contrast, number Knecht’se raised Upturn to the termModel of amendsa constant the multiplied learning curve by the exponent number ofterm cumulative b by multiplying units produced. b by Euler’s Mathematically, number e raised this is to expressed the term of a constantb exc multiplied by the number of cumulative units produced. Mathematically, this is Y = Ax · , where Y is the cumulative average unit cost, A is the theoretical first unit cost, x is the ∙ expressednumber of cumulative𝑌 = 𝐴𝑥 , units where produced, 𝑌 is the cumulativeb is the natural average logarithm unit cost, of the A learningis the theoretical curve slope first divided unit cost, by xthe is naturalthe number logarithm of cumulative of 2, and cunitsis a constant.produced, The b is forgetting the natural models logarithm stated of within the learning the manuscript curve slope also divided by the natural logarithm of 2, and c is a constant. The forgetting models stated within the Forecasting 2020, 2 435 amend the learning curve slope based indirectly on the number of cumulative units but only apply when interruptions to the production process occur. In response to these researchers’ findings, Boone [5] developed a learning curve model with a learning rate that diminishes as more units are produced. Conversely, the traditional learning curve theories diminish the rate of cost reductions as the number of units produced doubles. However, the existing literature provides evidence that the cost reductions with each doubling of units may not be constant as the number of units produced increases. Therefore, Boone [5] sought to attenuate the cost reductions that occur with each doubling of units produced by decreasing the learning rate as the number of units increases. Boone [5] devised a model that decreases the learning curve exponent b as the number of units produced x increases. He first considered a model without an additional parameter to reduce the learning curve exponent b directly by the unit number. However, he decided to temper the effect each additional unit has on the parameter b by adding an additional parameter c. The resulting learning curve is shown in Equation (3): b x Y = Ax 1+ c (3) where Y is the cumulative average cost of the first x units, A is the theoretical cost to produce the first unit, x is the cumulative number of units produced, b is the natural logarithm of the learning curve slope (LCS) divided by the natural logarithm of two, and c is a positive decay value. For example, a learning curve slope of 80%, first unit cost of 100 labor hours, and decay value of 100, Boone’s model yields a cumulative average cost at the second unit of 80.35 labor hours—or 60.70 labor hours for the second unit. What began as an 80% learning curve model has decayed to an 80.35% learning curve for the second unit. In comparison to Wright’s learning curve using the same parameters, the effect of learning has decreased slightly in the production of unit two. The inclusion of the decay value increases the learning curve slope, and hence decreases the learning rate as more units are produced. Note, Boone’s model can also be modified to incorporate Crawford’s unit theory–refer to Equation (3) for the necessary modifications. Boone’s learning curve diverges from the constant learning assumptions in both Wright’s and Crawford’s learning curve models by incorporating the unit number in the denominator of the exponent—thus decreasing the effect of b as the number of units produced increases. Furthermore, the decay value moderates this diminishing effect, so the amount of learning decreases more slowly. In general, Boone’s model is flatter near the end of production and steeper in the early stages compared to the traditional theories. Note, as the decay value approaches zero (holding other factors constant), the exponent term approaches zero representing a learning curve slope approaching 100%. As the decay value approaches infinity, the parameter b remains constant, and Boone’s learning curve simplifies to the traditional learning curve [5]. Boone [5] tested his learning curve using unit theory to provide a consistent comparison to Crawford’s learning curve. Based on the scope of his research and lack of comparison using cumulative average theory,a more robust examination and analysis of Boone’s learning curve should be accomplished.
3. Methodology One goal of this research is to examine the accuracy of Boone’s learning curve in comparison to the popular Wright and Crawford learning curve theories. In order to perform this analysis, production cost and quantity data from a diverse set of DoD systems was collected from government Functional Cost-Hour Reports, Progress Curve Reports, and the Air Force Life Cycle Management Center Cost Research Library. The dataset consisted of recurring costs (either in dollars or labor hours) by production lot for 169 unique end-items. Our data included end-items from a variety of systems (i.e., bomber, cargo, and fighter aircraft, missiles, and munitions), contractors, and time periods (1957–2018). Additionally, only production runs with at least four lots were included. The dataset for the Cumulative Average Theory analysis only includes 140 of the 169 end-items. This theory relies on Forecasting 2020, 2 436 continuous data because each lot’s cumulative average cost and cumulative quantity is a function of all previous lots’ costs and quantities. In order to compare Boone’s model to the traditional theories, each model will be fitted to data: (1) Boone’s and Wright’s models using cumulative average theory, and (2) Boone’s and Crawford’s models using unit theory. Then, the predicted values for each model will be compared to the actual costs using root mean squared error (RMSE) and mean absolute percentage error (MAPE). Labor costs were collected from the work breakdown structure (WBS) for the specific item being manufactured (e.g., aircraft frame) or from the documentation provided by the government. Our data included three broad functional cost categories: labor, material, and other. These costs are included in both forms of recurring and non-recurring costs. There are also four functional labor categories delineated that include manufacturing, tooling, engineering, and quality control labor. These four labor category costs, when summed with the material costs and other costs, comprise the total cost for each WBS element for recurring and non-recurring costs. The definition for the manufacturing labor cost category most clearly aligns with the extant literature to be the focus as the pertinent labor cost category for learning curve research. According to the WBS elements, the manufacturing labor category “includes the effort and costs expended in the fabrication, assembly, integration, and functional testing of a product or end item. It involves all the processes necessary to convert raw materials into finished items [28].” This manufacturing labor category aligns with the categories examined by Wright, which he called “assembly operations [2],” along with those cost categories Crawford studied, which he called “airframe-manufacturing processes [3].” Therefore, the manufacturing labor cost category as defined by the government is associated with the types of labor costs studied by traditional learning curve theorists and succeeding research. The learning curve parameters for each model (i.e., Equations (1)–(3)) will be estimated by minimizing the sum of squares error (SSE) using Excel’s generalized reduced gradient (GRG) nonlinear solver and evolutionary solver. The SSE is calculated by squaring the vertical difference of the observed data and predicted data for each lot and summing these squared differences across all lots. With lot data, cumulative theory models can be estimated directly. Conversely, when utilizing unit learning curve theory, Crawford’s and Boone’s models are estimated using an iterative process based on lot midpoints, adapted from Hu and Smith [29]. The algebraic lot midpoint is defined as “the theoretical unit whose cost is equal to the average unit cost for that lot on the learning curve” [6]. The lot midpoint supplants using sequential unit numbers when using lot cost data. Lot midpoints and model parameters are calculated iteratively due to the lack of a closed-form solution for the lot midpoint. First, an initial lot midpoint (for each lot) is determined using a parameter-free approximation formula [6]—see Equation (4):
F + L + 2 √FL Lot Midpoint(LMP) = (4) 4 where F is the first unit number in a lot and L is the last unit number in a lot. These lot midpoint estimates are then used to estimate the learning curve parameters for Crawford’s model (Equation (2)) using the GRG non-linear optimization algorithm. Next, using the estimated parameter b, a new set of lot midpoints are determined using a simple and popular formula—Asher’s Approximation [6]; see Equation (5): 1 + + ( ) 1 b 1 1 b 1 b L + F 2 − − 2 Lot Midpoint (5) ≈ (L F + 1)(b + 1) − where F is the first unit number in a lot, L is the last unit number in a lot, and b is the estimated value from Equation (2). Learning curve parameters will then be re-estimated using these more precise lot midpoint estimates. The iterative process is repeated until changes between successive values of the estimated lot midpoints and b are sufficiently small [29] (see AppendixA for a summary of Forecasting 2020, 2 437 this process). In order to use an iterative process for Boone’s model, Asher’s Approximation from Equation (5) was adapted to incorporate Boone’s decaying learning curve slope. This adaptation allows the lot costs of Boone’s learning curve to decrease as more units are produced which affects the lot midpoint estimates; the formula is shown in Equation (6):
( 1 ) b0+1 b0+1 b L + 1 F 1 0 2 2 Lot Midpoint − − (6) i ≈ (L F + 1)(b + 1) − 0
b where F is the first unit number in a lot, L is the last unit number in a lot, b = , and i is the 0 LMPi 1 1+ c − iteration number. This iterative process of calculating the lot mid-point then solving a non-linear least squares problem requires the execution of a series of non-linear optimization algorithms. Boone’s model requires the GRG algorithm which found solutions in a longer but still reasonable amount of time. While more burdensome than the traditional models due to the longer run time and the requirement to provide bounds for the parameters. For Boone’s model, the bounds for A and b have a fairly straightforward basis by which to define the bounds. In practice, the A parameter is often supported by a point estimate of the cost of the first theoretical unit. Thus, a bound can be built around this value with tools such as a confidence interval. The b parameter is defined by the learning curve slope which for all practical purposes will be in the (0, 1) interval—most likely on the higher end. As for the c parameter, the basis for the bound is more of a challenge. From a model implementation standpoint, the bound can be arbitrarily large if a long solve time is not limiting. Practically, the bound should be reasonably set; this aspect of the model is an avenue of future research which is discussed in the conclusion. This algorithm does allow the analyst to define stopping conditions such as convergence threshold, maximum number of iterations, or maximum amount of time. Additionally, there is an option called multi-start which uses multiple initial solutions to help locate a global solution verse possibly only finding a local solution. These options allow the user to mitigate the extra burden if necessary. Overall, the computing burden to calculate these models was on the order of minutes per weapon system. The final estimated parameters for Boone’s model and the traditional learning curves were used to create predicted learning curves. These predicted curves were then compared to observed data. Total model error was calculated by comparing the difference between observations and predicted values to understand how accurately the models explained variability in the data. Two measures were used to determine the overall model error. The first error measure was Root Mean Square Error (RMSE) that is calculated by taking the square root of the total SSE divided by the number of lots. RMSE is not robust to outliers—i.e., the effects of outliers may unduly influence this measure. RMSE is often interpreted as the average amount of error of the model as stated in the model’s original units. The second measure was mean absolute percentage error (MAPE). MAPE is calculated by subtracting the predicted value from the observed value, dividing this difference by the observed value, taking the absolute value, and multiplying by 100%. These absolute percent errors are then summed over all observations and divided by the total number of observations. MAPE provides a unit-less measure of accuracy and is interpreted as the average percent of model inaccuracy. Unlike RMSE, MAPE is robust to outliers. After calculating these measures of overall model error, a series of paired difference t-tests are conducted to determine if reductions in error from Boone’s learning curve are statistically significant. In order to conduct the first paired difference t-test, Boone’s learning curve RMSE using cumulative average theory will be subtracted from Wright’s learning curve RMSE, and the difference will be divided by Wright’s learning curve RMSE. This calculation will yield a percentage difference rather than raw difference to compare end-items of varying differences in magnitude equitably. The null hypothesis posits that Boone’s learning curve results in an equal amount (or more) of error in predicting Forecasting 2020, 2 438 observed values compared to Wright’s learning curve. The alternative hypothesis is that the percentage difference is greater than zero. Support for the alternative hypothesis signifies that Boone’s learning curve results in less error predicting observed values than Wright’s learning curve. This methodology will be repeated five times to examine each learning curve theory using the two error measures and the different units of production costs—see Table1.
Table 1. Paired Difference Hypothesis Tests Conducted.
Learning Curve Theory Error Measure Units of Measure Root Mean Squared Error Total Dollars(K) Percentage Difference Labor Hours Cumulative Average Theory Mean Absolute Percent Error Total Dollars(K)&Labor Percentage Difference Hours Combined Root Mean Squared Error Total Dollars(K) Percentage Difference Unit Theory Mean Absolute Percent Error Total Dollars(K)&Labor Percentage Difference Hours Combined
An assumption to utilize the paired difference t-test is that the data are approximately normally distributed. For hypothesis tests with large sample sizes, the central limit theorem can be invoked. Alternatively, a Shapiro–Wilk test will be used to evaluate the normality assumption for small samples. If the Shapiro–Wilk test does not support the normality assumption, the non-parametric Wilcoxon Rank Sum test will be used. A 0.05 level of significance will be used for all statistical tests.
4. Analysis & Results The detailed results for Wright’s and Boone’s learning curves using cumulative average theory are provided in AppendixB Tables A1 and A2. A total of 118 end-items in units of total dollars and 22 components in units of labor hours were analyzed. Each entry lists the program number, number of production lots, number of items produced, type of end-item, and units of the production costs. Additionally, each entry lists both error measures and the respective percent difference between the models. Positive (negative) differences indicate Boone’s model has less (more) error than Wright’s. Boone’s curve performs better for two reasons. First, Boone’s model can explain costs to at least the same degree of accuracy as the traditional learning curve theories due to the extra parameter. Second, increased accuracy could also be explained by Boone’s functional form. Despite these theoretical explanations, Boone’s model had more error than Wright’s for some observations; these negative percentage differences occur because an upper bound was placed on Boone’s decay value. An upper bound of 5000 was used for the decay value (same as Boone’s original paper). The practical effect of this particular bound can be observed by the number of end-items where the traditional models significantly outperformed Boone’s (i.e., a MAPE difference larger than 0.5%): 7 out of 140 for cumulative average theory and 15 out 169 for unit theory. Thus, the majority of the results were not affected by this artificial limitation which was chosen by trial and error. In practice, the bound could be set arbitrarily large so that it is not binding. Boone’s learning curve. This upper bound was necessary s since the GRG algorithm requires bounds on the estimated parameters. Some percentage error differences are approximately (but not exactly) zero. Observations with percentage error differences of approximately zero were defined as those within the bounds ( 0.25%, 0.25%). These bounds were used by the researchers to distinguish between observations with − approximately zero and non-zero percentage error differences in order to inform the descriptive statistics. Boone’s model had less error for 41% of observations, was approximately equal to Wright’s for 50% of observations, and had more error for 9% of observations. While Boone’s model is an improvement Forecasting 2020, 2 439 on Wright’s for some observations, many times the models fit the data equally well (i.e., an approximate zero difference). The results of the paired difference t-tests for cumulative average theory are shown in Table2 and a sample graph is shown in Figure4. No outliers, as defined by a value which fell more than three interquartile ranges from the upper 90% and lower 10% quantiles, were present in any of the tests.
Table 2. Cumulative Average Theory Descriptive and Inferential Statistics.
Hypothesis Test: H0: µ 0 H : µ > 0 ≤ A Learning Sample Units of Sample Number of Test Curve Error Measure Standard p-Value Result Measure ¯ Observations Statistic Theory Mean (x) Deviation (s) Root Mean Total Dollars(K) 19.3% 28.90% 118 7.23 <0.001 Reject H0 Squared Error Fail to Percentage Labor Hours 15.20% 31.20% 22 18.5 0.28 reject H0 Cumulative Difference Average Total Theory Mean Absolute Dollars(K)&Labor 18.60% 29.50% 140 7.45 <0.001 Reject H0 Percent Hours Combined
15000 14000 Observed Data 13000 12000 Wright's Predicted Curve 11000 Boone's Predicted 10000 Curve 9000 8000
7000 Cumulative Average Unit Cost Unit Average Cumulative 6000 Cumulative Units Produced 0 10 20 30 40 50 60 Figure 4. Comparison of Program 20 PME Air Vehicle.
The results of these hypothesis tests were mixed. For the RMSE percentage difference (measured in total dollars) and MAPE percentage difference, the paired difference t-tests led to rejection of the null hypothesis—indicating the increase in accuracy is statistically significant. Conversely, RMSE percentage difference (measured in hours) failed to reject the null hypothesis. Due to the small sample size, large sample theory could not be used, and the data failed a Shapiro–Wilk test (p-value = 0.721). Therefore, a Wilcoxon rank signed test was used. This indicates that Boone’s improvement in accuracy over Wright’s is not statistically significant when costs are measured in labor hours. However, small sample sizes can cause paired difference tests to have low power that may cause hypothesis tests to incorrectly fail to reject the null hypothesis [30]. Now considering unit theory, the results from Crawford’s and Boone’s learning curve models are presented in AppendixB. A total of 141 end-items (measured in total dollars) and 28 end-items (measured in labor hours) were analyzed. Similar to cumulative average theory, observations with percent error differences of approximately zero were defined as those within the bounds ( 0.25%, 0.25%). Boone’s model had less error for 43% − of observations across all percent difference error measures in comparison to crawford’s learning curve. Forecasting 2020, 2 440
Boone’s learning curve error was approximately equal for 52% of observations, and had more error for 5% of observations. The results of the paired difference testing for unit theory are provided in Table3 and a sample graph is shown in Figure5. Again, no outliers were present in any of the paired di fference t-tests.
Table 3. Unit Theory Descriptive and Inferential Statistics.
Hypothesis Test: H0: µ 0 H : µ > 0 ≤ A Learning Sample Units of Sample Number of Test Curve Error Measure Standard p-Value Result Measure ¯ Observations Statistic Theory Mean (x) Deviation (s) Total Root Mean Squared 13.80% 22.70% 141 7.23 <0.001 Reject H0 Error Percentage Dollars(K) Unit Difference Labor Hours 6.00% 14.80% 28 74.00 0.046 Reject H0 Theory Mean Absolute TotalDollars(K) Percent Error &Labor Hours 11.30% 23.10% 169 6.36 <0.001 Reject H0 Percentage Combined Difference Forecasting 2020, 2 FOR PEER REVIEW 13
120 110 100 Observed Data 90 80 Crawford's Predicted Curve
70 Boone's Predicted Curve 60 50 Unit Cost 40 30 20 Lot Midpoint 0 50 100 150 200 250 300 350 400 450
Figure 5. ComparisonComparison of of Program Program 1 PME Air Vehicle.
The results of these paired didifferencefference tests indicateindicate the improvement with Boone’s model is statistically significant. significant. Again, Again, the the RMSE RMSE percent percent difference difference (for (for labor hours) used a Wilcoxon rank sum test (due to the failure of the Shapiro–Wilk testtest withwith aa pp-value-value lessless thanthan 0.001).0.001).
5. Conclusions Conclusions A large, diverse dataset of DoD pr productionoduction programs was used to test if Boone’s learning curve more accurately explained error in comparisoncomparison toto traditionaltraditional learninglearning curvecurve theories.theories. The direct recurring costcost data data from from bomber, bomber, cargo, cargo, and fighterand fighter aircraft aircraft along withalong missiles with andmissiles munitions and munitions programs programsin units of in total units dollars of total and dollars labor and hours labor were hours analyzed were analyzed using Cumulative using Cumulative Average andAverage Unit Learningand Unit LearningCurve theories. Curve theories. Various Various components components of these of programs these programs were analyzed were analyzed from wingsfrom wings and dataand data link linksystems systems to the to airframes the airframes and air and vehicles. air vehicles. Boone’s learningBoone’s curvelearning was curve tested was against tested both against cumulative both cumulativeaverage and average unit learning and unit curve learning theories curve using theori two diesff usingerent measurestwo different of model measures error thatof model resulted error in thatsix paired resulted diff inerence six paired tests. difference This methodology tests. This resulted methodology in 998 total resulted observations in 998 total across observations all measures across and allensured measures the generalizabilityand ensured the of generalizability Boone’s learning of curveBoone’s was learning tested. curve was tested. Boone’s learning curve improved upon the traditional learning curve estimates for approximately 42% of the sampled program components while approximately equaling the traditional learning curve error for approximately 51% of program components. Boone’s learning curve resulted in a range of mean percentage difference reductions of 6% to 18.6% across all measures. The standard deviations of these improvements were high with coefficients of variation ranging from 150% to 247% across all measures. Absent additional analysis, these high amounts of variability make it challenging to conclude the degree to which Boone’s learning curve will improve the accuracy of explaining program component costs in comparison to the traditional estimation methods. Specifically, more research is needed to understand the shape of the learning curve and how it behaves related to production circumstances. It remains unclear which programs are more accurately modelled using Boone’s learning curve and to what degree Boone’s learning curve will more accurately model program component costs. The paired difference tests between Boone’s learning curve and the traditional theories indicate that Boone’s learning curve reduces error to a significant degree across a wide range of measures. Five of the six paired difference tests resulted in rejecting the null hypothesis that Boone’s learning curve had an equal amount or more error than the traditional theories at a significance level of 0.05. Due to data availability, program lot data was used instead of unitary data. Although Boone’s learning curve should perform just as well using either type of data, this research cannot conclusively state that Boone’s learning curve will more accurately explain programs in unitary data. Also, the Forecasting 2020, 2 441
Boone’s learning curve improved upon the traditional learning curve estimates for approximately 42% of the sampled program components while approximately equaling the traditional learning curve error for approximately 51% of program components. Boone’s learning curve resulted in a range of mean percentage difference reductions of 6% to 18.6% across all measures. The standard deviations of these improvements were high with coefficients of variation ranging from 150% to 247% across all measures. Absent additional analysis, these high amounts of variability make it challenging to conclude the degree to which Boone’s learning curve will improve the accuracy of explaining program component costs in comparison to the traditional estimation methods. Specifically, more research is needed to understand the shape of the learning curve and how it behaves related to production circumstances. It remains unclear which programs are more accurately modelled using Boone’s learning curve and to what degree Boone’s learning curve will more accurately model program component costs. The paired difference tests between Boone’s learning curve and the traditional theories indicate that Boone’s learning curve reduces error to a significant degree across a wide range of measures. Five of the six paired difference tests resulted in rejecting the null hypothesis that Boone’s learning curve had an equal amount or more error than the traditional theories at a significance level of 0.05. Due to data availability, program lot data was used instead of unitary data. Although Boone’s learning curve should perform just as well using either type of data, this research cannot conclusively state that Boone’s learning curve will more accurately explain programs in unitary data. Also, the majority of data utilized were end-item components in units of total dollars. The total dollar cost includes all cost categories rather than solely labor costs. These data are not ideal when applying learning curve theory and may bias learning curves to display diminishing rates of learning. Despite these potential issues, total dollar cost data are regularly utilized by cost estimators in the field due to data availability. Therefore, the practical applications of this analysis remain valid despite the limitations of using imperfect total dollar cost data in learning curve analysis. Boone’s learning curve was tested on programs whose lot costs were already known and whose parameters can be directly estimated. In other words, Boone’s learning curve was tested against the traditional theories on how well it explained rather than predicted program costs. In order to utilize Boone’s learning curve to predict costs, a decay value would be selected a priori. Similar to the learning curve slope, an analyst could use the decay value from similar programs to provide a range values to make predictions. Additionally, future research should investigate if Boone’s Decay Value can be predicted using various attributes of a program. Tests could be performed on how well Boone’s learning curve predicts costs for a program using analogous programs in comparison to the traditional theories. Lastly, additional labor hour data should be collected and analyzed in order to dispel the potential bias of learning curves displaying diminishing rates of learning when analyzed in units of total dollars.
Author Contributions: Conceptualization, D.H., J.E., C.K. and J.R.; methodology, D.H., J.E., C.K., J.R. and A.B.; software, D.H. and S.V.; validation, D.H., J.E., C.K. and J.R.; formal analysis, D.H., J.E., C.K. and J.R.; investigation, D.H., J.E., C.K., J.R. and S.V.; resources, D.H., A.B. and S.V.; data curation, D.H., J.E. and S.V.; writing—original draft preparation, D.H., J.E. and C.K.; writing—review and editing D.H., J.E., C.K., J.R. and A.B.; visualization, D.H. and A.B.; supervision, J.E., C.K., J.R. and A.B.; project administration, D.H., J.E., C.K. and J.R.; funding acquisition, J.E., J.R. and A.B. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Conflicts of Interest: The authors declare no conflict of interest.
Appendix A. Calculation Process for Lot Midpoint Estimation The following process was implemented to estimate parameters for lot midpoint estimation.
1. Parameter-free lot midpoint approximations (Equation (4)) were calculated for each production lot. 2. Crawford’s learning curve parameters A and b were initially estimated using OLS regression. Forecasting 2020, 2 442
a. Average unit cost was the dependent variable while lot midpoint, calculated in Step 1, was the independent variable. 3. These initial learning curve parameter estimates were used as starting values to more precisely estimate Crawford’s learning curve parameters using GRG non-linear solver. This process generated intermediate estimates of Crawford’s learning curve parameters. 4. The intermediate estimate of Crawford’s learning curve b parameter was used to calculate a more precise set of lot midpoints using Asher’s approximation (Equation (5)). 5. Applying these more precise lot midpoint approximations, Crawford’s learning curve parameters A and b were more accurately estimated using GRG nonlinear solver.
Steps 4 and 5 were repeated until the iterative process converged on a solution to produce final estimates of Crawford’s learning curve parameters and lot midpoint approximations.
Appendix B. Learning Curve Error Comparisons Using Cumulative Average and Unit Theories
Table A1. Error Comparison using Cumulative Average Theory.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference PME–Air Program 1 6 483 Dollars 557.9 111.7 80.0% 3.6% 0.7% 80.9% Vehicle PME–Air Program 1 6 483 Hours 15.5 0.3 98.0% 27.2% 0.5% 98.2% Vehicle Program 1 6 483 Airframe Dollars 411.2 114.1 72.3% 2.8% 0.7% 74.7% Program 1 6 483 Airframe Hours 21.7 1.5 93.0% 31.0% 1.7% 94.6% PME–Air Program 2 5 638 Dollars 129.8 6.5 95.0% 2.6% 0.1% 95.6% Vehicle PME–Air Program 3 5 500 Dollars 1630.3 291.1 82.1% 20.8% 3.9% 81.5% Vehicle PME–Air Program 4 19 205 Dollars 581.7 581.8 0.0% 3.1% 3.1% 0.0% Vehicle Program 4 19 205 Airframe Dollars 546.0 546.4 0.1% 3.2% 3.2% 0.1% − − PME–Air Program 5 7 459 Dollars 400.8 44.7 88.8% 2.7% 0.3% 88.2% Vehicle Electronic Program 5 7 459 Dollars 4.8 3.2 32.3% 7.2% 4.8% 33.7% Warfare (1) PME–Air Program 6 6 98 Dollars 99.3 32.2 67.6% 1.1% 0.3% 69.4% Vehicle Electronic Program 6 6 98 Dollars 12.7 1.7 86.8% 3.6% 0.6% 82.4% Warfare (1) Electronic Program 6 6 98 Dollars 15.0 13.3 11.4% 2.3% 2.0% 12.9% Warfare (2) Electronic Program 6 6 98 Dollars 1.8 1.1 40.3% 1.3% 0.8% 39.6% Warfare (3) PME–Air Program 7 7 110 Dollars 145.0 98.3 32.2% 1.0% 0.7% 32.6% Vehicle Electronic Program 7 7 110 Dollars 8.4 3.6 57.2% 2.7% 1.0% 61.3% Warfare (1) Electronic Program 7 7 110 Dollars 140.3 107.2 23.6% 1.2% 0.8% 27.5% Warfare (2) Electronic Program 7 7 110 Dollars 0.9 0.9 0.0% 0.5% 0.5% 0.1% Warfare (3) − Electronic Program 7 7 110 Dollars 140.7 111.3 20.9% 1.3% 1.0% 24.2% Warfare (4) Electronic Program 7 7 110 Dollars 21.3 21.0 1.1% 2.2% 2.1% 5.2% Warfare (5) PME–Air Program 8 8 3529 Dollars 27.7 23.6 14.8% 1.4% 1.3% 7.8% Vehicle PME–Air Program 8 8 3529 Hours 0.1 0.1 27.5% 1.1% 1.3% 27.9% Vehicle − − PME–Air Program 9 9 3798 Dollars 166.5 170.7 2.5% 8.4% 8.8% 3.7% Vehicle − − PME–Air Program 10 10 3803 Dollars 8.0 4.8 39.6% 2.5% 1.2% 51.7% Vehicle PME–Air Program 10 10 3803 Hours 24.4 14.0 42.7% 4.3% 2.0% 54.0% Vehicle Forecasting 2020, 2 443
Table A1. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference PME–Air Program 11 6 180 Dollars 514.0 508.4 1.1% 0.9% 0.8% 4.2% Vehicle PME–Air Program 12 10 20 Dollars 699.2 694.1 0.7% 5.8% 5.7% 1.0% Vehicle PME–Air Program 12 10 20 Hours 1042.5 906.5 13.1% 9.5% 8.4% 11.8% Vehicle Mission Program 12 7 11 Dollars 44.3 44.3 0.0% 2.5% 2.5% 0.0% Computer (1) PME–Air Program 13 5 100 Dollars 53,386.7 21,143.7 60.4% 12.8% 4.8% 62.1% Vehicle Program 13 5 100 Airframe Dollars 6569.7 6578.0 0.1% 3.7% 3.7% 0.0% − PME–Air Program 14 5 275 Dollars 3114.0 145.5 95.3% 3.8% 0.2% 95.5% Vehicle PME–Air Program 15 10 77 Dollars 44,386.0 44,390.2 0.0% 9.5% 9.5% 0.0% Vehicle PME–Air Program 15 12 83 Hours 79,242.0 79,247.5 0.0% 6.5% 6.5% 0.0% Vehicle Program 15 11 83 Airframe Dollars 39,624.4 39,628.0 0.0% 10.6% 10.6% 0.0% Mission Program 15 10 68 Dollars 1959.3 1959.4 0.0% 17.0% 17.0% 0.0% Computer (1) PME–Air Program 16 9 76 Dollars 436.3 144.4 66.9% 2.6% 1.0% 62.9% Vehicle PME–Air Program 17 5 50 Dollars 13,023.6 13,029.8 0.0% 2.8% 2.8% 0.1% Vehicle − PME–Air Program 18 9 31 Dollars 2942.5 2941.9 0.0% 1.0% 0.9% 0.0% Vehicle PME–Air Program 19 6 98 Dollars 313.3 313.4 0.0% 0.5% 0.5% 0.1% Vehicle − PME–Air Program 20 11 84 Dollars 1568.7 1121.9 28.5% 1.7% 1.5% 7.8% Vehicle Electronic Program 20 7 59 Dollars 452.8 143.0 68.4% 4.6% 1.3% 71.5% Warfare (1) Electronic Program 20 11 84 Dollars 98.7 76.5 22.5% 3.4% 3.6% 6.3% Warfare (2) − Electronic Program 20 7 59 Dollars 562.5 517.4 8.0% 1.8% 1.8% 1.7% Warfare (5) PME–Air Program 21 6 326 Dollars 5267.1 2408.8 54.3% 8.0% 4.2% 47.4% Vehicle Program 21 7 344 Airframe Dollars 4819.5 2544.3 47.2% 9.1% 5.4% 40.4% Program 21 7 344 Avionics Dollars 763.2 429.9 43.7% 6.6% 3.9% 40.8% PME–Air Program 21 14 453 Hours 3493.6 3495.9 0.1% 4.8% 4.8% 0.1% Vehicle − Program 21 14 453 Airframe Hours 4338.4 4339.7 0.0% 6.2% 6.2% 0.1% PME–Air Program 22 8 538 Hours 856.7 857.7 0.1% 2.5% 2.6% 0.1% Vehicle − − Program 22 8 538 Airframe Hours 5608.5 5609.7 0.0% 15.8% 15.9% 0.1% − PME–Air Program 23 5 469 Dollars 637.5 339.3 46.8% 5.4% 2.9% 47.3% Vehicle PME–Air Program 24 10 59 Dollars 3032.5 3033.0 0.0% 2.2% 2.2% 0.0% Vehicle PME–Air Program 25 9 348 Dollars 117.8 118.1 0.2% 0.9% 0.9% 0.2% Vehicle − − PME–Air Program 26 5 109 Dollars 3247.4 1676.8 48.4% 11.0% 6.0% 45.7% Vehicle PME–Air Program 26 5 109 Hours 607.1 453.5 25.3% 5.7% 4.2% 25.9% Vehicle PME–Air Program 27 18 631 Dollars 1669.6 913.3 45.3% 3.6% 1.9% 46.2% Vehicle PME–Air Program 28 6 425 Dollars 320.0 322.0 0.6% 0.9% 0.9% 0.6% Vehicle − − PME–Air Program 28 7 522 Hours 1776.1 1785.6 0.5% 1.8% 1.8% 0.1% Vehicle − − Program 28 7 522 Airframe Hours 1389.9 1393.9 0.3% 1.2% 1.2% 0.2% − − PME–Air Program 29 9 358 Hours 610.6 611.1 0.1% 0.9% 0.9% 0.4% Vehicle − Program 29 9 358 Airframe Hours 4804.8 2124.2 55.8% 7.3% 2.9% 60.1% PME–Air Program 30 5 204 Dollars 513.5 212.7 58.6% 1.2% 0.5% 56.1% Vehicle PME–Air Program 31 5 605 Dollars 1482.6 629.1 57.6% 6.1% 2.9% 53.1% Vehicle Forecasting 2020, 2 444
Table A1. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference PME–Air Program 32 5 870 Dollars 61.3 61.6 0.5% 0.4% 0.4% 0.3% Vehicle − − PME–Air Program 33 10 178 Dollars 7093.5 7101.1 0.1% 3.5% 3.5% 0.1% Vehicle − − PME–Air Program 33 10 178 Hours 8131.1 8144.1 0.2% 2.9% 2.9% 0.1% Vehicle − − Program 33 10 178 Airframe Dollars 1906.9 1910.8 0.2% 1.7% 1.7% 0.2% − − Program 33 10 712 Body Dollars 232.2 234.9 1.2% 1.5% 1.6% 1.3% − − Alighting Program 33 10 178 Dollars 76.6 76.6 0.0% 7.9% 7.9% 0.0% Gear Auxiliary Program 33 10 178 Dollars 90.7 90.7 0.1% 3.9% 3.9% 0.1% Power Plant − − Electronic Program 33 10 178 Dollars 775.5 776.1 0.1% 6.5% 6.5% 0.1% Warfare (1) − − Electronic Program 33 10 178 Dollars 360.1 273.4 24.1% 58.3% 46.0% 21.2% Warfare (2) Electronic Program 33 10 178 Dollars 62.5 62.4 0.2% 5.7% 5.7% 0.1% Warfare (3) Program 33 10 178 Empennage Dollars 352.2 352.3 0.0% 5.1% 5.1% 0.1% − Program 33 10 178 Hydraulic Dollars 22.7 22.7 0.1% 2.2% 2.2% 0.1% − − Program 33 10 178 Wing Dollars 296.5 296.9 0.1% 2.3% 2.3% 0.1% − − PME–Air Program 34 6 67 Dollars 11,059.1 11,061.2 0.0% 6.6% 6.6% 0.0% Vehicle PME–Air Program 34 6 67 Hours 9058.6 9061.7 0.0% 4.4% 4.4% 0.0% Vehicle Program 34 6 67 Airframe Dollars 2798.1 2004.6 28.4% 2.8% 1.7% 37.9% Program 34 6 201 Body Dollars 1924.5 828.9 56.9% 19.0% 8.7% 54.0% Alighting Program 34 6 67 Dollars 316.5 166.9 47.3% 17.2% 8.3% 51.9% Gear Program 34 6 67 Electrical Dollars 50.7 50.7 0.1% 1.9% 1.9% 0.1% − − Electronic Program 34 6 67 Dollars 428.3 428.4 0.0% 5.3% 5.3% 0.0% Warfare (1) Program 34 5 49 Empennage Dollars 202.2 202.2 0.0% 4.1% 4.1% 0.0% Program 34 6 67 EO/IR Dollars 45.6 36.6 19.7% 1.2% 1.1% 13.1% Program 34 6 67 EOTS Dollars 347.6 347.7 0.0% 6.5% 6.5% 0.0% Program 34 6 67 Hydraulic Dollars 122.3 101.5 17.0% 8.4% 6.2% 26.8% Mission Program 34 6 67 Dollars 484.8 484.9 0.0% 0.9% 0.9% 0.2% Computer (1) − Surface Program 34 6 67 Dollars 196.0 196.0 0.0% 4.9% 4.9% 0.0% Controls Program 34 6 67 Wing Dollars 998.4 998.6 0.0% 3.3% 3.3% 0.1% − PME–Air Program 35 5 41 Dollars 3578.6 3579.8 0.0% 1.5% 1.5% 0.0% Vehicle PME–Air Program 35 5 41 Hours 2003.7 2004.7 0.0% 1.1% 1.1% 0.0% Vehicle Program 35 5 50 Airframe Dollars 609.3 610.4 0.2% 0.6% 0.6% 0.3% − − Program 35 5 150 Body Dollars 235.8 156.5 33.6% 1.9% 1.4% 28.0% Alighting Program 35 5 50 Dollars 13.2 13.2 0.1% 0.5% 0.5% 0.0% Gear − Electronic Program 35 5 50 Dollars 259.6 259.7 0.0% 3.2% 3.2% 0.0% Warfare (1) Program 35 5 50 EO/IR Dollars 121.6 121.7 0.0% 1.3% 1.3% 0.1% − Program 35 5 50 EOTS Dollars 177.9 177.9 0.0% 2.8% 2.8% 0.1% − Program 35 5 50 Hydraulic Dollars 58.2 58.2 0.0% 3.1% 3.1% 0.0% Program 35 5 50 Radar Dollars 256.8 256.9 0.0% 3.2% 3.2% 0.0% Surface Program 35 5 50 Dollars 121.5 121.5 0.0% 2.6% 2.6% 0.0% Controls Program 35 5 50 Wing Dollars 1213.5 1213.6 0.0% 3.8% 3.8% 0.0% PME–Air Program 36 13 1285 Dollars 28.8 29.4 2.1% 0.6% 0.6% 2.2% Vehicle − − PME–Air Program 37 6 432 Dollars 791.3 793.8 0.3% 3.4% 3.4% 0.4% Vehicle − − PME–Air Program 38 6 52 Dollars 253.6 154.9 38.9% 1.2% 0.7% 41.6% Vehicle PME–Air Program 38 6 44 Hours 831.5 614.2 26.1% 1.3% 0.8% 42.8% Vehicle PME–Air Program 39 19 1023 Dollars 19.3 19.3 0.2% 0.7% 0.7% 0.2% Vehicle − − PME–Air Program 40 5 1725 Dollars 19.2 0.6 96.7% 2.0% 0.1% 97.0% Vehicle PME–Air Program 41 10 16 Dollars 14,787.6 14,787.8 0.0% 5.2% 5.2% 0.0% Vehicle Forecasting 2020, 2 445
Table A1. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference Data Program 41 10 16 Dollars 138.8 138.8 0.0% 3.7% 3.7% 0.0% Link (1) PME–Air Program 42 11 203 Dollars 1000.0 1000.1 0.0% 7.0% 7.0% 0.0% Vehicle Electronic Program 42 11 899 Dollars 67.5 67.7 0.2% 13.9% 13.9% 0.5% Warfare (1) − − PME–Air Program 43 11 203 Dollars 1121.7 1121.9 0.0% 5.5% 5.5% 0.0% Vehicle PME–Air Program 43 13 251 Hours 1944.2 1762.2 9.4% 3.4% 3.2% 6.1% Vehicle PME–Air Program 44 5 136 Dollars 57.1 16.3 71.4% 1.1% 0.3% 71.4% Vehicle PME–Air Program 45 9 155 Dollars 149.6 149.7 0.1% 0.3% 0.3% 0.1% Vehicle − − PME–Air Program 46 6 68 Dollars 3435.9 3436.0 0.0% 1.7% 1.7% 0.1% Vehicle PME–Air Program 46 6 68 Hours 2286.4 2286.6 0.0% 2.6% 2.6% 0.0% Vehicle Program 46 6 68 Airframe Dollars 539.1 527.6 2.1% 2.3% 2.1% 10.9% Data Program 46 6 68 Dollars 44.0 44.0 0.0% 3.0% 3.0% 0.0% Link (1) Electronic Program 46 6 68 Dollars 221.8 221.9 0.0% 5.4% 5.4% 0.0% Warfare (1) Electronic Program 46 6 68 Dollars 220.0 220.0 0.0% 6.5% 6.5% 0.0% Warfare (2) Electronic Program 46 6 68 Dollars 17.7 8.8 50.4% 2.2% 1.0% 54.6% Warfare (3) Electronic Program 46 6 68 Dollars 530.0 530.0 0.0% 5.2% 5.2% 0.0% Warfare (4) Program 46 6 68 EO/IR Dollars 120.7 120.8 0.0% 15.7% 15.7% 0.0% Mission Program 46 6 68 Dollars 477.9 478.0 0.0% 4.3% 4.3% 0.0% Computer (1) PME–Air Program 47 9 36 Dollars 1039.4 1039.4 0.0% 2.5% 2.5% 0.0% Vehicle PME–Air Program 47 9 36 Hours 8278.7 8278.6 0.0% 15.5% 15.5% 0.0% Vehicle Data Program 47 9 36 Dollars 170.2 170.2 0.0% 17.7% 17.7% 0.0% Link (1) PME–Air Program 48 5 179 Dollars 1858.3 391.3 78.9% 3.1% 0.6% 79.4% Vehicle PME–Air Program 49 6 180 Dollars 435.3 99.8 77.1% 4.4% 1.0% 76.5% Vehicle PME–Air Program 50 5 488 Dollars 349.3 350.7 0.4% 3.3% 3.4% 0.8% Vehicle − − PME–Air Program 51 6 663 Dollars 5.6 3.6 36.6% 0.6% 0.4% 24.8% Vehicle PME–Air Program 52 5 380 Dollars 456.9 454.6 0.5% 9.0% 8.9% 0.3% Vehicle PME–Air Program 53 6 749 Dollars 37.2 36.6 1.7% 0.5% 0.5% 4.3% Vehicle PME–Air Program 54 8 194 Dollars 28.8 28.8 0.1% 0.6% 0.6% 0.1% Vehicle − − PME–Air Program 55 9 677 Dollars 74.8 74.8 0.0% 1.6% 1.6% 0.0% Vehicle PME–Air Program 56 5 590 Dollars 6.6 6.6 0.5% 0.2% 0.2% 6.3% Vehicle PME–Air Program 57 5 579 Dollars 22.8 22.8 0.1% 0.8% 0.8% 0.0% Vehicle − Forecasting 2020, 2 446
Table A2. Error Comparison using Unit Theory.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference Program 1 7 503 Airframe Hours 4.6 3.5 23.4% 7.1% 5.0% 28.7% PME–Air Program 1 6 483 Hours 5.4 1.5 72.5% 11.3% 2.9% 74.0% Vehicle PME–Air Program 1 7 503 Dollars 2260.6 517.0 77.1% 12.9% 3.2% 75.2% Vehicle Program 1 7 503 Airframe Dollars 2383.2 857.9 64.0% 14.6% 4.9% 66.4% PME–Air Program 2 5 638 Dollars 315.4 195.3 38.1% 5.8% 4.3% 26.3% Vehicle PME–Air Program 3 5 500 Dollars 2984.5 1120.2 62.5% 49.4% 17.6% 64.4% Vehicle Program 4 7 357 Airframe Dollars 2662.2 2664.3 0.1% 13.1% 13.2% 0.1% − − PME–Air Program 4 9 424 Dollars 9323.3 4999.8 46.4% 37.9% 14.1% 62.8% Vehicle Program 5 19 205 Airframe Dollars 2446.1 2445.8 0.0% 12.6% 12.6% 0.3% − PME–Air Program 5 19 205 Dollars 3228.6 3228.9 0.0% 12.4% 12.4% 0.0% Vehicle Electronic Program 6 7 459 Dollars 20.9 20.9 0.0% 30.8% 30.8% 0.0% Warfare (1) PME–Air Program 6 7 459 Dollars 1439.9 738.1 48.7% 11.3% 5.9% 47.2% Vehicle PME–Air Program 7 5 321 Dollars 37.9 33.3 12.2% 3.8% 3.8% 1.1% Vehicle Electronic Program 8 6 98 Dollars 5.2 4.9 6.1% 4.8% 4.8% 1.4% Warfare (3) Electronic Program 8 6 98 Dollars 84.2 70.3 16.5% 11.1% 10.6% 4.7% Warfare (2) PME–Air Program 8 6 98 Dollars 375.2 339.5 9.5% 4.2% 3.7% 13.4% Vehicle Electronic Program 8 6 98 Dollars 27.5 18.7 31.9% 10.2% 5.9% 42.5% Warfare (1) Electronic Program 9 7 110 Dollars 102.9 99.2 3.5% 9.7% 10.4% 6.6% Warfare (5) − Electronic Program 9 7 110 Dollars 6.4 6.4 0.0% 4.7% 4.7% 0.0% Warfare (3) Electronic Program 9 7 110 Dollars 653.6 653.6 0.0% 6.2% 6.2% 0.0% Warfare (4) Electronic Program 9 7 110 Dollars 709.4 709.4 0.0% 6.1% 6.1% 0.0% Warfare (2) PME–Air Program 9 7 110 Dollars 668.5 668.5 0.0% 5.1% 5.1% 0.0% Vehicle Electronic Program 9 7 110 Dollars 31.6 29.1 8.0% 8.7% 8.0% 8.3% Warfare (1) PME–Air Program 10 9 1586 Dollars 115.5 115.6 0.2% 12.5% 12.5% 0.2% Vehicle − − PME–Air Program 10 10 1796 Hours 150.8 150.9 0.0% 12.5% 12.5% 0.1% Vehicle − PME–Air Program 11 8 3529 Hours 0.9 0.7 21.2% 27.5% 44.9% 63.4% Vehicle − PME–Air Program 11 8 3529 Dollars 97.1 97.5 0.4% 10.1% 10.4% 2.1% Vehicle − − PME–Air Program 12 16 7891 Hours 520.1 525.6 1.1% 86.2% 86.2% 0.0% Vehicle − PME–Air Program 12 21 10035 Dollars 243.8 239.2 1.9% 30.1% 28.8% 4.2% Vehicle Program 13 6 3385 EO Dollars 12.1 9.4 22.5% 10.7% 9.6% 10.0% PME–Air Program 13 10 3803 Dollars 33.6 24.8 26.1% 10.3% 7.5% 27.1% Vehicle PME–Air Program 13 10 3803 Hours 130.1 100.5 22.7% 21.5% 17.1% 20.7% Vehicle PME–Air Program 14 6 180 Dollars 2249.4 1008.9 55.2% 6.4% 2.3% 64.2% Vehicle PME–Air Program 15 10 20 Hours 3430.3 3430.4 0.0% 41.5% 41.5% 0.0% Vehicle PME–Air Program 15 10 20 Dollars 3013.9 3013.9 0.0% 17.4% 17.4% 0.0% Vehicle Mission Program 15 7 11 Dollars 213.9 213.9 0.0% 11.6% 11.5% 0.6% Computer (1) Program 16 5 100 Airframe Dollars 10,807.3 7455.4 31.0% 7.0% 4.1% 41.8% PME–Air Program 16 5 100 Dollars 137,225.9 81,884.9 40.3% 51.7% 26.9% 48.0% Vehicle PME–Air Program 17 5 275 Dollars 8837.5 1396.3 84.2% 17.6% 3.3% 81.6% Vehicle Forecasting 2020, 2 447
Table A2. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference PME–Air Program 18 12 83 Hours 266,012.8 266,015.3 0.0% 39.3% 39.3% 0.0% Vehicle Program 18 11 83 Airframe Dollars 89,956.0 89,961.1 0.0% 39.1% 39.1% 0.0% Mission Program 18 10 68 Dollars 4143.0 4143.2 0.0% 68.2% 68.2% 0.0% Computer (1) PME–Air Program 18 11 83 Dollars 82,138.6 82,143.3 0.0% 23.2% 23.2% 0.0% Vehicle Program 19 5 45 Airframe Dollars 501.2 501.2 0.0% 53.9% 53.9% 0.0% PME–Air Program 19 5 45 Dollars 649.0 649.0 0.0% 17.6% 17.6% 0.0% Vehicle Mission Program 19 5 45 Dollars 61.7 59.7 3.2% 9.8% 9.7% 1.2% Computer (1) PME–Air Program 20 9 76 Dollars 1108.7 522.5 52.9% 7.2% 3.6% 49.9% Vehicle PME–Air Program 21 5 50 Dollars 24,625.3 6362.0 74.2% 7.4% 2.3% 69.5% Vehicle PME–Air Program 22 9 31 Dollars 16,636.3 16,636.4 0.0% 6.6% 6.6% 0.0% Vehicle PME–Air Program 23 5 14 Dollars 14,475.8 14,476.0 0.0% 8.7% 8.7% 0.0% Vehicle PME–Air Program 24 6 98 Dollars 2259.9 2260.1 0.0% 3.3% 3.3% 0.0% Vehicle Electronic Program 25 7 59 Dollars 2808.4 2805.2 0.1% 14.8% 15.4% 4.0% Warfare (5) − PME–Air Program 25 11 84 Dollars 5083.2 4228.8 16.8% 8.7% 9.2% 5.2% Vehicle − Electronic Program 25 11 84 Dollars 248.9 248.6 0.1% 13.9% 14.3% 2.9% Warfare (2) − Electronic Program 25 7 59 Dollars 1259.1 653.3 48.1% 16.1% 7.1% 55.6% Warfare (1) Program 26 7 344 Airframe Dollars 11,474.7 8294.9 27.7% 22.7% 21.5% 5.3% Program 26 7 344 Avionics Dollars 2218.8 2102.8 5.2% 29.5% 26.9% 8.8% PME–Air Program 26 7 344 Dollars 12,898.4 8742.1 32.2% 20.7% 16.9% 18.4% Vehicle PME–Air Program 27 14 453 Hours 54,142.9 53,766.4 0.7% 59.9% 63.1% 5.4% Vehicle − Program 27 14 453 Airframe Hours 70,415.0 69,426.8 1.4% 58.8% 59.1% 0.5% − PME–Air Program 28 8 538 Hours 3828.8 3829.8 0.0% 9.8% 9.9% 0.0% Vehicle Program 28 8 538 Airframe Hours 3865.3 3866.2 0.0% 7.6% 7.6% 0.0% Program 29 8 529 Hydraulic Dollars 156.9 156.4 0.3% 22.3% 22.9% 2.8% − Program 29 12 477 Airframe Dollars 6490.2 5974.2 7.9% 14.2% 14.4% 1.8% − Program 29 12 477 Wing Dollars 712.3 712.7 0.1% 27.8% 27.8% 0.1% − − Electronic Program 29 11 433 Dollars 57.5 57.5 0.0% 13.5% 13.5% 0.1% Warfare (1) − Program 29 8 309 Electrical Dollars 230.6 230.7 0.1% 8.2% 8.2% 0.0% − Program 29 12 1045 Body Dollars 1922.2 1826.7 5.0% 26.0% 25.9% 0.7% Program 29 5 177 Empennage Dollars 32.3 22.0 31.8% 6.1% 4.6% 24.5% PME–Air Program 29 12 477 Dollars 8218.5 5525.3 32.8% 15.0% 10.2% 32.0% Vehicle Alighting Program 29 8 309 Dollars 205.7 42.2 79.5% 11.6% 2.0% 83.1% Gear PME–Air Program 30 5 469 Dollars 1283.8 891.8 30.5% 13.5% 8.3% 38.3% Vehicle PME–Air Program 31 10 59 Dollars 11,978.9 11,979.3 0.0% 8.6% 8.6% 0.0% Vehicle PME–Air Program 32 9 348 Dollars 430.6 430.8 0.0% 3.5% 3.5% 0.1% Vehicle − PME–Air Program 33 5 109 Hours 993.9 994.0 0.0% 9.5% 9.5% 0.0% Vehicle PME–Air Program 33 5 109 Dollars 6824.7 6824.8 0.0% 28.2% 28.2% 0.0% Vehicle PME–Air Program 34 18 631 Dollars 6926.7 2799.9 59.6% 17.0% 6.6% 61.0% Vehicle PME–Air Program 35 6 425 Dollars 1135.8 1137.5 0.2% 3.5% 3.5% 0.2% Vehicle − − PME–Air Program 35 7 522 Hours 4615.3 4458.5 3.4% 6.3% 6.1% 3.1% Vehicle Program 35 7 522 Airframe Hours 6757.0 6280.7 7.0% 5.7% 5.4% 4.8% PME–Air Program 36 9 358 Hours 5118.7 5120.1 0.0% 6.8% 6.8% 0.0% Vehicle Forecasting 2020, 2 448
Table A2. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference Program 36 9 358 Airframe Hours 12,155.2 11,257.1 7.4% 15.5% 14.3% 7.6% PME–Air Program 37 5 204 Dollars 1468.7 921.0 37.3% 2.9% 1.9% 36.4% Vehicle PME–Air Program 38 5 605 Dollars 2641.9 1527.7 42.2% 14.9% 8.1% 46.0% Vehicle PME–Air Program 39 5 870 Dollars 310.9 311.5 0.2% 2.3% 2.3% 0.2% Vehicle − − Electronic Program 40 10 178 Dollars 751.2 551.9 26.5% 69.7% 74.7% 7.1% Warfare (3) − Program 40 10 712 Body Dollars 617.6 577.6 6.5% 4.8% 5.1% 7.6% − Program 40 10 178 Airframe Dollars 4251.9 4226.4 0.6% 4.8% 4.9% 1.0% − Electronic Program 40 10 178 Dollars 721.7 721.7 0.0% 393.4% 393.4% 0.0% Warfare (2) Electronic Program 40 10 178 Dollars 1642.3 1643.0 0.0% 20.7% 20.7% 0.0% Warfare (1) PME–Air Program 40 10 178 Hours 13,454.5 13,466.8 0.1% 6.0% 6.0% 0.1% Vehicle − − Auxiliary Program 40 10 178 Dollars 385.1 385.1 0.0% 24.9% 24.9% 0.0% Power Plant PME–Air Program 40 10 178 Dollars 12,231.7 12,236.6 0.0% 7.9% 7.9% 0.0% Vehicle Alighting Program 40 10 178 Dollars 233.6 233.6 0.0% 30.1% 30.1% 0.0% Gear Program 40 10 178 Wing Dollars 607.4 607.6 0.0% 6.2% 6.2% 0.0% Program 40 10 178 Empennage Dollars 702.1 702.1 0.0% 17.4% 17.4% 0.0% Program 40 10 178 Hydraulic Dollars 72.2 70.2 2.8% 9.0% 8.8% 2.2% PME–Air Program 41 6 67 Hours 12,741.5 12,743.8 0.0% 9.5% 9.5% 0.0% Vehicle Program 41 5 49 Empennage Dollars 242.2 242.2 0.0% 5.8% 5.9% 0.0% PME–Air Program 41 6 67 Dollars 16,643.9 16,645.6 0.0% 10.7% 10.7% 0.0% Vehicle Surface Program 41 6 67 Dollars 281.7 281.7 0.0% 7.7% 7.7% 0.0% Controls Program 41 6 67 EOTS Dollars 442.3 442.4 0.0% 9.5% 9.5% 0.0% Program 41 6 67 Wing Dollars 1927.0 1927.3 0.0% 7.4% 7.4% 0.0% Program 41 6 67 Electrical Dollars 57.2 57.2 0.0% 2.1% 2.1% 0.0% Electronic Program 41 6 67 Dollars 547.3 547.3 0.0% 8.1% 8.1% 0.0% Warfare (1) Program 41 6 67 Hydraulic Dollars 281.5 274.6 2.4% 19.4% 19.0% 2.0% Mission Program 41 6 67 Dollars 1698.1 1542.4 9.2% 4.6% 3.7% 19.5% Computer (1) Program 41 6 67 Airframe Dollars 6877.8 5547.4 19.3% 8.7% 6.4% 26.8% Alighting Program 41 6 67 Dollars 582.3 521.1 10.5% 28.3% 25.0% 11.6% Gear Program 41 6 67 EO/IR Dollars 233.0 89.4 61.6% 9.3% 3.1% 66.8% Program 41 6 201 Body Dollars 3431.8 2343.2 31.7% 42.6% 29.9% 29.7% PME–Air Program 42 5 41 Dollars 8498.6 8499.6 0.0% 6.2% 6.2% 0.0% Vehicle PME–Air Program 42 5 41 Hours 15,696.5 15,696.9 0.0% 10.7% 10.7% 0.0% Vehicle Program 42 5 50 EOTS Dollars 593.3 593.3 0.0% 11.6% 11.6% 0.0% Program 42 5 50 EO/IR Dollars 578.4 578.4 0.0% 7.5% 7.5% 0.0% Program 42 5 50 Hydraulic Dollars 297.0 297.0 0.0% 15.4% 15.4% 0.0% Surface Program 42 5 50 Dollars 424.9 424.9 0.0% 11.0% 11.0% 0.0% Controls Program 42 5 50 Radar Dollars 733.8 733.8 0.0% 10.9% 10.9% 0.0% Program 42 5 50 Airframe Dollars 5222.7 5222.8 0.0% 5.9% 5.9% 0.0% Electronic Program 42 5 50 Dollars 746.5 746.5 0.0% 10.7% 10.7% 0.0% Warfare (1) Program 42 5 50 Wing Dollars 3726.6 3726.7 0.0% 16.5% 16.5% 0.0% Alighting Program 42 5 50 Dollars 78.6 77.4 1.5% 3.6% 3.5% 2.3% Gear Program 42 5 150 Body Dollars 1588.5 892.1 43.8% 12.6% 8.7% 30.8% PME–Air Program 43 13 1285 Dollars 88.1 88.8 0.8% 1.9% 1.9% 1.0% Vehicle − − PME–Air Program 44 6 432 Dollars 1621.0 1623.3 0.1% 10.0% 10.0% 0.2% Vehicle − − PME–Air Program 45 9 63 Dollars 2152.3 1557.1 27.7% 9.5% 6.4% 33.2% Vehicle PME–Air Program 46 6 44 Hours 7736.9 7255.3 6.2% 17.6% 16.7% 4.8% Vehicle PME–Air Program 46 10 113 Dollars 797.9 627.0 21.4% 3.8% 2.9% 22.7% Vehicle PME–Air Program 47 19 1023 Dollars 115.2 115.2 0.0% 4.3% 4.2% 0.2% Vehicle Forecasting 2020, 2 449
Table A2. Cont.
Number RMSE MAPE Number Component Traditional Boone Traditional Boone Program of Units Percentage Percentage of Lots Estimated RMSE RMSE MAPE MAPE Units Difference Difference PME–Air Program 48 5 1725 Dollars 59.8 3.1 94.9% 6.8% 0.3% 95.4% Vehicle Data Program 49 10 16 Dollars 470.3 470.3 0.0% 20.4% 20.4% 0.0% Link (1) PME–Air Program 49 10 16 Dollars 41,008.9 41,009.2 0.0% 14.1% 14.1% 0.0% Vehicle PME–Air Program 50 7 577 Dollars 1674.7 1224.7 26.9% 5.5% 4.6% 15.7% Vehicle PME–Air Program 51 12 244 Hours 625.6 612.8 2.0% 191.4% 191.8% 0.2% Vehicle − Electronic Program 52 11 899 Dollars 90.1 90.2 0.1% 29.2% 29.3% 0.1% Warfare (1) − − PME–Air Program 52 11 203 Dollars 2995.1 2992.0 0.1% 24.9% 23.6% 5.2% Vehicle PME–Air Program 53 13 251 Hours 4585.2 4585.2 0.0% 6.7% 6.7% 0.0% Vehicle PME–Air Program 53 11 203 Dollars 2459.9 2460.0 0.0% 9.6% 9.6% 0.0% Vehicle PME–Air Program 54 11 184 Hours 7010.4 7010.7 0.0% 18.0% 18.0% 0.0% Vehicle PME–Air Program 54 9 134 Dollars 1907.3 970.0 49.1% 11.8% 6.5% 44.9% Vehicle PME–Air Program 55 5 136 Dollars 321.6 277.7 13.7% 5.5% 4.7% 14.8% Vehicle PME–Air Program 56 9 155 Dollars 1356.5 1356.6 0.0% 3.9% 3.9% 0.0% Vehicle Program 57 6 68 EO/IR Dollars 326.0 326.0 0.0% 1261.8% 1261.8% 0.0% PME–Air Program 57 6 68 Dollars 8574.7 8470.9 1.2% 4.3% 4.3% 0.5% Vehicle − Electronic Program 57 6 68 Dollars 998.8 998.9 0.0% 58.9% 58.9% 0.0% Warfare (1) Electronic Program 57 6 68 Dollars 750.2 750.2 0.0% 31.3% 31.3% 0.0% Warfare (2) Data Program 57 6 68 Dollars 94.8 94.8 0.0% 7.2% 7.2% 0.0% Link (1) Electronic Program 57 6 68 Dollars 1156.3 1156.3 0.0% 12.2% 12.2% 0.0% Warfare (4) Mission Program 57 6 68 Dollars 1030.6 1030.6 0.0% 13.0% 13.0% 0.0% Computer (1) PME–Air Program 57 6 68 Hours 6435.9 6435.0 0.0% 12.3% 12.3% 0.3% Vehicle Program 57 6 68 Airframe Dollars 1443.2 1285.1 11.0% 6.7% 5.4% 18.5% Electronic Program 57 6 68 Dollars 53.4 21.8 59.1% 7.2% 3.0% 58.5% Warfare (3) PME–Air Program 58 9 36 Hours 60,347.2 60,347.3 0.0% 78.2% 78.2% 0.0% Vehicle Data Program 58 9 36 Dollars 227.8 227.8 0.0% 29.3% 29.3% 0.0% Link (1) PME–Air Program 58 9 36 Dollars 4570.2 4570.2 0.0% 10.9% 10.9% 0.0% Vehicle Program 58 5 18 EO/IR Dollars 3488.4 3469.8 0.5% 28.8% 28.7% 0.3% PME–Air Program 59 5 179 Dollars 4583.3 1334.5 70.9% 8.1% 2.8% 65.4% Vehicle PME–Air Program 60 6 180 Dollars 1010.5 333.9 67.0% 12.4% 4.6% 63.1% Vehicle PME–Air Program 61 5 488 Dollars 502.3 486.5 3.1% 9.2% 7.7% 16.3% Vehicle PME–Air Program 62 6 78 Hours 6027.1 5952.3 1.2% 33.8% 34.3% 1.6% Vehicle − Program 62 6 97 Airframe Hours 2648.5 2649.0 0.0% 20.5% 20.5% 0.0% PME–Air Program 62 9 110 Dollars 13,027.5 13,028.9 0.0% 24.0% 24.0% 0.0% Vehicle PME–Air Program 63 6 663 Dollars 23.2 21.1 9.2% 2.9% 2.6% 11.6% Vehicle PME–Air Program 64 5 380 Dollars 1520.9 1521.2 0.0% 57.4% 57.4% 0.0% Vehicle PME–Air Program 65 6 749 Dollars 116.6 115.9 0.6% 1.7% 1.8% 5.1% Vehicle − PME–Air Program 66 8 194 Dollars 128.3 119.3 7.0% 2.6% 2.4% 8.6% Vehicle PME–Air Program 67 9 677 Dollars 273.5 273.5 0.0% 5.1% 5.1% 0.0% Vehicle PME–Air Program 68 5 590 Dollars 87.1 87.2 0.0% 2.8% 2.8% 0.0% Vehicle PME–Air Program 69 5 579 Dollars 305.7 305.8 0.0% 9.5% 9.5% 0.0% Vehicle Forecasting 2020, 2 450
References
1. United States Government Accountability Office; Oakley, S.S. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends: Report to Congressional Committees; United States Government Accountability Office: Washington, DC, USA, 2018. 2. Wright, T.P. Factors Affecting the Cost of Airplanes. J. Aeronaut. Sci. 1936, 3, 122–128. [CrossRef] 3. Asher, H. Cost-Quantity Relationships in the Airframe Industry. Ph.D. Thesis, The Ohio State University, Columbus, OH, USA, 1956. 4. Boone, E.R.; Elshaw, J.J.; Koschnick, C.M.; Ritschel, J.D.; Badiru, A.B. A Learning Curve Model Accounting for the Flattening Effect in Production Cycles. Def. Acquis. Res. J. 2021. in-print. 5. Mislick, G.K.; Nussbaum, D.A. Cost Estimation: Methods and Tools; John Wiley & Sons: Hoboken, NJ, USA, 2015. 6. Argote, L.; Beckman, S.L.; Epple, D. The Persistence and Transfer of Learning in Industrial Settings. Manag. Sci. 1990, 36, 140–154. [CrossRef] 7. Argote, L. Group and Organizational Learning Curves: Individual, System and Environmental Components. Br. J. Soc. Psychol. 1993, 32, 31–51. [CrossRef] 8. Jaber, M.Y. Learning and Forgetting Models and Their Applications. Handb. Ind. Syst. Eng. 2006, 30, 30–127. 9. Glock, C.H.; Grosse, E.H.; Jaber, M.Y.; Smunt, T.L. Applications of Learning Curves in Production and Operations Management: A Systematic Literature Review. Comput. Ind. Eng. 2019, 131, 422–441. [CrossRef] 10. Jaber, M.Y.; Bonney, M. Production Breaks and the Learning Curve: The Forgetting Phenomenon. Appl. Math. Model. 1996, 2, 162–169. [CrossRef] 11. Nembhard, D.A.; Uzumeri, M.V. Experiential Learning and Forgetting for Manual and Cognitive Tasks. Int. J. Ind. Ergon. 2000, 25, 315–326. [CrossRef] 12. Sikström, S.; Jaber, M.Y. The Depletion–Power–Integration–Latency (DPIL) Model of Spaced and Massed Repetition. Comput. Ind. Eng. 2012, 63, 323–337. [CrossRef] 13. Anzanello, M.J.; Fogliatto, F.S. Learning Curve Models and Applications: Literature Review and Research Directions. Int. J. Ind. Ergon. 2011, 41, 573–583. [CrossRef] 14. Crossman, E.R. A Theory of the Acquisition of Speed-Skill . Ergonomics 1959, 2, 153–166. [CrossRef] ∗ 15. Moore, J.R.; Elshaw, J.J.; Badiru, A.B.; Ritschel, J.D. Acquisition Challenge: The Importance of Incompressibility in Comparing Learning Curve Models; US Air Force Cost Analysis Agency: Arlington, TX, USA, 2015. 16. Honious, C.; Johnson, B.; Elshaw, J.; Badiru, A. The Impact of Learning Curve Model Selection and Criteria for Cost Estimation Accuracy in the DoD; Air Force Institute of Technology: Wright Patterson AFB, OH, USA, 2016. 17. Baloff, N. Startups in Machine-Intensive Production Systems. J. Ind. Eng. 1966, 17, 25. 18. Baloff, N. Startup Management. IEEE Trans. Eng. Manag. 1970, 4, 132–141. [CrossRef] 19. Corlett, E.N.; Morecombe, V.J. Straightening Out Learning Curves. Pers. Manag. 1970, 2, 14–19. 20. Yelle, L.E. The Learning Curve: Historical Review and Comprehensive Survey. Decis. Sci. 1979, 10, 302–328. [CrossRef] 21. Hirschmann, W.B. Profit from the Learning-Curve. Harv. Bus. Rev. 1964, 42, 125–139. 22. Li, G.; Rajagopalan, S. A Learning Curve Model with Knowledge Depreciation. Eur. J. Oper. Res. 1998, 105, 143–154. [CrossRef] 23. Chalmers, G.; DeCarteret, N. Relationship for Determining the Optimum Expansibility of the Elements of a Peacetime Aircraft Procurement Program; Stanford Research Institute: Menlo Park, CA, USA, 1949. 24. De Jong, J.R. The Effects of Increasing Skill on Cycle Time and Its Consequences for Time Standards. Ergonomics 1957, 1, 51–60. [CrossRef] 25. Knecht, G.R. Costing, Technological Growth and Generalized Learning Curves. J. Oper. Res. Soc. 1974, 25, 487–491. [CrossRef] 26. Badiru, A.B. Computational Survey of Univariate and Multivariate Learning Curve Models. IEEE Trans. Eng. Manag. 1992, 39, 176–188. [CrossRef] 27. Office of the Secretary of Defense. 1921-1 Data Item Description. Available online: https://cade.osd.mil/ content/cade/files/csdr/dids/archive/1921-1.DI-FNCL-81566B.pdf (accessed on 29 September 2020). 28. Hu, S.-P.; Smith, A. Accuracy Matters: Selecting a Lot-Based Cost Improvement Curve. J. Cost Anal. Parametr. 2013, 6, 23–42. [CrossRef] 29. Cohen, J. Quantitative Methods in Psychology. Nature 1938, 141, 613. [CrossRef] Forecasting 2020, 2 451
30. Badiru, A.B. Half-Life Learning Curves in the Defense Acquisition Life Cycle. Def. Acquis. Res. J. 2012, 19, 283–308.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).