Using "InsiderEconometrics" to Study Productivity

By ANN BARTEL, CASEY ICHNIOWSKI, AND KATHRYN SHAW*

Great advances have been made in theory At the micro level, the firms or plants that and in econometric techniques, but these we analyze differ a greatdeal, even within will be wasted unless they are applied to what one might think of as a well-defined the right data. "industry".... They differ in the parti- -Zvi Griliches (1994 p. 2) cular assortment of products they may produce .... [and] in the inputs and tech- Griliches' 1994 presidential address consid- nologies that they use to producethem .... ers the limited success economists had in trying Unfortunately,standard census type data to account for the slowdown of do not provide enough additional infor- productivity relevant and char- the 1970's and 1980's, and us towardthe mation or product plant "urges acteristics to allow one to a task of observation and measurement."In the pursue substantive .... To make further 1990's, the rates of analysis high productivity growth we need to infuse [production the need for progress, emphasized new models of pro- functions] with new data and appropriate ductivity, this time turning to estimating theoretical and econometric models for organization-leveldeterminants of productivity dealing with the real heterogeneitythat is focusing on businesses' use of new computer- the hallmarkof the world we live in. based information technologies (IT), and new methods of work organization (Timothy Ichniowski and Shaw (2003) use the term "in- Bresnahanet al., 2002). In this paper, we take sider econometrics" to describe productivity up the charge to develop new data and new studies that combine extensive field work to methods for modeling the productivityof orga- assemble useful organization-level data sets nizations. We summarizethree methods for as- with rigorous econometrichypothesis testing of sembling data for an "insider econometrics" the effects of organization-specificdeterminants study of the productivityof organizations,and of productivity.This section summarizes three we illustrate one method that we refer to as approachesto "insidereconometrics" studies. "informedsurvey analysis." 1. Cross-Organization Studies Based on I. Three Methodsfor ConductingInsider Plant Visits.-Insider econometrics is defined Studiesof OrganizationalProductivity by two broad principles. First, it uses field work to generate a detailed understandingof a Griliches and Jacques Mairesse (1995 pp. specific production process, its technology, 22-24) describe why it is so challenging to and the nature of the work in a particular assemble the "rightdata" to investigate produc- industry. This field work in turn provides tivity determinantsof real organizations: valuable insights about how to model produc- tion in that industry and what data to collect to estimate those models. Second, detailed operating data from the industry are used to estimate econometric productivity models * Bartel: Columbia Business School, Columbia Univer- that permit convincing tests of hypotheses sity, 3022 Broadway, New York, NY 10027, and NBER; about the determinants of productivity. Ichniowski: Columbia Business School and NBER; Shaw, One method of implementing insider econo- StanfordBusiness School, StanfordUniversity, 518 Memo- metrics is to gather data from firms on the very rial Way, Stanford, CA 94305, and NBER. We thank that use in monitor- Ricardo Correa and Yoonsoo Lee for their excellent performancemeasures they research assistance and Lawrence Katz for his valuable ing production. Ichniowski, Shaw, and coau- comments. thors implementthis approachin their studies of

217 218 AEA PAPERSAND PROCEEDINGS MAY2004 the effects of human-resource management 3. Insider Productivity Research with "In- practices on productivity in the steel industry, formed Surveys."-A third approach for col- visiting about 85 plants in the steel industryto lecting "the right data" for organization-level conduct interviews and obtain data. The advan- productivitystudies is to obtain data from "in- tages of this approachare that researcherscan formed surveys."Plant visits and interviews are model very sensible cross-firmproduction func- conducted at a small sample of plants in an tions, and can model why some firms adopt new industryand then used to understandthe indus- human-resources practices and some do not. try's productionprocess and technology and to This approachis, unfortunately,also very costly develop a narrow industry-specificsurvey. We and time-consuming. illustrate this approach using our results from the valve industry below. Note however, that 2. Single-FirmStudies.-A second and more others have utilized "informed surveys" that common way to conduct insider productivity Census researchers with expertise in specific researchis to focus on the operationsof a single industrieshave tailored to specific industriesor firm. Insider insights about key productionpro- occupations.2This third approachis quite sim- cesses in the firm identify situations where in- ilar to the first above and is considerably dividual employees, teams of workers, or cheaper to undertake,but it suffers from poten- separate establishments inside the same com- tial recall bias or measurementerror. pany comprisethe productionunits. These with- in-firmstudies then provide convincing analysis II. Insider Insights into the U.S. Valve-Making of the effects of changing personnel practices Industry across these units. Examples include Edward Lazear's (2000) study of piece rates in wind- To pursuethis thirdapproach for plants in the shield installation, Barton Hamilton et al.'s U.S. valve-making industry (SICs 3491, 3492, (2003) study of team methods in apparel man- 3494, and 3593), we conducted site visits and ufacturing,Martin Gaynor et al.'s (2004) study interviews at five valve-making plants during of incentives in an HMO, Rosemary Batt's 1999-2000 and in 2002 (during survey devel- (1999) study of teams in a telecommunications opment). A valve is typically a metal device company, and studies by Bartel (2004) and Bar- attachedto pipes that regulates the flow of liq- tel et al. (2003) of employee satisfactionin bank uids or gases, such as the flow of naturalgas in branches of one Canadian company and one a heating system, or the control of liquids in a U.S. company, respectively. The advantage of chemical factory. The central production pro- this approach is that the research can often cess in valve-makingis the machiningphase. A model the sources of productivity change, in- simple valve would be made by taking a steel cluding changes in the selection of workers. Of block or pipe and completing several processes course, single-firm studies cannot model the on one or more machines, such as etching causes of the adoption of practices. grooves at each end for screwing the valve to pipes, boring holes at different spots to attach control devices, and then making and attach- ing the various devices that control the flow. Based on our visits and interviews at these sites, 1 Specifically, Ichniowski, Shaw, and coauthors visited we developed an industry-specific survey to 45 productionlines of 20 companies in the U.S. integrated steel industry(Ichniowski et al., 1997), five integratedsteel mills at two Japanese companies (Ichniowski and Shaw, 2 1999), and 34 productionlines operatedby 19 U.S. minimill Examples of the use of Census surveys are Thomas companies (Brent Boning et al., 2001). Otherinsider studies Hubbard (2004) and George Baker and Hubbard (2003) noteworthy for visits and data from many companies and for studies of the effects of information technologies in work sites include John Paul MacDuffie's (1995) analysis of trucking; Luis Garicano and Hubbard (2003) for their productivityeffects of human-resourcesmanagement prac- study of lawyers; and Chad Syverson (2003) for his study tices in automobile assembly plants and Kim B. Clark's of the cement industry. Maryellen Kelley (1994) con- (1984) study of unions and productivity in the cement ducts her own survey of machine operations in 21 indus- industry. tries to study the effects of work organization and IT. VOL. 94 NO. 2 NEW DATAAND NEW QUESTIONSIN PERSONNELECONOMICS 219 measure productivity, technologies, and work TABLE 1-SUMMARY STATISTICS ON PRODUCTION TIMES practices.3 IN VALVE MACHINING, NEW COMPUTER-BASED PRODUCTION TECHNOLOGIES, AND HUMAN-RESOURCE MANAGEMENT PRACTICES A. Measuring Efficiency in the Machining Process A Mean valuea itself involves time to Machining setup pro- Component 1997 2002 Log change gram machines so they will perform the right combination of tasks for the valve's Setup time 0.49 0.28 -0.681 specifica- Run time 0.45 0.39 -0.371 the actual run time to the ma- tion, complete Inspection time 0.05 0.03 -0.334 chining, and inspection time to verify the of the valves. We measure these three Total time 1.03 0.72 -0.481 quality Number of 5.63 4.97 -0.189 components by asking survey respondents to machines provide setup time, run time, and inspection time in 1997 and 2002 for the product they B the most over those Our sur- Fraction of observations produced years. Technology or vey results show that the production times for practiceb Using, 2002C Adopting, 1997-2002d these declined over the last five products years FMS 0.337 0.151 (Table 1). Auto sensors 0.283 0.137 3-D CADe 0.738 0.387 B. Technologies and Valve-MakingEfficiency Basic training 0.333 0.119 Technical training 0.726 0.211 Teams 0.647 0.298 Today, the central piece of equipment in the valve-making production process is a CNC a In fractions of a day, except for number of machines. b (computernumerically controlled) machine that See text for explanationsof abbreviations. c automates the machining process. While CNC Fraction of observations wherein the plant was using machines have been in use for about 30 the equipment or managementpractice in 2002. years, d Fractionof observationswherein the the the of individual CNC machines plant adopted capabilities equipment or managementpractice during 1997-2002. improved dramatically in the 1990's as com- e Three-dimensionalcomputer-assisted design. puter power increased. During our plant visits, managers described the primary way in which new CNC machines raise productivity:the in- precisely ("auto sensors"). In our survey, we creasing sophistication of the CNC machines asked if plants have these technologies and results directly in a decrease in the number of when they were introduced. As shown in machines needed to produce a given product. Table 1, these new technologies became in- Therefore,we use the numberof machines in a creasingly common over time. run of the plant's main product as our key measure of improvementsin CNC technology. C. Skills, Training, and Human-Resource Managers also identified two other tech- ManagementPractices nologies as importantsources of improved op- erational efficiency: flexible manufacturing These new technologies may be related to an systems (FMS) that coordinate the runs across increaseddemand for more-skilledworkers. We multiple machines through the use of sophisti- collect datain our valve-industrysurvey to mea- cated software, and new automated valve in- sure whether plants tried to increase worker spection equipment that uses laser probe skills througha trainingprogram in basic math technology to measure dimensions of valves and reading skills ("basic training")or through training in new technical skills for operating 3 new technologies ("technical training").Other The telephone survey was conducted during 2002- ask about the use of human- 2003 the Office for Research at the Institute for survey questions by Survey besides Public Policy and Social Research at Michigan State Uni- resource managementpractices training versity. The response rate was 43 percent. programs, such as problem-solving teams 220 AEA PAPERSAND PROCEEDINGS MAY2004

TABLE2-LRD PRODUCTIVITYREGRESSIONS TABLE3-1997-2002 FIRSTDIFFERENCE PRODUCTIVITY REGRESSIONSUSING SURVEY DATA Dependent variable Dependent variable (i) (ii) 1997 1992-1997 (i) (ii) (iii) Independentvariable Levels First differences Setup Run Inspection Independentvariable time time time Log(total hours) 0.384** 0.219** (0.040) (0.041) Change in number 0.546** Log(capital) -0.010 -0.015 of machines (0.176) (0.024) (0.026) New FMS -0.397t Log(materials) 0.610** 0.516** (0.243) (0.035) (0.036) New auto sensor -0.399** Number of observations: 178 145 (0.206) New technical -0.439** -0.381+ -0.183 R2: 0.938 0.721 training (0.208) (0.239) (0.193) New basic training -0.351 0.159 -0.071 Notes: The sample comprises plants in the authors' survey. Standarderrors are in (0.252) (0.294) (0.237) reported parentheses. New teams 0.264 0.300 -0.017 ** Statistically significant at the 1-percentlevel. (0.181) (0.199) (0.166) Number of observations: 140 146 155 R2: 0.15 0.17 0.04 ("teams").All of these practices increase over time (Table 1). Notes: All regressions include the age of the plant, the change in the numberof shop-floor employees at the plant III. Conventional Productivity Estimates Using and whether the plant is unionized. Standard errors are LRD Plant-Level Data reportedin parentheses. tStatistically significant at the 10-percentlevel. ** Statistically significant at the 1-percentlevel. As a contrastwith our own survey results for production, we introduce standardproduction- function results using the Census of Manufac- turers Longitudinal Research Database (LRD) endogenous. However, the simple estimates data for plants in the valve industry that re- presentedhere will highlight the difference be- sponded to our own survey. We estimate a tween standarddata applicationsand the use of standard production-function framework in our survey data below, suggesting perhapsthat which log of output (value of shipments minus "... measurement difficulties ... may in fact be a change in inventories) is a function of logs of major source of the failure ... to explain what labor hours, capital (gross value of depreciable has happenedto the economy" (Griliches, 1994 assets), and materials. p. 10). The results show that labor and materials inputs are always significant in these ordinary IV. Estimates of the Determinants of Productivity least-squares (OLS) regressions, and capital is Using an Informed Survey never significant (Table 2). Before interpreting these preliminaryresults as evidence thatcapital Using our survey data, we regress measures in valve-making plants is relatively unproduc- of productiontime on the technology measures tive, a number of alternativepossible interpre- described above. The results are straightfor- tations could be explored. One could argue that ward:the adoptionof new technologies reduces valve-making has fixed factor productionchar- production time in the stage of production acteristicsand that variationin the labor input is where the technology is of value (Table the constraining factor in production (e.g., if 3). Using fewer machines to produce a product some equipment lies idle due to lack of orders reduces setup times. Run time declines signifi- or labor shortages). One should also consider cantly in plants that adopt FMS technology. models that instrumentthe capital variable be- Inspection time declines with the introduction cause it is measured with error or because it is of new automated inspection equipment (auto VOL. 94 NO. 2 NEW DATAAND NEW QUESTIONSIN PERSONNELECONOMICS 221 sensors). New IT-based production machinery important effects of information technologies improves the efficiency of the stage of produc- and training that could not be identified with tion in which it is involved; new computertech- Census data. nologies do not improvethe efficiency of phases However, the question remains, given these of machining in which they are not involved. new survey data, what theoretical and econo- These results stand in sharp contrast to results metric models are now required?Note first that obtained with plant-level LRD data using sim- our survey data also reduces the likelihood of ilar OLS estimation methods that find that the endogeneity bias. Consider the setup-time re- partialcorrelation of capital and outputis insig- gression. The only way that setup time can be nificantly different from zero. Moreover, the reduced over time for the same productis if the estimated efficiency gains due to new technol- technology has changed, either because workers ogies in the survey data are sizable. are better able to use the existing technology The effects of human-resourcemanagement (perhapsdue to better training)or because there variablesare more mixed. Skills trainingrelated to is new technology. Based on plant visits and our new technologies(technical training) improves ef- understandingof the productionfunction, there ficiency in setup times and run times. The intro- is no reason for a decline in setup time to cause duction of teams and basic skills training are a decline in the number of machines in use. found to be uncorrelatedwith improvementsin Thus, some endogeneity problems are avoided any of the machining time components.These with these data. resultsconcerning the effects of improvedworker Two potential problems remain. First, there skills reveal that initiativesdesigned to improve may be some omitted-variablebias in our re- the specific skills needed to operatenew technol- sults, if, for example, a reductionin the number ogies in the plant are in fact the initiativesthat of machines used to produce a given productis improveoperational efficiency. correlated with unobserved contemporaneous changes in the organization. Here, the narrow V. New Data and AppropriateModels scope of the productivity model (spanning the operations of only a few machines) limits this When the researcher'sgoal is to uncover the problem, and direct contact with the plants effects of organizationalpractices or the effects and their managers allows the researcher to of specific computertechnologies on productiv- investigate whether such confounding factors ity, he should seek data that can be used to exist. estimate specific productivitymodels in which Second, there may be selectivity bias. It is the variablesof interestcan be expected to have likely that the adopters of new technologies direct effects that can be interpretedin a mean- (like new CNC machines that reduce the num- ingful way. As the Griliches and Mairesse ber of machines per product produced) are the (1995) passage quoted above warns, standard plants that have the most to gain from the new census data are usually not rich enough to per- technologies. Non-adoptersare droppingout of mit this. The problems that result from limita- our sample if they go out of business, or they tions of the Census data are well described in may not earn the same returnsto technological the literature:measurement error in the depen- change as the adopters do. The key question dent variable (which includes changes in prod- regardingthis plausible type of selectivity bias uct mix and requires appropriatedeflators to is: what is the goal of the study? If the goal is to translate nominal values into quantities) and estimate an unbiasedreturn to the randomadop- endogeneity and selection bias.4 We show that tion of new technologies (or the "averagetreat- models that express production-timeefficiency ment effect"), that is a difficult task with any as a function of specific technologies identify nonexperimental data, and firms rarely offer opportunities for natural experiments. If the goal is to understandthe gains for those who the "treatmentof the 4 For discussionof these see Grilichesand Mairesse are likely adopters (or issues, is to a (1995), Steve Olley and Ariel Pakes (1996), Syversen(2003), treated"), then the next step develop and JamesLevinsohn and Amil Petrin(2003). model that simultaneouslypredicts the adoption 222 AEA PAPERSAND PROCEEDINGS MAY2004 of new technologies and their likely gains Organizations Have an Attitude Problem? (Boning et al., 2003). The Impact of Workplaces on Employee Another concern is that, by focusing on the Attitudes and Economic Outcomes." Na- productionefficiency of producingone product, tional Bureau of Economic Research (Cam- we miss changes in product mix or product bridge,MA) WorkingPaper No. 9987, 2003. quality that may well contributeto the returnsto Batt, Rosemary."Work Organization,Technol- the adoption of new technologies. For this pur- ogy, and Performance in Customer Service pose, it may be wise to turn back to the Census and Sales." Industrial and Labor Relations data on value added. Review, July 1999, 52(4), pp. 539-64. Boning, Brent; Ichniowski, Casey and Shaw, VI. Conclusion Kathryn. "OpportunityCounts: Teams and the Effectiveness of Production Incentives." Insider econometric studies have typically National Bureau of Economic Research used one of three alternativetypes of appropri- (Cambridge,MA) Working Paper No. 8306, ate data for estimating organization-level pro- 2001. duction functions: data obtained from one firm Bresnahan,Timothy; Brynjolfsson, Erik and Hitt, to model productiondifferences across individ- Loren. "InformationTechnology, Work Or- uals or units of production (like teams or ganization, and the Demand for Skill Labor: branches) within that firm; productiondata ob- Firm-Level Evidence." QuarterlyJournal of tained directly from visits to many companies' Economics, February2002, 117(1), pp. 339- plants all employing a common productionpro- 76. cess; and finally, data from "informedsurveys" Clark, Kim B. "Unionization and Firm Per- that are tailored to elicit informationabout one formance: The Impact on Profits, Growth, specific production process. Using this "in- and Productivity." American Economic formed survey" approach, we show that there Review, December 1984, 74(5), pp. 893- appearto be gains from the use of information 919. technologies and personnel practices in the Garicano,Luis and Hubbard,Thomas. "Special- valve industry,gains that could not possibly be ization, Firms, and Markets:The Division of revealed using standardCensus of Manufactur- Labor Within and Between Law Firms." ing data. Moreover, field visits enabled us to Working paper, Graduate School of Busi- understand the production processes, output ness, University of Chicago, 2003. measures, and technologies in this industrial Gaynor,Martin; Rebitzer, James B. and Taylor, setting before econometric models of organiza- Lowell J. "Physician Incentives in HMO's." tion-level determinantsof productivitywere es- Journal of Political Economy, 2004 (forth- timated. Not only does getting "the right data" coming). mattera greatdeal, but so too does getting insid- Griliches,Zvi. "Productivity,R&D, and the Data ers' insightsabout what the right data really are. Constraint." American Economic Review, March 1994, 84(1), pp. 1-23. REFERENCES Griliches, Zvi and Mairesse, Jacques. "Pro- duction Functions: The Search for Iden- Baker, George and Hubbard, Thomas. "Make tification." National Bureau of Economic versus Buy in Trucking:Asset Ownership,Job Research (Cambridge, MA) Working Paper Design, and Information."American Economic No. 5067, 1995. Review,June 2003, 93(3), pp. 551-71. Hamilton, Barton; Nickerson, Jack and Owan, Bartel, Ann. "Human Resource Management Hideo. "Team Incentives and Worker Heter- and Organizational Performance: Evidence ogeneity: An Empirical Analysis of the from Retail Banking." Industrial and Labor Impact of Teams on Productivityand Partic- Relations Review, January 2004, 57(2), pp. ipation."Journal of Political Economy, June 181-203. 2003, 111(3), pp. 465-97. Bartel, Ann; Freeman, Richard; Ichniowski, Hubbard, Thomas. "Information, Decisions Casey and Kleiner, Morris. "Can Work and Productivity: On Board Computers VOL. 94 NO. 2 NEW DATAAND NEW QUESTIONSIN PERSONNELECONOMICS 223

and Capacity Utilization in Trucking." agement Science, November 1994, 40(11), American Economic Review, September pp. 1406-25. 2003, 93(4), pp. 1328-53. Lazear,Edward. "Performance Pay and Produc- Ichniowski,Casey and Shaw, Kathryn."The Ef- tivity." American Economic Review, Decem- fects of Human Resource Management Sys- ber 2000, 90(5), pp. 1346-61. tems on Productivity: An International Levinsohn,James and Petrin, Amil. "Estimating Comparison of U.S. and Japanese Plants." ProductionFunctions Using Inputsto Control Management Science, May 1999, 45(5), pp. for Unobservables." Review of Economic 704-22. Studies, April 2003, 70(2), pp. 317-42. "Beyond Incentive Pay: Insiders' MacDuffie,John Paul. "HumanResource Bun- Estimates of the Value of Complementary dles and ManufacturingPerformance: Orga- Human Resource Management Practices." nizational Logic and Flexible Production Journal of Economic Perspectives, Winter Systems in the World Auto Industry."Indus- 2003, 17(1), pp. 155-80. trial and Labor Relations Review, January Ichniowski, Casey; Shaw, Kathryn and Pren- 1995, 48(2), pp. 197-221. nushi, Giovanna. "The Effects of Human Olley, Steve and Pakes,Ariel. "The Dynamics of Resource ManagementPractices on Produc- Productivity in the Telecommunication tivity: A Study of Steel Finishing Lines." Equipment Industry." Econometrica, No- American Economic Review, June 1997, vember 1996, 64(6), pp. 1263-98. 87(3), pp. 291-313. Syverson, Chad. "Market Structure and Pro- Kelley,Maryellen. "Information Technology and ductivity: A Concrete Example." Working Productivity:The Elusive Connection."Man- paper, University of Chicago, 2003.