Tantalus on the Road to Asymptopia

Journal of Economic Perspectives—Volume 24, Number 2—Spring 2010—Pages 31–46

Edward E. Leamer

y fi rrstst reactionreaction toto “The“The CredibilityCredibility RevolutionRevolution inin EmpiricalEmpirical Economics,”Economics,” aauthoreduthored byby JoshuaJoshua DD.. AAngristngrist aandnd JJörn-Steffenörn-Steffen Pischke,Pischke, wwas:as: Wow!Wow! ThisThis M ppaperaper makesmakes a stunninglystunningly ggoodood ccasease fforor rrelyingelying oonn ppurposefullyurposefully random-random- iizedzed oror accidentallyaccidentally randomizedrandomized experimentsexperiments toto relieverelieve thethe doubtsdoubts thatthat afflaffl iictct iinferencesnferences ffromrom nnonexperimentalonexperimental ddata.ata. OOnn ffurtherurther rreflefl ection,ection, I realizedrealized thatthat I maymay hhaveave beenbeen overcomeovercome wwithith iirrationalrrational eexuberance.xuberance. MMoreover,oreover, wwithith tthishis ggreatreat hhonoronor bbestowedestowed oonn mmyy ““con”con” aarticle,rticle, I couldn’tcouldn’t easilyeasily throwthrow thisthis childchild ofof minemine overboard.overboard. WWee eeconomistsconomists ttrudgerudge rrelentlesslyelentlessly ttowardoward AAsymptopia,symptopia, wwherehere ddataata aarere uunlimitednlimited aandnd estimatesestimates areare consistent,consistent, wherewhere thethe lawslaws ofof largelarge numbersnumbers applyapply perfectlyperfectly andand wwherehere thethe fullfull intricaciesintricacies ofof tthehe eeconomyconomy aarere ccompletelyompletely rrevealed.evealed. BButut it’sit’s a frus-frus- ttratingrating journey,journey, ssince,ince, nono mattermatter howhow farfar wewe travel,travel, AAsymptopiasymptopia rremainsemains iinfinfi nnitelyitely ffarar aaway.way. WWorstorst ooff aall,ll, wwhenhen wwee ffeeleel ppumpedumped upup withwith ourour progress,progress, a tectonictectonic shiftshift cancan ooccur,ccur, llikeike thethe PanicPanic ofof 2008,2008, makingmaking itit seemseem asas thoughthough ourour longlong journeyjourney hashas leftleft usus ddisappointinglyisappointingly ccloselose toto thethe StateState ofof CompleteComplete IgnoranceIgnorance whencewhence wewe began.began. TThehe ppointlessnessointlessness ooff mmuchuch ooff oourur ddailyaily aactivityctivity mmakesakes uuss rreceptiveeceptive wwhenhen tthehe PPriestsriests ooff oourur ttriberibe ringring tthehe bellsbells andand announceannounce a shortenedshortened pathpath toto Asymptopia.Asymptopia. ((RememberRemember tthehe CowlesCowles FoundationFoundation offeringoffering asymptoticasymptotic propertiesproperties ofof simultaneoussimultaneous eequationsquations estimatesestimates andand structuralstructural parameters?)parameters?) WeWe maymay listen,listen, bbutut wwee ddon’ton’t hhear,ear, wwhenhen thethe PriestsPriests warnwarn thatthat tthehe newnew directiondirection isis onlyonly forfor thosethose withwith Faith,Faith, thosethose wwithith ccompleteomplete beliefbelief inin thethe AssumptionsAssumptions ofof thethe Path.Path. ItIt oftenoften takestakes yearsyears downdown thethe PPath,ath, butbut soonersooner oror later,later, someonesomeone articulatesarticulates thethe concernsconcerns thatthat gnawgnaw awayaway inin eacheach ofof

■ EEdwarddward EE.. LLeamereamer iiss PProfessorrofessor ooff EEconomics,conomics, MManagementanagement andand Statistics,Statistics, UniversityUniversity ofof CCaliforniaalifornia atat LosLos Angeles,Angeles, LosLos Angeles,Angeles, California.California. HisHis e-maile-mail addressaddress isis 〈[email protected]@ aanderson.ucla.edunderson.ucla.edu〉. doi=10.1257/jep.24.2.31 32 Journal of Economic Perspectives

us and asks if the Assumptions are valid. (T. C. Liu (1960) and Christopher Sims (1980) were the ones who proclaimed that the Cowles Emperor had no clothes.) Small seeds of doubt in each of us inevitably turn to despair and we abandon that direction and seek another. TTwowo ooff tthehe llatestatest pproducts-to-end-all-sufferingroducts-to-end-all-suffering areare nonparametricnonparametric eestimationstimation aandnd cconsistentonsistent standardstandard eerrors,rrors, wwhichhich promisepromise rresultsesults wwithoutithout aassumptions,ssumptions, asas ifif wwee wwereere aalreadylready iinn AAsymptopiasymptopia wwherehere datadata areare soso pplentifullentiful tthathat nnoo aassumptionsssumptions aarere nneeded.eeded. ButBut likelike proceduresprocedures thatthat relyrely explicitlyexplicitly onon assumptions,assumptions, thesethese newnew mmethodsethods workwork wellwell inin thethe circumstancescircumstances inin whichwhich explicitexplicit oorr hhiddenidden aassumptionsssumptions hholdold ttolerablyolerably wwellell aandnd ppoorlyoorly ootherwise.therwise. BByy ddisguisingisguising tthehe aassumptionsssumptions oonn wwhichhich nnonparametriconparametric mmethodsethods aandnd cconsistentonsistent sstandardtandard eerrorsrrors rrely,ely, thethe purveyorspurveyors ooff tthesehese mmethodsethods hhaveave mmadeade iitt iimpossiblempossible ttoo hhaveave aann iintelligiblentelligible cconversationonversation aaboutbout tthehe ccircumstancesircumstances iinn wwhichhich theirtheir ggimmicksimmicks ddoo nnotot wworkork wwellell aandnd ooughtught nnotot ttoo bbee uused.sed. AAss fforor mme,e, I ppreferrefer toto carrycarry parametersparameters onon mymy journeyjourney soso I knowknow wherewhere I amam andand wwherehere I amam going,going, nnotot ttravelravel sstonedtoned oonn tthehe llatestatest eeuphoriauphoria ddrug.rug. TThishis iiss a sstorytory ooff TTantalus,antalus, ggraspingrasping fforor kknowledgenowledge tthathat rremainsemains alwaysalways beyondbeyond rreach.each. IInn GGreekreek mythologymythology TantalusTantalus waswas favoredfavored amongamong allall mortalsmortals byby beingbeing askedasked ttoo ddineine wwithith tthehe ggods.ods. BButut hehe misbehaved—somemisbehaved—some ssayay bbyy tryingtrying ttoo ttakeake ddivineivine ffoodood bbackack toto thethe mortals,mortals, somesome saysay byby invitinginviting thethe godsgods toto a dinnerdinner forfor whichwhich TantalusTantalus bboiledoiled hishis sonson andand servedserved himhim asas thethe mainmain dish.dish. WhateverWhatever thethe etiquetteetiquette fauxfaux pas,pas, TTantalusantalus waswas punishedpunished byby beingbeing immersedimmersed upup toto hishis neckneck inin water.water. WhenWhen hehe bowedbowed hhisis headhead toto drink,drink, thethe waterwater draineddrained away,away, aandnd wwhenhen hehe stretchedstretched uupp ttoo eeatat thethe fruitfruit hhanginganging aabovebove him,him, wwindind wwouldould blowblow itit outout ofof reach.reach. ItIt wouldwould bebe muchmuch healthierhealthier fforor aallll ooff uuss iiff wwee ccouldould aacceptccept oourur ffate,ate, rrecognizeecognize tthathat pperfecterfect kknowledgenowledge willwill bebe fforeverorever beyondbeyond ourour reachreach andand fi nndd hhappinessappiness wwithith wwhathat wwee hhave.ave. IIff wwee sstoppedtopped ggraspingrasping fforor tthehe aapplepple ooff AAsymptopia,symptopia, wwee wwouldould ddiscoveriscover tthathat oourur ppoolool ooff TTantalusantalus iiss fullfull ofof smallsmall butbut enjoyableenjoyable iinsightsnsights andand wisdom.wisdom. CCanan wwee eeconomistsconomists agreeagree thatthat itit isis extremelyextremely hardhard workwork toto squeezesqueeze truthstruths ffromrom ourour datadata ssetsets andand whatwhat wwee ggenuinelyenuinely uunderstandnderstand wwillill remainremain uuncomfort-ncomfort- aablybly llimited?imited? WeWe needneed wordswords inin ourour methodologicalmethodological vocabularyvocabulary toto expressexpress thethe llimits.imits. WWee nneedeed ssensitivityensitivity analysesanalyses toto makemake thosethose limitslimits transparent.transparent. ThoseThose whowho tthinkhink otherwiseotherwise shouldshould bebe requiredrequired toto wearwear a scarlet-letterscarlet-letter O aroundaround theirtheir necks,necks, fforor “overconfi“overconfi ddence.”ence.” AngristAngrist aandnd PischkePischke obviouslyobviously knowknow this.this. TheirTheir ppaperaper isis ppepperedeppered withwith concernsconcerns aboutabout quasi-experimentsquasi-experiments andand withwith criticismscriticisms ofof instru-instru- mmentalental variablesvariables thoughtlesslythoughtlessly chosen.chosen. I tthinkhink wwee wwouldould mmakeake pprogressrogress ifif wewe sstoppedtopped uusingsing tthehe wwordsords ““instrumentalinstrumental vvariables”ariables” aandnd uusedsed iinsteadnstead ““surrogates”—surrogates”— mmeaningeaning ssurrogatesurrogates forfor thethe experimentexperiment thatthat wewe wishwish wwee ccouldould hhaveave cconducted.onducted. TThehe ppsychologicalsychological ppowerower ooff tthehe vvocabularyocabulary rrequiresequires a ““surrogate”surrogate” toto bebe chosenchosen wwithith mmuchuch ggreaterreater carecare thanthan anan ““instrument.”instrument.” AAss AngristAngrist aandnd PPischkeischke ppersuasivelyersuasively argue,argue, eithereither ppurposefullyurposefully randomizedrandomized eexperimentsxperiments oorr aaccidentallyccidentally rrandomizedandomized ““natural”natural” experimentsexperiments ccanan bbee eextremelyxtremely hhelpful,elpful, bbutut AAngristngrist aandnd PischkePischke seemseem toto meme toto overstateoverstate tthehe ppotentialotential benefibenefi tsts ooff tthehe aapproach.pproach. SSinceince hhardard aandnd iinconclusivenconclusive tthoughthought iiss nneededeeded toto transfertransfer thethe Edward E. Leamer 33

rresultsesults learnedlearned fromfrom randomizedrandomized experimentsexperiments intointo otherother domains,domains, therethere mustmust tthereforeherefore remainremain uncertaintyuncertainty andand ambiguityambiguity aboutabout thethe breadthbreadth ofof applicationapplication ooff aanyny fi ndingsndings fromfrom randomizedrandomized experiments.experiments. FForor example,example, hhowow doesdoes Card’sCard’s ((1990)1990) studystudy ooff tthehe eeffectffect onon thethe MiamiMiami laborlabor marketmarket ofof thethe MarielMariel boatliftboatlift ofof 1125,00025,000 CubanCuban refugeesrefugees inin 19801980 informinform usus ofof thethe effectseffects ofof a 20002000 milemile wallwall alongalong tthehe ssouthernouthern bborderorder ofof tthehe UUnitednited SStates?tates? ThoughtsThoughts areare aalsolso neededneeded toto justifyjustify tthehe cchoicehoice ofof iinstrumentalnstrumental vvariables,ariables, aandnd a criticalcritical elementelement ofof doubtdoubt andand ambiguityambiguity nnecessarilyecessarily aafflffl iictscts anyany instrumentalinstrumental vvariablesariables estimate.estimate. (You(You andand I knowknow thatthat ttrulyruly consistentconsistent estimatorsestimators areare imagined,imagined, notnot real.)real.) AngristAngrist andand PischkePischke under-under- sstandtand tthis.his. ButBut theirtheir studentsstudents andand theirtheir students’students’ studentsstudents mmayay ccomeome ttoo tthinkhink tthathat iitt iiss eenoughnough toto wavewave a cloveclove ofof garlicgarlic andand chantchant “randomization”“randomization” toto solvesolve allall oourur pproblemsroblems jjustust aass aann eearlierarlier ccohortohort ooff eeconometriciansconometricians hhaveave aactedcted aass iiff iitt wwereere eenoughnough ttoo cchanthant ““instrumentalinstrumental vvariable.”ariable.” I wwillill beginbegin tthishis ccommentomment withwith ssomeome thoughtsthoughts aboutabout thethe inevitableinevitable limitslimits ofof rrandomization,andomization, aandnd tthehe nneedeed fforor ssensitivityensitivity aanalysisnalysis iinn tthishis aarea,rea, asas inin allall areasareas ooff appliedapplied eempiricalmpirical wwork.ork. TToo bbee pprovocative,rovocative, I wwillill aarguergue hhereere tthathat tthehe fi nancialnancial ccatastropheatastrophe tthathat wewe havehave justjust experiencedexperienced ppowerfullyowerfully iillustratesllustrates a reasonreason whywhy eextrapolatingxtrapolating fromfrom nnaturalatural eexperimentsxperiments wwillill iinevitablynevitably bbee hhazardous.azardous. TThehe mmisin-isin- tterpretationerpretation ofof historicalhistorical datadata thatthat ledled ratingrating agencies,agencies, investors,investors, andand eveneven myselfmyself toto gguessuess tthathat hhomeome ppricesrices wwouldould ddeclineecline veryvery littlelittle andand ddefaultefault rratesates wwouldould bbee ttolerableolerable eevenven iinn a ssevereevere recessionrecession shouldshould sserveerve aass a ccautionaution forfor allall appliedapplied econometrics.econometrics. I wwillill alsoalso offeroffer ssomeome tthoughtshoughts aaboutbout hhowow tthehe ddiffiiffi ccultiesulties ofof appliedapplied eeconometricconometric workwork ccannotannot bebe evadedevaded withwith econometriceconometric iinnovations,nnovations, oofferingffering ssomeome uunder-recognizednder-recognized ddiffiiffi cultiesculties withwith instrumentalinstrumental variablesvariables andand robustrobust standardstandard errorserrors asas examples.examples. I cconcludeonclude wwithith ssomeome commentscomments aboutabout thethe shortcomingsshortcomings ofof anan experimentalistexperimentalist para-para- ddigmigm asas appliedapplied ttoo mmacroeconomics,acroeconomics, aandnd wwithith somesome warningswarnings aboutabout thethe willingnesswillingness ooff appliedapplied eeconomistsconomists ttoo aapplypply ppush-buttonush-button mmethodologiesethodologies wwithoutithout ssuffiuffi cientcient hhardard tthoughthought regardingregarding theirtheir applicabilityapplicability andand shortcomings.shortcomings.

RRandomizationandomization IsIs NotNot EnoughEnough

AAngristngrist aandnd PPischkeischke oofferffer a ccompellingompelling aargumentrgument tthathat rrandomizationandomization iiss oonene llargearge stepstep iinn tthehe rrightight ddirection.irection. WWhichhich itit iis!s! BButut likelike allall thethe otherother largelarge stepssteps wwee hhaveave alreadyalready taken,taken, thisthis oneone doesn’tdoesn’t getget uuss wwherehere wewe wantwant toto be.be. IInn additionaddition toto randomizedrandomized treatments,treatments, mostmost scientifiscientifi c experimentsexperiments alsoalso havehave ccontrolsontrols overover thethe iimportantmportant confoundingconfounding effects.effects. TheseThese controlscontrols areare neededneeded toto iimprovemprove thethe aaccuracyccuracy ooff tthehe eestimatestimate ooff tthehe ttreatmentreatment eeffectffect aandnd aalsolso ttoo ddetermineetermine cclearlylearly thethe rangerange ofof circumstancescircumstances ooverver wwhichhich thethe estimateestimate applies.applies. ((InIn a llaboratoryaboratory vvacuum,acuum, wewe wouldwould fi nndd thatthat a featherfeather ffallsalls asas fastfast aass a bowlingbowling ball.ball. InIn thethe realreal worldworld wwithith aair,ir, wind,wind, andand humidity,humidity, allall betsbets areare off,off, pendingpending furtherfurther study.)study.) IInn placeplace ooff eexperimentalxperimental ccontrols,ontrols, eeconomistsconomists ccan,an, sshould,hould, aandnd uusuallysually ddoo iincludenclude ccontrolontrol vvariablesariables iinn ttheirheir eestimatedstimated eequations,quations, wwhetherhether thethe datadata aarere 34 Journal of Economic Perspectives

nnonexperimentalonexperimental oror experimental.experimental. TToo mmakeake mmyy ppointoint aaboutbout thethe effecteffect ofof thesethese ccontrolsontrols itit willwill bebe helpfulhelpful toto referrefer toto thethe prototypicalprototypical model:model:

yt = α + (β0 + β1′ zt ) xt + θ ′ wt + εt , where x is the treatment, y the response, z is a set of interactive confounders, w is a set of additive confounders, where ε stands for all the other unnamed, unmeasured effects which we sheepishly assume behaves like a random variable, distributed independently of the observables. Here ( β0 + β1′ zt ) is the variable treatment effect that we wish to estimate. One set of problems is caused by the additive confounding variables w, which can be uncomfortably numerous. Another set of problems is caused by the interactive confounding variables z, which may include features of the experimental design as well as characteristics of the subjects. CConsideronsider fi rrstst tthehe pproblemroblem ooff tthehe aadditivedditive cconfounders.onfounders. WWee hhaveave bbeeneen ttaughtaught tthathat experimentalexperimental rrandomizationandomization ooff tthehe ttreatmentreatment eliminateseliminates thethe requirementrequirement ttoo includeinclude aadditivedditive controlscontrols iinn tthehe eequationquation becausebecause thethe correlationcorrelation bbetweenetween thethe ccontrolsontrols aandnd tthehe ttreatmentreatment iiss zzeroero bbyy ddesignesign aandnd rregressionegression eestimatesstimates wwithith oorr wwithoutithout thethe controlscontrols areare unbiased,unbiased, indeedindeed identical.identical. That’sThat’s truetrue inin Asymptopia,Asymptopia, bbutut iit’st’s nnotot ttruerue hhereere iinn tthehe LLandand ooff tthehe FFiniteinite SSampleample wwherehere correlationcorrelation isis anan ever-ever- ppresentresent ffactact ooff llifeife aandnd wwherehere iissuesssues ooff ssensitivityensitivity ooff cconclusionsonclusions ttoo aassumptionsssumptions ccanan aariserise eveneven wwithith randomizedrandomized treatmentstreatments ifif thethe correlationscorrelations betweenbetween thethe randomizedrandomized ttreatmentreatment andand thethe additiveadditive confounders,confounders, byby chance,chance, areare highhigh enough.enough. IIndeed,ndeed, ifif thethe numbernumber ofof additiveadditive confoundingconfounding variablesvariables isis equalequal toto oror largerlarger tthanhan thethe numbernumber ofof observations,observations, anyany treatmenttreatment x, randomizedrandomized oror not,not, willwill bebe pperfectlyerfectly collinearcollinear withwith thethe confoundingconfounding variablesvariables (the(the undersizedundersized samplesample problem).problem). TThen,hen, ttoo eestimatestimate tthehe ttreatmentreatment eeffect,ffect, wwee wwouldould nneedeed ttoo mmakeake jjudgmentsudgments aaboutbout wwhichhich ofof thethe confoundingconfounding variablesvariables toto exclude.exclude. ThatThat wouldwould ordinarilyordinarily requirerequire a sensi-sensi- ttivityivity aanalysis,nalysis, uunlessnless tthroughhrough DDivineivine rrevelationevelation eeconomistsconomists wwereere ttoldold eexactlyxactly wwhichhich ccontrolsontrols toto includeinclude andand wwhichhich ttoo eexclude.xclude. TThoughhough tthehe numbernumber ofof randomizedrandomized ttrialsrials mmayay bebe large,large, anan importantimportant sensitivitysensitivity questionquestion cancan stillstill arisearise becausebecause thethe numbernumber ooff cconfoundingonfounding vvariablesariables ccanan bebe iincreasedncreased withoutwithout limitlimit byby uusingsing llaggedagged valuesvalues andand nnonlinearonlinear fforms.orms. IInn otherother words,words, iiff yyouou ccannotannot ccommitommit toto somesome notionnotion ofof ssmoothnessmoothness ooff tthehe ffunctionalunctional formform ofof tthehe confoundersconfounders andand somesome notionnotion ofof limitedlimited oror smoothsmooth ttimeime delaysdelays inin response,response, yyouou wwillill nnotot bbee aableble ttoo eestimatestimate tthehe ttreatmentreatment eeffectffect eveneven wwithith a rrandomizedandomized experiment,experiment, unlessunless experimentalexperimental controlscontrols kkeepeep tthehe cconfoundingonfounding vvariablesariables constantconstant oror DivineDivine inspirationinspiration allowsallows youyou toto omitomit somesome ofof thethe variables.variables. YYouou aarere ffreeree toto dismissdismiss thethe precedingpreceding paragraphparagraph asas makingmaking a mountainmountain outout ooff a mmolehill.olehill. BByy rreducingeducing tthehe rrealizedealized correlationcorrelation betweenbetween thethe treatmenttreatment andand tthehe controls,controls, randomizationrandomization aallowsllows a llargerarger ssetet ooff aadditivedditive ccontrolontrol vvariablesariables toto bbee iincludedncluded bbeforeefore wwee cconfrontonfront tthehe ssensitivityensitivity iissuesssues ccausedaused bbyy ccollinearity.ollinearity. FForor tthathat rreason,eason, tthoughhough ccorrelationorrelation bbetweenetween thethe treatmenttreatment andand thethe confoundersconfounders withwith nnonexperimentalonexperimental ddataata isis a huge problem,problem, itit isis mmuchuch llessess importantimportant wwhenhen tthehe ttreat-reat- mmentent isis randomized.randomized. WithWith thatthat problemproblem neutralized,neutralized, concernconcern shiftsshifts elsewhere.elsewhere. Tantalus on the Road to Asymptopia 35

TThehe bigbig problemproblem withwith randomizedrandomized experimentsexperiments isis notnot additiveadditive confounders;confounders; iit’st’s tthehe interactiveinteractive confounders.confounders. ThisThis isis thethe heterogeneityheterogeneity issueissue thatthat especiallyespecially cconcernsoncerns HeckmanHeckman (1992)(1992) aandnd DDeatoneaton ((2008)2008) wwhoho eemphasizedmphasized thethe needneed toto studystudy ““causalcausal mmechanisms,”echanisms,” wwhichhich I amam summarizingsummarizing inin termsterms ofof thethe interactiveinteractive z vvariables.ariables. AngristAngrist andand PischkePischke completelycompletely understandunderstand thisthis point,point, butbut theythey seemseem iinappropriatelynappropriately dismissivedismissive whenwhen theythey accuratelyaccurately explainexplain “extrapolation“extrapolation ofof causalcausal eeffectsffects toto newnew settingssettings isis alwaysalways speculative,”speculative,” whichwhich isis true,true, butbut thethe extrapolationextrapolation sspeculationpeculation isis moremore transparenttransparent andand moremore worrisomeworrisome inin thethe experimentalexperimental casecase tthanhan iinn thethe nonexperimentalnonexperimental case.case. AAfterfter all,all, iinn nnonexperimentalonexperimental nnonrandomizedonrandomized ssettings,ettings, wwhenhen jjudiciousudicious cchoicehoice ofof additiveadditive confoundersconfounders allowsallows oneone toto obtainobtain justjust aaboutbout anyany estimateestimate ofof tthehe ttreatmentreatment eeffect,ffect, therethere isis littlelittle reasonreason toto worryworry aaboutbout ““extrapolationextrapolation ofof ccausalausal eeffectsffects toto newnew settings.”settings.” What’sWhat’s toto extrapolateextrapolate anyway?anyway? OurOur lacklack ofof knowledge?knowledge? GGreaterreater concernconcern aboutabout extrapolationextrapolation isis thusthus anan iindicatorndicator ooff tthehe pprogressrogress thatthat ccomesomes ffromrom randomization.randomization. WWhenhen thethe randomizationrandomization isis accidental,accidental, wewe maymay pretendpretend thatthat thethe instrumentalinstrumental vvariablesariables estimatorestimator isis consistent,consistent, butbut wewe allall knowknow thatthat thethe assumptionsassumptions thatthat justifyjustify tthathat conclusionconclusion cannotcannot possiblypossibly holdhold exactly.exactly. ThoseThose whowho useuse instrumentalinstrumental variablesvariables wwouldould ddoo wwellell ttoo aanticipatenticipate tthehe iinevitablenevitable barragebarrage ofof qquestionsuestions aaboutbout thethe appropriate-appropriate- nnessess ofof theirtheir iinstruments.nstruments. EEver-presentver-present asymptoticasymptotic biasbias castscasts a largelarge darkdark shadowshadow onon iinstrumentalnstrumental vvariablesariables eestimatesstimates aandnd iiss wwhathat limitslimits thethe applicabilityapplicability ofof thethe estimateestimate eevenven ttoo tthehe ssettingetting thatthat iiss oobserved,bserved, notnot toto mentionmention extrapolationextrapolation toto newnew settings.settings. InIn aaddition,ddition, smallsmall samplesample biasbias ofof instrumentalinstrumental variablesvariables estimators,estimators, eveneven inin thethe consis-consis- ttentent case,case, isis a hugehuge nneglectedeglected problemproblem withwith practice,practice, mademade worseworse byby thethe existenceexistence ofof mmultipleultiple wweakeak iinstruments.nstruments. TThishis sseemseems ttoo bbee oonene ooff tthehe ppointsoints ooff tthehe AAngristngrist aandnd PPischkeischke paper—purposefulpaper—purposeful randomizationrandomization isis betterbetter thanthan accidentalaccidental randomization.randomization. BButut whenwhen thethe randomizationrandomization iiss ppurposeful,urposeful, a wholewhole newnew setset ofof issuesissues arises—arises— eexperimentalxperimental contamination—whichcontamination—which iiss mmuchuch mmoreore sseriouserious wwithith hhumanuman ssubjectsubjects iinn a socialsocial systemsystem tthanhan wwithith cchemicalshemicals mixedmixed inin beakersbeakers oror partsparts assembledassembled intointo mmechanicalechanical structures.structures. AnyoneAnyone whowho designsdesigns anan experimentexperiment inin economicseconomics wouldwould dodo wwellell ttoo aanticipatenticipate thethe iinevitablenevitable barragebarrage ofof qquestionsuestions rregardingegarding thethe validvalid transferencetransference ooff tthingshings llearnedearned iinn tthehe lablab (one(one valuevalue ofof z) intointo thethe realreal worldworld ((aa ddifferentifferent valuevalue ofof z)).. WWithith iinteractiventeractive confoundersconfounders explicitlyexplicitly iincluded,ncluded, tthehe ooverallverall ttreatmentreatment effecteffect

β0 + β′ zt iiss nnotot a nnumberumber bbutut a vvariableariable tthathat ddependsepends oonn tthehe cconfoundingonfounding eeffects.ffects. AAbsentbsent oobservationbservation ooff tthehe iinteractiventeractive ccompoundingompounding eeffectsffects z, wwhathat iiss estimatedestimated iiss ssomeome kindkind ofof aaverageverage ttreatmentreatment eeffectffect wwhichhich isis calledcalled byby ImbensImbens andand AngristAngrist (1994)(1994) a “Local“Local AAverageverage TTreatmentreatment Effect,”Effect,” whichwhich isis a littlelittle likelike thethe lawyerlawyer whowho explainedexplained tthathat whenwhen hehe waswas a youngyoung manman hehe lostlost manymany casescases hehe shouldshould hhaveave wwonon bbutut aass hhee ggrewrew olderolder hehe wonwon manymany thatthat hehe shouldshould havehave lost,lost, soso thatthat onon thethe averageaverage justicejustice waswas ddone.one. IInn ootherther wwords,ords, ifif youyou actact asas ifif thethe treatmenttreatment effecteffect isis a randomrandom variablevariable byby ssubstitutingubstituting βt fforor β0 + β′ zt , thethe notationnotation inappropriatelyinappropriately relievesrelieves youyou ofof thethe heavyheavy bburdenurden ofof consideringconsidering wwhathat aarere tthehe iinteractiventeractive confoundersconfounders andand fi ndingnding somesome wwayay ttoo measuremeasure them.them. LessLess elliptically,elliptically, absentabsent observationobservation ofof z, tthehe eestimatedstimated treatmenttreatment 36 Journal of Economic Perspectives

eeffectffect shouldshould bebe transferredtransferred only intointo thosethose settingssettings iinn wwhichhich tthehe cconfoundingonfounding iinteractiventeractive variablesvariables havehave valuesvalues closeclose toto thethe meanmean valuesvalues inin thethe experiment.experiment. IIff llittleittle tthoughthought hashas gonegone iintonto iidentifyingdentifying tthesehese ppossibleossible cconfounders,onfounders, iitt sseemseems probableprobable tthathat llittleittle tthoughthought willwill bebe givengiven toto thethe limitedlimited applicabilityapplicability ofof thethe resultsresults inin otherother ssettings.ettings. ThisThis isis tthehe errorerror mademade byby thethe bondbond ratingrating aagenciesgencies inin tthehe rrecentecent fi nancialnancial ccrash—theyrash—they ttransferredransferred fi nndingsdings ffromrom oonene hhistoricalistorical eexperiencexperience toto a domaindomain inin wwhichhich theythey nono longerlonger appliedapplied bbecause,ecause, I wwillill ssuggest,uggest, socialsocial confoundersconfounders werewere notnot iincluded.ncluded. MoreMore oonn tthishis below.below.

SSensitivityensitivity AAnalysisnalysis aandnd SSensitivityensitivity CConversationsonversations aarere WWhathat WeWe NeedNeed

I thusthus standstand bbyy tthehe vviewiew iinn mmyy 11983983 essayessay thatthat econometriceconometric theorytheory promisespromises mmoreore thanthan itit cancan deliver,deliver, becausebecause itit requiresrequires a completecomplete commitmentcommitment toto assump-assump- ttionsions tthathat aarere aactuallyctually oonlynly hhalf-heartedlyalf-heartedly mmaintained.aintained. TheThe oonlynly wwayay ttoo ccreatereate ccredibleredible inferencesinferences wwithith doubtfuldoubtful assumptionsassumptions isis toto performperform a ssensitivityensitivity aanalysisnalysis tthathat sseparateseparates tthehe ffragileragile inferencesinferences fromfrom tthehe ssturdyturdy oones:nes: tthosehose tthathat ddependepend ssubstantiallyubstantially onon thethe doubtfuldoubtful assumptionsassumptions andand thosethose thatthat dodo not.not. SinceSince I wrotewrote mmyy ““concon iinn eeconometrics”conometrics” challengechallenge mmuchuch progressprogress hashas beenbeen mademade inin economiceconomic ttheoryheory andand inin econometriceconometric theorytheory andand inin experimentalexperimental design,design, butbut therethere hashas bbeeneen llittleittle pprogressrogress ttechnicallyechnically oorr pprocedurallyrocedurally oonn tthishis ssubjectubject ooff sensitivitysensitivity anal-anal- yysesses inin econometrics.econometrics. MostMost aauthorsuthors sstilltill ssupportupport ttheirheir cconclusionsonclusions wwithith tthehe rresultsesults iimpliedmplied bbyy sseveraleveral mmodels,odels, andand theythey leaveleave tthehe rrestest ooff uuss wwonderingondering hhowow hhardard theythey hhadad toto workwork toto fi nndd theirtheir ffavoriteavorite ooutcomesutcomes aandnd hhowow ssureure wwee hhaveave toto bebe aboutabout tthehe instrumentalinstrumental vvariablesariables assumptionsassumptions withwith accidentallyaccidentally randomizedrandomized treatmentstreatments aandnd aaboutbout thethe extentextent ofof thethe experimentalexperimental bbiasias wwithith ppurposefullyurposefully randomizedrandomized ttreatments.reatments. It’sIt’s likelike a courtcourt ofof lawlaw inin whichwhich wewe hearhear onlyonly thethe expertsexperts onon thethe plain-plain- ttiff’siff’s side,side, bbutut aarere wwiseise eenoughnough ttoo kknownow tthathat ttherehere aarere aabundantbundant aargumentsrguments fforor tthehe ddefense.efense. I havehave beenbeen makingmaking tthishis ppointoint iinn tthehe eeconometricsconometrics sspherephere ssinceince beforebefore I wrotewrote Specifi cation Searches: Ad Hoc Inference with Nonexperimental Data iinn 11978.978.1 TThathat bookbook wwasas sstimulatedtimulated byby mymy observationobservation ofof economistseconomists atat workwork whowho routinelyroutinely passpass theirtheir ddataata tthroughhrough thethe fi lterslters ofof mmanyany mmodelsodels aandnd tthenhen choosechoose a fewfew resultsresults forfor reportingreporting ppurposes.urposes. TheThe rangerange ooff mmodelsodels eeconomistsconomists aarere wwillingilling ttoo eexplorexplore ccreatesreates aambi-mbi- gguityuity iinn tthehe iinferencesnferences tthathat ccanan pproperlyroperly bbee ddrawnrawn ffromrom oourur ddata,ata, aandnd I hhaveave bbeeneen rrecommendingecommending mathematicalmathematical methodsmethods ofof ssensitivityensitivity aanalysisnalysis tthathat aarere iintendedntended toto ddetermineetermine thethe limitslimits ofof thatthat ambiguity.ambiguity.

1 Parenthetically, if you are alert, you might have been unsettled by the use of the word “with” in my title: Ad Hoc Inference with Nonexperimental Data, since inferences are made with tools but from data. That is my very subtle way of suggesting that knowledge is created by an interactive exploratory process, quite unlike the preprogrammed estimation dictated by traditional econometric theory. Edward E. Leamer 37

TThehe llanguageanguage tthathat I uusedsed ttoo mmakeake tthehe casecase fforor sensitivitysensitivity analysisanalysis seemsseems notnot toto hhaveave ppenetratedenetrated thethe consciousnessconsciousness ofof economists.economists. WWhathat I ccalledalled ““extremeextreme boundsbounds aanalysis”nalysis” inin mymy 19831983 essayessay isis a simplesimple exampleexample thatthat isis thethe best-knownbest-known aapproach,pproach, tthoughhough ppoorlyoorly uunderstoodnderstood aandnd inappropriatelyinappropriately applied.applied. EExtremextreme bboundsounds aanalysisnalysis iiss notnot anan “ad“ad hochoc butbut intuitiveintuitive approach,”approach,” asas describeddescribed byby AngristAngrist andand Pischke.Pischke. IItt isis a solutionsolution toto a clearlyclearly andand preciselyprecisely defidefi nedned sensitivitysensitivity qquestion,uestion, wwhichhich isis toto ddetermineetermine thethe rangerange ooff eestimatesstimates tthathat tthehe ddataata ccouldould ssupportupport ggiveniven a ppreciselyrecisely ddefiefi nedned rangerange ooff aassumptionsssumptions aaboutbout thethe priorprior distribution.distribution. IIt’st’s a ccorrespondenceorrespondence bbetweenetween thethe assumptionassumption spacespace andand thethe estimationestimation space.space. Incidentally,Incidentally, ifif youyou couldcould sseeee thethe wisdomwisdom iinn fi ndingnding thethe rangerange ooff eestimatesstimates tthathat tthehe ddataata aallow,llow, I wwouldould wworkork ttoo provideprovide ttoolsools thatthat identifyidentify thethe rangerange ofof t--values,values, a mmoreore iimportantmportant mmeasureeasure ofof tthehe fragilityfragility ofof tthehe inferences.inferences. A priorprior distributiondistribution iiss a fforeignoreign cconceptoncept fforor mmostost eeconomists,conomists, aandnd I ttriedried ttoo ccreatereate a bridgebridge betweenbetween thethe logiclogic ofof thethe analysisanalysis andand itsits applicationapplication byby expressingexpressing tthehe boundsbounds inin languagelanguage thatthat mostmost economistseconomists couldcould understand.understand. HereHere itit is:is: IIncludenclude iinn tthehe eequationquation thethe ttreatmentreatment variablevariable andand a singlesingle llinearinear ccombina-ombina- ttionion ooff tthehe aadditivedditive ccontrols.ontrols. TThenhen fi nndd tthehe linearlinear combinationcombination ofof controlscontrols thatthat pprovidesrovides tthehe ggreatestreatest eestimatedstimated ttreatmentreatment eeffectffect andand thethe linearlinear combinationcombination thatthat pprovidesrovides tthehe ssmallestmallest eestimatedstimated ttreatmentreatment eeffect.ffect. ThatThat correspondscorresponds toto thethe rangerange ooff eestimatesstimates tthathat ccanan bbee oobtainedbtained wwhenhen itit isis kknownnown tthathat tthehe ccontrolsontrols aarere ddoubtfuloubtful ((zerozero bbeingeing tthehe mostmost llikelyikely eestimate,stimate, a ppriori)riori) bbutut ttherehere iiss ccompleteomplete aambiguitymbiguity aaboutbout thethe probableprobable importanceimportance ofof thethe vvariables,ariables, arbitraryarbitrary scales,scales, andand arbitraryarbitrary ccoordinateoordinate systems.systems. WWhathat isis adad hochoc areare thethe follow-onfollow-on methods,methods, forfor example,example, ccomputingomputing thethe standardstandard errorserrors ofof thethe boundsbounds oror reportingreporting a distributiondistribution ofof esti-esti- mmatesates asas inin Sala-i-MartinSala-i-Martin (1997).(1997). A cultureculture thatthat insistsinsists onon statisticallystatistically ssignifiignifi ccantant eestimatesstimates isis notnot naturallynaturally rreceptiveeceptive toto anotheranother reasonreason ourour datadata areare uninformativeuninformative (too(too muchmuch dependencedependence onon aarbitraryrbitrary assumptions).assumptions). OneOne reasonreason thesethese methodsmethods areare rarelyrarely usedused isis theirtheir honestyhonesty sseemseems ddestructive;estructive; oor,r, toto putput iitt anotheranother way,way, a fanaticalfanatical commitmentcommitment toto fancifulfanciful fformalormal mmodelsodels iiss ooftenften nneededeeded ttoo ccreatereate tthehe aappearanceppearance ooff pprogress.rogress. ButBut wewe needneed ttoo changechange tthehe ccultureulture aandnd rregardegard thethe fi nndingding ooff ““nono persuasivepersuasive evidenceevidence iinn tthesehese ddata”ata” onon thethe samesame footingfooting asas a “statistically“statistically ssignifiignifi ccantant andand sturdysturdy eestimate.”stimate.” KeepKeep iinn mindmind thatthat tthehe ssensitivityensitivity ccorrespondenceorrespondence bbetweenetween aassumptionsssumptions andand inferencesinferences ccanan gogo iinn eeitherither ddirection.irection. WWee ccanan aasksk wwhathat setset ofof inferencesinferences correspondscorresponds toto a pparticulararticular ssetet ooff aassumptions,ssumptions, bbutut wwee ccanan aalsolso aasksk wwhathat aassumptionsssumptions aarere nneededeeded ttoo supportsupport a hoped-forhoped-for inference.inference. ThatThat isis exactlyexactly whatwhat anan economiceconomic theoremtheorem does.does. TThehe iintellectualntellectual valuevalue ofof thethe FFactoractor PPricerice EEqualizationqualization TTheoremheorem doesdoes notnot dderiveerive ffromrom iitsts ttruthfulness;ruthfulness; iit’st’s vvaluealue ccomesomes ffromrom tthehe ffactact tthathat iitt pprovidesrovides a mminimalinimal ssetet ooff aassumptionsssumptions thatthat implyimply FFactoractor PricePrice Equalization,Equalization, thusthus focusingfocusing attentionattention oonn why ffactoractor pricesprices areare notnot eequalized.qualized. TToo tthosehose ooff yyouou wwhoho dodo datadata analysis,analysis, I tthushus pposeose ttwowo qquestionsuestions tthathat I tthinkhink eeveryvery empiricalempirical eenterprisenterprise shouldshould bbee aableble ttoo aanswer:nswer: WWhathat ffeatureeature ofof thethe datadata leadsleads ttoo thatthat cconclusion?onclusion? WhatWhat ssetet ooff aassumptionsssumptions iiss eessentialssential ttoo ssupportupport thatthat iinference?nference? 38 Journal of Economic Perspectives

TThehe TroublesTroubles oonn WWallall SStreettreet aandnd TThree-Valuedhree-Valued LogicLogic

WWithith tthehe ashesashes ofof thethe mathematicalmathematical modelsmodels usedused toto raterate mortgage-backedmortgage-backed ssecuritiesecurities stillstill ssmolderingmoldering oonn WWallall SStreet,treet, nownow isis anan idealideal ttimeime ttoo rrevisitevisit thethe sensi-sensi- ttivityivity issues.issues. JustinJustin FFoxox ((2009)2009) ssuggestsuggests inin hishis titletitle The Myth of the Rational Market, A History of Risk, Reward, and Delusion on Wall Street tthathat wewe havehave beenbeen makingmaking a mmodelingodeling eerrorrror aandnd tthathat tthehe pproblemsroblems llieie iinn tthehe aassumptionssumption ooff rrationalational aactors,ctors, wwhichhich presumablypresumably cancan bebe remediedremedied byby business-as-usualbusiness-as-usual afterafter addingadding a “behav-“behav- iioral”oral” variablevariable oror twotwo intointo thethe model.model. I thinkthink thethe rootsroots ofof thethe problemproblem areare deeper,deeper, ccallingalling forfor a changechange iinn tthehe wwayay wwee ddoo bbusinessusiness andand callingcalling forfor a bookbook thatthat mightmight bbee ttitled:itled: The Myth of the Data Generating Process: A History of Delusion in Academia. RRationalityationality ofof fi nnancialancial marketsmarkets isis a prettypretty straightforwardstraightforward consequenceconsequence ofof tthehe aassumptionssumption thatthat fi nnancialancial rreturnseturns aarere ddrawnrawn ffromrom a ““datadata ggeneratingenerating pprocess”rocess” wwhosehose propertiesproperties areare apparentapparent toto experiencedexperienced investorsinvestors andand econometricians,econometricians, aafterfter studyingstudying thethe historicalhistorical data.data. IfIf wewe don’tdon’t knowknow thethe datadata generatinggenerating process,process, tthenhen thethe eeffiffi ccientient mmarketsarkets eedifidifi ccee ffallsalls apart.apart. EvenEven simple-mindedsimple-minded fi nancenance ideas,ideas, llikeike thethe benefibenefi tsts ffromrom ddiversifiiversifi ccation,ation, bbecomeecome ssuspectuspect iiff wwee ccannotannot rreliablyeliably aassessssess ppredictiveredictive mmeans,eans, variances,variances, andand covariances.covariances. BButut itit isn’tisn’t justjust fi nnanceance tthathat rrestsests oonn tthishis mmythyth ooff a ddata-generatingata-generating pprocess.rocess. IItt isis thethe wwholehole eedifidifi ccee ooff empiricalempirical eeconomics.conomics. LLet’set’s ffaceace iit.t. TThehe evolving,evolving, iinno-nno- vvating,ating, sself-organizing,elf-organizing, sself-healingelf-healing hhumanuman ssystemystem wwee ccallall tthehe eeconomyconomy iiss nnotot wwellell ddescribedescribed bbyy a fi ctionalctional “data-generating“data-generating pprocess.”rocess.” TThehe ppointoint ofof thethe sensitivitysensitivity aanalysesnalyses tthathat I hhaveave bbeeneen aadvocatingdvocating bbeginsegins withwith thethe admissionadmission thatthat thethe historicalhistorical ddataata areare compatiblecompatible withwith countlesscountless alternativealternative data-generatingdata-generating models.models. IfIf therethere isis oone,ne, tthehe bestbest wwee ccanan ddoo iiss ttoo ggetet cclose;lose; wwee aarere nneverever ggoingoing ttoo kknownow iit.t. CConfrontedonfronted wwithith oourur ccollectiveollective colossalcolossal failurefailure toto anticipateanticipate thethe problemsproblems wwithith mmortgage-backedortgage-backed securities,securities, wewe economistseconomists havehave beenbeen stampedingstampeding shame-shame- llesslyessly bbackack ttoo KKeynesianeynesian thinkingthinking aaboutbout mmacroeconomics,acroeconomics, scurryingscurrying ttoo rrereaderead KKeynes’eynes’ (1936)(1936) General Theory of Employment, Interest and Money. WWee wouldwould ddoo wwellell ttoo gogo bbackack a littlelittle furtherfurther inin timetime toto 19211921 whenwhen bothboth Keynes’Keynes’ Treatise on Probability aandnd FrankFrank Knight’sKnight’s Risk, Uncertainty and Profi t wwereere ppublished.ublished. BBothoth ooff tthesehese bbooksooks aarere aboutabout thethe mythmyth ofof thethe datadata generatinggenerating process.process. BothBoth dealdeal withwith thethe llimitsimits ofof tthehe eexpectedxpected utilityutility maximizationmaximization paradigm.paradigm. BothBoth serveserve asas foundationsfoundations fforor thethe argumentsarguments iinn ffavoravor ooff ssensitivityensitivity aanalysisnalysis iinn mmyy ““con”con” ppaper.aper. BBothoth aarere aaboutbout ““three-valuedthree-valued logic.”logic.” HHereere iiss hhowow tthree-valuedhree-valued llogicogic wworks.orks. SSupposeuppose yyouou ccanan cconfionfi dentlydently ddetermineetermine tthathat itit isis a goodgood ideaidea toto bringbring youryour umbrellaumbrella ifif thethe chancechance ofof rainrain isis 1010 percentpercent oror hhigher,igher, butbut carryingcarrying thethe umbrellaumbrella involvesinvolves moremore costcost thanthan expectedexpected benefibenefi t ifif tthehe cchancehance ofof rrainain isis less.less. WhenWhen thethe datadata pointpoint clearlyclearly toto a probabilityprobability eithereither inin excessexcess ofof 1100 percentpercent oror lessless thanthan 1010 percent,percent, thenthen wewe areare inin a worldworld ofof KnightianKnightian riskrisk inin whichwhich tthehe ddecisionecision cancan bebe basedbased onon expectedexpected utilityutility mmaximization.aximization. BButut supposesuppose therethere isis oonene mmodelodel tthathat ssuggestsuggests thethe probabilityprobability isis 1515 percentpercent whilewhile anotheranother equallyequally goodgood mmodelodel suggestssuggests thethe probabilityprobability isis 5 percent.percent. ThenThen wewe areare inin a worldworld ofof KnightianKnightian Tantalus on the Road to Asymptopia 39

uuncertaintyncertainty inin whichwhich expectedexpected utilityutility maximizationmaximization doesn’tdoesn’t produceproduce a decision.decision. WWhenhen anyany nnumberumber betweenbetween 0.050.05 andand 0.150.15 isis anan equallyequally goodgood2 aassessmentssessment ofof thethe cchancehance ofof rrainain wewe areare dealingdealing withwith epistemicepistemic pprobabilitiesrobabilities thatthat areare notnot numbersnumbers butbut iintervals,ntervals, iinn tthehe sspiritpirit ooff KKeyneseynes ((1921).1921). WWhilehile tthehe ddecisionecision iiss ttwo-valued—youwo-valued—you eithereither ttakeake youryour umbrellaumbrella oorr yyouou ddon’t—theon’t—the sstatetate ooff mmindind iiss tthree-valued:hree-valued: yesyes (take(take thethe uumbrella),mbrella), nnoo ((leaveleave iitt bbehind),ehind), oorr I ddon’ton’t kknow.now. TThehe pointpoint ofof aadoptingdopting tthree-valuedhree-valued llogicogic llikeike KKeyneseynes aandnd KKnightnight iiss ttoo eencouragencourage uuss ttoo tthinkhink cclearlylearly aaboutbout tthehe llimitsimits ooff oourur knowledgeknowledge andand thethe limitslimits ofof thethe expectedexpected utilityutility maximizationmaximization paradigm.paradigm. WithWith aallll ooff tthehe ffocusocus oonn ddecisionecision mmakingaking wwithith ttwo-valuedwo-valued llogic,ogic, tthehe pprofessionrofession hhasas ddoneone ppreciousrecious littlelittle workwork onon decisiondecision underunder ambiguityambiguity andand three-valuedthree-valued logic.logic.3 IInn otherother wwords,ords, whenwhen I askedasked usus toto “take“take thethe concon outout ofof eeconometrics”conometrics” I wwasas onlyonly sayingsaying tthehe oobvious:bvious: IIff thethe rangerange ooff iinferencesnferences tthathat ccanan rreasonablyeasonably bebe ssupportedupported byby thethe datadata wewe havehave isis tootoo widewide toto pointpoint toto oneone aandnd oonlynly oonene ddeci-eci- ssion,ion, wewe needneed toto admitadmit thatthat thethe datadata leaveleave uuss cconfused.onfused. ThusThus mmyy ccontributionontribution ttoo eeconometricsconometrics hhasas bbeeneen cconfusion!onfusion! You,You, though,though, refuserefuse toto admitadmit thatthat youyou areare justjust aass confusedconfused aass II.. TThree-valuedhree-valued logiclogic seemsseems especiallyespecially pertinentpertinent whenwhen extendingextending thethe resultsresults ofof eexperimentsxperiments iintonto ootherther ddomains—thereomains—there iiss a llotot ooff ““wewe ddon’ton’t kknow”now” there.there. EvenEven inin tthehe casecase ofof mechanicalmechanical systems,systems, withwith statisticalstatistical propertiesproperties muchmuch betterbetter understoodunderstood tthanhan eeconomics/ficonomics/fi nancialnancial humanhuman systems,systems, iitt iiss nnotot assumedassumed thatthat modelsmodels willwill workwork uundernder eextremextreme cconditions.onditions. EngineersEngineers maymay designdesign aircraftaircraft thatthat accordingaccording toto theirtheir ccomputeromputer modelsmodels cancan fl yy,, butbut uuntilntil rrealeal airplanesairplanes areare actuallyactually testedtested inin normalnormal aandnd stressfulstressful cconditions,onditions, thethe aircraftaircraft areare notnot certificertifi eded toto carrycarry passengers.passengers. IIndeed,ndeed, BBoeing’soeing’s ccompositeomposite pplasticlastic 778787 DDreamlinerreamliner hhasas bbeeneen ssufferinguffering pproductionroduction ddelayselays bbecauseecause wingwing damagedamage hhasas shownshown uupp ““whenwhen tthehe stressstress oonn tthehe wingswings wwasas wellwell belowbelow tthehe loadload thethe wingswings mustmust bbearear ttoo bbee ffederallyederally certificertifi eded toto carrycarry passengers”passengers” (Gates,(Gates, 22009).009). TooToo badbad wewe couldn’tcouldn’t havehave stress-testedstress-tested tthosehose mmortgage-backedortgage-backed ssecuritiesecurities bbeforeefore wewe startedstarted fl yyinging aaroundround tthehe wworldorld wwithith tthem.hem. TToooo bbadad tthehe rratingating aagen-gen- cciesies ddidid nnotot useuse three-valuedthree-valued logiclogic withwith anotheranother bondbond rating:rating: “AAA-H,”“AAA-H,” meaningmeaning hhypotheticallyypothetically AAAAAA accordingaccording toto a modelmodel butbut notnot yetyet certificertifi eded aass rrecession-proof.ecession-proof. TTherehere iiss ooff ccourseourse mmoreore tthanhan oonene rreasoneason whywhy mortgage-backedmortgage-backed securitiessecurities ddidn’tidn’t fl y wwhenhen tthehe wweathereather ggotot rrough,ough, bbutut iitt wwillill ssuituit mmyy ppurposesurposes hhereere iiff I mmakeake tthehe aargumentrgument thatthat itit waswas anan inappropriateinappropriate extrapolationextrapolation ofof datadata fromfrom oneone acci-acci- ddentalental eexperimentxperiment ttoo a ddifferentifferent ssettingetting tthathat iiss aatt tthehe rrootoot ooff tthehe pproblem.roblem. ToTo makemake tthishis argument,argument, iitt isis enoughenough toto looklook atat thethe datadata inin LosLos Angeles.Angeles. Don’tDon’t expectexpect a fullfull eeconometricconometric housinghousing model.model. It’sIt’s onlyonly a provocativeprovocative illustration.illustration. FFigureigure 1 contrastscontrasts nnonfarmonfarm ppayrollsayrolls iinn LLosos AAngelesngeles aandnd iinn tthehe UUnitednited StatesStates ooverallverall duringduring tthehe llastast tthreehree rrecessions.ecessions. InIn 20012001 andand inin 2008,2008, thethe L.A.L.A. jobjob marketmarket

2 Please don’t think I am assigning a uniform distribution over this interval, since if that were the case, the probability would be precisely equal to the mean: 0.10. 3 Bewley (1986), Klibanoff, Marinacci, and Mukerji (2005), and Hanany and Klibanoff (2009) and references therein are exceptions. 40 Journal of Economic Perspectives

Figure 1 Payroll Jobs in Los Angeles and U.S. Overall (recessions shaded)

28 24 20 16 12 U.S. overall 8 4 0 Los Angeles −4 −8 −12 Dec. 1993 trough Percent change in jobs (relative to Jan. 1990) 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008

N o te: The fi gure shows the change in number of employees on nonfarm payrolls, in the Los Angeles area and in the U.S. overall, relative to January 1990, seasonally adjusted. The Los Angeles area is the Los Angeles–Long Beach–Santa Ana, California, Metropolitan Statistical Area. ddeclinedeclined inin parallelparallel withwith thethe U.S.U.S. decline,decline, butbut thethe recessionrecession ofof 1990–911990–91 waswas espe-espe- cciallyially ssevereevere inin LosLos Angeles.Angeles. UU.S..S. ppayrollayroll jjobsobs ddroppedropped bbyy 11.5.5 ppercentercent duringduring tthishis rrecession,ecession, wwhilehile jobsjobs iinn tthehe LosLos AngelesAngeles mmetropolitanetropolitan sstatisticaltatistical aarearea ddeclinedeclined byby 1111 percentpercent aandnd diddid notnot returnreturn toto theirtheir 11990990 llevelsevels untiluntil 1999.1999. ThisThis waswas a naturalnatural eexperimentxperiment kknownnown aass tthehe eendnd ooff tthehe CColdold WWar,ar, wwithith LLosos AAngelesngeles ttreatedreated aandnd wwith,ith, fforor example,example, SSanan FFranciscorancisco iinn tthehe ccontrolontrol ggroup.roup. ((TheThe 22001001 rrecessionecession hadhad thethe treat-treat- mmentent reversed,reversed, withwith SanSan FranciscoFrancisco treatedtreated toto a techtech bustbust butbut LosLos AngelesAngeles inin thethe ccontrolontrol group.)group.) TThus,hus, I suggest,suggest, thethe L.A.L.A. datadata inin thethe earlyearly 1990s1990s isis a testtest ccasease ooff tthehe eeffectffect ooff a severesevere rrecessionecession onon mortgagemortgage defaults.defaults. FigureFigure 2 illustratesillustrates thethe numbernumber ofof homeshomes ssoldold aandnd tthehe mmedianedian ppricesrices iinn LLosos AAngelesngeles ffromrom JJanuaryanuary 11985985 ttoo AAugustugust 22009.009. TThehe hhorizontalorizontal lineline segmentssegments iindicatendicate tthehe pperioderiod ooverver wwhichhich aallll hhomesomes ppurchasedurchased wwithith a loanloan toto valuevalue ratioratio ofof 9090 percentpercent (indicating(indicating a 1010 percentpercent downpayment)downpayment) andand iinterest-onlynterest-only ppaymentsayments wwillill bbee uunderwaternderwater aatt tthehe nnextext trough.trough. IfIf allall thesethese homeshomes wwereere rreturnedeturned toto thethe banksbanks underunder thethe worstworst casecase scenarioscenario whenwhen thethe priceprice hitshits bbottom,ottom, tthehe bbankank llossesosses aarere 9900 ppercentercent ooff tthehe ggapap bbetweenetween tthathat llineine ssegmentegment aandnd tthehe priceprice atat origination.origination. CClearlylearly tthehe uunderwaternderwater pproblemroblem iiss mmuchuch mmoreore iintensentense iinn tthehe latestlatest ddownturnownturn tthanhan iitt wwasas iinn tthehe 11990s.990s. IInn thethe fi rrstst eepisode,pisode, volumevolume peakedpeaked muchmuch beforebefore prices,prices, fallingfalling byby 5050 percentpercent iinn 2828 months.months. ThoughThough volumevolume waswas crashing,crashing, thethe medianmedian priceprice continuedcontinued toto iincrease,ncrease, peakingpeaking inin MayMay 1991,1991, 101101 percentpercent aboveabove itsits valuevalue atat thethe startstart ofof thesethese ddataata iinn JanuaryJanuary 11985.985. TThenhen ccommencedommenced a sslowlow ppricerice ddeclineecline wwithith a ttroughrough iinn Edward E. Leamer 41

Figure 2 Los Angeles Housing Market: Median Home Price and Sales Volume

700,000 500,000 400,000 Median price 300,000 (right axis) 6,000 Median price ($) 5,000 200,000 4,000 Sales volume

Sales volume 100,000 3,000 (left axis)

2,000

1985198619871989198819901991199219931994 19951996199719981999200020012002 20032004 200520062007 20082009

Line segments identify underwater homes at cycle troughs assuming 90% loan to value

Source: California Association of Realtors. Note: The median price and sales volume are a 3-month moving average, seasonally adjusted. The line segments indentify homes that will be underwater at the coming cycle trough assuming 90 percent loan to value at origination and interest only payments. (Denoting the price at time t by pt and the price at the trough by pmin, with 10 percent down, the loan balance is 0.9 pt , and the home is underwater at the trough if 0.9 pt > pmin or if pt > pmin /0.9.)

DDecemberecember 11996,996, withwith thethe medianmedian priceprice 2929 percentpercent belowbelow itsits previousprevious peak,peak, a ddeclineecline ofof 5 percentpercent perper year.year. I hhaveave bbeeneen fondfond ofof summarizingsummarizing thesethese datadata byby sayingsaying thatthat forfor homes,homes, it’sit’s a vvolumeolume cycle,cycle, notnot a priceprice cycle.cycle. ThisThis veryvery slowslow price-discoveryprice-discovery occursoccurs becausebecause ppeopleeople celebratecelebrate investmentinvestment gains,gains, butbut denydeny losses.losses. Owner-occupantsOwner-occupants ofof homeshomes ccanan llikewiseikewise hholdold oontonto long-agolong-ago vvaluationsaluations aandnd iinsistnsist oonn ppricesrices tthathat tthehe mmarketarket ccannotannot support.support. BecauseBecause ofof thatthat denial,denial, therethere aarere mmanyany fewerfewer transactions,transactions, andand tthehe ttransactionsransactions thatthat dodo occuroccur tendtend toto bebe aatt tthehe sseller’seller’s pprices,rices, nnotot eequilibriumquilibrium mmarketarket pprices.rices. TThehe slowslow ppricerice ddiscoveryiscovery actsacts likelike a time-out,time-out, allowingallowing thethe fundamentalsfundamentals toto ccatchatch uupp ttoo vvaluationsaluations aandnd keepingkeeping fforeclosureoreclosure rratesates aatt mminimalinimal llevels.evels.4 InIn thethe eearlyarly phasephase ofof thethe currentcurrent housinghousing correction,correction, historyhistory seemedseemed toto bebe repeatingrepeating iitself,tself, sincesince volumevolume waswas droppingdropping rapidlyrapidly eevenven aass ppricesrices ccontinuedontinued ttoo rrise.ise. BButut thenthen bbeganegan a rapidrapid 5511 ppercentercent ddroprop iinn hhomeome ppricesrices bbetweenetween AAugustugust 20072007 toto MarchMarch 2009,2009, ccreatingreating a hugehuge amountamount ofof underwaterunderwater valuation.valuation.

4 The fi rst underwater problem illustrated in Figure 2 could be almost completely remedied by a switch from 90 to 80 percent loan-to-value ratios starting when volumes began to drop, because most of the underwater loans came after that break in the market. 42 Journal of Economic Perspectives

Figure 3 Southern California Foreclosures

20,000

18,000 Inland Empire 16,000 Los Angeles 14,000 12,000 10,000 8,000 6,000 4,000 2,000 0 1987 1989 1991 1993 1995 199719992001 2003 2005 2007 2009

Source: MDA DataQuick. Note: The concentration of foreclosures in time and space in Southern California in the latest housing correction is illustrated in Figure 3, which displays quarterly foreclosures since 1987 for both Los Angeles County and “The Inland Empire,” composed of the two “peripheral” counties east of Los Angeles County (Riverside and San Bernadino), where subprime lending was very prevalent. In both Los Angeles and the Inland Empire in the 1990s, the rise in foreclosures trailed the price movement by several years. In contrast, the time-concentrated spike in foreclosures in 2007 occurred at the very start of the price erosion and has been much more extreme in the Inland Empire than in Los Angeles.

WWhyhy diddid thethe LL.A..A. 1990s1990s ddataata mmisleadislead wwithith rregardegard toto thethe currentcurrent housinghousing ccorrection?orrection? OneOne possiblepossible answeranswer isis untesteduntested socialsocial effectseffects andand unmeasuredunmeasured subjectsubject eeffects—twoffects—two veryvery importantimportant interactiveinteractive confounders.confounders. InnovationsInnovations inin mortgagemortgage ooriginationrigination iinn 22003–2005003–2005 extendedextended thethe home-ownershiphome-ownership peripheriesperipheries ofof ourour citiescities bbothoth inin termsterms ooff iincomencome andand location.location. WhenWhen thethe subprimesubprime mortgagemortgage windowwindow shutshut ddown,own, tthehe demanddemand forfor owner-occupiedowner-occupied homeshomes atat thethe extendedextended peripheriesperipheries waswas eeliminatedliminated virtuallyvirtually overnight.overnight. SinceSince thesethese propertiesproperties werewere fi nnancedanced withwith loansloans tthathat ccouldould oonlynly wworkork iiff tthehe hhousesouses ppaidaid fforor tthemselveshemselves vviaia aappreciation,ppreciation, tthehe bbanksanks bbecameecame tthehe newnew oowners.wners. AccountingAccounting rrulesules ddoo nnotot aallowllow banksbanks ttoo ddenyeny llossesosses tthehe wwayay oowner-occupantswner-occupants do,do, andand bbanksanks iimmediatelymmediately dumpeddumped thethe foreclosedforeclosed homeshomes ontoonto tthehe marketmarket atat tthehe wworstorst ttime,ime, inin tthehe worstworst way,way, withwith brokenbroken wwindowsindows aandnd bburned-upurned-up llawns,awns, cconcentratedoncentrated inin timetime andand space,space, causingcausing veryvery rapidrapid (or(or eveneven exaggerated)exaggerated) ppricerice ddiscoveryiscovery inin thethe aaffectedffected peripheries.peripheries. InIn contrast,contrast, fforeclosuresoreclosures inin thethe 1990s,1990s, iillustratedllustrated inin FigureFigure 3 forfor thethe casecase ofof SouthernSouthern California,California, werewere delayed,delayed, andand werewere ddispersedispersed inin timetime andand space.space. WWee aallll needneed toto learnlearn bothboth narrowernarrower andand broaderbroader lessonslessons here.here. I expectedexpected priceprice ddiscoveryiscovery inin housinghousing marketsmarkets toto bebe slow,slow, asas itit hadhad beenbeen afterafter thethe burstingbursting ofof previousprevious bbubbles,ubbles, aandnd I wwasas ccompletelyompletely wwrong.rong. I wwillill bbee mmoreore ccarefulareful aaboutbout interactiveinteractive socialsocial cconfoundersonfounders inin thethe future.future. Tantalus on the Road to Asymptopia 43

WWhite-Washinghite-Washing

IItt sshouldhould notnot bbee a ssurpriseurprise aatt tthishis ppointoint iinn tthishis eessayssay tthathat I ppartart wwaysays wwithith AAngristngrist andand PischkePischke inin theirtheir aapparentpparent eendorsementndorsement ooff WWhite’shite’s ((1980)1980) ppaperaper oonn howhow ttoo ccalculatealculate robustrobust sstandardtandard errors.errors. AngristAngrist andand PischkePischke write:write: “Robust“Robust standardstandard eerrors,rrors, aautomatedutomated cclustering,lustering, aandnd llargerarger ssamplesamples hhaveave aalsolso ttakenaken tthehe ssteamteam ooutut ooff iissuesssues llikeike hheteroskedasticityeteroskedasticity aandnd sserialerial ccorrelation.orrelation. A llegacyegacy ooff WWhite’shite’s ((1980)1980) ppaperaper onon robustrobust sstandardtandard eerrors,rrors, oneone ofof thethe mostmost hhighlyighly ccitedited fromfrom tthehe pperiod,eriod, iiss tthehe nnear-deathear-death ofof generalizedgeneralized lleasteast squaressquares inin cross-sectionalcross-sectional appliedapplied work.”work.” AAnn earlierearlier generationgeneration ofof econometricianseconometricians correctedcorrected thethe heteroskedasticityheteroskedasticity pproblemsroblems withwith weightedweighted leastleast squaressquares usingusing weightsweights suggestedsuggested byby anan explicitexplicit hetero-hetero- sskedasticitykedasticity model.model. TheseThese earlierearlier econometricianseconometricians understoodunderstood thatthat reweightingreweighting thethe oobservationsbservations ccanan hhaveave ddramaticramatic eeffectsffects onon thethe actualactual estimates,estimates, butbut ttheyhey ttreatedreated tthehe effecteffect onon thethe standardstandard errorserrors asas a secondarysecondary matter.matter. A “robust“robust standard”standard” errorerror ccompletelyompletely turnsturns thisthis around,around, leavingleaving thethe estimatesestimates thethe samesame butbut changingchanging thethe ssizeize ofof tthehe cconfionfi dencedence interval.interval. WhyWhy shouldshould oonene wworryorry aaboutbout thethe lengthlength ooff thethe cconfionfi ddenceence interval,interval, butbut notnot thethe location?location? ThisThis mistakenmistaken adviceadvice reliesrelies onon asymp-asymp- ttoticotic propertiesproperties ofof eestimators.stimators.5 I callcall itit “White-washing.”“White-washing.” BBestest toto rememberremember thatthat nnoo mmatteratter hhowow ffarar wewe travel,travel, wwee rremainemain aalwayslways iinn tthehe LLandand ooff tthehe FFiniteinite SSample,ample, iinfinfi nitelynitely farfar fromfrom AAsymptopia.symptopia. RRatherather tthanhan mmathematicalathematical musingsmusings aboutabout lifelife inin AAsymptopia,symptopia, wwee sshouldhould bbee ddoingoing tthehe hhardard wworkork ooff mmodelingodeling tthehe hheteroskedasticityeteroskedasticity aandnd thethe timetime dependencedependence ttoo ddetermineetermine iiff sensiblesensible rreweightingeweighting ofof tthehe observationsobservations mmateriallyaterially changeschanges tthehe llocationsocations ofof tthehe estimatesestimates ooff iinterestnterest aass wwellell aass tthehe wwidthsidths ooff tthehe cconfionfi dencedence intervals.intervals. EEstimationstimation wwithith iinstrumentalnstrumental vvariablesariables iiss aanothernother ccasease ooff iinappropriatenappropriate reli-reli- aancence onon asymptoticasymptotic properties.properties. InIn fi nitenite samples,samples, thesethese estimatorsestimators cancan seriouslyseriously _ _ ddistortistort tthehe eevidencevidence fforor tthehe ssameame rreasoneason tthathat tthehe rratioatio ofof twotwo samplesample means,means, y / x , isis a ppooroor ssummaryummary ofof thethe datadata evidenceevidence aboutabout thethe ratioratio ofof thethe means,means, E(E(y))/E(/E(x)),, wwhenhen _ tthehe fi nniteite samplesample lleaveseaves a ““dividing-by-zero-problem”dividing-by-zero-problem” bbecauseecause tthehe ddenominatorenominator x iiss nnotot statisticallystatistically ffarar fromfrom zzero.ero. ThisThis problemproblem isis greatlygreatly amplifiamplifi eedd wwithith multiplemultiple wweakeak instruments,instruments, a ssituationituation thatthat isis quitequite common.common. ItIt isis actuallyactually quitequite feasiblefeasible ffromrom a BBayesianayesian pperspectiveerspective ttoo pprogramrogram anan alertalert intointo oourur iinstrumentalnstrumental vvariablesariables eestimationstimation togethertogether withwith rremedies,emedies, thoughthough tthishis ddependsepends oonn AAssumptions.ssumptions. InIn ootherther wwords,ords, yyouou hhaveave toto ddoo ssomeome hhardard thinkingthinking ttoo uusese iinstrumentalnstrumental vvariablesariables mmethodsethods iinn fi nniteite ssamples.amples. AAss thethe samplesample ssizeize grows,grows, concernconcern shouldshould shiftshift fromfrom small-samplesmall-sample biasbias toto aasymptoticsymptotic bbiasias causedcaused byby thethe failurefailure ofof tthehe aassumptionsssumptions neededneeded toto makemake instru-instru- mmentalental vvariablesariables work.work. SinceSince itit iiss tthehe uunnamed,nnamed, unobservedunobserved variablesvariables thatthat areare thethe ssourceource ooff tthehe pproblem,roblem, tthishis iisn’tsn’t eeasyasy ttoo tthinkhink aabout.bout. TThehe ppercentageercentage bbiasias iiss ssmallmall iiff

5 The change in length but not location of a confi dence interval is appropriate for one specialized covariance structure. 44 Journal of Economic Perspectives

tthehe vvariablesariables youyou hhaveave fforgottenorgotten aarere uunimportantnimportant ccomparedompared withwith thethe variablesvariables thatthat yyouou havehave remembered.remembered. That’sThat’s easyeasy toto determine,determine, right?right?

A WWordord oonn MMacroeconomicsacroeconomics

FFinally,inally, I thinkthink thatthat AngristAngrist andand PischkePischke areare wayway tootoo optimisticoptimistic aboutabout thethe pprospectsrospects forfor anan experimentalexperimental approachapproach toto macroeconomics.macroeconomics. OurOur understandingunderstanding ooff ccausalausal eeffectsffects inin macroeconomicsmacroeconomics isis virtuallyvirtually nil,nil, aandnd wwillill rremainemain sso.o. DDon’ton’t wwee kknownow that?that? ThoughThough manymany membersmembers ofof ourour professionprofession havehave jumpedjumped upup toto supportsupport tthehe $787$787 billionbillion stimulusstimulus programprogram inin 20092009 asas ifif theythey knewknew thatthat waswas anan appropriateappropriate rresponseesponse ttoo tthehe PPanicanic ofof 22008,008, tthehe iintellectualntellectual basisbasis forfor thatthat opinionopinion isis veryvery thin,thin, eespeciallyspecially ifif youyou taketake a closeclose looklook atat howhow thatthat stimulusstimulus billbill waswas written.written. TThehe eeconomistsconomists wwhoho ccoinedoined thethe DDSGESGE acronymacronym combinedcombined inin threethree termsterms thethe tthingshings economistseconomists leastleast understand:understand: “dynamic,”“dynamic,” standingstanding forfor forward-lookingforward-looking deci-deci- ssionion making;making; “stochastic,”“stochastic,” standingstanding fforor ddecisionsecisions uundernder uuncertaintyncertainty aandnd aambiguity;mbiguity; aandnd “general“general equilibrium,”equilibrium,” standingstanding fforor tthehe ssocialocial pprocessrocess tthathat ccoordinatesoordinates aandnd iinflnfl uencesuences tthehe actionsactions ofof allall thethe players.players. I havehave triedtried toto makemake thisthis pointpoint inin thethe titletitle ooff mymy recentrecent book:book: Macroeconomic Patterns and Stories (Leamer,(Leamer, 22009).009). TThat’shat’s wwhathat wwee ddo.o. WeWe seekseek patternspatterns andand telltell stories.stories.

CConclusiononclusion

IIgnorancegnorance iiss a formidableformidable foe,foe, andand toto havehave hopehope ofof eveneven modestmodest victories,victories, wewe eeconomistsconomists nneedeed toto useuse everyevery resourceresource andand everyevery weaponweapon wewe cancan muster,muster, iincludingncluding tthoughthought experimentsexperiments ((theory),theory), andand thethe analysisanalysis ofof datadata fromfrom nnonexperiments,onexperiments, aaccidentalccidental experiments,experiments, aandnd ddesignedesigned experiments.experiments. WWee sshouldhould bbee ccelebratingelebrating thethe ssmallmall genuinegenuine victoriesvictories ofof thethe economistseconomists whowho useuse theirtheir toolstools mostmost eeffectively,ffectively, andand wwee shouldshould ddialial bbackack oourur aadorationdoration ooff tthosehose wwhoho ccanan ccarryarry tthehe bbiggestiggest andand brightestbrightest aandnd least-understoodleast-understood weapons.weapons. WeWe wouldwould benefibenefi t fromfrom ssomeome sseriouserious hhumility,umility, aandnd ffromrom bburningurning oourur ““MissionMission Accomplished”Accomplished” banners.banners. It’sIt’s nevernever gonnagonna happen.happen. PPartart ooff tthehe pproblemroblem iiss thatthat wewe datadata analystsanalysts wwantant iitt aallll aautomated.utomated. WWee wwantant aann aanswernswer aatt tthehe ppushush ofof a buttonbutton onon a keyboard.keyboard. WeWe knowknow intellectuallyintellectually thatthat tthought-hought- llessess choicechoice ofof anan instrumentinstrument ccanan bbee a ssevereevere pproblemroblem aandnd tthathat ssummarizingummarizing thethe ddataata withwith thethe “consistent”“consistent” instrumentalinstrumental vvariablesariables estimateestimate whenwhen thethe instrumentsinstruments aarere weakweak isis anan equallyequally llargearge eerror.rror.6 TThehe substantialsubstantial literatureliterature onon estimationestimation withwith wweakeak instrumentsinstruments hhasas nnotot yyetet producedproduced a seriousserious practicalpractical competitorcompetitor toto thethe usualusual

6 Bayesians have a straightforward solution in theory to this problem: describe the marginal likelihood function, marginalized with respect to all the parameters except the coeffi cient being estimated. If the instruments are strong, this marginal likelihood will have its mode near the instrumental variables estimate. If the instruments are weak, the central tendency of the marginal likelihood will lie elsewhere, sometimes near the ordinary least-squares estimate. Edward E. Leamer 45

iinstrumentalnstrumental vvariablesariables eestimator.stimator. OOurur kkeyboardseyboards nownow comecome withwith a hhighlyighly sseduc-educ- ttiveive bbuttonutton fforor iinstrumentalnstrumental vvariablesariables eestimates.stimates. TToo ddecideecide hhowow bbestest ttoo aadjustdjust tthehe iinstrumentalnstrumental vvariablesariables eestimatesstimates forfor small-samplesmall-sample distortionsdistortions requiresrequires somesome hardhard tthought.hought. ToTo decidedecide howhow muchmuch asymptoticasymptotic biasbias afflaffl ictsicts ourour so-calledso-called consistentconsistent esti-esti- mmatesates requiresrequires somesome veryvery hardhard thoughtthought andand dozensdozens ofof alternativealternative buttons.buttons. FacedFaced wwithith tthehe choicechoice betweenbetween thinkingthinking longlong andand hardhard versusversus pushingpushing thethe instrumentalinstrumental vvariablesariables button,button, thethe singlesingle buttonbutton iiss wwinninginning byby a veryvery largelarge margin.margin. LLet’set’s nnotot aadddd a ““randomization”randomization” bbuttonutton ttoo oourur iintellectualntellectual kkeyboards,eyboards, toto bebe ppushedushed wwithoutithout hardhard reflrefl ectionection andand thought.thought.

■ Comments from Angus Deaton, James Heckman, Guido Imbens, and the editors and referees for the JEP (Timothy Taylor, James Hines, David Autor, and Chad Jones) are gratefully acknowledged, but all errors of fact and opinion, of course, remain my own.

References

Bewley, Truman F. 1986. “Knightian Deci- Heckman, James J. 1992. “Randomization sion Theory.” Parts 1 and 2. Cowles Foundation and Social Program Evaluation.” In Evaluating Discussion Papers No. 807 and 835. Welfare and Training Programs, eds. Charles Manski Card, David. 1990. “The Impact of the Mariel and Irwin Garfi nkel, 201–230. Cambridge, MA: Boatlift on the Miami Labor Market.” Industrial Harvard University Press. (Also available as NBER and Labor Relations Review, 43(2): 245–57. Technical Working Paper 107.) Chamberlain, Gary, and Guido Imbens. Imbens, Guido W. 2009. “Better LATE than 1996. “Hierarchical Bayes Models with Many Nothing: Some Comments on Deaton (2009) and Instrumental Variables.” NBER Technical Heckman and Urzua (2009).” NBER Working Working Paper 204. Paper No. w14896. The Economist. 2009. “Cause and Defect.” Imbens, Guido W., and Joshua D. Angrist. August 13, 2009. 1994. “Identifi cation and Estimation of Local Deaton, Angus. 2008. “Instruments of Devel- Average Treatment Effects.” Econometrica, 62(2): opment: Randomization in the Tropics, and the 467–75. Search for the Elusive Keys to Economic Develop- Keynes, John Maynard. 1921. A Treatise on ment.” The Keynes Lecture, British Academy, Probability. London: Macmillan. October, 2008. Keynes, John Maynard. 1936. The General Fox, Justin. 2009. The Myth of the Rational Theory of Employment, Interest and Money. Macmillan. Market. A History of Risk, Reward and Delusion on Klibanoff, Peter, Massimo Marinacci, and Wall Street. New York: HarperCollins. Sujoy Mukerji. 2005. “A Smooth Model of Deci- Gates, Dominic. 2009. “Boeing 787 Wing sion Making Under Ambiguity.” Econometrica, Flaw Extends Inside Plane.” Seattle Times, July 73(6): 1849–1892. 30. http://seattletimes.nwsource.com/html Knight, Frank. 1921. Risk, Uncertainty and /boeingaerospace/2009565319_boeing30.html. Profi t. New York: Houghton Miffl in. Hanany, Eran, and Peter Klibanoff. 2009. Leamer, Edward E. 1978. Specifi cation Searches: “Updating Ambiguity Averse Preferences.” The Ad Hoc Inference with Nonexperimental Data. John B.E. Journal of Theoretical Economics, vol. 9, issue 1 Wiley and Sons. (Advances), article 37. Leamer, Edward E. 1983. “Let’s Take the Con 46 Journal of Economic Perspectives

Out of Econometrics.” American Economic Review, Econometrica, 28(4): 855–65. 73(1): 31–43. Sala-i-Martin, Xavier. 1997. “I Just Ran Two Leamer, Edward E. 1985. “Vector Autoregres- Million Regressions.” American Economic Review, sions for Causal Inference?” In Karl Brunner and 87(2): 178–83. Allan H. Meltzer, eds., Carnegie-Rochester Conference Sims, Christopher. 1980. “Macroeconomics Series on Public Policy, vol. 22, pp. 255–304. and Reality.” Econometrica, 48(1): 1–48. Leamer, Edward E. 2009. Macroeconomic White, Halbert. 1980. “A Heteroskedasticity- Patterns and Stories. Springer Verlag. Consistent Covariance Matrix Estimator and a Liu, Ta-Chung. 1960. “Underidentifi ca- Direct Test for Heteroskedasticity.” Econometrica, tion, Structural Estimation, and Forecasting.” 48(4): 817–38.