Management Half-Truths and Nonsense: How to Practice Evidence- Based Management
JeffreyPfeffer Robert I. Sutton
"Instead of being interestedin what is new,we ought to be interestedin what is true." - Pfeffer's Law
"Ifyou thinkthat you have a new idea, you are wrong. Someone probably already had it. This idea isn't originaleither; I stole it fromsomeone else." - Sutton'sLaw
questfor information and research-basedinsight is an obsession in thecapital markets. There is a veritableindustry of analysts, investmentbankers, portfolio managers, and investorswho seek anyinformational advantage, which is one reasonthat academics who studyfinance - suchas NobelPrize winners Myron S choies, William Sharpe,and MichaelSpence - have been recruitedto workon WallStreet and withmoney managers. The value ofevidenced-based investment decisions also explainswhy acting on privateinformation - insider trading - is regulatedso tightly,and whyU.S. companiesare forbiddenfrom releasing information to elitegroups of investors and analysts.Instead, they must make conference calls and announcementssimultaneously available to thegeneral public. Quantitative researchon capitalmarkets abounds, and companiessuch as Vanguard,Fidelity, BarclaysGlobal Investors, and hundredsmore have used it to developinvest- mentproducts and strategies. The potentialpayoff for using valid evidence is even greaterwhen it comesto manaßinßorganizations. Capital markets are amongthe most efficient
Adapted by permissionof HarvardBusiness School Press. Hard Facts, Dangerous Half-Truths,and TotalNonsense: Profitingfrom Evidence-Based Management, by JeffreyPfeffer and RobertI. Sutton. Copyright© 2006 JeffreyPfeffer and RobertI. Sutton.All RightsReserved.
CALIFORNIAMANAGEMENT REVIEW VOL48.NO.3 SPRING2006 77 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
in theworld, so itis hardto gainan enduringinformation advantage. Innova- tionssuch as junkbonds, indexed mutual funds, and derivatives,for example, werecopied with lightning speed. Imitation is muchslower and less effectivein theworld of management practices, in partbecause such practices depend on tacitknowledge and implementationskill, on knowingnot just whatto do but howto do it.And management practices and logicresist copying because of the powerof precedent and ideology.Consider how longSouthwest Airlines had its businessmodel and an amazingrecord of unparalleled profitability in the airline industrypretty much to itselfbefore JetBlue and a fewothers began to seriously copyit. Or witnessthe failure of competitors to catchToyota in productivity, quality,or timeto marketwith new productinnovations, even thoughToyota givestours to competitors,and thefundamentals of its system have been describedin numerouswritings.1 We have foundthat most managers actually try to act on thebest evidence.They follow the business press, buy business books, hire consultants, and attendseminars featuring business experts. Companies do sometimesbene- fitfrom these efforts. Yet there is surprisinglylittle rigorous use or seriousappre- ciationof what we call "evidence-basedmanagement." Firstand foremost,evidence -based management is a way ofseeing the worldand thinkingabout the craft of management. Evidence-based manage- mentproceeds from the premise that using better, deeper logic and employing factsto theextent possible permits leaders to do theirjobs better.Evidence-based managementis based on thebelief that facing the hard facts about what works and whatdoesn't, understanding the dangerous half-truths that constitute so muchconventional wisdom about management, and rejectingthe total non- sensethat too oftenpasses for sound advice will help organizationsperform better.
Obstacles on the Road to Implementing Evidence-Based Management Implementingevidence-based management is a journey,not a quick-fix technique,and alongthe road you willencounter hindrances and obstacles.It is ourjob to tellyou aboutsome ofthe most pernicious potential roadblocks and providesuggestions about how to avoidand overcomethem, or at leastmitigate theireffects.
UsingData ChangesPower Dynamics A formerstudent who workedat Netscapereported that James Barksdale, a formerCEO ofthat company, once remarkedat a meetingat thecompany somethingto theeffect of: "If the decision is goingto be made bythe facts, [then]anyone's facts, as longas theyare relevant,are equal. Ifthe decision is goingto be made on thebasis of people's opinions, then mine [he was the CEO at thetime] counts for a lotmore." What that anecdote illustrates is thatfacts and evidenceare greatlevelers of hierarchy. Therefore, some ofthe resistance to
78 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
evidence-basedpractice arises because, when done right,it does changepower dynamics,replacing formal authority, reputation, and intuitionwith data. One pundit,referring to evidence-basedmedicine, called it replacingwarriors with accountants.2 Adoptingevidence -based management could send similar reverberations throughthe corporate world. Senior leaders are oftenseen as heroesand vener- atedfor their wisdom and decisiveness.CEOs and theirbrethren could lose statureas theirintuitions are replaced,at leastat times,by judgments based on evidenceavailable to virtuallyany educatedperson with access to thedata. But as recentresearch by Rakesh Khurana suggests, less heroicCEOs and leaders who operateon thebasis of the best data and insightsmight actually fuel better organizationalperformance.3 The implicationis thatleaders need to makea fundamentaldecision: do theywant to be toldthey are alwaysright, or do theywant to lead organizations thatactually perform well? When Gary Loveman of Harrah's told a groupof Stanfordstudents that he frequentlymade mistakes,that he was willingto listen to all thefacts and analysis,and thathis facts and insightswere not privileged over anyone JeffreyPfeffer ¡s Professorof Organizational else's,this was notsome politically correct Behaviorat the Graduate School of Business posture.Loveman is a verycompetitive per- at StanfordUniversity. son,and he wantsHarrah's to win- and win- RobertI. Sutton is Professorof Management Science and at Stanford ningrequires mustering the truth and the Engineering University. bestinformation for making decisions, not deferringto peoplebased on title,rank, or anythingelse. Thatsort of egalitarian culturewas supposedlythe norm in the SiliconValley - and is stillevident at placessuch as Google,with its more academic and long-termorientation. But egosloom large even in high-technologycompanies, and evidence oftencuts inflated egos down to size.There is a clearimplication in all ofthis for selectingleaders - avoidat all coststhe people who thinkthey know everything. Theydon't. But worsethan that, they are unlikelyto embraceany factsthat disagreewith their preconceptions. This is whyone ofour favoritesayings is, "Whentwo people always agree, one ofthem is unnecessary."This is a principle bothof us have appliedwhen advising leaders how to interactwith others, help- ingcompanies hire new people,or lookingfor a coauthorto workwith.
PeopleOften Don't Wantto Hear theTruth The phrasedon 't shoot the messenger contains an enormousamount of truth,namely, that delivering bad newsis notsomething that typically wins you manyfriends. People like to delivergood news,regardless of its validity, in large partbecause most people seem to preferhearing good news.The important insighthere is thata lie takestwo parties - theperson who tellsthe lie and quite frequentlythe listener who signalsin a numberof ways that she or he wantsto be liedto. As GaryLoveman explained to us, sayhe goes to a casinothat isn't doing well.If the leadership at thefacility tells him that they understand the problem
CALIFORNIAMANAGEMENT REVIEW VOL 48, NO.3 SPRING2006 79 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
and knowhow to fixit, he can flyoff feeling good thatthings will get better. If insteadthey tell him that they've tried a bunchof things, basically everything theycan thinkof, and thecasino is stilllosing out to thecompetition, Loveman and his teamhave to actuallyfix the problem, possibly deliver bad news to theirbosses, the board of directors, and can'tgo offinto the sunset content and secure.But theimportant thing is thatthey can fix it, because they've been giventhe facts. Building a cultureof truth telling and actingon thehard facts requiresan enormousamount of self-discipline in orderto notonly be willingto hearthe truth, however unpleasant, but to actuallyencourage people to deliver bad news.Kent Thiry, the CEO ofDaVita, told us thatsenior managers at his companyactively seek out problemsand bad news.That's because the good newsdoesn't require any decisionsor action;it's the bad news thatcreates the need to do somethingto fixthe failure. And you can'tfix things or bringadvice and talentto bearon problemsunless you knowabout them. Thereis reallyonly one way aroundthis reluctance to confrontthe hard facts,and thatis to consciouslyand systematicallyunderstand the psychological propensityto wantto bothdeliver and heargood news and to activelywork againstit. To practiceevidence-based management you firstneed to knowthe realtruth. And it's better to knowthe truth early, when situationscan be reme- died,than later when it maybe too lateto do much.
TheMarketplace for Business Ideas Is Messyand Inefficient Thereis yetanother barrier to practicingevidence -based management, and thatis thesorry state of the business idea marketplace.The sad factis that any sane manager,consultant, or changeagent who wades intothe market for businessknowledge is soon overwhelmedby vast amounts of clashing and mis- leadingadvice. The business-ideamarketplace is plaguedby severalintertwined problemsthat confront anyone who seriouslytries to practiceevidence-based management. First,there is simplytoo muchinformation for any singleperson to consume.There are at leasta hundredmagazines and newspapersdevoted to businessissues.4 There are at least30,000 business books in print5and approximately3,500 new ones are publishedeach year* Second, disparate and disconnectedrecommendations about management practice are seldomwoven togetherin a way thatmakes remembering them or thinkingabout them easy, or maybeeven possible.Consider for instance Business: The Ultimate Resource, an encyclopedictome that weighs over eight pounds and runs2,172 oversized pages.Business claims "it will become the 'operatingsystem' for any organization or anyonein business."This claim is faultybecause a good operatingsystem fits togetherin a seamlessand logicalmanner. Unfortunately, this collection of over 150 essaysand articlesreads like a nearlyrandom collection of disconnected bits ofadvice. No discernableeffort is made to connectany ofthese bits. Business offersadvice on a dizzyingarray of topics from creating a funplace to work,to calculatingworking capital, to creatingpowerful brands, to designinga website. Beyondthat, the reader is givenalmost no informationabout what evidence,
80 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 Management Half-Truthsand Nonsense: How to Practice Evidence-Based Management
TABLE I . Whom ShouldYou Believe?Clashing Business BookTHJes
In Search of Excellence: The Myth of Excellence: Why Great Companies Never Try Lessons fromAmerica's Best-RunCompanies to Be the Best at Everything
Charisma: Seven Keys to Developing the Magnetism Leading Quietly. That Leads to Success An UnorthodoxGuide to Doing the RigfitThing
Leading the Revolution:How to Thrivein TurbulentTimes Managing forthe ShortTerm.The New Rules forRunning a by Making Innovationa Way of Life Business in a Day-to-Day World
Love Is the KillerApp: Business Is Combat A FighterPilot's Guide to Winningin How to Win Business and InfluenceFriends Modern Business Warfare
The Peaceable Kingdom: Capitalizingon Conflict Buildinga Company WithoutFactionalism, Fiefdoms, Strategiesand Practices forTurning Conflict to Synergy Fear and Other Staples of Modem Business in Organizations
Managing by Measuring.How to ImproveYour Organization's Managing withPassion: PerformanceThrough Effective Benchmarking Making the Most ofYourJoband YourLife
The Quest forAuthentic Power: What Would Machiavelli Do? GettingPast Manipulation,Control, and Self-LimitingBeließ The Ends Justifythe Meanness
and ThinkingInside the Box: Out of the Box: Strategiesfor Achieving Profits Today The 12 Timeless Rules forManaging a SuccessfulBusiness GrowthTomorrow through Web Services
Builtto Last SuccessfulHabits of VisionaryCompanies Corporate Failureby Design: Why OrganizationsAre Builtto Fail
theory,or logic(if any) supportsthe thousands of "do's" and "don'ts"listed, makingit impossibleto judge thequality of all thatadvice.7 Third,the advice managers get from the vast and ever-expandingsupply ofbusiness books, articles, gurus, and consultantsis remarkablyinconsistent. Considerthe following clashing recommendations, drawn directly from popular businessbooks: Hire a charismaticCEO; hirea modestCEO.8 Embrace complex- itytheory; strive for simplicity.9 Become a strategy-focusedorganization; don't wastemuch time on strategicplanning because it is oflittle value.10 The deeper you look,the more confusing and bewilderingit all becomes.Table 1 illustrates justa smallsample of the conflicting advice managers obtain from the business book marketplace. Even worse,because good adviceis frequentlyhard to distinguishfrom bad,managers are constantlyenticed to believein and implementflawed busi- nesspractices. This happens partly because consultants and otherswho sellideas and techniquesare alwaysrewarded for getting work, only sometimes rewarded fordoing good work,and hardlyever rewarded for whether their advice actually enhancesperformance.11 The incentivesare ofteneven moreperverse than that,because if a clientcompany's problems are onlypartly solved, that leads to morework for the consulting firm. The seniorexecutive of a humanresources
CALIFORNIA MANAGEMENT REVIEW VOL 48, NO.3 SPRING 2006 8 1 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
consultingfirm, for example, told us thatbecause pay forperformance programs almostnever work that well, you usuallyget asked back again and againto repairthe programs your clients bought from you. Similarly,while we were writingour lastbook, a seniorpartner in a largeconsulting firm commented that thebusiness process reengineering work his firmhad done was one ofthe best thingsthat had everhappened. First the firm made a lot ofmoney doing the reengineeringconsulting; then it made even moremoney from the same clients becauseit turned out thatmany of the "unnecessary"people removed during reengineeringefforts had in factbeen doingnecessary work. The resultwas that his own consultantswere then hired to do thatsame work- ofcourse, at a far higherwage ratethan the people they replaced. Ifyou thinkour chargeis too harsh,ask yourfavorite consulting firm whatevidence they have thattheir advice or techniquesactually work - and pay attentionto theevidence they offer using some ofthe guidelines we willprovide a littlelater on in thisarticle. A fewyears ago, seniorBain consultantDarrell Rigbybegan conducting the only survey we have encounteredon theuse and persistenceof various management techniques and practices.12As Rigbytold us, it struckhim as odd thatyou couldget good informationon productssuch as toothpasteand cerealbut there was almostno informationabout interventions thatcompanies were spending literally millions of dollars to implement.Even theBain survey,noteworthy as it is,measures only the presence and persistence ofvarious programs and subjectiveassessments of them. Yetanother flaw with the marketplace for business ideas is thatit is filled withsloppy analogies that somehow win managersover. Two of our favorites are: thosethat have been used to justifyforced curve ranking systems, made famousby GeneralElectric; and thebusiness is waror combatanalogy, which has been used to arguefor harsh steps toward the competition and, occasionally, towardone's own people (who are viewedas "necessaryevils"). Jack Welch, the formerCEO ofGE, made theargument for forced curve ranking, a quitecontro- versialmanagement practice, this way: people get graded in school,so why shouldn'tthey get graded at work?This leads to the (reasonable)implication thatgrades in schoolare typicallyassigned on a comparativebasis.13 First of all - and ironically,given the use ofthis analogy - theevidence strongly suggests that studentslearn better when theyare notgraded and certainlynot when they are gradedon a curve.14But settingthat fact aside, consider a crucialdifference betweenschool and work.In school,there is relativelylittle interdependence in performance- if you learnchemistry and yourcolleagues don't, it does not affectyou in anyway. Learning is a matterof you, individually, mastering the particularsubject matter. Cooperation or teamworkin school,at leaston tests, is calledcheating. By contrast,work organizations are typicallyfilled with inter- dependentaction, where your ability to accomplishsomething is crucially dependenton thehelp and cooperationof others. So, ifgrading on thecurve causescompetition and conflict,the consequences will be vastlydifferent for interdependentas opposedto noninterdependentsystems.
82 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 Management Half-Truthsand Nonsense: How to Practice Evidence-Based Management
Followingthe business as war analogycan be similarlymisleading. The analogyimplies that you wouldalways aim to hobbleand destroycompetitors and avoidcooperating with other firms in yourindustry. If such logic had been followed,however, the Napa Valleymight never have becomesuch a prestigious winegrowingregion. When Robert Mondavi started his winery in 1966,he workedto enhancethe reputation and qualityof every Napa winery,not just his own. Such cooperationset the stage for the famous "Judgment in Paris"in 1976, whereprestigious French wine criticsconsistently mistook the Californiawines forFrench and rankedthe California wines higher. Although no Mondaviwines weretasted, California winemakers from Chateau Montelena (the top white) and Stag'sLeap (thetop red) werequick to thankMondavi for helping them succeed.Indeed, both winners - Mike Grgichand WarrenWiniarski - had workedfor Mondavi before leaving with his blessing to starttheir own wineries. Mondavi'sgenerosity paid off:he and his companyprofited when the prices of ¿//Napawines skyrocketed after the 1976 "Judgment."15Yet if you reada Har- vardcase bystrategy researcher Michael Porter about Robert Mondavi Winery, it onlyconsiders how Mondavicompetes with other California winemakers like Kendall-Jacksonand Gallo.Apparently, Porter's narrow focus on theuseful but incomplete"competitive strategy" analogy led himto avoidnoticing (or men- tioning)the cooperation that benefited Mondavi's company as he enjoyedthe reputationalspillover from the Napa Valley'sgrowing prestige.16
Use Sound Logic and Analysis So, whatis an executivewho is committedto implementingevidence- based managementto do? We have twobroad categories of recommendations. First,familiarize yourself with analytical and logicalissues so thatyou can be a moresophisticated and informedconsumer of business ideas and research.And second,develop a differentset of criteria for evaluating business writing and research.We considereach ofthese suggestions in turn. Usingsound logic and analysisdoes notmean you have to takea course in statisticsor thescientific method. It just means you need to pay closeatten- tionto problemsin exposition,logic, and inferencethat are reasonablyobvious once identifiedand thatbedevil much of what passes for business research. For instance,some popular business books, such as TheWar for Talent, collect infor- mationon theindependent variables, in thiscase practicesfor managing talent, afterthe time period covered by the data on theperformance that talent man- agementpresumably causes.17 This is notthe only book thathas thisproblem ofcausal order. But it is importantto recognizethat in mostdiscussions of causality,the cause needsto occurbefore the effect. Or considera bigproblem that plagues managers who tryto learnfrom experienceas well as manybusiness writers who drawlessons from currently successfulcompanies. What both the managers and thewriters don't seem to realizeis thatcrucial evidence is lostif they ignore the practices and strategies used byfailed companies. As a consequence,the steps taken by failed companies
CALIFORNIA MANAGEMENT REVIEW VOL 48, NO.3 SPRING 2006 83 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
are consistentlyundersampled.18 It is hardto overstatethe breadth of this prob- lem,and how muchit distortsthe conclusions that people reach.For example, in 1922 theU.S. tireindustry had 274 firms,but by 1936 thenumber of compa- 19 nieswas downby 80 percent,to 49. Todayonly two U.S.-owned tire compa- niesare stillstanding. One studyestimated that there were 2,197 automobile producersin theUnited States between 1885 and 1981; less than 1 percentstill exist.20Yet few study the histories and practicesused byall those"losers" and how theydiffer from surviving companies. Studyingand imitatingonly surviving companies, especially the most successfulones, can lead to flawedand dangerousconclusions about what are thebest and safestpractices. For instance, companies that use risky,unusual practicesperform either much better or muchworse than average, especially comparedto thosethat do whatmost other companies do. But "ifonly the best, notthe worst, performers are observed,performance will seem to be associated withstrategies that are farmore likely to killa companythan to resultin supe- riorperformance."21 Similarly, concentrating resource allocations on a narrow rangeof products, strategies, and marketswill expose companies to greaterrisk thana strategyof diversification, because they will have putall oftheir eggs in fewerbaskets. But ifwe just observethe survivors and topperformers - those thathave managedto concentratetheir efforts and resourcesin preciselythe rightway - itwill appear as ifconcentrated, focused strategies win out mostof thetime, even thoughthe exact opposite is usuallytrue. The onlyway to avoid learningsuch flawed lessons is to devotemore attention to studyingcompanies thatfail and whythey fail, not just thosethat succeed. It is also usefulto getin thehabit of running small experiments and thinkingabout the inferences you are drawingfrom your observations and from thedata yourorganization is constantlygenerating. In medicalresearch, two mainresearch methodologies are used to generateknowledge. The goldstandard is thedouble-blind, placebo-controlled study, in whichpatients in one groupare randomlyassigned to geta treatment,patients in anothergroup are randomly assignedto geta placebo,and neitherdoctors nor patients know who is in which group,so priorexpectations don't affect the results. It is oftenimpossible, how- ever,to use suchstudies. For example,it would be unethicalto randomlyassign nonsmokingteenagers to conditionswhere they did and did notstart smoking. The alternative,less rigorousapproach is to use surveysor observationsto mea- surebehaviors like exercise, vitamin consumption, or smoking;assess health outcomeswith medical records; and use statisticaltechniques to relatethe behaviorand outcomes,controlling for as manyalternative explanations as possible. Similarly,in managementit oftenisn't feasible to use experiments,espe- ciallydouble-blind designs. People will often know, for instance, if they are get- tingcertain incentives or not.Yet field experiments are valuable,and field experimentsare to be foundin some ofthe most compelling and usefulacade- micresearch. As one example,a randomizedfield experiment with Israeli sol- diersconfirmed the Pygmalion Effect, that high or low performanceexpectations
84 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
can becomeself-fulfilling. When drill instructors were tricked into believing that certainrandomly selected soldiers would achieve superior performance, those soldierssubsequently performed far better on taskslike firing weapons and read- ingmaps than soldiers in controlconditions who didnot have thehigher perfor- manceexpectations. This experiment and numerousother studies show that leadersoften get the performance they expect from subordinates.22 In otherfield experiments,incentives have been used in someparts of an organizationand notothers, as have changesto makejobs morechallenging and to compareopen and closedoffices.23 A bigbarrier to usingexperiments to buildmanagement knowledge is thatcompanies tend to adoptpractices in an all or nothingway - eitherthe CEO is behindit so everyonedoes it,or at leastclaims to do it,or it isn'ttried at all. Thistendency to do thingseverywhere or nowhereseverely limits a company's abilityto learnby trying things in someplaces but not in others,much as firms routinelydo whentest-marketing a product or evaluatingan advertisingcam- paign.In particular,multi-site organizations like restaurants, hotels, and manu- facturerswith multiple plants can learnby experimenting in selectedsites and comparingthe results to controllocations. The wisdomof starting with a smallexperiment is illustratedby research we didyears ago withthe SouthlandCorporation, which operates and franchises 7-Elevenstores. Southland executives were enamored with Peters and Water- man'sIn Searchof Excellence, which exhorted managers to "be closeto thecus- tomer"and developa "serviceobsession." This infatuation was translatedinto a companywideeffort to improvecustomer service, which aimed to getevery clerkin everystore in NorthAmerica to offera greeting,smile, eye contact, and a thanksto everycustomer. Millions of dollars of management bonuses were linkedto thesecourtesy measures and, in 1987,Southland held the "Thanksa Million"contest where 17 storemanagers who won regionalcourtesy contests qualifiedto entera drawingfor $1 million.The contestculminated in a slick mediaevent hosted by game show host Monty Hall ofLet's Make a Deal fame. DebraWilson, a storemanager from Piano, Texas, won themillion bucks. Thiswas good fun,but was itworth the money? We workedwith Larry Ford,then Southland's director of field research, to do research(including a randomizedexperiment with 15 stores)to discoverif courteous clerks fueled sales.Unfortunately, executives hadn't bothered to tryany pilot studies or experimentsbefore spending all thatmoney on courtesyprograms, although Fordurged them to do so. We ultimatelyfound little if any evidencethat cour- tesyincreased store sales. Yes, it was possibleto increasecourtesy. We used trainingand coachingduring our 10-weekfield experiment to increasethe per- centageof customers who receivedgreetings from 33 percentto 58 percentand smilesfrom 32 percentto 49 percent.But themain finding, including results fromlarge-scale studies by Ford's group, was thatclerks in storeswith more saleswere actually less courteous. Apparently, the crowding and longlines in busystores made clerksand customersgrouchy. This research ultimately helped convinceexecutives to scaleback courtesy programs and realizethat, for most
CALIFORNIAMANAGEMENT REVIEW VOL48.NO.3 SPRING2006 85 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
7-Elevencustomers, good servicemeant getting out ofthe store fast, not fake smilesand insinceresocial amenities. Southland could have savedmillions by doingsome pilot studies first. In managementresearch, studies that use surveysor data fromcompany recordsto correlatepractices with various performance outcomes are farmore commonthan experiments. Such nonexperimentalresearch is useful,but careis requiredto controlstatistically for alternative explanations, which arise in even thebest studies. Managers who consumesuch knowledge need to understand thelimitations and thinkcritically about the results. Anyone who everhad a statisticsclass knows that correlation is notcausation, but it is surprisinghow oftenpurveyors of business knowledge are fooledor tryto foolcustomers. We admireBain's consulting and have made favorablecomments here about their research.We do wonder,however, why they put a tableon theirhome page that brags:"Our Clients Outperform the Market 3 to I/'24The smartpeople at Bain knowthis correlation doesn't prove or even implythat their advice transformed clientsinto top performers. For starters, top performers may simply have more moneyfor hiring consultants. Indeed, any claimthat Bain deservescredit for suchperformance is conspicuouslyabsent. Perhaps they are hopingthat visitors willmomentarily forget what they learned in theirstatistics classes. Althoughquantitative data are important,it is crucialto also learnfrom clinicalpractice and observation,and to understandthat management, like med- icine,is bothan artand a science.We rejectthe notion that only quantitative data are acceptablefor evidence-based management. As Einsteinput it, "Not everythingthat can be countedcounts, and noteverything that counts can be counted."By focusingonly on whatcan be quantified,we can lose sightof what mattersmost. John Steinbeck explained the limits of quantification in TheLog fromthe Sea ofCortei, his book abouta scientificexpedition: The MexicanSierra has "XVII-15-IX"spines on thedorsal fin. These can be easily counted.But ifthe sierra strikes hard so thatour handsare burned,if the fish soundsand nearlyescapes and finallycomes over the rail, his colorspulsing and histail beating the air, a wholenew relationalreality comes into being - an entity whichis morethan the sum of the fish plus the fisherman. The onlyway to count thespines of the sierra unaffected by this second relational reality is to sitin a laboratory,open an evilsmelling jar, remove a stiffcolorless fish from the forma- lin solution,count the spines, and writethe truth "D.XVII-15-IX." There you have recordeda realitythat cannot be assailed- probablythe least important reality concerningeither the fish or yourself.It is good to knowwhat you are doing.The man withthe pickled fish has setdown one truthand has recordedin hisexperi- ence manylies. The fishis notthat color, that texture, that dead, nor does he smellthat way.25 Whenused correctly,stories and cases are powerfultools for building managementknowledge. Many quantitative studies are publishedon developing new products,but few compare to TracyKidder's Pulitzer Prize-winning Soul of a NewMachine for capturing how engineersdevelop products and how managers can contributeto - or undermine- theirsuccess. Gordon McKenzie's Orbiting the
86 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
TABLE 2. CurrentStandards versus Evidence-BasedManagement
Evidence-Based Current Practice Management
Treatold ideas as ifthey are brand-new. Treatold ideas likeold ideas.
Glorify,celebrate, and applybreakthrough ideas and Be suspiciousof breakthroughideas and studies- they studies. almostnever happen.
Celebratebrilliant individuals like management gurus, Celebratecommunities of smartpeople and collective thoughtleaders, and starperformers. brilliance,not lone geniusesor gurus.
Emphasizeonly the virtuesof the researchmethods and Emphasizethe virtuesand drawbacks(and uncertainties) the managementpractices you use. Don't mention of yourresearch and proposed practices, drawbacksor uncertainties.
Use success and failurestories about companies,teams, Use success and failurestories to illustratepractices and people to uncoverbest and worstpractices. supportedby other evidence, not necessarilyas valid evidence.
Use popularideologies and theoriesto generateand Take a neutralapproach to ideologiesand theories.Base justifymanagement practices. Ignore or rejectall clashing managementpractices on the best evidence,not what is evidence(no matterhow strong). invogue.
GiantHairball is themost charming and usefulbook on corporatecreativity we know.Hairball for example, argues that teasing is "a disguisedform of shaming" and illustrateshow it stiflescreativity and damagespeople, a hypothesiscon- firmedby recent experiments.26 We are enthusiasticabout such writings because theyare engaging,bolster other (often quantitative) research, and suggestnew practicesthat companies can try.
Some Guidelines for Evaluating Management Ideas and Knowledge Our firstset of suggestions dealt with learning enough about the logic ofanalysis to putclaims and data to thetest. Our secondset of suggestions addressesa differentbut related problem: the existing standards for assessing managementknowledge are deeplyflawed and oftencompletely dysfunctional. To partiallyremedy this problem, we proposesix standardsfor generating, eval- uating,selling, and applyingbusiness knowledge. Table 2 contraststhese new (or moreprecisely old, but less common)standards with current standards. Unfortunately,although deeply flawed, the reigning standards are reinforced bythe actions of virtually every major player in themarketplace for business knowledge.Those who generatesuch knowledge - gurus,consultants, and acad- emics-need to thinkharder about whether anyone can comprehendor use theirwares. They need to stoppretending that proven- or debunked-old ideas are shinynew cures.And they need to groundtheir recommendations in better evidence.The businesspress, purveyor of so manypractices, needs to reduceits
CALIFORNIAMANAGEMENT REVIEW VOL48.NQ3 SPRING2006 87 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
fixationon individualheroes and what'snew, and talkmore about what's true. Finally,as thosewho buyand use suchknowledge, and imposeit on others, managerswill have thegreatest say overif and how evidence-basedapproaches spread.Yes, dear managers, you are alreadybeleaguered with other burdens. But you and yourcompanies can reapgreat benefits - and ultimatelysave much timeand money- bybecoming more sophisticated at judgingideas in the plethoraof books, seminars, and advicethat rains down on you. We didn't writeHard Facts,Dangerous Half-Truths, and TotalNonsense just to identifyincomplete and flawedbeliefs that shape organizationalaction. An even moreimportant aim is to helpteach managers, and thosewho writefor and advisethem, how to makebetter judgments about the virtues and flawsof the evidencethey generate and encounter.
TreatOld Ideas As If TheyAre Old Ideas The managementideas marketedby consultants,publishers, and the businesspress are oftentreated like laundry detergent - theonly thing they are interestedin is what'snew and presumablyimproved. This pursuit of novelty for itsown sake maybe charmingand profitablewhen applied to fashion,dress, and popularculture or food,but is costlyand destructivewhen used to guideman- agementpractice. Consider exhibit A, thequality movement at theFord Motor Company:"Twenty years ago, when an invasionof Japanese imports threatened theAmerican automobile industry, the Ford Motor Company led a quality revivalbased on themanagement philosophy of W. EdwardsDeming, who was controversialthen and is out offashion now. The resultsof the movement . . . werestunning at Ford.After racking up $3 billionin lossesbetween 1979 and 1982 ... by 1986 Fordhad becomethe most profitable American auto company."27 Yet,once implemented,total quality management was yesterday'snews, an old idea. Andmaking cars was so yesterday.Under Jacques Nasser, the CEO at thetime who was subsequentlyfired, Ford pursued innovation and "revolu- tion."It was laudedin FastCompany - because the business press is enamored withnovelty - forits emphasis on doingnew things:"[A manager's]team rec- ommendedthat Ford enter the parts-recycling business, start a for-profitdriver educationprogram, and developa chainof branded maintenance and repair shops.The companyhas done all three."28 Thereis nothingwrong with innovation, encouraging creativity, or forthat matter, introducing new productsand services,except that in thiscase Fordlost its focus on quality.Costs and defectsincreased, competitors developed superiordesigns, and sales slipped.By 2001, Fordwas in thesame fixit had been in twodecades earlier. The solution:re-embrace an old idea, quality,but thistime under the Six Sigmabanner so thatit would presumably look "new."A NewYork Times business reporter concluded, "Had Fordstuck with Total Quality Management,it mighthave avoidedmany of the problems that have plaguedit recently."29But how couldit? Stickingwith old ideas is boring.Managers, and
88 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 Management Half-Truthsand Nonsense: How to Practice Evidence-Based Management
forthat matter, academics and thebusiness press, pursue novelty often for its own sake.30 Perhapssticking with proven practices is boring,but we need to acknowl- edge- even glorify- old ideas ifwe wantto debunkbad managementpractices and improvegood ones. Afterall, isn't bland old excellencea betterfate than an excitingnew failure?It soundsironic, but even creativityis mostlysparked by old ideas.Both major creative leaps and incrementalimprovements come from fiddlingwith ideas fromother places and blendingthem in new ways.Better ideasresult when people act like"nothing is inventedhere" and seek new uses forothers' ideas.31 This holds for even themost creative companies like Apple, 3M, IDEO, Genentech,Google, Capital One, and Cirquedu Soleil.Unfortu- nately,too manycompanies are plaguedby the not invented here syndrome, where peopleinsist on usinghomegrown ideas, especially ideas thatcan be ballyhooed as new and different.There are, after all, substantialrewards for pretending that thesame old ideas are brandnew. Managers can impressbosses with cutting- edgeideas. Consultants can sellclients unique services.Gurus can land lucrative book contractsand speakingfees by peddling the next big thing. And journalists can sellnewspapers and magazinesby giving readers the latest scoop. Theseincentives fuel a bizarrecollective amnesia. The same thingsare "discovered,"or at leastreported, over and over,which wastes much effort and time.The HarvardBusiness Review has publishedat leastthree articles on incen- tivepay and organizationalperformance in thepast decade. Each makesa simi- larpoint: compensating people for only individual performance creates more problemsthan it solves,so rewardsshould emphasize organizational, not just individual,performance. Alfie Kohn wroteabout this idea in 1993,Jeffrey Pfef- ferdid it 1998,and Egon Zehnderdid it yetagain in 2001. Notone ofthese arti- - clesrefers to theprior article, because HBR policy precludes footnotes and based on our experiencein publishingseven HBR articles- discourages refer- encesto priorwork.32 How do we breakfree from this cycle where again and againbusiness writers,experts, and managersact likeold ideas are brand-new?The short answeris to treatold ideas as old ideas.The longeranswer is morecomplex, but somesimple steps would help. People who spreadmanagement knowledge can tellus wherethey got their ideas. They can also reviewpast work to avoid rein- ventingthe past. Although footnotes are notright for all publications,as they slowand distractreaders, there are alternatives.We are consistentlyimpressed withhow theEconomist and NewYorker weave referencesto pastwork into the text,especially in MalcolmGladwell's deft New Yorker contributions.33 No matterhow it is done,people who spreadideas oughtto acknowledge keysources, as it discouragesclothing old ideas as new and encourageswriters and managersto buildon and blendexisting ideas. Doing so isn'tjust intellectu- allyhonest and polite.It leads to betterideas. The 17th-centuryphysicist Sir Isaac Newtonis oftencredited as saying,"If I have seen farther,it is by standing on theshoulders of giants." Of course Newton didn't invent this saying. The originatoris impossibleto find,but versions of it go backat least 1,000years
CALIFORNIA MANAGEMENT REVIEW VOL 48, NO.3 SPRING 2006 89 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
beforeNewton's time. Our favoriteis, "A dwarfstanding on theshoulder of a giantmay see fartherthan the giant himself."34 " Be Suspiciousof "BreakthroughIdeas and Studies Relatedto thedesire for new is thedesire for big - thebig idea, thebig study,the big innovation. Unfortunately, they rarely if ever happen. Even in the physicalsciences, close examinationof so-called breakthroughs nearly always revealspainstaking, incremental work that finally is recognizedas a biginsight. Forinstance, the development of the integrated circuit was thework of numer- ous peoplein a varietyof companies interacting over a numberof years, even thoughone or a fewpeople oftengot most of the public credit.35 Yet many man- agersstill yearn for breakthroughs and hordesof purveyors pretend to bring themthe magic they crave. The resultof this search for the big thing is whatwe saw at a largebank: theidea ofthe moment. The formerCEO ofthis bank fancied himself (and was) quitean intellectual.He was alwayslearning about new techniques,hoping each wouldbe theconcept to propelthe bank to thenext performance level. The bankshifted from one programto anotherand managerslearned to bracethem- selvesfor the next flavor of the month. When a new initiativewas announced, veteransdid nothing,recognizing that by the time their inaction was noticed, theCEO wouldbe on to thenext big thing. We repeatedlytalked to and heard ofmanagers who perfectedthe art of pretending to use thelatest flavor, while makingno actualchanges in how theyor theirpeople worked. The upshotwas thatthe company churned through talking about one idea afteranother, but had littleelse to showfor it. The temptationto searchfor the silver bullet, that breakthrough study or concept,occurs far more often in theworld of management than in academic research.We constantlytravel between academia and thereal world of practic- ingmanagers and theiradvisers. Breakthroughs hardly ever seem to happen in academicresearch, even thoughthousands of studies are publishedin peer- reviewedjournals each year.Academics hesitate to place too muchweight on any one study,preferring to findpatterns across many studies and to base con- clusionson theweight of the evidence. In contrast,managers face an onslaught ofwriters, gurus, and consultantswho claimor implyone breakthroughafter another.Even whenpeople who do a studyor coina phrasedon't call it a breakthrough,the press often reinterprets modest progress (or none at all) as "shocking"and "revolutionary"ideas. In 2001 through2005, theFebruary issue ofHarvard Business Review has publishedlists of "Breakthrough Ideas forToday's BusinessAgenda," described as "boldand unsettlingnew ideas."36We've had threeof our ideas selectedfor the list. One ofour HBRarticles that was selected forthe list in 2002, Sutton's"The Weird Rules of Creativity," really isn't a break- through.The publicitywas nice,but embarrassingbecause these rules were actuallyderived from past, sometimes quite old, research. And in 2005, Pfeffer's "breakthroughidea" was evidence-based management, which might be unset- tlingto somepeople, but we are thefirst to admitit isn'tnew. As Stanford's
90 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
JamesMarch - amongthe most renowned organizational theorists - putit, "Mostclaims of originality are testimonyto ignoranceand mostclaims of magic are testimonyto hubris/'37
Celebrateand DevelopCollective Brilliance, Not Lone Geniusesor Gurus The businessworld is amongthe few places where the term guru appar- entlyhas primarilypositive connotations. In religionand politics,gurus are portrayedas extraordinarybut often dangerous leaders, who attractfanatical discipleswho bendto theirwishes, even whendoing so harmsthemselves and others.Yet despite the occasional cynical article or book,gurus are stillportrayed in largelypositive terms by the press, where they are routinelylisted, rank ordered,and gossipedabout. What's the Big Idea? by Thomas Davenport and LaurencePrusak uses pressmentions, Google Internet searches, and citations in academicjournals to rankthe top 100 businessgurus (Michael Porter was numberone and TomPeters was numbertwo). In 2001 and 2002, Business2.0 notonly listed top gurus like Tom Peters and LarryBossidy, they inserted card- board"Business Guru Trading Cards" for each, with a colorpicture and informa- tionincluding star power (from one to fourstars), fees, the guru's "big idea," and a "littleknown fact." Certainly,people who have good ideas and communicatethem well can helporganizations. But a focuson gurusmasks how businessknowledge is and oughtto be developedand used. Knowledgeisn't generated by lone geniuses who magicallyproduce brilliant new ideas in theirgigantic brains. This is a dangerousfiction. Writers and consultantsneed to be morecareful about describingthe teams and communitiesof researchers who developideas. Even moreimportant, implementing practices, accomplishing organizational change, and executingstrategy require the coordinated actions of many people. This is partlybecause commitment to an idea or programis amplifiedwhen people feel ownershipfor it. Learning also happensfaster when people take an activerole in developinga practiceor program.Sitting and listeningto even themost insightfulspeaker does none ofthis - thereis no ownershipof the ideas and no involvementin buildinga solution,so littleinformation is retained and lessis learnedabout how to actuallyuse theideas. Moreover,gurus too oftenoversimplify management challenges. The commonimplication that business practices can be installedlike a new machine is especiallydangerous. As Wharton'sRussell Ackoff put it, "Gurus provide readymade solutions,but educators provide ways that one can findsolutions foroneself . . . The outputof a guruis a closedsystem of thought, closed to externalinfluences and notsubject to change;the output of an educatoris an open systemof thought, open to externalinfluences and subjectto change."38 Leadinga revolution,reengineering, adopting Six Sigma,or becomingstrategy- focusedcan helpsome companies, but no approachis rightfor everyone. Those ofus who hawkbusiness knowledge need to come clean.We need to denythat we have magicanswers. We need to confessthat we arejust suggestingideas thatmight make managers' hard jobs a biteasier. We also need to followthose
CALIFORNIAMANAGEMENT REVIEW VOL48.NO.3 SPRING2006 91 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
fewgurus who rebukedangerous oversimplification. Consider C. K. Prahalad, who (withGary Hamel) wrote the blockbuster Competing for the Future and ranks highon manyguru lists. A fewyears back, he endedan engaging45 -minute speechto a hugeaudience by warning, "Don't forget, if someone tells you they have theanswer, they probably don't understand the question."39
EmphasizeVirtues and Drawbacks No drugis withoutside effects.Most surgical procedures have risksand even whenperformed perfectly may have downsides.Doctors are gettingbetter at explainingrisks to patientsand, in thebest circumstances, enabling them to join in a decisionprocess where risks and potentialproblems are considered. Thisrarely happens in management,where too manysolutions are presentedas costlessand universallyapplicable, with little acknowledgment of potential pit- falls.Yet all managementpractices and programshave strongand weak points, and even thebest have costs.This doesn't mean companiesshouldn't implement thingslike Six Sigmaor balancedscorecards, just thatthey should recognize the hazards.That way, managers won't become disenchanted when known setbacks occur,or worseyet, abandon a valuableprogram or practice. Lastyear, one ofus (Sutton)gave a speechat a companythat is renownedfor innovation. The companyhad spentthe prior year implementing a Six Sigmaprogram to improveefficiency and customersatisfaction. Sutton mentionedduring his speechthat process improvement programs like Six Sigma and TQM had been shownto driveout errorsand improveefficiency, but also to stifleinnovation.40 A self-described"internal quality guru" and "qualityevange- list"came up afterthe talk and arguedthat Six Sigmacould be used to improve anyprocess, including creativity. This guru kept insisting this was so, even though(after working in thefield for 10 years)he couldn'tname a single instancewhere Six Sigmaor TotalQuality Management programs had improved innovation.Sutton finally asked, "Can you thinkof any drawbacks to usingSix Sigmaor TQM?" Therewas a longsilence until the guru said, "Other than irra- tionalfear and resistance,I can't think of any." Thisevangelist illustrates how businessknowledge is routinelysold. Unlikemedicine, where physicians are ethicallyobligated to revealrisks and drawbacks,advocates of business practices rarely describe risks, problems that ariseeven in successfulcases, or occasionswhen their wares are likelyto be ineffective.A few exceptions illustrate a betterapproach. Consider J. Richard Hackman'sarticle "On the ComingDemise of Job Enrichment," published at theheight of a managementfad in the 1970s,a fadbased largelyon Hackman's research.Hackman was troubledbecause he couldonly find published success storiesabout companies that had redesignedwork to be moremotivating and meaningful.Yet in his experience,most redesign efforts were failing. Hackman warnedmanagers that enrichment programs were time-consuming and expen- siveto implement;they couldn't simply be installedlike a new machine.He predictedthat job enrichmentwould be just anotherfailed fad (a prediction thatpartly came true)if companies didn't start diagnosing work systems before
92 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
tryingto enrichjobs, kept pretending that jobs had changedwhen they hadn't, and didnot change the organization (especially incentives and supervision)to supportenriched work.41 The lessonis that,although some "experts"will make less money in the shortterm, business knowledge will advance and cynicismabout "flavors of the month"will wane ifthose who peddlebusiness practices routinely admit flaws and uncertainties.This means, among other things, admitting that their wares are thebest they can buildright now and,like all good ideas,will require con- stantmodification as moreis learnedalong the way.
Use Success(and Failure) Storiesto IllustrateSound Practices, Not as a Valid ResearchMethod One ofthe most common research methods entails sorting teams or orga- nizationsinto "high" and "low"performers, and thendigging into their pasts withinterviews, questionnaires, and publishedpress reports that rely on fuzzy and flawedhuman memory. These recollections are used to explainthe differ- encesbetween winners and losers,and to identifywhich practices to use (what winnersdid) and whichto avoid (whatlosers did). A related,and even more suspect,method is to gatherrecollections from and aboutonly winners, and to assumethat similarities explain success - even though(unstudied) losers may have actedthe same way. In Searchof Excellence used preciselythis approach - whichmay explain why a studyby Michael Hitt and Duane Irelandfound no significantperformance differences between Peters and Waterman's"excellent" companiesand a representativesample of Fortune 1000 companies.42 Academicresearchers also makeextensive use ofretrospective success and failurestories, although they usually offer ritualistic apologies. But such apologiesdon't prevent what is learnedfrom being based on suspectevidence. Manyinfluential studies, for example, have askedmanagers to identifyone suc- cessfuland one unsuccessfulpast product development effort, and thento recall differencesbetween the two. These studies find dramatic differences between winnersand losers.Successful efforts are rememberedas havingmore top man- agementsupport, more competent engineers, superior planning and execution, morefrequent interaction between key players, and on thewhole using a processthat was superiorto failedefforts in nearlyevery way.43 Thereis an enormousproblem with inferences based on recollection. In 1911,Ambrose Bierce's The Devil's Dictionary defined recollect as "to recallwith additionssomething not previously known,"44 foreshadowing much research on humanmemory.45 It turns out thateyewitness accounts are notoriouslyunreli- able. We humanshave terriblememories, regardless of how confidentwe are in our recollections.In particular,we remembervery different things when we are anointedwinners versus losers, and whatwe recallhas littleto do withwhat happened.Barry Staw, for example, assigned 20 teamsof MBAs to workon a financialpuzzle game. These three-person teams used a company'sannual reportto predictits sales and stockprice a yearlater. Staw then gave them falseperformance feedback. Ten teams were told their predictions were almost
CALIFORNIAMANAGEMENT REVIEW VOL 48, NO.3 SPRING2006 93 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
perfect;the other 10 weretold their estimates were far off target. In reality, therewere no significantdifferences between how well teamsin thetwo groups predictedperformance. Yet these false beliefs led to profounddifferences in what membersrecalled. The 30 winnersreported that their teams were more moti- vated,more cohesive, more open to others'ideas, had moreconstructive con- flict,and had greaterability compared to whatthe 30 losersremembered about theirteams. This research and similarstudies showing that false information causeswinning team members to recallhaving more effective leaders demon- stratethat winners and loserstell predictable stories, no matterwhat actually happened.So usingreports from winners or losersto findways to turnyour companyor teaminto a winneris questionableat best.46 We aren'tadvocating a ban on successand failurestories. Vivid cases grab our attention,show us whatto do, and inspireus to do it.The challengeis to tell moretrue stories. This means that storytellers (and listenerstoo) mustremind themselvesthat once theyknow who winsand who loses,they will believe pre- dictablethings that are notnecessarily true. It also meansthat we can learn moreby studying what people do in realtime rather than by studyingwhat they rememberdoing. And when we do telland heartales of triumph and despair,if we wantto learndespite our biases,we mightlook forfailures embedded in success storiesand successesembedded in failure stories. This could mean, forexample, that we shouldlook backat whatEnron and WorldComdid rightrather than what theydid wrong!
Take a Neutral, Dispassionate Approach to Ideologiesand Theories Ideologyis amongthe more widespread, potent, and vexingimpediments to usingevidence-based management. People routinely ignore evidence about managementpractices that clash with their political convictions or idiosyncratic personalhistories. Simon and Garfunkelwere right when they sang, "A man hearswhat he wantsto hearand disregardsthe rest." Academics and other thoughtleaders may worship and believein theirown theoriesso ferventlythat it rendersthem incapable of learning from new evidence.This happens partly becausepeople "see whatthey believe." It also happensbecause theories can becomeself-fulfilling: when we act as our pettheories suggest we should,we can producethe very behavior we expectin ourselvesand thosearound us. So, ifwe expectpeople to be untrustworthy,we willclosely monitor their behavior, whichmakes it impossibleto developtrust. After all, how can I knowif you can be trustedunless I provideyou an opportunityto show you can be trusted?And experimentalevidence shows that when people are placedin situationswhere authorityfigures expected them to cheat,more of them do in factcheat.47 Manyeconomists, including Gary Becker and OliverWilliamson, believe thathuman beings act primarilyto enhancetheir individual self-interest. The assumptionthat people are wiredto be selfishisn't just used to explainthe powerof financial incentives; economists use it to explainwhy people fall in love,get married (including why they may or maynot prefer polygamy), and have children.Yet these economists ignore evidence that being selfish - or not-
94 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
is learned,not a hardwiredhuman trait.48 Indeed, economists appear to teach studentsto be selfish.Research by Cornell'sRobert Frank and his colleagues showedthat the percentage of students choosing unethical options on an hon- estytest increased dramatically among students taking microeconomics courses, butnot among students in astronomyclasses. Other researchers asked students to recommenda plumberfor a filmclub from offers that varied by how much moneythe plumber charged and how muchmoney the student would get if the recommendedplumber were hired. They found that "economists are morecor- ruptiblethan others." Such studiessuggest that self-interest is learned and varies widelyacross people, groups, and countries.Yet most economic theories still reston theassumption that everyone is selfish.The lessonis thateven people trainedto do rigorousresearch are stillprone to rejectand ignoreevidence that contradictstheir most precious beliefs.49 Anotherexample of how ideologytrumps evidence comes from educa- tion.At the moment, U.S. schoolsare applyingand enforcingthe most stringent performancestandards ever, especially holding students, teachers, and adminis- tratorsaccountable for performance. It is hardto argueagainst high standards. Butwhen an ideal is appliedwithout regard to evidenceabout which tough practiceswork and whichdon't, performance suffers. For instance, there is great pressureto end socialpromotion, advancing a childto thenext grade even ifhis or herwork isn't up to par- in otherwords, to startflunking more kids who don'tmeet certain standards. Social promotion was endedin Chicagoin 1996 at MayorDaley's urging, in New Yorkin 1999 at MayorGiuliani's insistence, and in numerousother cities including Baltimore and Philadelphia.Former president BillClinton's 1999 Stateof the Union address called for an end to socialpromo- tion,because "when we promotea childfrom grade to gradewho hasn'tmas- teredthe work, we do thechild no favors."And President Bush is just as vehementas Clintonwas abouteliminating social promotion. On thesurface, it is hardto arguewith Clinton's conclusion. But it is a half-truth.Clinton's statement is partlyright because when students fail to meet minimumstandards during the school year, going to summerschool before startingthe next grade may enhance test scores. And advancing kids who can't performat thenext grade level certainly creates frustration and maylower moti- vationof kids who putforth the effort to meetstandards. But endingsocial pro- motionharms students and schools,and thestrongest negative effects are found in thebest, most rigorous studies. At least 55 studiesshow that when flunked studentsare comparedto sociallypromoted students, flunked students perform worseand dropout ofschool at higherrates. One ofthe most careful studies foundthat, after controlling for numerous alternative explanations including race,gender, family income, and schoolcharacteristics, students held back one gradewere 70 percentmore likely to dropout ofhigh school. Holding students backalso leavesschools crowded with older students, and costsskyrocket as moreteachers and otherresources are neededbecause the average student spendsmore years in school.50Clinton, Bush, Giuliani, and manyother politi- ciansignored the overwhelming evidence that ending social promotion would
CALIFORNIAMANAGEMENT REVIEW VOL48.NQ3 SPRING2006 95 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
fail,even thoughrenowned educational researcher Robert Hauser warned in 1999: "We shouldknow that a new policyworks before trying it out on a large scale.In itsplan to end socialpromotion the administration appears to have [included]... an enforcementprovision - flunkingkids by the carload lot - aboutwhich the great mass of evidence is stronglynegative. And thispolicy willhurt poor and minoritychildren most of all."51 Hauser's predictionscame true.New Yorkeliminated social promotion in 1999,but reinstated it in 2002 becausethe numbers of holdovers mounted to 43,000 and reacheda disastrous100,000 by 2004. Thisdrove up costs,forcing theelimination of numerous programs, including those for helping underachiev- ers.There was also no evidencethat flunking students helped them learn more. Thiswas nota new lesson:the exact same problemshappened when New York Cityended social promotion 20 yearsearlier.52 Yet as ofthis writing, Mayor Bloombergis once againmoving to eliminatesocial promotion in New York. Whycan't smart people like Mayor Bloomberg and MayorDaley learn fromsuch clear evidence? Because learningis difficultwhen leaders or anyone else is drivenby ideology rather than evidence.
Wisdom: The Most ImportantThing The guidelinesdeveloped here can helpmanagers and theiradvisers to do a betterjob evaluatingand applyingbusiness knowledge. But somethingelse, somethingbroader, is moreimportant than any singleguideline for reaping the benefitsof evidence-based management: the attitude people have towardbusi- nessknowledge. The idea thatwisdom is reflectedin theattitude people have towardwhat they know, not in how muchor how littlethey know, goes back at leastto Plato'swritings. Plato described Socrates' visit to "a man witha high reputationfor wisdom." Socrates "gave a thoroughexamination of this person" and concluded:
[I] formedthe impression that although in manypeople's opinion, and especially his own,he appearedto be wise,in fact,he was not ... I reflectedas I walked away,well I certainlyam wiserthan this man. It is onlytoo likelythat neither of us has anyknowledge to boastof, but he thinksthat he knowssomething which he does notknow, whereas I am quiteconscious of my ignorance. At any rate it seemsthat I am wiserthan him to thissmall extent, that I do notthink that I knowwhat I do notknow.
The powerof Plato's ancient insight persists because wisdom (or thelack ofit) shapeshow peoplethink, feel, and act in so manyways. Psychologists includingJohn Meacham and RobertSternberg have studiedthe nuances of wisdom,especially how beingwise is differentfrom being smart. One oftheir mostimportant insights is that,as Plato'squote suggests,wisdom means "know- ingwhat you knowand knowingwhat you don'tknow," especially striking a balancebetween arrogance (assuming you knowmore than you do) and insecu- rity(believing that you knowtoo littleto act). Thisattitude enables people to act
96 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 Management Half-Truthsand Nonsense: How to Practice Evidence-Based Management
on theirpresent knowledge while doubting what they know. It meansthey can do thingsnow, as well as keeplearning along the way.53 The attitudeof wisdom pervades the standards we've proposedhere for consumersand purveyorsof business knowledge. To illustrate,we urgedpeople to compensatefor their limited knowledge by building on old ideas and to join communitiesof smart people rather than relying only on theirown insights. Ourperspective implies that the unacknowledged ignorance or arrogance behindmost apparent breakthroughs stems from an absenceof wisdom. And we emphasizedthat embracing an ideologyor theorycan be comforting,but can makeit difficultfor devotees to acknowledgeflaws in theirideas - letalone to seek,accept, and learnfrom criticism. In essence,practicing evidence-based managementmeans adopting beliefs and designingsettings that enable people to keepacting with knowledge while doubting what they know, and to openly acknowledgethe imperfections in even theirbest ideas alongthe way. To turnto evidence-basedmedicine, pioneer David Sackett'sreaction to theaccolades he has receiveddemonstrates the attitude of wisdom. He worries openlythat people will treat his ideas as gospelrather than an initialeffort that mustbe constantlyrevised and challenged.Sackett even despiseswhen people call himan experton evidence-basedmedicine since "[he] dislikesthe concept ofexperts and expertisebecause so-called experts can't help but be biased towardtheir own publishedviews." Dr. Sacketthas gone to extremelengths to conveythat he is nota lone genius,that he was partof a teamat McMasterUni- versitythat developed modern evidence-based medicine, and so doesn'tdeserve to be singledout as theguru. Sackett refused an invitationto be inductedinto theCanadian Medical Hall ofFame, and onlyrelented after they allowed him to acceptthe honor on behalfof all ofhis colleagues.Sackett's words and deedsnot onlydemonstrate the right attitude for sparking constant improvement in med- ical knowledge,they provide a modelfor management researchers and writers.54 Unfortunately,there are fewDavid Sackettsin themanagement idea - marketplace.Modesty is in shortsupply and absolutesabound in recommen- dationsas to whatto do, in conclusionsabout what affects individual and orga- nizationalperformance, and in beliefsabout what is trueand whatis false.There is littlewonder then that half-truths fill management lore, causing all sortsof problemsfor those who are seducedby their appeal. That's why investigating, understanding,and decidingwhat to do aboutsome ofthe most important dan- geroushalf-truths is so crucialfor implementing evidence-based management.
Notes 1. AlexTaylor III, "HowToyota Defies Gravity," Fortune, December 8, 1997,pp. 100-108; Michael N. Kennedy and Allen Ward, ProductDevelopment for the Lean Enterprise:Why Toyota's SystemIs Four TimesMore Productive and How YouCan ImplementIt (Richmond,VA: Oaklea Press,2003). 2. KevinPatterson, "What Doctors Don't Know (AlmostEverything)," New York Times Magazine,May 5, 2002,p. 77. 3. Rakesh Khurana, Searchingfor the Corporate Savior: The Irrational Quest for Charismatic CEOs (Princeton,NJ: Princeton University Press, 2002).
CALIFORNIA MANAGEMENT REVIEW VOL 48, NO.3 SPRING 2006 97 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
4. This estimateis based on counts of business magazines and newspapers listed on Yahoo.com fora search done in October 2002. 5. This estimateis based on informationprovided on barnesandnoble.comand Amazon .com. 6. PeterMeyers, "CrankyConsumer: RatingBusiness-Book Summaries," WallStreet Journal, November 5, 2002. 7. Perseus Publishing,Business: The UltimateResource (Cambridge, MA: Perseus, 2002). 8. Compare, forinstance, the recommendationsthat come fromJim Collins, Goodto Great (New York,NY: HarperCollins,2001) with those fromTony Alessandra, Charisma:Seven Keys to Developingthe Magnetism That Leads toSuccess (New York,NY: Warner,1998). 9. Bill Jensen,Simplicity: The New CompetitiveAdvantage in a Worldof More, Better, Faster (New York,NY: Perseus, 2001); RobertAxelrod and Michael D. Cohen, HarnessingComplexity: OrganizationalImplications of a ScientificFrontier (New York,NY: Free Press,2000). 10. RobertS. Kaplan and David P. Norton,The Strategy-Focused Organization (Boston, MA: Har- vard Business School Press,2000); HenryMintzberg, The Rise and Fall ofStrategic Planning (New York,NY: Free Press, 1993). 1 1. On small New York- and San Francisco-based consultingfirms specializing in organizational and culturalchange and strategyimplementation, The TriumGroup does do follow-up surveysto see if changes in knowledge, attitudes,behavior, and resultscome fromits work with clients.Also, the firmhas a three-tierpricing model in which clientsvoluntarily decide whetherto pay more ifthe projecthas exceeded theirgoals and expectations,an agreed upon amount ifit has met objectives,or less ifthe project did not deliver all that was expected. This providesan incentivefor the company to do good work, not just sell busi- ness. 12. Darrell Rigby,"Management Tools and Techniques: A Survey,"California Management Review, 43/2 (Winter2001): 139-160. 13. JackWelch quote fromWorkforce Management magazine, available on line at
98 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006 Management Half-Truthsand Nonsense: How to Practice Evidence-Based Management
Daniel J. Brass, "'EmployeeReactions to an Open-Plan Office:A NaturallyOccurring Quasi- Experiment/Administrative Science Quarterly, 24 (1979): 267-284. 24. See
CALIFORNIA MANAGEMENT REVIEW VOL. 48, NO.3 SPRING 2006 99 ManagementHalf-Truths and Nonsense: How to PracticeEvidence-Based Management
46. BarryM. Staw, "Attributionof the 'Causes' of Performance:An AlternativeInterpretation of Cross-SectionalResearch on Organizations/ OrganizationalBehavior and Human Performance, 13 (1975): 414-432. 47. RobertB. Cialdini,Influence: Säence and Practice,4th ed. (Boston, MA: Allyn & Bacon, 2001 ). 48. D. T. Miller,"The Norm of Self-interest,"American Psychologist, 54 (1999): 1053-1060. 49. B. Frank and G.G. Schulze, "Does Economics Make Citizens Corrupt?"Journal of Economic Behavior& Organization,43 (2000): 101-113; H. Frank,T.D. Gilovich,and D.T. Regan, "Does StudyingEconomics InhibitCooperation?" Journalof Economic Perspectives, 7 (1993): 159-171. 50. This researchis summarizedin RobertM. Hauser, "What If We Ended Social Promotion?" EducationWeek, April 7, 1999, pp. 64-66. 51. Ibid. 52. SherylMcCarthy, "Schools Repeat Social PromotionProblems," Newsday, March 28, 2002. 53. See JohnA. Meacham, "The Loss of Wisdom," in RobertJ. Sternberg,ed., Wisdom:Its Nature, Origins,and Development(Cambridge, England: CambridgeUniversity Press, 1990), 181-211; JohnA. Meacham, "Wisdom and the Context of Knowledge: Knowing What One Doesn't Know One Doesn't Know," in D. Huhn and J. A. Meacham, eds., On theDevelopment of Devel- opmentalPsychology (Basel, Switzerland:Krager, 1983), 111-134; see RobertJ. Sternberg, "ImplicitTheories of Intelligence,Creativity, and Wisdom,"Journal of Personality and Social Psychology,49 (1985): 607-627. 54. GabrielleBauer, "A ReluctantPolicy Wonk: Dr. David Sackett,a Pioneer in Evidence-Based Medicine, Has No Time to Play King, but the World Seems Intenton CrowningHim," The MedicalPost, August 22, 2002.
100 UNIVERSITY OF CALIFORNIA, BERKELEY VOL 48, NO. 3 SPRING 2006