SeriesSeriesSeries ISSN: ISSN:ISSN: 1935-3235 1935-32351935-3235 BARROSO • CLIDARAS • HÖLZLE BARROSO • CLIDARAS • HÖLZLE BARROSO • CLIDARAS • HÖLZLE SSSYNTHESISYNTHESISYNTHESIS LLLECTURESECTURESECTURES ONONON MMM MorganMorganMorgan& & & ClaypoolClaypoolClaypool PublishersPublishersPublishers CCCOMPUTEROMPUTEROMPUTER AAARCHITECTURERCHITECTURERCHITECTURE &&&CCC SeriesSeriesSeries Editor: Editor:Editor: MarkMarkMark D.D.D. Hill,Hill,Hill, UniversityUniversityUniversity of ofof Wisconsin WisconsinWisconsin TheTheThe DatacenterDatacenterDatacenter asasas aaa ComputerComputerComputer AnAnAn IntroductionIntroductionIntroduction tototo thethethe DesignDesignDesign ofofof Warehouse-ScaleWarehouse-ScaleWarehouse-Scale Machines,Machines,Machines, SecondSecondSecond EditionEditionEdition TheTheThe DatacenterDatacenterDatacenter LuizLuizLuiz André AndréAndré Barroso, Barroso,Barroso, J JJimmyimmyimmy Clidaras, Clidaras,Clidaras, and andand Urs UrsUrs Hölzle, Hölzle,Hölzle, Google, Google,Google, Inc. Inc.Inc. THE DATACENTER AS A COMPUTER, SECOND EDITION AS A COMPUTER, SECOND EDITION THE DATACENTER AS A COMPUTER, SECOND EDITION THE DATACENTER THE DATACENTER AsAsAs computation computationcomputation continues continuescontinues to toto move movemove into intointo the thethe cloud, cloud,cloud, the thethe computing computingcomputing platform platformplatform of ofof interest interestinterest no nono longer longerlonger resembles resemblesresembles a aa pizzapizzapizza box boxbox or oror a aa refrigerator, refrigerator,refrigerator, but butbut a aa warehouse warehousewarehouse full fullfull of ofof computers. computers.computers. These TheseThese new newnew large largelarge datacenters datacentersdatacenters are areare quite quitequite different differentdifferent asasas aaa ComputerComputerComputer fromfromfrom traditional traditionaltraditional hosting hostinghosting facilities facilitiesfacilities of ofof earlier earlierearlier times timestimes and andand cannot cannotcannot be bebe viewed viewedviewed simply simplysimply as asas a aa collection collectioncollection of ofof co-located co-locatedco-located servers.servers.servers. Large LargeLarge portions portionsportions of ofof the thethe hardware hardwarehardware and andand software softwaresoftware resources resourcesresources in inin these thesethese facilities facilitiesfacilities must mustmust work workwork in inin concert concertconcert to toto AnAnAn IntroductionIntroductionIntroduction tototo thethethe DesignDesignDesign efficientlyefficientlyefficiently deliver deliverdeliver good goodgood levels levelslevels of ofof Internet InternetInternet service serviceservice performance, performance,performance, something somethingsomething that thatthat can cancan only onlyonly be bebe achieved achievedachieved by byby a aa holistic holisticholistic approachapproachapproach to toto their theirtheir design designdesign and andand deployment. deployment.deployment. In InIn other otherother words, words,words, we wewe must mustmust treat treattreat the thethe datacenter datacenterdatacenter itself itselfitself as asas one oneone massive massivemassive warehouse-scalewarehouse-scalewarehouse-scale computer computercomputer (WSC). (WSC).(WSC). We WeWe describe describedescribe the thethe architecture architecturearchitecture of ofof WSCs, WSCs,WSCs, the thethe main mainmain factors factorsfactors influencing influencinginfluencing their theirtheir ofofof Warehouse-ScaleWarehouse-ScaleWarehouse-Scale MachinesMachinesMachines design,design,design, operation, operation,operation, and andand cost costcost structure, structure,structure, and andand the thethe character charactercharacteristicsisticsistics of ofof their theirtheir software softwaresoftware base. base.base. We WeWe hope hopehope it itit will willwill be bebe useful usefuluseful tototo architects architectsarchitects and andand programmers programmersprogrammers of ofof today’s today’stoday’s WSCs, WSCs,WSCs, as asas we wewellllll as asas those thosethose of ofof future futurefuture many-core many-coremany-core platforms platformsplatforms which whichwhich may maymay oneoneone day dayday implement implementimplement the thethe equivalent equivalentequivalent of ofof today’s today’stoday’s WSCs WSCsWSCs on onon a aa single singlesingle board. board.board. SecondSecondSecond EditionEditionEdition NOTESNOTESNOTES FOR FORFOR THE THETHE SECOND SECONDSECOND EDITION EDITIONEDITION AfterAfterAfter nearly nearlynearly four fourfour years yearsyears of ofof substantial substantialsubstantial academic academicacademic and andand industrial industrialindustrial developments developmentsdevelopments in inin warehouse-scale warehouse-scalewarehouse-scale computing, computing,computing, wewewe are areare delighted delighteddelighted to toto present presentpresent our ourour first firstfirst major majormajor update updateupdate to toto this thisthis lecture. lecture.lecture. The TheThe increased increasedincreased popularity popularitypopularity of ofof public publicpublic clouds cloudsclouds hashashas made mademade WSC WSCWSC software softwaresoftware techniques techniquestechniques relevant relevantrelevant to toto a aa larger largerlarger pool poolpool of ofof programmers programmersprogrammers since sincesince our ourour first firstfirst edition. edition.edition. Therefore, Therefore,Therefore, LuizLuizLuiz AndréAndréAndré BarrosoBarrosoBarroso wewewe expanded expandedexpanded Chapter ChapterChapter 2 22 to toto reflect reflectreflect our ourour better betterbetter understanding understandingunderstanding of ofof WSC WSCWSC software softwaresoftware systems systemssystems and andand the thethe toolbox toolboxtoolbox of ofof softwaresoftwaresoftware techniques techniquestechniques for forfor WSC WSCWSC programming. programming.programming. In InIn Chapter ChapterChapter 3, 3,3, we wewe added addedadded to toto our ourour coverage coveragecoverage of ofof the thethe evolving evolvingevolving landscape landscapelandscape JimmyJimmyJimmy ClidarasClidarasClidaras ofofof wimpy wimpywimpy vs. vs.vs. brawny brawnybrawny server serverserver trade-offs, trade-offs,trade-offs, and andand we wewe now nownow present presentpresent an anan overview overviewoverview of ofof WSC WSCWSC interconnects interconnectsinterconnects and andand storage storagestorage systemssystemssystems that thatthat was waswas promised promisedpromised but butbut lacking lackinglacking in inin the thethe original originaloriginal edition. edition.edition. Thanks ThanksThanks largely largelylargely to toto the thethe help helphelp of ofof our ourour new newnew co-author, co-author,co-author, UrsUrsUrs HölzleHölzleHölzle GoogleGoogleGoogle Distinguished DistinguishedDistinguished Engineer EngineerEngineer Jimmy JimmyJimmy Clidaras, Clidaras,Clidaras, the thethe material materialmaterial on onon facility facilityfacility mechanical mechanicalmechanical and andand power powerpower distribution distributiondistribution designdesigndesign hashashas beenbeenbeen updatedupdatedupdated andandand greatlygreatlygreatly extendedextendedextended (see(see(see ChaptersChaptersChapters 444 andandand 5).5).5). ChaptersChaptersChapters 666 andandand 777 havehavehave alsoalsoalso beenbeenbeen revampedrevampedrevamped significantly. significantly.significantly. We WeWe hope hopehope this thisthis revised revisedrevised edition editionedition continues continuescontinues to toto meet meetmeet the thethe needs needsneeds of ofof educators educatorseducators and andand professionals professionalsprofessionals ininin this thisthis area. area.area. ABOUTABOUTABOUT SYNTHESISSYNTHESISSYNTHESIS MORGANMORGANMORGAN ThisThisThis volume volumevolume is isis a aa printed printedprinted version versionversion of ofof a aa work workwork that thatthat appears appearsappears in inin the thethe Synthesis SynthesisSynthesis DigitalDigitalDigital Library LibraryLibrary of ofof Engineering EngineeringEngineering and andand Computer ComputerComputer Science. Science.Science. Synthesis SynthesisSynthesis Lectures LecturesLectures provideprovideprovide concise, concise,concise, original originaloriginal presentations presentationspresentations of ofof important importantimportant research researchresearch and andand development developmentdevelopment & & & topics,topics,topics, published publishedpublished quickly, quickly,quickly, in inin digital digitaldigital and andand print printprint formats. formats.formats. For ForFor more moremore information informationinformation CLAYPOOLCLAYPOOLCLAYPOOL visitvisitvisit www.morganclaypool.com www.morganclaypool.comwww.morganclaypool.com ISBN:ISBN:ISBN: 978-1-62705-009-8978-1-62705-009-8978-1-62705-009-8 999000000000000 SSSYNTHESISYNTHESISYNTHESIS LLLECTURESECTURESECTURES ONONON MorganMorganMorgan & & & ClaypoolClaypoolClaypool PublishersPublishersPublishers CCCOMPUTEROMPUTEROMPUTER AAARCHITECTURERCHITECTURERCHITECTURE www.morganclaypool.comwww.morganclaypool.comwww.morganclaypool.com 999777888111666222777000555000000999888 MarkMarkMark D.D.D. Hill,Hill,Hill, SeriesSeriesSeries EditorEditorEditor The Datacenter as a Computer An Introduction to the Design of Warehouse-Scale Machines Second Edition iii Synthesis Lectures on Computer Architecture Editor Mark D. Hill, University of Wisconsin, Madison Synthesis Lectures on Computer Architecture publishes 50- to 100-page publications on topics pertaining to the science and art of designing, analyzing, selecting, and interconnecting hardware components to create computers that meet functional, performance, and cost goals. The scope