PerformanceComparisonofDynamicWebPlatforms TonyHansen,VarshaMainkarandPaulReeser AT&TLabs 200LaurelAve,RoomD5-3C12,Middletown,NJ07748USA Phone1732-420-3714,Fax1732-368-1919 Email{tony,mainkar,preeser}@att.com Keywords:Dynamic,Web,CGI,FastCGI,C++,Java,Servlets,JSP,Performance,Comparison Over the last few years, the World Wide Web has transformed itself from a static content-distribution mediumtoaninteractive,dynamicmedium.TheWebisnowwidelyusedasthepresentationlayerfora hostofon-lineservicessuchase-mailandaddressbooks,e-cards,e-calendar,shopping,banking,and stock trading. As a consequence, HTML files are now typically generated dynamically after the server receives the request. From the Web-site providers’ point of view, dynamic generation of HTML pages impliesalesserunderstandingoftherealcapacityandperformanceoftheirWebservers.FromtheWeb developers’ point of view, dynamic content implies an additional technology decision: the Web programmingtechnologytobeemployedincreatingaWeb-basedservice.SincetheWebisinherently interactive,performanceisakeyrequirement,andoftendemandscarefulanalysisofthesystems.Inthis paper,wecomparefourdynamicWebprogrammingtechnologiesfromthepointofviewofperformance. Thecomparisonisbasedtestingandmeasurementoftwocases:oneisacasestudyofarealapplication thatwasdeployedinanactualWeb-basedservice;theotherisatrivialapplication.Thetwocasesprovide uswithanopportunitytocomparetheperformanceofthesetechnologiesattwoendsofthespectrumin terms of complexity. Our focus in this paper is on how complex vs. simple applications perform when implementedusingdifferentWebprogrammingtechnologies.Thepaperdrawscomparisonsandinsights basedonthisdevelopmentandperformancemeasurementeffort. 1. INTRODUCTION The World Wide Web (WWW) first emerged a decade ago as a medium to render hypertext documentsthatwerestoredontheInternet,onauser’scomputer,usingspecialsoftware(thebrowser) andanewprotocol(HTTP:HyperTextTransferProtocol).Forthefirstfewyears,theWWWgrewprimarily asanewmediuminwhichstaticcontentcouldbepublished,andinformationshared.Thecontentwas publishedintheformofHTML(HyperTextMarkupLanguage)files,whichwereservedbyWebservers, on requests from browsers. However, over the last few years the WWW has transformed itself from a staticcontent-distributionmediumtoaninteractive,dynamicmedium.ContentontheWebisnowoften personalized,andthereforedynamicallygenerated.TheWebisnowwidelyusedasthepresentationlayer forahostofon-lineservicessuchase-mail,e-cards,e-calendar,andaddressbooks,shopping,banking, andstocktrading.Asaconsequence,theHTMLfilesthatarerenderedbytheclient’sbrowserarenow typicallygenerateddynamicallyaftertheWebserverhasprocessedtheuser’srequest. This dynamic generation of HTML files has nothappenedwithoutanassociatedperformancecost. JustwhenInternetusersweregettingaccustomedto“click-and-wait”ondial-uplinesduetographics-rich 1 Web-sites,dynamicallygeneratedcontentstartedproliferatingontheWeb.Nowusersmustwaitnotonly for the network delay but also for the server-side processing delay associated with serving a request dynamically.Inmanycases,thisisturningouttobethelargestcomponentofthedelay.FromtheWeb- siteproviders’pointofview,dynamicgenerationofHTMLpagesimpliesalesserunderstandingofthereal capacity of their Web servers. The vendor-provided “hits-per-second” capacity of the Web server is no longerenough,asthisonlypertainstostaticHTMLfiles. From the Web developers’ point of view, dynamic Web content implies an additional technology decision:theWebprogrammingtechnologytobeemployedincreatingaWeb-basedserviceorproduct. This decision is based on several factors. Among the factors considered are ease of programming, richnessoffeatures,maintainability,reliability,andperformance.SincetheWebisinherentlyinteractive, performanceisakeyrequirement,andoftendemandscarefulanalysisofthesystems. In this paper, we compare the performance of four Web programming technologies, namely Java Servlets,JavaServerPages,CGI/C++andFastCGI/C++.Thecomparisonisbasedontwocases:oneis acasestudyofacomplexapplicationthatwasdeployedinanactualWeb-basedservice;theotherisa “trivial” application. The methodology of performance analysis was stress testing and measurement. Performance measurement (rather than modeling) made most sense in this effort, since a quick turnaroundofresultswasnecessaryandtheaccuracyofresultswasrequiredtobehigh. The two cases (i.e., the complex and the trivial) provided us with an opportunity to compare performance of these technologies attwoendsofthespectrumofapplications,intermsofcomplexity. Theperformance“order”ofdifferenttechnologiesisseldomabsolute–itdependsgreatlyonthenatureof the application. Our focus in this paper is on how complex vs. simple applications perform when implementedusingdifferentWebprogrammingtechnologies.Thepaperdrawscomparisonsandinsights basedonthisdevelopmentandperformancemeasurementeffort. Themainobservationsfromthisworkwereasfollows:Ingeneral,FastCGIoutperformedCGI,while JSP outperformed Java servlets. In the case of a complex application, the CGI-based technologies outperformed the Java-based technologies by 3-4x, with Java performance limited by software bottlenecksintheJVM.Inthecaseofatrivialapplication,therelativeperformancewasreversed,asJava outperformedCGIby2-3x. 2 Therestofthepaperisorganizedasfollows:inSection2weprovidethemotivationforconducting such a comparative study, and in Section 3 we describe briefly the technologies that we compared. Section 4 describes the performance measurement and analysismethodology,Section5describesthe case-study testing results in detail, and Section 6 describes the results of testing a trivial application. Finally,Section7summarizesourresults,andSection8providessomeconcludingremarks. 2. MOTIVATION TheapplicationcontextforthisstudywasanewWeb-basedmessagingservice.Theeffortcentered on the development of a “page generation engine” to perform dynamic Web page construction in a distributedenvironment(seeFigure2-1).Givenacceleratedtime-to-marketgoalsandlimiteddevelopment time,thenaturalchoiceoftechnologywastheoneperceivedtobepowerful,feature-rich,andyeteasyto useanddeploy–Javaservlets. The distributed execution environment consists of a front-end sub-system running a standard Web server and a Java virtual machine (JVM) servlet engine, plus a back-end sub-system of servers implementing the businesslogicanddata(e.g.,POP/IMAPmailservers,SMTPgateways,LDAP/X.500 directoryservers,etc.).ThepageengineconsistsofanumberofWebpagetemplatesandJavaservlets. TheWebpagetemplatesareHTMLfiles,butwithtagsfromaproprietaryscriptinglanguagethatspecify howtocollectdynamicdatathatcustomizestheseHTMLfilestotheparticularrequest.Collectively,these servlets: 3 ••• ••• LOGIC DATA HTTPTHREADS ServletTHREADS APPLICATION1 ••• WEBPAGE POP TEMPLATES SERVLETS IMAP SMTP LOGIC LDAP DATA X.500 WEBSERVER JAVAVM APPLICATION2 FRONT-ENDSERVER BACK-ENDSERVER Figure2-1:SystemArchitecture • parsetherequestedtemplateforscriptinglanguage(dynamic)content, • issuetheappropriateback-endrequeststoprocessthebusinesslogicorretrievetherequireddata, • processthereturneddataintothenecessarydatastructures, • performprotocolconversion(asnecessary),and • constructtheWebpagetoreturntotheuser. Theprotocol/languagebetweentheend-userandtheWebserverisHTTP/HTML,andthatbetween theJavaservletsandtheback-endisdeterminedbytheparticularapplication(e.g.,IMAP,LDAP,etc.). Within the JVM environment, data is passed between the various servlets as eXtensible Markup Language (XML) data structures. Hence, the scripts perform a variety of protocol and language conversionsduringthecourseofrequestexecution. 2.1. InitialMeasurements Weconductedaseriesofstresstestsusingacommercialloaddrivertogeneraterepeatedrequests totheservletengineatvariouslevelsofconcurrency(simulatedusers).Thetestconfigurationconsisted ofaWindowsNTserverrunningtheloadgenerationscripts(driver),aSolarisserverrunningthefront-end software,andaSolarisserverrunningtheback-endapplication(forthistest,aPOP3/IMAP4mailserver andanLDAPdirectoryserver).Hardware(CPU,memory,disk,I/O)andsoftwareresourceconsumptions weremeasuredonallmachines.Inaddition,end-to-enduser-perceivedresponsetimesweremeasured. 4 Thedriverscriptsemulatedaprescribednumberofconcurrentusersrepeatedlygeneratingthesame request(e.g.,readmessage,sendmessage,etc.).Thenumberofconcurrentsimulateduserswasvaried from1to20.Thenumberofrepeatedrequestsperuserateachconcurrencylevel(typically2000)was sufficient to achieve statistical stability. The tests wererunin“stress”mode;thatis,assoonasauser receivesaresponse,itimmediatelysubmitsthenextrequest(i.e.,withnegligibleclientdelay).Eachofthe N simulated users does so independently and in parallel. As a result, the concurrency level (i.e., the numberofrequestsinthesystem)equalsNatalltimes. The results of the stress tests for a particular request type (read a 20KB message) are shown in Figures2-2and2-3.Inparticular,Figure2-2plotstheend-to-endresponsetimeontheleft-handaxis, andthefront-endCPUutilizationontheright-handaxis,asafunctionoftheconcurrencylevel.Ascanbe seen,theresponsetimecurvebeginstoridealongalinearasymptote(shownbythedottedline)afteronly 7concurrentusers.Thatis,responsetimeincreasesproportionallywiththenumberofusers,indicating saturation in a closed system [1]. Additionally, CPU utilization levels off after 11 users at 65-70% (indicatinganon-CPUsystembottleneck). 1 0 0 R e s p o n s e t i m e 9 0 C P U u t i l i z a t i o n 8 0 E M I 7 0 ) T % ( E S N N 6 0 O I O T P A S 5 0 Z E I L R I T d e 4 0 U z i U l a P C m 3 0 r o N 2 0 1 0 0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1