An Open Testbed for Realistic Benchmarking of Web Applications

Total Page:16

File Type:pdf, Size:1020Kb

Load more

BenchLab: An Open Testbed for Realistic Benchmarking of Web Applications Emmanuel Cecchet, Veena Udayabhanu, Timothy Wood, Prashant Shenoy University of Massachusetts Amherst {cecchet,veena,twood,shenoy}@cs.umass.edu Abstract Web applications have evolved from serving static content to dynamically generating Web pages. Web 2.0 applica- tions include JavaScript and AJAX technologies that manage increasingly complex interactions between the client and the Web server. Traditional benchmarks rely on browser emulators that mimic the basic network functionality of real Web browsers but cannot emulate the more complex interactions. Moreover, experiments are typically conduct- ed on LANs, which fail to capture real latencies perceived by users geographically distributed on the Internet. To address these issues, we propose BenchLab, an open testbed that uses real Web browsers to measure the perfor- mance of Web applications. We show why using real browsers is important for benchmarking modern Web applica- tions such as Wikibooks and demonstrate geographically distributed load injection for modern Web applications. 1. Introduction vendors. CloudStone [15] is a recently proposed open- source cloud/Web benchmark that addresses some of Over the past two decades, Web applications have the above issues; it employs a modern Web 2.0 applica- evolved from serving primarily static content to com- tion architecture with load injectors relying on a Mar- plex Web 2.0 systems that support rich JavaScript and kov model to model user workloads. Cloudstone, how- AJAX interactions on the client side and employ so- ever, does not capture or emulate client-side JavaScript phisticated architectures involving multiple tiers, geo- or AJAX interactions, an important aspect of today’s graphic replication and geo-caching on the server end. Web 2.0 applications and an aspect that has implica- From the server backend perspective, a number of Web tions on the server-side load. application frameworks have emerged in recent years, In this paper, we present BenchLab, an open testbed for such as Ruby on Rails, Python Django and PHP Cake, realistic Web benchmarking that addresses the above that seek to simplify application development. From a drawbacks. BenchLab’s server component employs client perspective, Web applications now support rich modern Web 2.0 applications that represent different client interactivity as well as customizations based on domains; currently supported server backends include browser and device (e.g., laptop versus tablet versus Wikibooks (a component of Wikipedia) and Cloud- smartphone). The emergence of cloud computing has Stone’s Olio social calendaring application, with sup- only hastened these trends---today’s cloud platforms port for additional server applications planned in the (e.g., Platform-as-a-Service clouds such as Google Ap- near future. BenchLab exploits modern virtualization pEngine) support easy prototyping and deployment, technology to package its server backends as virtual along with advanced features such as autoscaling. appliances, thereby simplifying the deployment and To fully exploit these trends, Web researchers and de- configuration of these server applications in laboratory velopers need access to modern tools and benchmarks clusters and on public cloud servers. BenchLab sup- to design, experiment, and enhance modern Web sys- ports Web performance benchmarking “at scale” by tems and applications. Over the years, a number of leveraging modern public clouds---by using a number Web application benchmarks have been proposed for of cloud-based client instances, possibly in different use by the community. For instance, the research com- geographic regions, to perform scalable load injection. munity has relied on open-source benchmarks such as Cloud-based load injection is cost-effective, since it TPC-W [16] and RUBiS [13] for a number of years; does not require a large hardware infrastructure and however these benchmarks are outdated and do not also captures Internet round-trip times. In the design of fully capture the complexities of today’s Web 2.0 ap- BenchLab, we make the following contributions: plications and their workloads. To address this limita- We provide empirical results on the need to capture tion, a number of new benchmarks have been proposed, the behavior of real Web browsers during Web load such as TPC-E, SPECWeb2009 or SPECjEnter- injection. Our results show that traditional trace re- prise2010. However, the lack of open-source or freely play methods are no longer able to faithfully emu- available implementations of these benchmarks has late modern workloads and exercise client and serv- meant that their use has been limited to commercial er-side functionality of modern Web applications. features of modern Web applications. These features Based on this insight, we design BenchLab to use include: real Web browsers, in conjunction with automated Multi-tier architecture: Web applications commonly tools, to inject Web requests to the server applica- use a multi-tier architecture comprising at least of a tion. As noted above, we show that our load injec- database backend tier, where persistent state is stored, tion process can be scaled by leveraging inexpensive and a front-end tier, where the application logic is im- client instances on public cloud platforms. plemented. In modern applications, this multi-tier archi- Similar to CloudStone’s Rain [2], BenchLab pro- tecture is often implemented in the form of a Model- vides a separation between the workload model- View-Controller (MVC) architecture, reflecting a simi- ing/generation and the workload injection during lar partitioning. A number of platforms are available to implement such multi-tier applications. These include benchmark execution. Like Rain, BenchLab sup- traditional technologies such as JavaEE and PHP, as ports the injection of real Web traces as well as syn- well as a number of newer Web development frame- thetic ones generated from modeling Web user be- works such as Ruby on Rails, Python Django and PHP havior. Unlike Rain, however, BenchLab uses real Cake. Although we are less concerned about the idio- browsers to inject the requests in these traces to syncrasies of a particular platform in this work, we faithfully capture the behavior of real Web users. must nevertheless pay attention to issues such as the BenchLab is designed as an open platform for real- scaling behavior and server overheads imposed by a istic benchmarking of modern Web applications us- particular platform. ing real Web browsers. It employs a modular archi- Rich interactivity: Regardless of the actual platform tecture that is designed to support different backend used to design them, modern Web applications make server applications. We have made the source code extensive use of JavaScript, AJAX and Flash to enable for BenchLab available, while also providing virtual rich interactivity in the application. New HTML5 fea- appliance versions of our server application and cli- tures confirm this trend. In addition to supporting a rich ent tools for easy, quick deployment. application interface, such applications may incorporate The rest of this document is structured as follows. Sec- functionality such as “auto complete suggestions” tion 2 explains why realistic benchmarking is an im- where a list of completion choices is presented as a user portant and challenging problem. Section 3 introduces types text in a dialog or a search box; the list is contin- BenchLab, our approach to realistic benchmarking uously updated as more text is typed by the user. Such based on real Web browsers. Section 4 describes our functions require multiple round trip interactions be- current implementation that is experimentally evaluated tween the browser and the server and have an implica- in section 5. We discuss related work in section 6 be- tion on the server overheads. fore concluding in section 7. Scaling behavior: To scale to a larger number of users, an application may incorporate techniques such as rep- 2. Why Realistic Benchmarking Matters lication at each tier. Other common optimizations in- A realistic Web benchmark should capture, in some clude use of caches such as memcached to accelerate reasonable way, the behavior of modern Web applica- and scale the serving of Web content. When deployed tions as well as the behavior of end-users interacting on platforms such as the cloud, it is even feasible to use with these applications. While benchmarks such as functions like auto-scaling that can automatically provi- TPC-W or RUBiS were able to capture the realistic sion new servers when the load on existing ones crosses behavior of Web applications from the 1990s, the fast a threshold. paced technological evolution towards Web 2.0 has Domain: Finally, the “vertical” domain of the applica- quickly made these benchmarks obsolete. A modern tion has a key impact on the nature of the server-side Web benchmark should have realism along three key workload and application characteristics. For example, dimensions: (i) a realistic server-side application, (ii) a “social” Web applications incorporate different features realistic Web workload generator that faithfully emu- and experience a different type of workload than say, lates user behavior, and (iii) a realistic workload injec- Web applications in the financial and retail domains. tor that emulates the actual “browser experience.” In Although it is not feasible for us to capture the idiosyn- this section, we describe the key issues that must
Recommended publications
  • NASA-LMCSOC SN and GN Interoperability Testbed

    NASA-LMCSOC SN and GN Interoperability Testbed

    Consolidated Space Operations Contract NASA/Lockheed Martin-CSOC Ground Network and Space Network Interoperability Testbed Lindolfo Martinez & Larry Muzny Lockheed Martin Space Operations Consolidated Space Operations Contract PO Box 58980, Mail Code L1C Houston, TX 77258-8980 (281) 853-3325 [email protected] [email protected] 1.0 Introduction Lockheed Martin-CSOC, a prime contractor for NASA ground systems, has been supporting in the development of plans for the evolution of NASA’s Ground Network (GN) and Space Network (SN), and where possible, synchronizing those plans with plans for the evolution of the Deep Space Network (DSN). Both organizations want those networks to have certain key attributes. Among the desired attributes are: 1. A common interface for all NASA networks to allow users to reduce development and operations costs by using common standards-based system interfaces; 2. Network interoperability to allow the sharing of resources among space agencies and other government agencies; and 3. Provide a common interface for integrating commercial ground network providers and network users. A multi-center and multi-contractor study team led by Lockheed Martin-CSOC concluded that the Space Link Extension (SLE) set of Consultative Committee for Space Data Systems (CCSDS) recommendations has the desired attributes. The team recommended that NASA and Lockheed Martin-CSOC set up an interoperability testbed to show how most future NASA missions can use CCSDS SLE and that SLE could meet NASA’s interoperability goals. NASA’s involvement with SLE started with its support for the development of SLE as the primary interface for future Deep Space Network missions.
  • Using XML, XSL, and CSS in a Digital Library

    Using XML, XSL, and CSS in a Digital Library

    Using XML, XSLT, and CSS in a Digital Library Timothy W. Cole, William H. Mischo, Robert Ferrer, and Thomas G. Habing Grainger Engineering Library Information Center University of Illinois at Urbana-Champaign Abstract The functionality of formats available for the online representation of text continues to evolve. ASCII and bit-mapped image representations of text objects (e.g., journal articles) have been superceded by more functional representations such as application- specific word processing formats and proprietary and non-proprietary information interchange formats (e.g., Adobe PDF, Microsoft RTF, TeX, SGML). Standards like SGML and now XML, which support the representation of a text as an “ordered hierarchy of content objects,” are arguably the best and most sophisticated models available for representing text objects.1 Ratified as an international standard in 1989, SGML has become a well-established approach for encoding text. However SGML’s complexity and requirements for specialized and expensive tools to implement SGML- based systems have limited its scope as an information interchange standard, particularly in today’s Web-dominated environment. XML, established as a W3 Consortium Recommendation in February 1998, strives to make the best features of SGML more accessible to Web authors and publishers, but the XML specification itself doesn't explicitly deal with the presentation of content, nor does it address document object transformation. These issues must be addressed through the use of CSS and XSLT (both more recent W3 Consortium Recommendations). It is necessary to use these three technologies in concert to create powerful and robust text applications on the Web. This paper describes how XML, XSLT, and CSS can be used in a digital library application.
  • Introducing Control System Security Center (Cssc)

    Introducing Control System Security Center (Cssc)

    Control System Security Center INTRODUCING CONTROL SYSTEM SECURITY CENTER (CSSC) Control System Security Center h(p://www.css-center.or.jp/en/index.html CSSC Promotion Video About 8 Minutes If Tokyo city falls into wide-area blackout, ・・・・・・・・ h(p://www.youtube.com/watch?v=qgsevPqZpAg&feature=youtu.be 2 Control System Security Center Where is Tagajo? l Jo = castle; since 8th century l Historically famous and important place in Japan l Tsunami (2-4 m height) caused by the earthquake has covered the 33% of the city land (Mar. 11.2011) l After the earthquake, Tagajo city launched “Research Park for Disaster Reduction” plan. – Internationally prominent effort for achieving disaster reduction – Development of distinct technologies and products – Policies for disaster reduction “The testbed of CSSC truly suits the concept of Research park for disaster reduction.” (Mayor of Tagajo) Source: hp://www.city.tagajo.miyagi.jp/ 3 Control System Security Center Industrial Control System Network Internet Maintenance/services, related factories, sales Office network Firewall Infrastructure Industrial Control System network (factories, building, filter plant, sewage plant, disaster control center) DCS PLC opening/closing valve Monitoring room(SCADA) controlling temperature, Engineering PC pressure and robot Parameter configuraon Evaluaon DCS: Distributed Control System PLC: Programmable Logic Controller SCADA: Supervisory Control And Data AcquisiXon 4 4 Control System Security Center PLC and DCS DCS PLC Usually, a DCS configuration comprises three PLC comprises a combination of PC monitoring and elements: an HMI (Human Machine Interface) control software and performs process monitoring used by the operator for control and monitoring and control. PLC is used, for example, in assembly and a control network that connects the HMI plants or for building control.
  • Software System for the Testbed

    Software System for the Testbed

    European Research 7th Framework Programme Project title: Community Networks Testbed for the Future Internet Software system for the testbed Deliverable number: D2.7 Version 1.0 This project has received funding from the European Union’s Seventh Programme for research, technological development and demonstration under grant agreement No 288535 Project Acronym: CONFINE Project Full Title: Community Networks Testbed for the Future Internet Type of contract: Integrated project (IP) contract No: 288535 Project URL: http://confine-project.eu Editor: Felix Freitag, UPC Deliverable nature: Report (R) Dissemination level: Public (PU) Contractual Delivery Date: September 30, 2015 Actual Delivery Date September 24, 2015 Suggested Readers: Project partners Number of pages: 98 Keywords: WP2, testbed design, developments, requirement analysis, experimental research, community networks, testbed Authors: Roger Baig (Guifi.net), Christoph Barz (FKIE), Christoph Fuchs (FKIE), Jonathan Kirchhoff (FKIE), Julia Niewiejska (FKIE), Bart Braem (iMinds), Emmanouil Dimogerontakis (UPC), Pau Escrich (Guifi.net), Felix Freitag (UPC), Esunly Medina (UPC), Aaron Kaplan (Funkfeuer), Ralf Schlatterbeck (Funkfeuer), Ivan Vilata (Pangea) Peer review: Leandro Navarro, Ester Lopez´ (UPC) Abstract This document reports on the enhancements made in the system architecture, design and development for the CONFINE testbed software system (Community-Lab) done during the forth year of the project along with an overall summary of the final systems. It builds on the work reported in D2.5 during the third year of the project and previous deliverables D2.1, D2.2 and D2.3 of the first and second year of the project, respectively. The software can be found in the project repository at http://redmine.confine-project.eu.
  • Testbed-14 Call for Participation (CFP)

    Testbed-14 Call for Participation (CFP)

    Testbed-14 Call for Participation (CFP) Version 2.4.1 - 21 March, 2018 Table of Contents Introduction . 5 Benefits of Participation . 6 Testbed Policies and Procedures . 7 Initiative Roles . 7 1. General Proposal Submission Guidelines . 9 1.1. Questions and Clarifications . 10 2. Proposal Evaluation Criteria . 11 2.1. Management Criteria . 11 2.2. Technical Criteria . 11 3. Master Schedule . 12 3.1. Sequence of Events, Phases, and Milestones . 12 4. Summary of Testbed 14 Deliverables. 16 4.1. CFP Deliverables and Funding Status . 17 Appendix A: Management Requirements . 20 A.1. Proposal Submission Procedures. 20 A.2. Conditions for Participation . 23 A.3. Proposal Evaluation and Invitations to Selected Bidders . 23 A.4. Kickoff Workshop . 25 A.5. Participant Communication and Reporting . 26 A.6. General Requirements for Proposing Deliverables . 29 A.7. Specific Requirements for Proposing Document Deliverables . 29 A.8. Specific Requirements for Proposing Component Implementation Deliverables . 30 A.9. Project Monitoring and Control . 32 A.10. Tips for New Bidders. 33 Appendix B: Technical Architecture . 36 B.1. Introduction . 36 B.2. Testbed Baseline . 36 B.3. Testbed Threads . 40 B.4. Tasks . 41 B.5. Machine Learning, Deep Learning & Artificial Intelligence . 41 B.6. Information Registries & Semantic Enablement . 47 B.7. Next Generation OGC Web Services, Federated Clouds, Security & Workflows. 50 B.8. Complex Feature Handling . 66 B.9. Swath Data and the Climate Forecast Convention . 68 B.10. Application Schema Modeling and Conversion . 75 B.11. LiDAR Point Cloud Data Handling . 79 B.12. CityGML and Augmented Reality . 82 B.13. Portrayal . 83 B.14.
  • The Openuav Swarm Simulation Testbed: a Collaborative Design Studio for Field Robotics

    The Openuav Swarm Simulation Testbed: a Collaborative Design Studio for Field Robotics

    The OpenUAV Swarm Simulation Testbed: a Collaborative Design Studio for Field Robotics Harish Anand, Zhiang Chen, Jnaneshwar Das Abstract— In this paper, we showcase a multi-robot design data analysis [5, 6]. An example application of sUAS- studio where simulation containers are browser accessible based imaging technology in geology involves generating Lubuntu desktops. Our simulation testbed, based on ROS, orthomosaic of a region and estimating rock traits such Gazebo, PX4 flight stack has been developed to tackle higher- level challenging tasks such as mission planning, vision-based as diameter and orientation. To extend such UAS based problems, collision avoidance and multi-robot coordination for sampling methods to other domains, we need software tools Unpiloted Aircraft Systems (UAS). The new architecture is that help in algorithms development and testing. Single UAS built around TurboVNC and noVNC WebSockets technology, to based methods encounter limitations in tasks such as wide- seamlessly provide real-time web performance for 3D rendering area mapping, freight transportation, and cooperative search in a collaborative design tool. We have built upon our previous work that leveraged concurrent multi-UAS simulations, and and recovery. Therefore, there is a need for a simulation extended it to be useful for underwater, airship and ground framework that provides flight stack and communication vehicles. This opens up the possibility for both rigorous Monte network to develop efficient swarm algorithms. Carlo styled software testing of heterogeneous swarm sim- With our belief that hardware abstraction and end-to-end ulations, as well as sampling-based optimization of mission simulation tools will accelerate innovation and education, we parameters.
  • A Thesis Entitled a Light-Weight, Flexible, and Web-Based Testbed for Computer & Network Security Experiments by Wei Liu Su

    A Thesis Entitled a Light-Weight, Flexible, and Web-Based Testbed for Computer & Network Security Experiments by Wei Liu Su

    A Thesis entitled A Light-weight, Flexible, and Web-based Testbed for Computer & Network Security Experiments by Wei Liu Submitted to the Graduate Faculty as partial fulfillment of the requirements for the Master of Science Degree in Engineering with Concentration on Computer Science _________________________________________ Dr. Weiqing Sun, Committee Chair _________________________________________ Dr. Ahmad Javaid,Committee Co-Chair _________________________________________ Dr. Hong Wang, Committee Member _________________________________________ Dr. Amanda Bryant-Friedrich , Dean College of Graduate Studies The University of Toledo August 2016 Copyright 2016, Wei Liu This document is copyrighted material. Under copyright law, no parts of this document may be reproduced without the expressed permission of the author. An Abstract of A Light-weight, Flexible, and Web-based Testbed for Computer & Network Security Experiments by Wei Liu Submitted to the Graduate Faculty as partial fulfillment of the requirements for the Master of Science Degree in Engineering with Concentration on Computer Science The University of Toledo August 2016 In this thesis, we implement and deploy a light-weight, flexible, and web-based testbed to host multiple virtual networks on a single underlying physical network infrastructure for network and security experiments in an academic set-up. Creating and deleting virtual machines (VMs) and virtual networks (VNs) are time-consuming tasks. To make those tasks easier, much virtualization software is used by universities and institutions that enable the VMs and VNs creation and deletion efficiency on a single physical infrastructure. However, these technologies are heavy-weight. In contrast, we use Linux Containers (LXC), Open vSwitch (OVS), and Shell In A Box (ShellInABox) to deploy a light-weight and user-friendly web-based system.
  • D6.3 Final Demo and Testbed Experimentation Results

    D6.3 Final Demo and Testbed Experimentation Results

    5G Programmable Infrastructure Converging disaggregated network and compUte REsources D6.3 Final Demo and Testbed experimentation results This project has received funding from the European Union’s Framework Programme Horizon 2020 for research, technological development and demonstration 5G PPP Research and Validation of critical technologies and systems Project Start Date: June 1st, 2017 Duration: 36 months Call: H2020-ICT-2016-2 Date of delivery: 31st May 2020 Topic: ICT-07-2017 Version 1.0 Project co-funded by the European Commission Under the H2020 programme Dissemination Level: Public Grant Agreement Number: 762057 5G Programmable Infrastructure Converging Project Name: disaggregated network and compUte Resources Project Acronym: 5G-PICTURE Document Number: D6.3 Document Title: Final Demo and Testbed experimentation results Version: 1.0 Delivery Date: 31 May 2020 Responsible: COMSA Editor(s): Manuel Alfageme (COMSA) Authors: Manuel Alfageme, Juan Agustí, Sergio Martínez, Martí Roset, Romeo Santiago (COMSA) Jim Zou (ADVA) Peter Legg, Richard Graham (BWT) Salvatore Pontarelli (CNIT) Paula Ciria (FGC) Amin Emami, Arash Farhadi Beldachi, Anna Tzanakaki, Hamid Falaki (UNIVBRIS-HPN) Nebojsa Maletic, Darko Cvetkovski, Meysam Goodarzi, Marcus Ehrig, Eckhard Grass, Jesús Gutiérrez (IHP) Jorge Paracuellos, Alan Holt, Rodrigo Mompó (ZN) Ferran Cañellas, Daniel Camps (I2CAT) Hadi Razzaghi (UPB), Kostas Choumas, Paris Flegkas (UTH), Vaia Kalokidou (UNIVBRIS-CSN), Ioanna Mesogiti (COS) Final demonstration, Use Cases, 5GUK smart city, Rail Keywords: Demo, Stadium Demo, Final Deployment, KPIs evaluation Status: Final Dissemination Level Public Project URL: http://www.5g-picture-project.eu/ 5G-PICTURE Deliverable Revision History Rev. N Description Author Date Railway Demonstration of Railway Use Case. Initial v0.1 COMSA 29/11/2019 draft.
  • Bianca's Thesis

    Bianca's Thesis

    Improving the performance of static and dynamic requests at a busy web site Bianca Schroeder CMU-CS-05-167 August 30, 2005 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Mor Harchol-Balter, Chair Anastassia Ailamaki Todd C. Mowry Erich M. Nahum (T. J. Watson Research Center, IBM Research) Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy This research was sponsored by the National Science Foundation under grant nos. CCR-0133077 and CCR-9457766, by a generous grant from the Pittsburgh Digital Greenhouse and by an IBM graduate student fellowship. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity. Abstract Running a high-volume Web site is a challenging task. Web traffic is bursty with peak request rates rising far above average rates and likely phenomena such as flash crowds and hot spots. Yet Web users are very demanding: they expect Web sites to be quickly accessible from around the world 24 hours a day, 7 days a week. Recent studies show that even a short period of slowdown or service interruption can have severe effects: it not only sends customers to the “just a click-away” competitor; it also reflects negatively on the corporate image as a whole. The broad goal of this thesis is to improve the user-perceived performance of both static and dynamic requests at a busy Web site.
  • SCIONLAB: a Next-Generation Internet Testbed

    SCIONLAB: a Next-Generation Internet Testbed

    SCIONLAB: A Next-Generation Internet Testbed Jonghoon Kwon∗, Juan A. García-Pardo∗, Markus Legner∗, François Wirz∗, Matthias Frei∗, David Hausheery, Adrian Perrig∗ ∗Department of Computer Science, ETH Zürich, Zürich, Switzerland yDepartment of Computer Science, Universität Magdeburg, Magdeburg, Germany Abstract—Network testbeds have empowered networking re- Path-aware networking enables end-hosts to obtain informa- search and facilitated scientific progress. However, current tion about network paths leading to the destination, and to se- testbeds focus mainly on experiments involving the current Inter- lect the path among a set of paths offered by the network [20], net. In this paper, we propose SCIONLAB, a novel global network testbed that enables exciting research opportunities and experi- [52]. Path awareness offers exciting properties to applica- mentation with the SCION next-generation Internet architecture. tions, such as path transparency, fine-grained path control, New users can join SCIONLAB as a full-fledged autonomous fast failover or route optimization to alternate paths, or geo- system with minimal effort and administrative overhead, and fencing [50], [51], [63]. These properties enable the creation directly gain unfettered access to its inter-domain routing system. of new transport protocols and advanced application features. Based on a well-connected network topology consisting of globally distributed nodes, SCIONLAB enables new experiments, such as Path-aware networking also enables multipath communication: inter-domain multipath communication, path-aware networking, in case the end-host obtains multiple paths from the network, it exploration of novel routing policies, and new approaches for can select the path on a per-packet basis. Multipath communi- DDoS defense. SCIONLAB has been operational since 2016 and cation can provide increased bandwidth, improved reliability, has supported diverse research projects.