Open-Source Virtualization Master Thesis Jan Magnus Granberg Opsahl

Total Page:16

File Type:pdf, Size:1020Kb

Open-Source Virtualization Master Thesis Jan Magnus Granberg Opsahl UNIVERSITY OF OSLO Department of Informatics Open-source virtualization Functionality and performance of Qemu/KVM, Xen, Libvirt and VirtualBox Master Thesis Jan Magnus Granberg Opsahl Spring 2013 Abstract The main purpose of this thesis is to evaluate the most common open-source virtualization technologies available. Namely Qemu/KVM, Xen, Libvirt and VirtualBox. The thesis investigates the various virtualization platforms in terms of architecture and overall usage. Before further investigating the platforms through a series of benchmarks. The results gathered from the benchmarks presents Qemu/KVM as the bet- ter in terms of performance in most of the benchmarks. Of these we can count the CPU- and memory intensive benchmarks. For the file-systems benchmarks, Xen delivers performance that is above the other examined virtualization plat- forms. The results also highlight the performance gained with processor ad- ditions such as Intel Extended Page Tables and AMD Rapid Virtualization Indexing, to enable hardware assisted paging. i ii Acknowledgments First and foremost, I thank my thesis supervisor Knut Omang, for his insights, directing me in the right direction when I have lost my way, and most impor- tantly for being incredibly patient. I would also like to thank my fellow students at the Dmms laboratory for a thriving environment, inspiring discussions and their feedback. Last I thank my family for their patience, understanding and endless sup- port during my thesis work. Most importantly I thank my wonderful girlfriend Ingebjørg Miljeteig for believing in me and her enduring support and love. May 2. 2013. Jan Magnus Granberg Opsahl iii iv Contents 1 Introduction 1 1.1 Introduction . .1 1.2 Motivation . .2 1.3 Previous work . .2 1.4 Thesis structure . .3 2 Background 5 2.1 Introduction . .5 2.2 Terms and definitions . .5 2.2.1 On Intel VT and AMD-V . .6 2.3 What is virtualization? . .6 2.3.1 Characteristics . .6 2.3.2 Virtualization Theorems . .7 2.3.3 Types of VMMs . .8 2.3.4 Types of virtualization . .9 2.4 Background for virtualization . 11 2.4.1 Historic background for virtualization . 11 2.4.2 Modern background for virtualization . 12 2.5 A brief history of virtualization . 13 2.5.1 Early history of virtualization . 13 2.5.2 X86 virtualization and the future . 17 2.6 Benefits and different solutions . 21 2.6.1 Advantages and the disadvantages of virtualization tech- nology . 21 2.6.2 Virtualization technology and solutions . 23 2.7 Conclusion . 26 3 Virtualization software 27 3.1 Introduction . 27 3.2 Qemu/KVM . 27 3.2.1 KVM . 27 3.2.2 Qemu . 29 3.3 Xen . 32 3.4 Libvirt . 33 3.4.1 User tools and usage . 34 3.5 VirtualBox . 37 3.5.1 About . 37 3.5.2 Usage . 38 v 3.6 Comparison . 39 4 Benchmarks 43 4.1 Introduction . 43 4.2 Motivation and previous work . 43 4.2.1 Summary . 46 4.3 Virtual Machine Monitors . 47 4.3.1 KVM . 48 4.3.2 QEMU . 48 4.3.3 QEMU-KVM . 48 4.3.4 Virtual Machine Manager and libvirt . 48 4.3.5 Xen . 49 4.3.6 Virtualbox . 49 4.3.7 Equipment and operating system . 49 4.4 Benchmarking suites . 49 4.4.1 Context Switches . 50 4.4.2 Cachebench . 50 4.4.3 LMBench . 51 4.4.4 Linpack . 51 4.4.5 IOZone . 52 4.5 Experiment design . 53 4.5.1 CPU-based tests . 53 4.5.2 Memory-based tests . 54 4.5.3 I/O-based tests . 54 4.5.4 Platform configurations . 55 5 Results 57 5.1 Introduction . 57 5.1.1 Regarding the Host benchmarks . 58 5.2 CPU-based benchmarks . 58 5.2.1 High Performance Linpack . 58 5.2.2 LMBench Context Switch (CTX) . 62 5.2.3 Context Switching . 67 5.2.4 Comments to the CPU benchmarks . 71 5.3 Memory-based benchmarks . 72 5.3.1 Cachebench . 72 5.3.2 LMBench . 75 5.3.3 Comments upon the memory benchmarks . 79 5.4 I/O-based benchmarks - IOZone . 80 5.4.1 Comments . 89 6 Conclusion 91 6.1 About the conclusion . 91 6.2 Virtualization software . 91 6.3 Benchmarking results . 92 6.3.1 CPU-based benchmarks . 92 6.3.2 Memory-based benchmarks . 92 6.3.3 I/O-based benchmarks . 93 6.3.4 Final words . 94 6.4 Shortcomings . 94 vi 6.5 Future work . 95 A Additional results 97 A.1 About . 97 A.2 LMBench CTX . 97 A.3 LMBench MEM . 99 B Installation of used software 101 B.1 Introduction . 101 B.2 KVM . 101 B.3 QEMU . 102 B.4 QEMU-KVM . 102 B.5 High Performance Linpack (HPL) . 102 C Virtualization suite configuration 105 C.1 Qemu/KVM . 105 C.2 Xen . 106 vii viii List of Tables 2.1 Instructions that cause traps. 18 2.2 Intel and AMD new and modified instructions for the X86 hard- ware virtualization extensions. 20 4.1 Table showing the various hypervisors to be tested. 47 4.2 Various process grid configurations for HPL benchmark. 52 4.3 CPU-based tests . 53 4.4 Memory-based tests . 54 4.5 File-based tests . 55 ix x List of Figures 2.1 The typical architecture of virtualization software. Hardware at the bottom and an abstract layer to expose a VM, which runs its own operating system on what it thinks is real hardware. .9 2.2 Paravirtualization abstraction showing the modified drivers that need be present in the OS. 10 2.3 Operating system level virtualization. 11 2.4 Virtual memory abstraction with pointers to RAM memory and the disk. 14 2.5 An IBM System/360-67 at the University of Michigan. Image courtesy of Wikimedia Commons. 16 2.6 Hypervisor and guests with regard to processor rings. 18 3.1 The KVM basic architecture. 28 3.2 Simplified view of Qemu with regard to the operating system. 30 3.3 The basic flow of a KVM guest in Qemu. 30 3.4 Qemu-kvm command-line example. 31 3.5 Xen architecture with guest domains. 32 3.6 Libvirt with regard to hypervisors and user tools. 34 3.7 Guest creation in virt-manager. 36 3.8 Guest installation using virt-install. 36 3.9 virt-viewer commands. 37 3.10 VirtualBox main screen. 38 3.11 xl.cfg file for a Xen-HVM guest. 40 3.12 Comparison of the various virtualization suites. 41 4.1 The different QEMU configurations and abbreviations. 55 4.2 Xen configurations and abbreviations. 56 4.3 Libvirt configurations and abbreviations . 56 4.4 Virtualbox configuration and abbreviation. 56 5.1 HPL benchmark for 1 cpu core. 59 5.2 HPL benchmark for 2 cpu cores. 59 5.3 HPL benchmark for 4 cpu cores. 60 5.4 HPL benchmark for 8 cpu cores. 61 5.5 LMBench CTX with 2 processes. ..
Recommended publications
  • Understanding Full Virtualization, Paravirtualization, and Hardware Assist
    VMware Understanding Full Virtualization, Paravirtualization, and Hardware Assist Contents Introduction .................................................................................................................1 Overview of x86 Virtualization..................................................................................2 CPU Virtualization .......................................................................................................3 The Challenges of x86 Hardware Virtualization ...........................................................................................................3 Technique 1 - Full Virtualization using Binary Translation......................................................................................4 Technique 2 - OS Assisted Virtualization or Paravirtualization.............................................................................5 Technique 3 - Hardware Assisted Virtualization ..........................................................................................................6 Memory Virtualization................................................................................................6 Device and I/O Virtualization.....................................................................................7 Summarizing the Current State of x86 Virtualization Techniques......................8 Full Virtualization with Binary Translation is the Most Established Technology Today..........................8 Hardware Assist is the Future of Virtualization, but the Real Gains Have
    [Show full text]
  • Paravirtualization (PV)
    Full and Para Virtualization Dr. Sanjay P. Ahuja, Ph.D. Fidelity National Financial Distinguished Professor of CIS School of Computing, UNF x86 Hardware Virtualization The x86 architecture offers four levels of privilege known as Ring 0, 1, 2 and 3 to operating systems and applications to manage access to the computer hardware. While user level applications typically run in Ring 3, the operating system needs to have direct access to the memory and hardware and must execute its privileged instructions in Ring 0. x86 privilege level architecture without virtualization Technique 1: Full Virtualization using Binary Translation This approach relies on binary translation to trap (into the VMM) and to virtualize certain sensitive and non-virtualizable instructions with new sequences of instructions that have the intended effect on the virtual hardware. Meanwhile, user level code is directly executed on the processor for high performance virtualization. Binary translation approach to x86 virtualization Full Virtualization using Binary Translation This combination of binary translation and direct execution provides Full Virtualization as the guest OS is completely decoupled from the underlying hardware by the virtualization layer. The guest OS is not aware it is being virtualized and requires no modification. The hypervisor translates all operating system instructions at run-time on the fly and caches the results for future use, while user level instructions run unmodified at native speed. VMware’s virtualization products such as VMWare ESXi and Microsoft Virtual Server are examples of full virtualization. Full Virtualization using Binary Translation The performance of full virtualization may not be ideal because it involves binary translation at run-time which is time consuming and can incur a large performance overhead.
    [Show full text]
  • Vmware Wants to Reign Supreme
    August/September 2010 | Vol. 3 | No. 5 VirtualizationReview.com King of Clouds VMware wants to reign supreme. + VDI Update Acronis Backup & Recovery Q&A with Amir Sharif of Parallels Thethingwithdata:ifyoudon’t getinfrontofit,you’llsoonbe buried under it. To help you keep pace with your ever-increasing amounts of data, CDW’s storage specialistscanhelpyousimplifyyourstoragesystemsfor increased flexibility and optimization. We can assist you with everything from product recommendations to designing and implementing complete storagesolutions—including multi-tiered storagecomponents,deduplicationandvirtualizationtechnology.Noneedtoworry aboutyourdata;we’llkeepyouontopof it. VMware® vSphere™ IBM® System x3650 IBM System Storage™ Enterprise Plus M3 Rack-mount Server DS5020 Acceleration Kit1 $ .99 CALL FOR PRICING Single licenses available. 4743 License for 8 processors CDW 2051288 CDW 1863874 CALL FOR PRICING CDW 1753279 Hard drives sold separately Smarterdatastoragestartshere. 800.399.4CDW | CDW.com/infrastructure 1Requiresaminimumofone-yearsupportandsubscription(SaS)atthetimeofpurchase;callyourCDWaccountmanagerfordetails.Offersubject toCDW’sstandardtermsandconditionsofsale,availableatCDW.com.©2010CDWLLC VISIT VIRTUALIZATIONREVIEW.COM contents 16 “If you look at apples to apples, we come out actually lower on cost per application. Not only do we come out lower on cost per application, we can run more applications on a given server than any of our competition.” Raghu Raghuram, General Manager and VP, Server Business Unit, VMware Inc. The ease
    [Show full text]
  • Full Virtualization on Low-End Hardware: a Case Study
    Full Virtualization on Low-End Hardware: a Case Study Adriano Carvalho∗, Vitor Silva∗, Francisco Afonsoy, Paulo Cardoso∗, Jorge Cabral∗, Mongkol Ekpanyapongz, Sergio Montenegrox and Adriano Tavares∗ ∗Embedded Systems Research Group Universidade do Minho, 4800–058 Guimaraes˜ — Portugal yInstituto Politecnico´ de Coimbra (ESTGOH), 3400–124 Oliveira do Hospital — Portugal zAsian Institute of Technology, Pathumthani 12120 — Thailand xUniversitat¨ Wurzburg,¨ 97074 Wurzburg¨ — Germany Abstract—Most hypervisors today rely either on (1) full virtua- a hypervisor-specific, often proprietary, interface. Full virtua- lization on high-end hardware (i.e., hardware with virtualization lization on low-end hardware, on the other end, has none of extensions), (2) paravirtualization, or (3) both. These, however, those disadvantages. do not fulfill embedded systems’ requirements, or require legacy software to be modified. Full virtualization on low-end hardware Full virtualization on low-end hardware (i.e., hardware with- (i.e., hardware without virtualization extensions), on the other out dedicated support for virtualization) must be accomplished end, has none of those disadvantages. However, it is often using mechanisms which were not originally designed for it. claimed that it is not feasible due to an unacceptably high Therefore, full virtualization on low-end hardware is not al- virtualization overhead. We were, nevertheless, unable to find ways possible because the hardware may not fulfill the neces- real-world quantitative results supporting those claims. In this paper, performance and footprint measurements from a sary requirements (i.e., “the set of sensitive instructions for that case study on low-end hardware full virtualization for embedded computer is a subset of the set of privileged instructions” [1]).
    [Show full text]
  • Taxonomy of Virtualization Technologies Master’S Thesis
    A taxonomy of virtualization technologies Master’s thesis Delft University of Technology Faculty of Systems Engineering Policy Analysis & Management August, 2010 Paulus Kampert 1175998 Taxonomy of virtualization technologies Preface This thesis has been written as part of the Master Thesis Project (spm5910). This is the final course from the master programme Systems Engineering, Policy Analysis & Management (SEPAM), educated by Delft University of Technology at the Faculty of Technology, Policy and Management. This thesis is the product of six months of intensive work in order to obtain the Master degree. This research project provides an overview of the different types of virtualization technologies and virtualization trends. A taxonomy model has been developed that demonstrates the various layers of a virtual server architecture and virtualization domains in a structured way. Special thanks go to my supervisors. My effort would be useless without the priceless support from them. First, I would like to thank my professor Jan van Berg for his advice, support, time and effort for pushing me in the right direction. I am truly grateful. I would also like to thank my first supervisor Sietse Overbeek for frequently directing and commenting on my research. His advice and feedback was very constructive. Many thanks go to my external supervisor Joris Haverkort for his tremendous support and efforts. He made me feel right at home at Atos Origin and helped me a lot through the course of this research project. Especially, the lectures on virtualization technologies and his positive feedback gave me a better understanding of virtualization. Also, thanks go to Jeroen Klompenhouwer (operations manager at Atos Origin), who introduced me to Joris Haverkort and gave me the opportunity to do my graduation project at Atos Origin.
    [Show full text]
  • Virtualization Basics: Understanding Techniques and Fundamentals
    Virtualization Basics: Understanding Techniques and Fundamentals Hyungro Lee School of Informatics and Computing, Indiana University 815 E 10th St. Bloomington, IN 47408 [email protected] ABSTRACT example of growing involvement in virtualization technolo- Virtualization is a fundamental part of cloud computing, gies with the cloud. Early technologies and developments especially in delivering Infrastructure as a Service (IaaS). in the virtualization have been accomplished by some com- Exploring different techniques and architectures of the vir- panies such as IBM from 1967 and VMware from 1998. In tualization helps us understand the basic knowledge of virtu- open source communities, Xen, KVM, Linux-vServer, LXC alization and the server consolidation in the cloud with x86 and others have supported virtualization in different plat- architecture. This paper describes virtualization technolo- forms with different approaches. In this paper, x86 archi- gies, architectures and optimizations regarding the sharing tecture virtualization will be discussed with these historical CPU, memory and I/O devices on x86 virtual machine mon- changes. itor. In cloud computing, Infrastructure-as-a-Service (IaaS) pro- vides on-demand virtual machine instances with virtualiza- Categories and Subject Descriptors tion technologies. IaaS has been broadly used to provide re- C.0 [General]: Hardware/software interface; C.4 [Performance quired compute resources in shared resource environments. of systems]: Performance attributes; D.4.7 [Operating Amazon Web Services (AWS), Google Compute Engine, Mi- Systems]: Organization and design crosoft Windows Azure, and HP Cloud offer commercial cloud services. OpenStack, Eucalyptus, SaltStack, Nimbus, General Terms and many others provide private open source cloud plat- Performance, Design forms with community support in development.
    [Show full text]
  • NIST 800-125: Guide to Security for Full Virtualization Technologies
    Special Publication 800-125 Guide to Security for Full Virtualization Technologies Recommendations of the National Institute of Standards and Technology Karen Scarfone Murugiah Souppaya Paul Hoffman NIST Special Publication 800-125 Guide to Security for Full Virtualization Technologies Recommendations of the National Institute of Standards and Technology Karen Scarfone Murugiah Souppaya Paul Hoffman C O M P U T E R S E C U R I T Y Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 January 2011 U.S. Department of Commerce Gary Locke, Secretary National Institute of Standards and Technology Patrick D. Gallagher, Director GUIDE TO SECURITY FOR FULL VIRTUALIZATION TECHNOLOGIES Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-125 Natl. Inst. Stand. Technol. Spec. Publ. 800-125, 35 pages (January 2010) Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately.
    [Show full text]
  • Parallels Server Bare Metal 5.0 User's Guide
    Parallels Server Bare Metal 5.0 User's Guide Copyright © 1999-2014 Parallels Holdings, Ltd. and its affiliates. All rights reserved. Parallels Holdings, Ltd. c/o Parallels International GMbH. Parallels International GmbH Vordergasse 49 CH8200 Schaffhausen Switzerland Tel: + 41 526320 411 Fax: + 41 52672 2010 www.parallels.com Copyright © 1999-2012 Parallels Holdings, Ltd. and its affiliates. All rights reserved. This product is protected by United States and international copyright laws. The product’s underlying technology, patents, and trademarks are listed at http://www.parallels.com/trademarks. Microsoft, Windows, Windows Server, Windows NT, Windows Vista, and MS-DOS are registered trademarks of Microsoft Corporation. Apple, Mac, the Mac logo, Mac OS, iPad, iPhone, iPod touch, FaceTime HD camera and iSight are trademarks of Apple Inc., registered in the US and other countries. Linux is a registered trademark of Linus Torvalds. All other marks and names mentioned herein may be trademarks of their respective owners. Contents Introduction ............................................................................................................... 9 About This Guide ............................................................................................................ 9 Organization of This Guide .......................................................................................................9 Documentation Conventions ................................................................................................. 10 Getting
    [Show full text]
  • Intro Comparing and Contrasting Virtualization Technologies
    Intro Comparing and Contrasting Virtualization Technologies 9304 Do you knowAbstract the differences Part 1 between Xen and VMware? Abstract Part Two Do you know when it is more advantageous to use one over the other? Abstract Part Three Virtualization can be a complicated subject with many different facets. It is not always easy to choose the strategy that best fits your needs. I am here to help [email protected] Worldwide Design Centers Poughkeepsie, NY Montpellier, France [email protected] [email protected] Makuhari, Japan Boeblingen, Germany 6 [email protected] [email protected] Worldwide Design Centers Our mission – To architect & design innovative end to end solutions with selected worldwide clients that leverage leading-edge IBM technologies to accelerate their IT transformation . Analysis Define and analyze various solution options that meet existing and future requirements . Assessment Review and validate a planned solution, strategy, IT transformation, or architecture, and provide recommendations and roadmaps . Design Define an architecture and high level design for an IT infrastructure to meet business requirements 7 There is no There is no free free lunch lunch Types of Virtualization Hardware Virtualization Partitioned Partitioned OS OS Partition Controller Application Application Physical Machine OS FullHardware Virtualization OS OS Application Application Application Fully Virtualized Hypervisor Physical Machine ModifiedPara Virtualization Modified Modified OS OS OS Application Application Application Para Virtualized
    [Show full text]
  • The Do's and Don'ts of Server Virtualization
    The Do’s and Don’ts of Server Virtualization Back to basics tips for Australian IT professionals Server Virtualization Reminders: Basic Do’s and Don’ts Virtualization is a well-established in today’s IT environments, but it still remains a top priority among data center managers and admins. Therefore, Contents our experts have compiled an essential guide that highlights Hosted vs. bare- some of the key elements of an effective virtualization mental hypervisor strategy from choosing the right hypervisor technology to types improving VM performance to keep top of mind. Top 10 hypervisors technologies Understanding hosted and bare-metal virtualization 4 mistakes that kill VM hypervisor types performance By: Eric Siebert Additional Resources When choosing a virtualization hypervisor, you have to consider both hypervisor types. A virtualization hypervisor comes in one of two forms: a bare-metal hypervisor, also known as Type 1; or a hosted hypervisor, also known as Type 2. There are important differences between a hosted and bare-metal virtualization hypervisor, and each has pretty specific use cases. Bare-metal virtualization hypervisors A bare-metal virtualization hypervisor does not require admins to install a server operating system first. Bare-metal virtualization means the hypervisor has direct access to hardware resources, which results in better performance, scalability and stability. One disadvantage of a bare-metal virtualization hypervisor, however, is that hardware support is typically more limited, because the hypervisor usually has limited device drivers built into it. Bare-metal virtualization is well suited for enterprise data centers, because it usually comes with advanced features for resource management, high availability and security.
    [Show full text]
  • Virtualization Technology
    Virtualization Technology Zhonghong Ou Data Communications Software Lab, Aalto University 1 Definition • Virtualization refers to a concept in which access to a single underlying piece of hardware, like a server, is coordinated so that multiple guest operating systems can share that single piece of hardware, with no guest operating system being aware that it is actually sharing anything at all. (A guest operating system is an operating system that’s hosted by the underlying virtualization software layer, which is often, you guessed it, called the host system.) • A guest operating system appears to the applications running on it as a complete operating system (OS), and the guest OS itself is completely unaware that it’s running on top of a layer of virtualization software rather than directly on the physical hardware. Source: Virtualization for dummies 2 Four drivers of virtualization • Hardware is underutilized – Moore’s Law: “The complexity for minimum component costs has increased at a rate of roughly a factor of two per year…”, (actually, most people estimate the timeframe at around 18 months) i.e. the number of transistors that can be placed inexpensively on an integrated circuit has doubled approximately every two years. Moore’s law • Data centers run out of space – The net effect of explosion of data is that huge numbers of servers have been put into use over the past decade, which is causing a real -estate problem for companies. • Energy costs go through the roof – The cost of running computers, coupled with the fact that many of the machines filling up data centers are running at low utilization rates, means that virtualization’s ability to reduce the total number of physical servers can significantly reduce the overall cost of energy for companies.
    [Show full text]
  • This Combined Issue Brings You a Virtual Cornucopia of Products and Vendors
    December 2010/January 2011 | Vol. 4 | No. 1 VirtualizationReview.com This combined issue brings you a virtual cornucopia of products and vendors. Can you guess who dominates the RCA awards? The 2011 Virtualization Review Readers Choice Awards Ultimateand Buyers READERS CHOICE AWARDS Guide Our readers rate products, and we reward them with the hands-down, most-informative, must-read print guide to virtualization and cloud products. By Bruce Hoard 2 | December 2010/January 2011 | Virtualization Review | VirtualizationReview.com FEATURE | The 2011 Ultimate Virtualization Buyers Guide and Virtualization Review Readers Choice Awards elcome to the combined 2011 Application Virtualization Virtualization Buyers Guide and Virtualization Review Readers Readers Choice Award Winner VMware ThinApp 4.5 – VMware Inc. Choice Awards. By combining (43.6%) W $5,000 n vmware.com n 877-486-9273; these two editorial efforts, we are able to 650-427-5000 showcase both the virtualization and cloud This is a big win for VMware in a category where Citrix is extremely well established with its XenApp product, which products that you voted as your favorites has been around for many years. ThinApp no doubt benefited from its ability to ease the migration of existing apps to and many others that will be vying for your Windows 7. Merit Award winner Citrix was joined by Parallels, attention during the coming year. All told, which finished a distant third, but is favorably viewed as a technology trendsetter. we amassed some 211 products from 106 VMware ThinApp isolates applications from the operating vendors, and placed them in 15 categories for environment, enabling enterprises to manage applications from a central source and deliver them instantly to nearly both the Readers Choice Awards and Ultimate any device, while streamlining security, updates, patches and performance.
    [Show full text]