Data Center Transformation with Google Cloud
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Data Centers and Cloud Computing Data Centers
Data Centers and Cloud Computing Data Centers • Large server and storage farms – 1000s of servers • Intro. to Data centers – Many TBs or PBs of data • Virtualization Basics • Used by – Enterprises for server applications – Internet companies • Intro. to Cloud Computing • Some of the biggest DCs are owned by Google, Facebook, etc • Used for – Data processing – Web sites – Business apps Computer Science Lecture 22, page 2 Computer Science Lecture 22, page 3 Inside a Data Center MGHPCC Data Center • Giant warehouse filled with: • Racks of servers • Storage arrays • Cooling infrastructure • Power converters • Backup generators • Data center in Holyoke Computer Science Lecture 22, page 4 Computer Science Lecture 22, page 5 Modular Data Center Virtualization • ...or use shipping containers • Each container filled with thousands of servers • Can easily add new containers • Virtualization: extend or replace an existing interface to – “Plug and play” mimic the behavior of another system. – Just add electricity – Introduced in 1970s: run legacy software on newer mainframe hardware • Allows data center to be easily • Handle platform diversity by running apps in VMs expanded – Portability and flexibility • Pre-assembled, cheaper Computer Science Lecture 22, page 6 Computer Science Lecture 22, page 7 Types of Interfaces Types of OS-level Virtualization • Different types of interfaces – Assembly instructions – System calls • Type 1: hypervisor runs on “bare metal” – APIs • Type 2: hypervisor runs on a host OS • Depending on what is replaced /mimiced, -
F1 Query: Declarative Querying at Scale
F1 Query: Declarative Querying at Scale Bart Samwel John Cieslewicz Ben Handy Jason Govig Petros Venetis Chanjun Yang Keith Peters Jeff Shute Daniel Tenedorio Himani Apte Felix Weigel David Wilhite Jiacheng Yang Jun Xu Jiexing Li Zhan Yuan Craig Chasseur Qiang Zeng Ian Rae Anurag Biyani Andrew Harn Yang Xia Andrey Gubichev Amr El-Helw Orri Erling Zhepeng Yan Mohan Yang Yiqun Wei Thanh Do Colin Zheng Goetz Graefe Somayeh Sardashti Ahmed M. Aly Divy Agrawal Ashish Gupta Shiv Venkataraman Google LLC [email protected] ABSTRACT 1. INTRODUCTION F1 Query is a stand-alone, federated query processing platform The data processing and analysis use cases in large organiza- that executes SQL queries against data stored in different file- tions like Google exhibit diverse requirements in data sizes, la- based formats as well as different storage systems at Google (e.g., tency, data sources and sinks, freshness, and the need for custom Bigtable, Spanner, Google Spreadsheets, etc.). F1 Query elimi- business logic. As a result, many data processing systems focus on nates the need to maintain the traditional distinction between dif- a particular slice of this requirements space, for instance on either ferent types of data processing workloads by simultaneously sup- transactional-style queries, medium-sized OLAP queries, or huge porting: (i) OLTP-style point queries that affect only a few records; Extract-Transform-Load (ETL) pipelines. Some systems are highly (ii) low-latency OLAP querying of large amounts of data; and (iii) extensible, while others are not. Some systems function mostly as a large ETL pipelines. F1 Query has also significantly reduced the closed silo, while others can easily pull in data from other sources. -
Deploy, Operate, and Evolve Your Data Center: HP Datacenter Care
Brochure Deploy, operate, and evolve your data center HP Datacenter Care The data center is evolving—shouldn’t Improve agility, scalability—react at the your support keep pace? speed of business Historically, business-critical IT has been delivered on dedicated, HP Datacenter Care service is designed to support this homogenous, and proprietary infrastructures. In this siloed results-oriented approach by providing an environment-wide IT model, performance, uptime, and security outweighed support solution tailored to your needs. HP Datacenter Care considerations of speed, agility, and cost. Now, however, the trends is a flexible, comprehensive, relationship-based approach to of mobility, big data, and cloud computing are fundamentally personalized support and management of heterogeneous data changing how you deliver information, and how technology is centers. Datacenter Care is a structured framework of repeatable, implemented, moving closer to unconstrained access to IT. tested, and globally available services “building blocks.” You have an account support manager (ASM) who knows your business and your IT is delivered as services anywhere, across hybrid deployments of IT environment, and who can help you select the services you need private, managed, and public cloud, as well as traditional IT. It’s what from an extensive portfolio of support and consulting services. The HP is developing as the Converged Cloud, and it can finally allow ASM leverages our experience in supporting complex environments, enterprises to achieve business agility, with choice and improved global support partnerships, and technical expertise. ROI. Today, HP is a leader in the industry for support services, for customers worldwide who need to keep their systems running, So you get exactly the services you need—when and where you reduce costs, and avoid issues. -
Containers at Google
Build What’s Next A Google Cloud Perspective Thomas Lichtenstein Customer Engineer, Google Cloud [email protected] 7 Cloud products with 1 billion users Google Cloud in DACH HAM BER ● New cloud region Germany Google Cloud Offices FRA Google Cloud Region (> 50% latency reduction) 3 Germany with 3 zones ● Commitment to GDPR MUC VIE compliance ZRH ● Partnership with MUC IoT platform connects nearly Manages “We found that Google Ads has the best system for 50 brands 250M+ precisely targeting customer segments in both the B2B with thousands of smart data sets per week and 3.5M and B2C spaces. It used to be hard to gain the right products searches per month via IoT platform insights to accurately measure our marketing spend and impacts. With Google Analytics, we can better connect the omnichannel customer journey.” Conrad is disrupting online retail with new Aleš Drábek, Chief Digital and Disruption Officer, Conrad Electronic services for mobility and IoT-enabled devices. Solution As Conrad transitions from a B2C retailer to an advanced B2B and Supports B2C platform for electronic products, it is using Google solutions to grow its customer base, develop on a reliable cloud infrastructure, Supports and digitize its workplaces and retail stores. Products Used 5x Mobile-First G Suite, Google Ads, Google Analytics, Google Chrome Enterprise, Google Chromebooks, Google Cloud Translation API, Google Cloud the IoT connections vs. strategy Vision API, Google Home, Apigee competitors Industry: Retail; Region: EMEA Number of Automate Everything running -
The Cloud-Enabled Data Center
A Revolutionary Technology Extending the Ultra-Broad Functionality of Ethernet White Paper May 2013 The Cloud-Enabled Data Center The Cloud-Enabled Data Center Introduction A growing number of organizations are adopting some form of cloud computing to meet the challenges of rapidly deploying their IT services and addressing their dynamic workload environments while maximizing their IT return on investments (ROIs). Cloud computing gives users the ability to access the IT resources faster than traditional and virtual servers. It also provides improved manageability and requires less maintenance. By deploying technology as a service, your users access only the resources they need for a specific task, which prevents you from incurring costs for computing resources not in use. In addition, developing a cloud-enabled data center can improve operational efficiencies while reducing operational expenses by 50%. Panduit, the leader in Unified Physical Infrastructure offerings, and IBM, a pioneer in delivering cloud computing solutions to clients, have joined forces to deliver optimized, custom and preconfigured solutions for the cloud- enabled data center. This white paper describes how network, storage, compute and operations are important factors to consider when deploying private and public cloud computing in the data center. Specifically, it examines integrated stack/pre- configured and custom cloud deployments. It also explains the importance of the physical infrastructure as the foundation in successful cloud deployment and maintenance. Finally, it showcases how IBM’s depth of experience in delivering systems and software for cloud solutions when combined with Panduit’s physical infrastructure expertise, provides a tremendous impact on room-level data center environmental and new-age topology. -
Spanner: Becoming a SQL System
Spanner: Becoming a SQL System David F. Bacon Nathan Bales Nico Bruno Brian F. Cooper Adam Dickinson Andrew Fikes Campbell Fraser Andrey Gubarev Milind Joshi Eugene Kogan Alexander Lloyd Sergey Melnik Rajesh Rao David Shue Christopher Taylor Marcel van der Holst Dale Woodford Google, Inc. ABSTRACT this paper, we focus on the “database system” aspects of Spanner, Spanner is a globally-distributed data management system that in particular how query execution has evolved and forced the rest backs hundreds of mission-critical services at Google. Spanner of Spanner to evolve. Most of these changes have occurred since is built on ideas from both the systems and database communi- [5] was written, and in many ways today’s Spanner is very different ties. The first Spanner paper published at OSDI’12 focused on the from what was described there. systems aspects such as scalability, automatic sharding, fault tol- A prime motivation for this evolution towards a more “database- erance, consistent replication, external consistency, and wide-area like” system was driven by the experiences of Google developers distribution. This paper highlights the database DNA of Spanner. trying to build on previous “key-value” storage systems. The pro- We describe distributed query execution in the presence of reshard- totypical example of such a key-value system is Bigtable [4], which ing, query restarts upon transient failures, range extraction that continues to see massive usage at Google for a variety of applica- drives query routing and index seeks, and the improved blockwise- tions. However, developers of many OLTP applications found it columnar storage format. -
Comparison of Availability Between Local and Cloud Storage Leviathan Security Group
Comparison of Availability Between Local and Cloud Storage Leviathan Security Group limitless innovation. no compromise. Introduction The U.S. National Institute for Standards and Technology denes cloud computing as having ve essential characteristics: on-demand self- service for provisioning, network access, the use of virtual machines to provide servers, rapid elastic provisioning and release of servers, and metering capability on the service provided.1 This paper seeks to answer whether, from the perspective of availability of data, especially in the face of natural or manmade threats to data and access, cloud computing is able to provide a greater benet than local data storage. For the purposes of this discussion, local data storage will include arrangements where data is kept on business premises or in a nearby datacenter, such as a colocation facility or other rigidly-provisioned resource. We will rst examine the signicant dierentiator that cloud computing provides in this space, and then examine dierent scales of threats to availability to see whether local data storage or cloud data storage fares better against known historical threats. 2 The Cloud Dierentiator: Geographic Redundancy At its heart, cloud computing and cloud storage are exactly the same as computing and storage provided through a datacenter or SOHO server; the cloud is made of computers. The dierence between cloud- based storage and local storage is that using cloud-based storage allows replication between separate geographic regions. Indeed, the key feature of -
Economic and Social Impacts of Google Cloud September 2018 Economic and Social Impacts of Google Cloud |
Economic and social impacts of Google Cloud September 2018 Economic and social impacts of Google Cloud | Contents Executive Summary 03 Introduction 10 Productivity impacts 15 Social and other impacts 29 Barriers to Cloud adoption and use 38 Policy actions to support Cloud adoption 42 Appendix 1. Country Sections 48 Appendix 2. Methodology 105 This final report (the “Final Report”) has been prepared by Deloitte Financial Advisory, S.L.U. (“Deloitte”) for Google in accordance with the contract with them dated 23rd February 2018 (“the Contract”) and on the basis of the scope and limitations set out below. The Final Report has been prepared solely for the purposes of assessment of the economic and social impacts of Google Cloud as set out in the Contract. It should not be used for any other purposes or in any other context, and Deloitte accepts no responsibility for its use in either regard. The Final Report is provided exclusively for Google’s use under the terms of the Contract. No party other than Google is entitled to rely on the Final Report for any purpose whatsoever and Deloitte accepts no responsibility or liability or duty of care to any party other than Google in respect of the Final Report and any of its contents. As set out in the Contract, the scope of our work has been limited by the time, information and explanations made available to us. The information contained in the Final Report has been obtained from Google and third party sources that are clearly referenced in the appropriate sections of the Final Report. -
Ovirt Storage Overview
Ovirt Storage Overview Jun 8 th, 2012 Barak Azulay Storage overview – oVirt workshop 2011 1 Agenda ● Defining the problem ● Roadmap ● Storage Domain ● How to contribute ● Storage Pool ● Q&A ● Example ● Domain Classes ● Domain Types ● File Domains ● Block Domains ● SPM ● Thin Provisioning Storage overview – oVirt workshop 2011 2 Defining the problem ● The requirements ● Manage tens of thousands of virtual disk images ● Each image potentially accessible from hundreds (thousands?) of nodes ● Support both file based storage as well as raw block devices ● The assumptions ● Exclusive access when used ● Big (X GBs) ● Centralized manager (can be HA in and by itself) Storage overview – oVirt workshop 2011 3 Defining the problem - guidelines ● High level, image centered API ● Cluster safe ● High performance (not in data path) ● Highly available ● no single point of failure ● Continues working in the absence of the manager ● Backing storage agnostic Storage overview – oVirt workshop 2011 4 Storage Domain ● A standalone storage entity ● Stores the images and associated metadata (but not VMs) Only true persistent storage for VDSM Domain Storage overview – oVirt workshop 2011 5 Domain Classes ● Data ● Master ● ISO (NFS only) ● Backup (NFS only) ● Domain classes are planned for deprecation Storage overview – oVirt workshop 2011 6 Domain Types ● File Domains (NFS, local dir) ● Use file system features for segmentation ● Block Domains (iSCSI, FCP, FCoE, SAS, ...) ● Use LVM for segmentation ● Very specialized use of LVM Storage overview – oVirt workshop -
The Cloud V. Data Center Decision
SPECIAL REPORT The cloud v. data center decision COPYRIGHT ©2017 CBS INTERACTIVE INC. ALL RIGHTS RESERVED. THE CLOUD V. DATA CENTER DECISION TABLE OF CONTENTS 03 Cloud v. data center: Key trends for IT decision-makers 11 Infographic: Cloud use is growing, but the data center isn’t dead yet 12 Understanding the pros and cons of five cloud types 16 Five major pitfalls to avoid in a cloud migration 19 Multi-cloud is a messy reality, but there’s hope 23 Six reasons why companies hang on to their data centers 27 Has IT’s default setting switched from data center to cloud? 30 How hybrid cloud is strengthening Fitness First 2 COPYRIGHT ©2017 CBS INTERACTIVE INC. ALL RIGHTS RESERVED. THE CLOUD V. DATA CENTER DECISION CLOUD V. DATA CENTER: KEY TRENDS FOR IT DECISION-MAKERS BY CHARLES MCLELLAN Cloud-based compute, networking and storage infrastructure, and cloud-native applications, are now firmly on the radar of IT managers—be they in startups, small businesses or large enterprises. So much so that, whereas a few years ago the question facing CIOs was “Which workloads should I move to the cloud?”, it’s now becoming “Which, if any, workloads should I keep on-premises?”. While most organisations will probably end up pursuing a hybrid cloud strategy, it’s worth examining this turnaround, and the reasons behind it. The general background, as ZDNet has explored in recent special features, is the competitive pressure for organisations to undergo a digital transformation based on cloud-native applications and methods such as DevOps, in pursuit of improved IT and organisational performance. -
Amazon Web Services (AWS) Is Committed
The Cloud Amazon Web Services (AWS) is committed to running our business in the most environmentally friendly way possible and achieving 100% renewable energy usage for our global infrastructure. AMAZON SUSTAINABILITY ENVIRONMENT 44 Learn how AWS is working to achieve Amazon’s goal of 100% renewable energy. JAN 2016 JAN Amazon launches Wind Farm JUL 2015 AWS announces Fowler Ridge, marking the first that it has contract- of our four announced renew- APR 2015 Amazon announces joining ed with Iberdrola able energy projects to move the American Council on Renewables, LLC into full operation. Renewable Energy (ACORE) to construct and op- and will participate in erate Amazon Wind JUN 2016 AWS and Dominion the U.S. Partnership for Farm U.S. East. Virginia Power join forces Renewable Energy Finance on a landmark renewable (U.S. PREF) to increase our energy delivery deal. work with state and federal NOV 2015 NOV With this, Dominion Vir- policymakers and other AWS announces that it ginia Power will manage stakeholders to enable more has contracted with EDP and integrate the energy renewable energy opportu- Renewables to construct produced from various nities for cloud providers. and operate Amazon Wind Amazon wind and solar OCT 2016 Farm U.S. Central. Amazon Solar Farm U.S. farm projects onto the East is now in production grid that serves AWS in Accomack County, data centers. Virginia. 2014SEP 2015 2015 2016 Amazon joins The Buyers’ Princi- NOV 2016 NOV ples to collaborate with more than AWS announces five new solar farms across the Common- NOV 2014 NOV AWS shares 40 other companies on making wealth of Virginia. -
All About Cloud
Everything as a service HP’s point of view on cloud computing and cloud services Rebecca Lawson April 2010 1 Everything as a Service A world of information, opportunities and experiences—from computing power to business processes to personal interactions, are delivered wherever, however and whenever you need it. 2 It’s All About the Service Service providers Service consumers Field engineer Business Finance services executive Campaign director Application Sales person services Logistics manager Data Call center services representative App developer Infrastructure QA engineer services Service manager 3 Driving Outcomes That Matter Technology- Business enabled Business Outcomes Strategy services Improve value Become more agile Enable innovation 4 Source and Govern Services That Result in Outcomes That Matter Cloud service IT organization Outcomes that providers Strategic service broker matter Hosted/managed Accelerate growth service providers Services Service Services Lower costs Network service sourced portfolio delivered providers Mitigate risk Internal service providers 5 Perspective on the IT Service Supply Chain Cloud service provider IT organization Global-class Service portfolio software Business users Massive scale-out infrastructure S S S S S Business S outcome Enterprise-class Hosted, managed, software outsourced service provider S Dedicated / shared Enterprise-class infrastructure software Dedicated / shared infrastructure 6 The Real Question: Which Services, From Which Source? Corporate website service Sales forecasting service E-mail service Linux Dev lab service Supply Chain service 7 Cloud Services Deliver New Kinds of Value Service providers Service consumers Cloud services are highly scalable and elastic technology-enabled services, delivered and consumed over the internet through an as-needed, pay-per-use business model.