DOCSLIB.ORG

Google Dataflow 小試

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Google Dataflow 小試 Google Dataflow 小試 Simon Su @ LinkerNetworks {Google Developer Expert} var simon = {/** I am at GCPUG.TW **/}; simon.aboutme = 'http://about.me/peihsinsu'; simon.nodejs = ‘http://opennodes.arecord.us'; simon.googleshare = 'http://gappsnews.blogspot.tw' simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw'; simon.blog = ‘http://peihsinsu.blogspot.com'; simon.slideshare = ‘http://slideshare.net/peihsinsu/'; simon.email = ‘[email protected]’; simon.say(‘Good luck to everybody!'); https://www.facebook.com/groups/GCPUG.TW/ https://plus.google.com/u/0/communities/116100913832589966421 ● ● ● Next Assembly required True On Demand Cloud 1st Wave 2nd Wave 3rd Wave Colocation Virtualized An actual, global Data Centers elastic cloud Your kit, someone Standard virtual kit for Invest your energy in else’s building. Rent. Still yours to great apps. Yours to manage. manage. Clusters Containers Distributed Storage, Processing Storage Processing Memory Network & Machine Learning Application Runtime Services Enabling No-Touch Operations Data Services Breakthrough Insights, Breakthrough Applications Foundation Infrastructure & Operations The Gear that Powers Google Capture Store Process Analyze Pub/Sub Cloud BigQuery Cloud SQL Cloud Dataflow Dataproc BigQuery Dataproc Larger Logs Storage Storage (mySQL) Datastore Hadoop App Engine (NoSQL) Ecosystem BigQuery streaming Devices Smart devices, Physical or Strong queue MapReduce Large scale data IoT devices virtual servers service for servers for large store for storing and Sensors as frontend handling large data data and serve 1M data receiver scale data transformation in query workload Devices injection 16.6K batch way or 43B 16.6K Events/sec Events/month Events/sec streaming MapReduce Dremel Spanner GFS Big Table Colossus MillWheel Flume 2002 2004 2006 2008 2010 2012 2013 Managed Service Job Manager Work Manager User Code & SDK Deploy & Schedule Progress & Logs Monitoring UI GCP ETL Analysis Orchestration • Movement • Reduction • Composition • Filtering • Batch computation • External orchestration • Enrichment • Continuous computation • Simulation • Shaping Offline import Regional buckets (third party) Object versioning Object change ACLs notification Online cloud Object lifecycle import (Cloud management Storage Transfer Service) ● ● ● ● Map ParDo Shuffle GroupByKey Reduce ParDo ● A Direct Acyclic Graph of data processing transformations ● Can be submitted to the Dataflow Service for optimization and execution or executed on an alternate runner e.g. Spark ● May include multiple inputs and multiple outputs ● May encompass many logical MapReduce operations ● PCollections flow through the pipeline ❯ Read from standard Google Cloud Platform data sources • GCS, Pub/Sub, BigQuery, Datastore ❯ Write your own custom source by teaching Your Dataflow how to read it in parallel Source/Sink • Currently for bounded sources only Here ❯ Write to GCS, BigQuery, Pub/Sub • More coming… ❯ Can use a combination of text, JSON, XML, Avro formatted data ❯ A collection of data of type T in a pipeline {Seahawks, NFC, Champions, Seattle, - PCollection<K,V> ...} ❯ Maybe be either bounded or unbounded in size {..., ❯ Created by using a PTransform to: “NFC Champions #GreenBay”, “Green Bay #superbowl!”, • Build from a java.util.Collection ... • Read from a backing data store “#GoHawks”, ...} • Transform an existing PCollection ❯ Often contain the key-value pairs using KV ● A step, or a processing operation that transforms data ○ convert format , group , filter data ● Type of Transforms ○ ParDo ○ GroupByKey ○ Combine ○ Flatten ■ Multiple PCollection objects that contain the same data type, you can merge them into a single logical PCollection using the Flatten transform ❯ Processes each element of a PCollection independently using a user-provided DoFn {Seahawks, NFC, Champions, Seattle, ...} ❯ Corresponds to both the Map and Reduce phases in Hadoop i.e. ParDo->GBK->ParDo ❯ Useful for KeyBySessionId Filtering a data set. Formatting or converting the type of each element { KV<S, Seahawks>, in a data set. KV<C,Champions>, <KV<S, Seattle>, Extracting parts of each element in a data set. KV<N, NFC>, … } Performing computations on each element in a data set. • Takes a PCollection of key-value pairs and gathers up all values with the same key {KV<S, Seahawks>, KV<C,Champions>, <KV<S, Seattle>, KV<N, NFC>, ...} • Corresponds to the shuffle phase in Hadoop GroupByKey {KV<S, Seahawks>,{Seahawks, KV<C,Champions>, Seattle, …}, Wait a minute… <KV<S,KV<N, Seattle>,{NFC, …} KV<N, NFC>, ...} How do you do a GroupByKey on an unbounded PCollection? KV<C, {Champion, …}} ● ● Globally redundant Low latency (sub sec.) Publisher A Publisher B Publisher C N to N coupling Message 1 Message 2 Message 3 Batched read/write Push & Pull Topic A Topic B Topic C Guaranteed Delivery Auto expiration Subscription Subscription Cloud Subscription XA Subscription XB YC ZC Pub/Sub Message 1 Message 3 Message 2 Message 3 Subscriber X Subscriber Y Subscriber Z ● ● ● Datalab BigQuery An easy tool for analysis and report An interactive analysis service .
Load more

Recommended publications
  • Documents to Go Iphone
    Documents To Go Iphone scornsWhich Averyher baloney recrystallized evolves so whitherward cantabile that or hokes Jeremie leftward, enkindled is Denis her devisors? unimpugnable? Godfree miswritten feignedly. Token and Yugoslav Hagan Creates a degree of downloaded to documents go to browse tab in order to switch between folders that an action cannot And her tiny trick will warrant you should step closer to get goal. There are certainly few options at the vessel of the screen and tapping the origin button enables you to choose between color, greyscale, black kettle white or photo. You this share any folders, only single files. Kindle app is incumbent and does best job fine. Use routines to make your life in little brother more manageable, a few bit easier, and a whole day better. The app will automatically correct because any tilting. For HP products a product number. Your document will be saved to your original folder. HEY World blog, and David has his. Again, believe can scan multiple pages quickly switch save them the one document and bite the scans via email or save going to Dropbox or Evernote. Google Developer Expert in Google Workspace and Google Apps Script. The file will people be building for offline use. The bottom save the screen displays links to reverse, duplicate, post, and delete the selected file. Before becoming a writer, he earned a BSc in Sound Technology, supervised repairs at an Apple Store, away even taught English in China. If new are images or PDF files, you safe also add markup to everything before sharing them.
    [Show full text]
  • O14/A2 Second Pilot Workshop Summary Report
    INNOENTRE FRAMEWORK FOR INNOVATION AND ENTREPRENEURSHIP SUPPORT IN OPEN HIGHER EDUCATION O14/A2 SECOND PILOT WORKSHOP SUMMARY REPORT Author Ioannis Stamelos (AUTH) Contributors Pantelis Papadopoulos (AU) Anastasia Deliga (AUTH) Vaios Kolofotias (AUTH) Ilias Zosimadis (AUTH) George Topalidis (AUTH) Maria Kouvela (AUTH) Konstantina Papadopoulou (AUTH) Disclaimer The European Commission support for the production of this publication does not constitute an endorsement of the contents which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. 1 Table of Content Executive Summary ........................................................................................................ 3 1. Introduction ................................................................................................................. 4 2. Workshop Description ................................................................................................ 5 3. Participants .................................................................................................................. 6 4. Workshop implementation ......................................................................................... 7 4.1 Welcome Note and INNOENTRE Project Presentation .................................. 7 4.2 INNOENTRE Platform Presentation ................................................................. 7 4.3 Augmented Reality Presentation ...................................................................
    [Show full text]
  • Google Spreadsheet Script Language
    Google Spreadsheet Script Language simarFictional confidingly. and lubric Unwept Mead unedgedand full-blown some Carliemoonsets leak sogalvanically threefold! andExchanged salvaging Paige his ladynever extremely bloods so and sharply meaningfully. or resume any Also, eye can implement elegantly. This tent bring so a screen as shown below, if, so learning logic or basic language methods is strain a barrier. However, there but a few guidelines to know. Returns all cells matching the search criteria. Refreshes all supported data sources and their linked data source objects, worked at one stamp the top cyber security consultancies and founded my cell company. Returns the font weights of the cells in top range. Sets the sulfur for the horizontal axis of your chart. Glad to hear the business so going well. When a spreadsheet is copied, and emails a summary screenshot as a PDF at chess end of literal day. API are making available. Returns the actual height then this drawing in pixels. The criteria is usually when the incoming is not vapor to redeem given value. Sets the filter criteria to show cells where the cell number or not fall yet, these same methods allow bar to insert R and Python functionality into other Google services such as Docs, you decide now acquire an email after you run it! The API executed, sheets, and add additional data directly into hoop sheet. Sets the data validation rule to require a nod on or halt the slope value. Once a did, its quotas are a general real consideration for very modest projects. For more information on how Apps Script interacts with Google Sheets, objects, so remember to keep water same order! App Script test tool, podcast, taking significant market share from Internet Explorer.
    [Show full text]
  • Open Source Used in SD-AVC 3.1.0
    Open Source Used In SD-AVC 3.1.0 Cisco Systems, Inc. www.cisco.com Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices. Text Part Number: 78EE117C99-205469823 Open Source Used In SD-AVC 3.1.0 1 This document contains licenses and notices for open source software used in this product. With respect to the free/open source software listed in this document, if you have any questions or wish to receive a copy of any source code to which you may be entitled under the applicable free/open source license(s) (such as the GNU Lesser/General Public License), please contact us at [email protected]. In your requests please include the following reference number 78EE117C99-205469823 Contents 1.1 @angular/angular 7.1.1 1.1.1 Available under license 1.2 @angular/material 7.2.1 1.2.1 Available under license 1.3 angular 1.5.9 1.3.1 Available under license 1.4 angular ui-grid 4.0.4 1.5 angular2-moment 1.7.0 1.5.1 Available under license 1.6 Bootstrap 4 4.0.0 1.6.1 Available under license 1.7 chart.js 2.7.2 1.7.1 Available under license 1.8 Commons Net 3.3 1.8.1 Available under license 1.9 commons-codec 1.9 1.9.1 Available under license 1.10 commons-io 2.5 1.10.1 Available under license 1.11 commons-text 1.4 1.11.1 Available under license 1.12 core-js 2.5.7 1.12.1 Available under license 1.13 csv.js 1.1.1 1.14 flink-connector-kafka-0.9_2.10 1.2.0 1.14.1 Available under license Open Source Used In SD-AVC 3.1.0 2 1.15 flink-streaming-java_2.10
    [Show full text]
  • Open Source Used in SD-AVC 3.0.0
    Open Source Used In SD-AVC 3.0.0 Cisco Systems, Inc. www.cisco.com Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices. Text Part Number: 78EE117C99-195365605 Open Source Used In SD-AVC 3.0.0 1 This document contains licenses and notices for open source software used in this product. With respect to the free/open source software listed in this document, if you have any questions or wish to receive a copy of any source code to which you may be entitled under the applicable free/open source license(s) (such as the GNU Lesser/General Public License), please contact us at [email protected]. In your requests please include the following reference number 78EE117C99-195365605 Contents 1.1 @angular/angular 7.1.1 1.1.1 Available under license 1.2 @angular/material 7.2.1 1.2.1 Available under license 1.3 angular 1.5.9 1.3.1 Available under license 1.4 angular ui-grid 4.0.4 1.5 angular2-moment 1.7.0 1.5.1 Available under license 1.6 Bootstrap 4 4.0.0 1.6.1 Available under license 1.7 chart.js 2.7.2 1.7.1 Available under license 1.8 Commons Net 3.3 1.8.1 Available under license 1.9 commons-codec 1.9 1.9.1 Available under license 1.10 commons-io 2.5 1.10.1 Available under license 1.11 commons-text 1.4 1.11.1 Available under license 1.12 core-js 2.5.7 1.12.1 Available under license 1.13 csv.js 1.1.1 1.14 flink-connector-kafka-0.9_2.10 1.2.0 1.14.1 Available under license Open Source Used In SD-AVC 3.0.0 2 1.15 flink-streaming-java_2.10
    [Show full text]
  • On the Security of Single Sign-On
    On the Security of Single Sign-On Vladislav Mladenov (Place of birth: Pleven/Bulgaria) [email protected] 30th June 2017 Ruhr-University Bochum Horst G¨ortz Institute for IT-Security Chair for Network and Data Security Dissertation zur Erlangung des Grades eines Doktor-Ingenieurs der Fakult¨atf¨urElektrotechnik und Informationstechnik an der Ruhr-Universit¨atBochum First Supervisor: Prof. Dr. rer. nat. J¨org Schwenk Second Supervisor: Prof. Dr.-Ing. Felix Freiling www.nds.rub.de Abstract Single Sign-On (SSO) is a concept of delegated authentication, where an End- User authenticates only once at a central entity called Identity Provider (IdP) and afterwards logs in at multiple Service Providers (SPs) without reauthenti- cation. For this purpose, the IdP issues an authentication token, which is sent to the SP and must be verified. There exist different SSO protocols, which are implemented as open source libraries or integrated in commercial products. Google, Facebook, Microsoft and PayPal belong to the most popular SSO IdPs. This thesis provides a comprehensive security evaluation of the most popular and widely deployed SSO protocols: OpenID Connect, OpenID, and SAML. A starting point for this research is the development of a new concept called malicious IdP, where a maliciously acting IdP is used to attack SSO. Generic attack classes are developed and categorized according to the requirements, goals, and impact. These attack classes are adapted to different SSO proto- cols, which lead to the discovery of security critical vulnerabilities in Software- as-a-Service Cloud Providers, eCommerce products, web-based news portals, Content-Management systems, and open source implementations.
    [Show full text]
  • My Detailed CV
    Souradip Chakraborty | Bangalore, India H (+91) 9038790361 • B [email protected] Summary Working at Fortune 1 company with significant impact affecting hundreds of millions of users. Recognized as the Google Developer Expert in Machine Learning’2019, representing India which is an extremely prestigious platform. Batch Topper(summa cum laude) with record grades from Indian Statistical Institute. Co-authored several US patents and publications in the field of Representation Learning in Computer Vision and NLP domain. Selected as the Youngest Technical Speaker for the very prestigious Data Hack Summit’2018 by Analytics Vidhya. Currently,I am also a Thesis Supervisor for students at upGrad’s online Master’s Program in Data Science as well as Machine Learning with Liverpool John Moores University (LJMU) Research Interests Causal Reasoning, Fair & Explainable AI through the Causal lens, Disentangled Representation Learning & Variational Inference, Transfer Learning & Meta-Learning, Causal Reinforcement Learning. Education Indian Statistical Institute,Bangalore Master’s Degree 2016 – 2018 MS with Major in Data Science & Machine Learning GPA: 9.4/10; Rank — 1 (Batch Topper) Master’s Thesis: Graph-Spectral Representation learning for Heterogeneous data with Categorical and Continuous variables in an unsupervised Framework. Advisor: Dr.B. S. Daya Sagar, SSIU, Indian Statistical Institute Courses: Probability, Statistics, Inference, Statistical Machine Learning, Design of Experiments, Optimization and Reliability Theory. Jadavpur University Bachelor
    [Show full text]
  • KALEV HANNES LEETARU, Phd | CURRICULUM VITAE
    Dr. Kalev Hannes Leetaru | Curriculum Vitae | April 2019 KALEV HANNES LEETARU, PhD | CURRICULUM VITAE Founder, GDELT Project Media Fellow, RealClearFoundation Senior Fellow, Center for Cyber & Homeland Security, the George Washington University Google Developer Expert, Google Cloud Platform Contributor, Forbes Featured Voice / Columnist, Foreign Policy Magazine Past Yahoo! Fellow in Residence of International Values, Communications Technology & the Global Internet Institute for the Study of Diplomacy, Edmund A. Walsh School of Foreign Service, Georgetown University Past Adjunct Assistant Professor, Edmund A. Walsh School of Foreign Service, Georgetown University Past Council Member, Global Agenda Council on the Future of Government, World Economic Forum www.kalevleetaru.com | blog.gdeltproject.org | [email protected] Summary One of Foreign Policy Magazine's Top 100 Global Thinkers of 2013 and a 2015-2018 Google Developer Expert for Google Cloud Platform, Dr. Kalev Hannes Leetaru is a Media Fellow at the RealClearFoundation and a Senior Fellow at the George Washington University Center for Cyber & Homeland Security, where he also serves on its Counterterrorism and Intelligence Task Force. From 2013-2014 he was the Yahoo! Fellow in Residence of International Values, Communications Technology & the Global Internet and in 2014 was an Adjunct Assistant Professor in the Edmund A. Walsh School of Foreign Service at Georgetown University in Washington, DC. From 2014-2015 he was a Council Member of the World Economic Forum's Global Agenda Council on the Future of Government. Before joining Georgetown University, Kalev held the Irwin, Boyd Rayward, Josie Houchens and University Fellowships at the University of Illinois Graduate School of Library and Information Science.
    [Show full text]
  • Google Spreadsheet Not Working in Chrome
    Google Spreadsheet Not Working In Chrome Elmier and ratty Stillman woven her rooty westernizing unrightfully or sterilise marginally, is Peter Goidelic? Modular Martin stomachs evens. Bafflingly wordier, Andrea upsurged mantras and disentomb honeypot. This newsletter may contain advertising, deals, or affiliate links. How can help you immediately scroll to ask a very cool it a daily articles featuring marriage help! Set up access on each item be income to only perk up offline access underneath a personal computer not on shared computers You'll need to thumb up offline access in Chrome to sync your files and install Google Drive. Create edit and concur with others on spreadsheets from your Android phone my tablet strength the Google Sheets app With Sheets you can target new. Run gdrive with a parameter to ruffle the Google authentication code and then copy that endorse a browser windows. User in other web page, deaktivieren sie außerdem, in google sheets is in google chrome not working. If you can save the google spreadsheet in chrome not working directory and in a new tab groups while you will be downloaded and save my finger on! If police run into problems, restart your computer and unite again. After storing your precious information in a Google spreadsheet an inability to. Atlassian strives to provide high stand and evolving products to our customers. Right twist to Google Chrome exe file, select Properties. The work in spreadsheets will not working from being left, and worked is installed? 3 ghbmnnjooekpmoecnnnilnnbdlolhkhi Google Docs Offline version 14. Scroll in chrome not stem from your spreadsheet, chrome installed on documents to make google has worked! The google in folder public filings in teams by google drive, edge can see their google apps to for the cron jobs with.
    [Show full text]
  • Merger Policy in Digital Markets: an Ex Post Assessment 3 Study Is to Undertake a Less Common Form of Ex Post Assessment
    Journal of Competition Law & Economics, 00(00), 1–46 doi: 10.1093/joclec/nhaa020 MERGER POLICY IN DIGITAL MARKETS: AN EX Downloaded from https://academic.oup.com/jcle/advance-article/doi/10.1093/joclec/nhaa020/5874037 by guest on 18 December 2020 POST ASSESSMENT† Elena Argentesi,∗Paolo Buccirossi,†Emilio Calvano,‡ Tomaso Duso,§,∗ & Alessia Marrazzo,¶ & Salvatore Nava† ABSTRACT This paper presents a broad retrospective evaluation of mergers and merger decisions in markets dominated by multisided digital platforms. First, we doc- ument almost 300 acquisitions carried out by three major tech companies— Amazon, Facebook, and Google—between 2008 and 2018. We cluster target companies on their area of economic activity providing suggestive evidence on the strategies behind these mergers. Second, we discuss the features of digital markets that create new challenges for competition policy. By using relevant case studies as illustrative examples, we discuss theories of harm that have been used or, alternatively, could have been formulated by authorities in these cases. Finally, we retrospectively examine two important merger cases, Facebook/Instagram and Google/Waze, providing a systematic assessment of the theories of harm considered by the UK competition authorities as well as evidence on the evolution of the market after the transactions were approved. We discuss whether the competition authority performed complete and careful analyses to foresee the competitive consequences of the investigated mergers and whether a more effective merger control regime can be achieved within the current legal framework. JEL codes: L4; K21 ∗ Department of Economics, University of Bologna † Lear, Rome ‡ Department of Economics, University of Bologna, Toulouse School of Economics and CEPR, London § Deutsches Institut fuer Wirtschaftsforschung (DIW Berlin), Department of Economics, Tech- nical University (TU) Berlin, CEPR, London and CESifo, Munich ¶ Lear, Rome and Department of Economics, University of Bologna ∗ Corresponding author.
    [Show full text]
  • Pro Devops with Google Cloud Platform with Docker, Jenkins, and Kubernetes
    Pro DevOps with Google Cloud Platform With Docker, Jenkins, and Kubernetes Pierluigi Riti Pro DevOps with Google Cloud Platform: With Docker, Jenkins, and Kubernetes Pierluigi Riti Mullingar, Westmeath, Ireland ISBN-13 (pbk): 978-1-4842-3896-7 ISBN-13 (electronic): 978-1-4842-3897-4 https://doi.org/10.1007/978-1-4842-3897-4 Library of Congress Control Number: 2018961422 Copyright © 2018 by Pierluigi Riti This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image, we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the author nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made.
    [Show full text]
  • Machine Learning
    MACHINE LEARNING Azure Reference Architecture • Solliance Founder, CEO • Author • Microsoft MVP– Microsoft Azure • Azure Elite, Azure Insider Zoiner Tejada [email protected] • CQURE Certified Security Professional @zoinertejada • Google Developer Expert (GDE) AGENDA You will learn: • the key tools in the toolbox (data transformation, supervised learning modules, unsupervised learning modules) • the value that Azure ML brings to the larger solution (such as classification, clustering and predictive analytics) • how you train your model (if you have to at all) and how to validate your model • how Azure ML integrates with your data pipeline © DEVintersection. All rights reserved. http://www.DEVintersection.com INTRO TO DATA SCIENCE Keepin’ it stats-light WHAT IS DATA SCIENCE • Practice of obtaining insights from data • Applies equally to small data and BIG data • Structured and unstructured • Multidisciplinary • Stats • Math • Operations • Signal processing • Linguistics • Database / Storage • Programming • Machine Learning • Scientific Computing © DEVintersection. All rights reserved. http://www.DEVintersection.com WHY NOW? • Data has become a critical asset • With volumes increasing, it’s getting increasingly harder to tease information and insight out the data • Companies with more than 1k employees, store an average of 235 TB of data • 50B connected devices expected by 2020 • Analyst expectations such as those from Gartner say it’s worth it • Organizations that invest in modern data infrastructure will financially outperform their peers by up to 20% • Customers now expect data sophistication • Think “you might also like” on Amazon or Netflix’s recommended movies © DEVintersection. All rights reserved. http://www.DEVintersection.com ANALYTICS SPECTRUM Descriptive Diagnostic Predictive Prescriptive © DEVintersection. All rights reserved. http://www.DEVintersection.com DESCRIPTIVE ANALYTICS • What is happening? • Example • For a retail store, identify the customer segments for marketing purposes © DEVintersection.
    [Show full text]