DOCSLIB.ORG

Bigquery Audit Logs Overview | Google Cloud

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Bigquery Audit Logs Overview | Google Cloud 8/23/2020 BigQuery audit logs overview | Google Cloud BigQuery audit logs overview Overview Cloud Audit Logs are a collection of logs provided by Google Cloud that provide insight into operational concerns related to your use of Google Cloud services. This page provides details about BigQuery specic log information, and it demonstrates how to use BigQuery to analyze logged activity. Versions The audit log message system relies on structured logs, and the BigQuery service provides three distinct kinds of messages: AuditData (#auditdata_examples): The old version of logs, which reports API invocations. BigQueryAuditMetadata (#bigqueryauditmetadata_examples): The new version of logs, which reports resource interactions such as which tables were read from and written to by a given query job and which tables expired due to having an expiration time congured. In general, users will want to leverage the new BigQueryAuditMetadata logs functionality. AuditLog (#auditlog_examples): The logs that BigQuery Reservations (/bigquery/docs/reservations-intro) and BigQuery Connections (/bigquery/docs/reference/bigqueryconnection/rest) use when reporting requests. Message Formats AuditData format The AuditData (/bigquery/docs/reference/auditlogs/rest/Shared.Types/AuditData) messages are communicated within the protoPayload.serviceData submessage within the Cloud Logging LogEntry (/logging/docs/reference/v2/rest/v2/LogEntry) message. https://cloud.google.com/bigquery/docs/reference/auditlogs 1/11 8/23/2020 BigQuery audit logs overview | Google Cloud BigQueryAuditMetadata format You can nd BigQueryAuditMetadata (/bigquery/docs/reference/auditlogs/rest/Shared.Types/BigQueryAuditMetadata) details in the protoPayload.metadata submessage that is in the Cloud Logging LogEntry (/logging/docs/reference/v2/rest/v2/LogEntry) message. In the Cloud Logging logs, the protoPayload.serviceData information is not set or used. In BigQueryAuditMetadata messages, there is more information: resource.type is set to one of the following values: bigquery_project for jobs resource.labels.location contains the location of the job. bigquery_dataset for storage resource.labels.dataset_id contains the encapsulating dataset. protoPayload.methodName is set to one of the following values: google.cloud.bigquery.v2.TableService.InsertTable google.cloud.bigquery.v2.TableService.UpdateTable google.cloud.bigquery.v2.TableService.PatchTable google.cloud.bigquery.v2.TableService.DeleteTable google.cloud.bigquery.v2.DatasetService.InsertDataset google.cloud.bigquery.v2.DatasetService.UpdateDataset google.cloud.bigquery.v2.DatasetService.PatchDataset google.cloud.bigquery.v2.DatasetService.DeleteDataset google.cloud.bigquery.v2.TableDataService.List google.cloud.bigquery.v2.JobService.InsertJob google.cloud.bigquery.v2.JobService.Query google.cloud.bigquery.v2.JobService.GetQueryResults InternalTableExpired https://cloud.google.com/bigquery/docs/reference/auditlogs 2/11 8/23/2020 BigQuery audit logs overview | Google Cloud protoPayload.resourceName now contains the URI for the referenced resource. For example, a table created by using an insert job reports the resource URI of the table The earlier format reported the API resource (the job identier). protoPayload.authorizationInfo only includes information relevant to the specic event. With earlier AuditData messages, you could merge multiple records when source and destination tables were in the same dataset in a query job. AuditLog format BigQuery Reservations (/bigquery/docs/reservations-intro) uses the AuditLog (/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog) format when reporting requests. Logs contain information such as: resource.type is set to: bigquery_project for jobs resource.labels.location contains the location of the reservation-related resource. protoPayload.methodName is set to one of the following values: google.cloud.bigquery.reservation.v1beta1.ReservationService.CreateReserva tion google.cloud.bigquery.reservation.v1beta1.ReservationService.DeleteReserva tion google.cloud.bigquery.reservation.v1beta1.ReservationService.UpdateReserva tion google.cloud.bigquery.reservation.v1beta1.ReservationService.CreateCapacit yCommitment google.cloud.bigquery.reservation.v1beta1.ReservationService.DeleteCapacit yCommitment google.cloud.bigquery.reservation.v1beta1.ReservationService.CreateAssignm ent google.cloud.bigquery.reservation.v1beta1.ReservationService.DeleteAssignm ent https://cloud.google.com/bigquery/docs/reference/auditlogs 3/11 8/23/2020 BigQuery audit logs overview | Google Cloud google.cloud.bigquery.reservation.v1beta1.ReservationService.MoveAssignmen t BigQuery Connections (/bigquery/docs/reference/bigqueryconnection/rest) uses the AuditLog (/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog) format when reporting requests. Logs contain information such as: resource.type is set to: audited_resource resource.labels.method contains the full method name. resource.labels.project_id contains the project name. resource.service contains service name. protoPayload.methodName is set to one of the following values: google.cloud.bigquery.connection.v1.ConnectionService.CreateConnection google.cloud.bigquery.connection.v1.ConnectionService.DeleteConnection google.cloud.bigquery.connection.v1.ConnectionService.UpdateConnection google.cloud.bigquery.connection.v1.ConnectionService.SetIamPolicy Mapping audit entries to log streams Audit logs are organized into the following three streams. For more information about the streams, see the Cloud Audit Logs (/logging/docs/audit) documentation. Data access System event Admin activity Data access (data_access) The data_access stream contains entries about jobs by using the JobInsertion and JobChange events and about table data modications by using the TableDataChange and TableDataRead https://cloud.google.com/bigquery/docs/reference/auditlogs 4/11 8/23/2020 BigQuery audit logs overview | Google Cloud events. For example, when a load job appends data to a table, the data_access stream adds a TableDataChange event. A TableDataRead event indicates when a consumer reads a table. Note: BigQuery does not emit data access log entries in the following scenarios: Data appended to a table by using the streaming insert mechanism does not generate TableDataChange log entries. Recursive dataset deletions, such as removing a dataset and its contents in a single API call, do not yield deletion entries for each resource contained in the dataset. The dataset removal is present in the activity log. Partitioned tables do not generate TableDataChange entries for partition expirations. Wildcard tables access generates a single TableDataRead entry and doesn't write a separate entry for each queried table. System event (system_event) You can set an expiration time on tables to remove them at a specied time. The system_event stream reports a TableDeletion event when the table expires and is removed. Admin activity (activity) The main activity stream reports all remaining activities and events such as table and dataset creation. Creating, deleting, and updating resources related to BigQuery Reservations (/bigquery/docs/reservations-concepts) are reported in the admin activity stream. Visibility and access control BigQuery audit logs can include information that users might consider sensitive, such as SQL text, schema denitions, and identiers for resources such as table and datasets. For information about managing access to this information, see the Cloud Logging access control documentation (/logging/docs/access-control). https://cloud.google.com/bigquery/docs/reference/auditlogs 5/11 8/23/2020 BigQuery audit logs overview | Google Cloud Caller identities and resource names Caller identities and IP addresses are redacted from the audit logs if all of the following conditions are true: This is a read-only access. The resource is public. The identity is not a service account that belongs to the project. The identity does not belong to the same customer as the project. For cross-project access, there are additional rules that apply: The billing project must be the project that sends the request, and the data project must be the project whose resources are also accessed during the job. For example, a query job in a billing project reads some table data from the data project. The billing project resource ID is redacted from the data project log unless the projects have the same domain associated with them or are in the same organization. Identities and caller IP addresses are not redacted from the data project log if either one of the preceding conditions apply or the billing project and the data project are in the same organization and the billing project already includes the identity and caller IP address. Cloud Logging expos BigQuery automatically sends audit logs to Cloud Logging. Cloud Logging lets users lter and export messages to other services (/logging/docs/export), including Pub/Sub, Cloud Storage, and BigQuery. With long term log retention and log exports to BigQuery, you can do aggregated analysis on logs data. Cloud Logging documents how messages are transformed (/logging/docs/export/bigquery) when exported to BigQuery. Filtering expos To lter relevant BigQuery Audit messages, you can express lters as part of the export. https://cloud.google.com/bigquery/docs/reference/auditlogs 6/11 8/23/2020 BigQuery audit logs overview | Google Cloud For example, the following advanced lter represents an export that only includes the newer BigQueryAuditMetadata format: Payload.metadata."@type"="type.googleapis.com/google.cloud.audit.BigQueryAuditMetada
Load more

Recommended publications
  • Data Warehouse Offload to Google Bigquery
    DATA WAREHOUSE OFFLOAD TO GOOGLE BIGQUERY In a world where big data presents both a major opportunity and a considerable challenge, a rigid, highly governed traditional enterprise data warehouse isn’t KEY BENEFITS OF MOVING always the best choice for processing large workloads, or for applications like TO GOOGLE BIGQUERY analytics. Google BigQuery is a lightning-fast cloud-based analytics database that lets you keep up with the growing data volumes you need to derive meaningful • Reduces costs and business value, while controlling costs and optimizing performance. shifts your investment from CAPEX to OPEX Pythian’s Data Warehouse Offload to Google BigQuery service moves your workload from an existing legacy data warehouse to a Google BigQuery data • Scales easily and on demand warehouse using our proven methodology and Google experts–starting with a fixed-cost Proof of Concept stage that will quickly demonstrate success. • Enables self-service analytics and advanced analytics GETTING STARTED The Pythian Data Warehouse Offload to Google BigQuery service follows a proven methodology and delivers a Proof of Concept (POC) that demonstrates viability and value within three to four weeks. The POC phase will follow this workflow: 1. Assess existing data warehouse environment to identify tables and up to two reports that will be offloaded in this phase 2. Provision GCP infrastructure including Cloud storage, Bastion hosts, BigQuery, and Networking 3. Implement full repeatable extract/load process for selected tables 4. Implement selected reports on BigQuery 5. Produce report PYTHIAN DELIVERS By the end of the first stage of our engagement, you can expect to have: • Working prototype on BigQuery • Up to two reports • Demonstrated analysis capabilities using one fact with five associated dimensions www.pythian.com • Report that includes: an assessment of your current setup and support you need to plan and maintain your full (including a cost analysis for BigQuery), performance/ Google BigQuery data warehouse and enterprise analytics usability analysis of POC vs.
    [Show full text]
  • Frequently Asked Questions for Google Bigquery Connector
    Frequently Asked Questions for Google BigQuery Connector © Copyright Informatica LLC 2017, 2021. Informatica, the Informatica logo, and Informatica Cloud are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at https:// www.informatica.com/trademarks.html. Abstract This article describes frequently asked questions about using Google BigQuery Connector to read data from and write data to Google BigQuery. Supported Versions • Cloud Data Integration Table of Contents General Questions............................................................ 2 Performance Tuning Questions................................................... 5 General Questions What is Google Cloud Platform? Google Cloud Platform is a set of public cloud computing services offered by Google. It provides a range of hosted services for compute, storage, and application development that run on Google hardware. Google Cloud Platform services can be accessed by software developers, cloud administrators, and other enterprise IT professionals over the public internet or through a dedicated network connection. Google Cloud Platform provides Google BigQuery to perform data analytics on large datasets. How can I access Google Cloud Platform? You must create a Google service account to access Google Cloud Platform. To create a Google service account, click the following URL: https://cloud.google.com/ What are the permissions required for the Google service
    [Show full text]
  • What's New for Google in 2020?
    Kevin A. McGrail [email protected] What’s new for Google in 2020? Introduction Kevin A. McGrail Director, Business Growth @ InfraShield.com Google G Suite TC, GDE & Ambassador https://www.linkedin.com/in/kmcgrail About the Speaker Kevin A. McGrail Director, Business Growth @ InfraShield.com Member of the Apache Software Foundation Release Manager for Apache SpamAssassin Google G Suite TC, GDE & Ambassador. https://www.linkedin.com/in/kmcgrail 1Q 2020 STORY TIME: Google Overlords, Pixelbook’s Secret Titan Key, & Googlesplain’ing CES Jan 2020 - No new new hardware was announced at CES! - Google Assistant & AI Hey Google, Read this Page Hey Google, turn on the lights at 6AM Hey Google, Leave a Note... CES Jan 2020 (continued) Google Assistant & AI Speed Dial Interpreter Mode (Transcript Mode) Hey Google, that wasn't for you Live Transcripts Hangouts Meet w/Captions Recorder App w/Transcriptions Live Transcribe Coming Next...: https://mashable.com/article/google-translate-transcription-audio/ EXPERT TIP: What is Clipping? And Whispering! Streaming Games - Google Stadia Android Tablets No more Android Tablets? AI AI AI AI AI Looker acquisition for 2.6B https://www.cloudbakers.com/blog/why-cloudbakers-loves-looker-for-business-intelligence-bi From Thomas Kurian, head of Google Cloud: “focusing on digital transformation solutions for retail, healthcare, financial services, media and entertainment, and industrial and manufacturing verticals. He highlighted Google's strengths in AI for each vertical, such as behavioral analytics for retail,
    [Show full text]
  • Getting Started with Microsoft Azure Virtual Machines
    G “ M A V M Ιντροδυχτιον Ψου χαν υσε α Μιχροσοφτ Αζυρε ςιρτυαλ Μαχηινε ωηεν ψου νεεδ α σχαλαβλε, χλουδ−βασεδ σερϖερ ρυννινγ α Wινδοωσ ορ Λινυξ οπερατινγ σψστεm ανδ ανψ αππλιχατιον οφ ψουρ χηοοσινγ. Βψ τακινγ αδϖανταγε οφ Μιχροσοφτ Αζυρε Ινφραστρυχτυρε ασ α Σερϖιχε (ΙααΣ), ψου χαν ρυν α ϖιρτυαλ mαχηινε ον αν ονγοινγ βασισ, ορ ψου χαν στοπ ανδ ρεσταρτ ιτ λατερ ωιτη νο λοσσ το ψουρ δατα ορ σερϖερ σεττινγσ. Ψου χαν θυιχκλψ προϖισιον α νεω ϖιρτυαλ mαχηινε φροm ονε οφ τηε ιmαγεσ αϖαιλαβλε φροm Μιχροσοφτ Αζυρε. Ον τηε οτηερ ηανδ, ιφ ψου αρε αλρεαδψ ρυννινγ αππλιχατιονσ ιν α ςΜωαρε ορ Ηψπερ−ς ϖιρτυαλιζεδ ενϖιρονmεντ, ψου χαν εασιλψ mιγρατε ψουρ ϖιρτυαλ mαχηινε το Μιχροσοφτ Αζυρε. Ονχε ψου ηαϖε χρεατεδ ψουρ ϖιρτυαλ mαχηινε ανδ αδδεδ ιτ το Μιχροσοφτ Αζυρε, ψου χαν ωορκ ωιτη ιτ mυχη λικε αν ον−πρεmισεσ σερϖερ βψ ατταχηινγ mορε δισκσ φορ δατα στοραγε ορ βψ ινσταλλινγ ανδ ρυννινγ αππλιχατιονσ ον ιτ. ςιρτυαλ mαχηινεσ ρελψ ον Μιχροσοφτ Αζυρε Στοραγε φορ ηιγη αϖαιλαβιλιτψ. Wηεν ψουρ ϖιρτυαλ mαχηινε ισ προϖισιονεδ, ιτ ισ ρεπλιχατεδ το τηρεε σεπαρατε λοχατιονσ ωιτηιν τηε δατα χεντερ το ωηιχη ψου ασσιγν ιτ. Ψου ηαϖε τηε οπτιον το εναβλε γεο−ρεπλιχατιον το ηαϖε χοπιεσ οφ ψουρ ϖιρτυαλ mαχηινε αϖαιλαβλε ιν α ρεmοτε δατα χεντερ ρεγιον. Χονσιδερινγ Σχεναριοσ φορ α ςιρτυαλ Μαχηινε Μιχροσοφτ Αζυρε γιϖεσ ψου νοτ ονλψ τηε φλεξιβιλιτψ το συππορτ mανψ αππλιχατιον πλατφορmσ, βυτ αλσο τηε φλεξιβιλιτψ το σχαλε υπ ανδ σχαλε δοων το συιτ ψουρ ρεθυιρεmεντσ. Φυρτηερmορε, ψου χαν θυιχκλψ προϖισιον α νεω ϖιρτυαλ mαχηινε ιν α φεω mινυτεσ. Α Μιχροσοφτ Αζυρε ϖιρτυαλ mαχηινε ισ σιmπλψ α φρεση mαχηινε πρελοαδεδ ωιτη αν οπερατινγ σψστεm οφ ψουρ χηοιχεψου χαν αδδ ανψ νεεδεδ αππλιχατιον εασιλψ.
    [Show full text]
  • Understanding Alphabet and Google, 2017
    This research note is restricted to the personal use of [email protected]. Understanding Alphabet and Google, 2017 Published: 24 February 2017 ID: G00297707 Analyst(s): Tom Austin, David Mitchell Smith, Yefim V. Natis, Isabelle Durand, Ray Valdes, Bettina Tratz-Ryan, Roberta Cozza, Daniel O'Connell, Lydia Leong, Jeffrey Mann, Andrew Frank, Brian Blau, Chris Silva, Mark Hung, Adam Woodyer, Matthew W. Cain, Steve Riley, Martin Reynolds, Whit Andrews, Alexander Linden, David Yockelson, Joe Mariano Google's size, market differentiation, rapid pace of innovation and ambitions can complicate fully understanding the vendor and its fit to current digital business needs. CIOs and IT leaders can use this report to explore in detail selected topics from the Gartner Vendor Rating. Key Findings ■ Two outcomes are apparent more than a year after the creation of the Alphabet-Google structure: Google is beginning to show increased momentum and has made significant investments in its enterprise offerings (most of its 2016 acquisitions were focused on this); and it is applying more discipline in Alphabet's "Other Bets." ■ Google is flourishing despite challenging external market factors: adverse publicity, competitors, government regulators and law enforcement. ■ Google values data, encourages bold investments in long-term horizons, pivots plans based on results in near real time, and reveres user-oriented engineering excellence. ■ Google is fully committed to 100% cloud-based and web-scale infrastructure, massive scaling, the maximum rate of change, and stream-lined business processes for itself and its customers. Recommendations CIOs and IT leaders managing vendor risk and performance should: ■ Plan for a long-term strategic relationship with Google based on an assumption that "what you see is what you get." Major vendor changes to core culture and fundamental operating principles in response to customer requests usually come slowly, if at all.
    [Show full text]
  • Powered by the Sun, Hosted in the Cloud: Sunrun Creates a Brighter Future with Solar-Powered Plants
    Powered by the Sun, Hosted in the Cloud: Sunrun Creates a Brighter Future with Solar-powered Plants “ Thanks to Informatica Intelligent Cloud Services and Google Cloud, we now have much more capacity for advanced analytics, giving us the insights we need to compete in the fast-changing solar power industry.” Harish Ramachandraiah Director, Engineering & Analytics, Sunrun Goals Solution Results Achieve faster, more accurate reporting and Migrate from on-premises infrastructure to Google Enables reporting and visualization development in visualization of business metrics and analytics Cloud and use Cloud Storage and BigQuery as a weeks versus multiple quarters, delivering high-quality cloud data lake and warehouse analytics to business users 3x faster Accelerate cloud migration and build data pipelines Use Informatica Cloud Data Integration to ingest Saves time for IT, reducing data warehouse design faster to save time for IT and improve refresh cycles data from dozens of source systems and build time by 50 percent and infrastructure building time for data consumers hundreds of pipelines into Google Cloud by 75 percent Offer business intelligence (BI) and analytics Deploy Looker, a business intelligence platform Expands base of analytics power users 7x, from capabilities to more business users to drive adoption that connects directly to BigQuery, to simplify data approximately 100 employees to 700 throughout the company visualization and provide self-service Business Requirements: Informatica Success Story: Sunrun • Simplify processes and architecture For a growing number of Americans, solar energy promises a more sustainable future. However, many are unsure how to go about a residential solar deployment and need assistance in navigating the fast-changing • Reduce costs and improve scalability solar marketplace.
    [Show full text]
  • Tamr on Google Cloud Platform: Walkthrough
    Tamr on Google Cloud Platform: Walkthrough Tamr on Google Cloud Platform: Walkthrough Overview Tamr on Google Cloud Platform empowers users to manage and publish data without learning a new SDK or coding in Java. This preview version of Tamr on Google Cloud Platform allows users to move data from Google Cloud Storage to BigQuery via a visual interface for selection and transformation of data. The preview of Tamr on Google Cloud Platform covers: + Attribute selection from CSV files + Joining CSV sources + Transformation of missing values, and + Publishing a table in BigQuery, Google’s fully managed, NoOps, data analytics service Signing Into Tamr & Google Cloud Platform To get started, register with Tamr and sign into Google Cloud Platform (using a Gmail account) by going to gcp-preview.tamr.com + If you don’t have an account with Google Cloud Platform, you can go through the Tamr portion of the offering, but will not be able to push your dataset to BigQuery. + If you don’t have a Google Cloud Platform account but would like to register for one, select the “Free Trial” option at the top of the Google Cloud Platform sign-in page. Selecting Sources Once you have signed in: + Select the project and bucket on Google Cloud Platform from which you would like to pull data into Tamr. Tamr on Google Cloud Platform: Walkthrough Adding / Subtracting Attributes Now that a data source has been added, attributes related to that data source should now appear on the left side of the screen. At this point, you have the option to add all of the attributes to a preview (via ‘Add All’ button) or add some of the attributes of interest to the preview (via click-and-drag functionality).
    [Show full text]
  • Data Sheet Informatica for Google Bigquery
    Data Sheet Informatica for Google BigQuery Benefits Accelerate Your Cloud Data Warehouse Modernization • Quickly transform raw data from multiple sources into actionable insights Organizations are universally looking to unleash value from the abundance and variety of • Integrate Google BigQuery data they produce to gain new insights that will guide their data-driven digital transformations. with data from multi-cloud They are striving to transform every aspect of their businesses, from customer experience and and on-premises environments operational excellence, to creating new products or services built on data. These efforts are futile with hundreds of out-of-the-box connectors without first rapidly integrating and analyzing—in meaningful ways—the massive volumes of • Improve productivity for disparate data residing across your organization. developers and citizen integrators with role-based, metadata-centric Your data warehouse is only as good as the quality of the data within it. To fully realize the visual tools, out-of-the-box prebuilt templates, and wizards benefits of Google BigQuery enterprise data warehouse, you must load it with trustworthy, integrated, relevant, and timely data from all your cloud and on-premises systems. With Informatica Intelligent Cloud ServicesSM, a multi-tenant, fully-managed integration platform as a service (iPaaS), you can rapidly access, synthesize, and load data into and out of Google BigQuery from numerous sources: cloud; SaaS applications; social media; sensors; machine sources; on-premises systems like SAP; and relational databases, including Netezza, Teradata, and Oracle. Informatica® gives you the agility needed to rapidly kick off a cloud analytics BigQuery project and seamlessly scale it up or down as data volume and needs vary.
    [Show full text]
  • Google Cloud Platform: Healthcare Solutions Playbook
    Google Cloud Platform: Healthcare Solutions Playbook Cloud training, tools, and reference architectures for researchers - Prepared for the National Institutes of Health (NIH) STRIDES. Author(s):​ Adrienne Wilkins, Trenton Leslie Title:​ Technical Account Manager (TAM), Intern PROFESSIONAL SERVICES Contents Introduction 2 Learning 2 Data Management Solutions 4 Reference Architectures 5 Genomics and Secondary Analysis 5 Patient Monitoring 9 Variant Analysis 11 Healthcare API for Machine Learning and Analytics 14 Radiological Image Extraction 18 Appendix 21 Page 1 PROFESSIONAL SERVICES Introduction The National Institutes of Health (NIH) established The Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability (STRIDES) initiative to provide biomedical researchers with access to advanced, cost-effective, cloud-based computational infrastructure, tools, and services. Through STRIDES, researchers can take advantage of emerging data management methodologies, technological expertise, computational platforms, and tools to support cutting-edge experimentation and innovation. NIH has partnered with Google Cloud to support the STRIDES initiative through cloud services. In support of STRIDES, we’ve developed sets of playbooks to help enable researchers to build healthcare and life sciences solutions on Google Cloud Platform (GCP). The goal of this playbook it to aid researchers in developing healthcare systems and workflows on GCP. This playbook will provide GCP architecture and solutions examples for implementing
    [Show full text]
  • SAP on Google Cloud with Bigquery
    Enterprise Strategy Group | Getting to the bigger truth.™ Technical Review Move SAP to the Cloud for Faster Time to Innovation: SAP on Google Cloud with BigQuery Date: March 2020 Author: Tony Palmer, Senior Validation Analyst Abstract This ESG Technical Review highlights how Google Cloud enables organizations using SAP to manage, safely store, and securely analyze their data to generate in-depth insights for greater business intelligence. A combination of hands-on analysis and customer use cases were used to validate the advantages of leveraging Google Cloud and BigQuery for robust advanced analytics and AI/ML capabilities. ESG also examined how SAP customers leverage Google Cloud to deploy enterprise-ready applications on which to run their mission-critical operations. The Challenges Collecting and using data in real time is transforming and empowering organizations. A growing variety of sources generate real-time data from devices and machines, customers, suppliers, partners, and market interactions. Messaging applications are now real-time, and sensor-enabled machines deliver constant streams of data. Social media delivers real-time feedback and insight to consumers, and clickstream data from digital commerce can deliver predictive value to companies. All these data sources present the opportunity to add significant business value. This does not come without challenges; the volume of data has been increasing at an accelerating pace for a long time. In a recent survey, two-thirds (66%) of organizations reported that they are managing a petabyte of data or more, with nearly one-third (31%) managing 5PB of data or more (see Figure 1). This explosion in the volume of data makes it difficult to manage, safely store, securely analyze, and generate robust insights.
    [Show full text]
  • Make Petabytes of Data Instantly Accessible Across Your Organization
    Make Petabytes of Data Instantly Accessible Across Your Organization Google BigQuery and Looker combine to create a solution that helps everyone in your business build the workflows they need to deliver value. Google Cloud Whitepaper 04.17.20 The business problem In today’s business world, it’s common to collect data from every part of your business—data about events, about user behavior, about the customer journey, about your marketing efforts, BigQuery about your product or service. How do you make sense of it all? As you continue to invest in collecting and storing gigabytes, Flexible architecture terabytes, maybe even petabytes of data, you need a solution that Only pay for what you use translates your data into meaningful insights about every aspect of your business. Scalable storage and compute Instantly scale horizontally Looker works with BigQuery to enable self-service discovery on all and vertically the data your business is collecting. This complete solution makes massive quantities of data intuitive to explore by everyone in your Modern workflows company - not just the data scientist—so it informs and empowers Machine learning, marketing, your business decision-makers. And since both Looker and BigQuery and web analytics workflows are architected to support analytics at any scale, you’ll always have that are quick to deploy fast access to every terabyte (or petabyte) of data. and scale Google BigQuery + Looker: Looker A joint solution Powerful integration with Together, BigQuery and Looker represent a fundamentally different Google Cloud approach to Data Analytics. BigQuery relies on a multi-tenant Directly leverage critical distributed architecture that uses a small portion of a super-large features of BigQuery compute cluster.
    [Show full text]
  • (SOC) 3 Report on the Google Cloud Platform System Relevant To
    System and Organization Controls (SOC) 3 Report on the Google Cloud Platform System Relevant to Security, Availability, and Confidentiality For the Period 1 May 2018 to 30 April 2019 Google LLC 1600 Amphitheatre Parkway Mountain View, CA, 94043 650 253-0000 main Google.com Management’s Report of its Assertion on the Effectiveness of Its Controls Over the Google Cloud Platform System Based on the Trust Services Criteria for Security, Availability, and Confidentiality We, as management of, Google LLC (“Google” or “the Company”) are responsible for: · Identifying the Google Cloud Platform System (System) and describing the boundaries of the System, which are presented in Attachment A · Identifying our principal service commitments and system requirements · Identifying the risks that would threaten the achievement of its principal service commitments and system requirements that are the objectives of our system, which are presented in Attachment B · Identifying, designing, implementing, operating, and monitoring effective controls over the System to mitigate risks that threaten the achievement of the principal service commitments and system requirement · Selecting the trust services categories that are the basis of our assertion We assert that the controls over the system were effective throughout the period 1 May 2018 to 30 April 2019, to provide reasonable assurance that the principal service commitments and system requirements were achieved based on the criteria relevant to security, availability, and confidentiality set forth in the AICPA’s
    [Show full text]