DOCSLIB.ORG

Troubleshooting and Debugging | Cloud Dataflow | Google Cloud

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Troubleshooting and Debugging | Cloud Dataflow | Google Cloud 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud Troubleshooting and debugging This page provides troubleshooting tips and debugging strategies that you might nd helpful if you're having trouble building or running your Dataow pipeline. This information can help you detect a pipeline failure, determine the reason behind a failed pipeline run, and suggest some courses of action to correct the problem. Dataow provides real-time feedback on your job, and there is a basic set of steps you can use to check the error messages, logs, and for conditions such as your job's progress having stalled. For guidance on common errors you might encounter when running your Dataow job, see the Common error guidance (/dataow/docs/guides/common-errors) page. If a job fails, Dataow might not automatically clean up your temporary and staged les. In this case, manually e these les from any Cloud Storage buckets that are used as temporary buckets. Checking your pipeline's status You can detect any errors in your pipeline runs by using the Dataow monitoring interface (/dataow/pipelines/dataow-monitoring-intf). 1. Go to the Google Cloud Console (https://console.cloud.google.com/). 2. Select your Google Cloud project from the project list. 3. Click the menu in the upper left corner. 4. Navigate to the Big Data section and click Dataow. A list of running jobs appears in the right-hand pane. 5. Select the pipeline job you want to view. You can see the jobs' status at a glance in the Status eld: "Running," "Succeeded," or "Failed." https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 1/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud Figure 1: A list of Dataow jobs in the Developers Console with jobs in the running, succeeded, and failed states. Basic troubleshooting workow If one of your pipeline jobs fails, you can select the job to view more detailed information on errors and run results. When you select a job, you can view key charts for your pipeline, the execution graph, the Job info panel, and a panel for Job logs, Worker logs, andJob error reporting. Checking job error messages To expand the Job Logs generated by your pipeline code and the Dataow service, click . You can lter the messages that appear in Job logs by clicking Any log level. To only display error messages, click Filter and select Errors. To expand an error message, click the expandable section . https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 2/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud Alternatively, you can click the Job error reporting panel. This panel shows where errors occurred along the chosen timeline and a count of all logged errors. Viewing step logs for your job When you select a step in your pipeline graph, the logs panel toggles from displaying Job Logs generated by the Dataow service to showing logs from the Compute Engine instances running your pipeline step. https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 3/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud Cloud Logging (/logging) combines all of the collected logs from your project's Compute Engine instances in one location. See Logging pipeline messages (/dataow/pipelines/logging) for more information on using Dataow's various logging capabilities. Handling automated pipeline rejection In some cases, the Dataow service identies that your pipeline might trigger known SDK issues (/dataow/docs/resources/release-notes-service). To prevent pipelines that will likely encounter issues, from being submitted, Dataow will automatically reject your pipeline and display the following message: workflow was automatically rejected by the service because it may trigger an ified bug in the SDK (details below). If you think this identification is ror, and would like to override this automated rejection, please re-submit workflow with the following override flag: [OVERRIDE FLAG]. https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 4/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud etails: [BUG DETAILS]. ct Google Cloud Support for further help. e use this identifier in your communication: [BUG ID]. After reading the caveats in the linked bug details, if you want to try to run your pipeline anyway, you can override the automated rejection. Add the ag --experiments=<override- flag> and resubmit your pipeline. Determining the cause of a pipeline failure Typically, a failed Apache Beam pipeline run can be attributed to one of the following causes: Graph or pipeline construction errors. These errors occur when Dataow runs into a problem building the graph of steps that compose your pipeline, as described by your Apache Beam pipeline. Errors in job validation. The Dataow service validates any pipeline job you launch. Errors in the validation process can prevent your job from being successfully created or executed. Validation errors can include problems with your Google Cloud project's Cloud Storage bucket, or with your project's permissions. Exceptions in worker code. These errors occur when there are errors or bugs in the user- provided code that Dataow distributes to parallel workers, such as the DoFn instances of a ParDo transform. Slow-running pipelines or lack of output. If your pipeline runs slowly or runs for a long period of time without reporting results, you might check your quotas for streaming data sources and sinks such as Pub/Sub. There are also certain transforms that are better- suited to high-volume streaming pipelines than others. Errors caused by transient failures in other Google Cloud services. Your pipeline may fail because of a temporary outage or other problem in the Google Cloud services upon which Dataow depends, such as Compute Engine or Cloud Storage. Detecting graph or pipeline construction errors A graph construction error can occur when Dataow is building the execution graph for your pipeline from the code in your Dataow program. During graph construction time, Dataow https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 5/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud checks for illegal operations. If Dataow detects an error in graph construction, keep in mind that no job will be created on the Dataow service. Thus, you won't see any feedback in the Dataow monitoring interface. Instead, you'll see an error message similar to the following in the console or terminal window where you ran your Apache Beam pipeline: Java: SDK 2.xPython (#python)Java: SDK 1.x (#java:-sdk-1.x) For example, if your pipeline attempts to perform an aggregation like GroupByKey on a globally windowed, non-triggered, unbounded PCollection, you'll see an error message similar to the following: ... ... Exception in thread "main" java.lang.IllegalStateException: ... GroupByKey cannot be applied to non-bounded PCollection in the GlobalWindow with ... Use a Window.into or Window.triggering transform prior to GroupByKey ... Should you encounter such an error, check your pipeline code to ensure that your pipeline's operations are legal. Detecting errors in Cloud Dataow job validation Once the Dataow service has received your pipeline's graph, the service will attempt to validate your job. This validation includes the following: Making sure the service can access your job's associated Cloud Storage buckets for le staging and temporary output. Checking for the required permissions in your Google Cloud project. Making sure the service can access input and output sources, such as les. If your job fails the validation process, you'll see an error message in the Dataow monitoring interface, as well as in your console or terminal window if you are using blocking execution. The error message will look similar to the following: https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline/ 6/10 8/23/2020 Troubleshooting and debugging | Cloud Dataflow | Google Cloud INFO: To access the Dataflow monitoring console, please navigate to https://console.developers.google.com/project/google.com%3Aclouddfe/dataflow/job/2 Submitted job: 2016-03-08_18_59_25-16868399470801620798 ... ... Starting 3 workers... ... Executing operation BigQuery-Read+AnonymousParDo+BigQuery-Write ... Executing BigQuery import job "dataflow_job_16868399470801619475". ... Stopping worker pool... ... Workflow failed. Causes: ...BigQuery-Read+AnonymousParDo+BigQuery-Write failed. Causes: ... BigQuery getting table "non_existent_table" from dataset "cws_demo" in p Message: Not found: Table x:cws_demo.non_existent_table HTTP Code: 404 ... Worker pool stopped. ... com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner run INFO: Job finished with status FAILED Exception in thread "main" com.google.cloud.dataflow.sdk.runners.DataflowJobExecutio Job 2016-03-08_18_59_25-16868399470801620798 failed with status FAILED at com.google.cloud.dataflow.sdk.runners.DataflowRunner.run(DataflowRunner.java: at com.google.cloud.dataflow.sdk.runners.DataflowRunner.run(DataflowRunner.java: at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:180) at com.google.cloud.dataflow.integration.BigQueryCopyTableExample.main(BigQueryC Detecting an exception in worker code While your job is running, you may encounter errors or exceptions in your worker code. These errors generally mean that the DoFns in your pipeline code have generated unhandled exceptions, which result in failed tasks in your Dataow job.
Load more

Recommended publications
  • Google Cloud Platform Integration
    Solidatus FACTSHEET Google Cloud Platform Integration The Solidatus Google Cloud Platform (GCP) integration suite helps to discover data structures and lineage in GCP and automatically create and maintain Solidatus models describing these assets when they are added to GCP and when they are changed. As of January 2019, the GCP integration supports the following scenarios: • Through the Solidatus UI: – Load BigQuery dataset schemas as Solidatus objects on-demand. • Automatically using a Solidatus Agent: – Detect new BigQuery schemas and add to a Solidatus model. – Detect changes to BigQuery schemas and update a Solidatus model. – Detect new files in Google Cloud Storage (GCS) and add to a Solidatus model. – Automatically detect changes to files in GCS and update a Solidatus model. • Automatically at build time: – Extract structure and lineage from a Google Cloud Dataflow and create or update a Solidatus model. FEATURES BigQuery Loader Apache Beam (GCP Dataflow) Lineage A user can import a BigQuery table definition, directly Mapper from Google, as an object into a Solidatus model. A developer can visualise their Apache Beam job’s The import supports both nested and flat structures, pipeline in a Solidatus model. The model helps both and also includes meta data about the table and developers and analysts to see that data from sources dataset. Objects created via the BigQuery Loader is correctly mapped through transforms to their sinks, can be easily updated by a right-clicking on an providing a data lineage model of the pipeline. object in Solidatus. Updating models using this Generating the models can be ad-hoc (on-demand by feature provides the ability to visualise differences in the developer) or built into a CI/CD process.
    [Show full text]
  • Google's Mission
    & Big Data & Rocket Fuel Dr Raj Subramani, HSBC Reza Rokni, Google Cloud, Solutions Architect Adrian Poole, Google Cloud, Google’s Mission Organize the world’s information and make it universally accessible and useful Eight cloud products with ONE BILLION Users Increasing Marginal Cost of Change $ Traditional Architectures Prohibitively Expensive change Marginal cost of 18 years of Google R&D / Investment Google Cloud Native Architectures (GCP) Increasing complexity of systems and processes Containers at Google Number of running jobs Enabled Google to grow our fleet over 10x faster than we grew our ops team Core Ops Team 2004 2016 4 Google’s innovation in data Millwheel F1 Spanner TensorFlow MapReduce Dremel Flume GFS Bigtable Colossus Megastore Pub/Sub Dataflow 2002 2004 2006 2008 2010 2012 2013 2016 Proprietary + Confidential5 Google’s innovation in data Dataflow Spanner NoSQL Spanner Cloud ML Dataproc BigQuery Dataflow GCS Bigtable GCS Datastore Pub/Sub Dataflow 2002 2004 2006 2008 2010 2012 2013 2016 Proprietary + Confidential6 Now available on Google Cloud Platform Compute Storage & Databases App Engine Container Compute Storage Bigtable Spanner Cloud SQL Datastore Engine Engine Big Data Machine Learning BigQuery Pub/Sub Dataflow Dataproc Datalab Vision API Machine Speech API Translate API Learning Lesson of the last 10 years... ● Democratise ML ● Big datasets beat fancy algorithms ● Good Models ● Lots of compute Google BigQuery BigQuery is Google's fully managed, petabyte scale, low cost enterprise data warehouse for analytics. BigQuery is serverless. There is no infrastructure to manage and you don't need a database administrator, so you can focus on analyzing data to find meaningful insights using familiar SQL.
    [Show full text]
  • Are3na Crabbé Et Al
    ARe3NA Crabbé et al. (2014) AAA for Data and Services (D1.1.2 & D1.2.2): Analysing Standards &Technologies for AAA ISA Action 1.17: A Reusable INSPIRE Reference Platform (ARE3NA) Authentication, Authorization & Accounting for Data and Services in EU Public Administrations D1.1.2 & D1.2.2– Analysing standards and technologies for AAA Ann Crabbé Danny Vandenbroucke Andreas Matheus Dirk Frigne Frank Maes Reijer Copier 0 ARe3NA Crabbé et al. (2014) AAA for Data and Services (D1.1.2 & D1.2.2): Analysing Standards &Technologies for AAA This publication is a Deliverable of Action 1.17 of the Interoperability Solutions for European Public Admin- istrations (ISA) Programme of the European Union, A Reusable INSPIRE Reference Platform (ARE3NA), managed by the Joint Research Centre, the European Commission’s in-house science service. Disclaimer The scientific output expressed does not imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of this publication. Copyright notice © European Union, 2014. Reuse is authorised, provided the source is acknowledged. The reuse policy of the European Commission is implemented by the Decision on the reuse of Commission documents of 12 December 2011. Bibliographic Information: Ann Crabbé, Danny Vandenbroucke, Andreas Matheus, Dirk Frigne, Frank Maes and Reijer Copier Authenti- cation, Authorization and Accounting for Data and Services in EU Public Administrations: D1.1.2 & D1.2.2 – Analysing standards and technologies for AAA. European Commission; 2014. JRC92555 1 ARe3NA Crabbé et al. (2014) AAA for Data and Services (D1.1.2 & D1.2.2): Analysing Standards &Technologies for AAA Contents 1.
    [Show full text]
  • Economic and Social Impacts of Google Cloud September 2018 Economic and Social Impacts of Google Cloud |
    Economic and social impacts of Google Cloud September 2018 Economic and social impacts of Google Cloud | Contents Executive Summary 03 Introduction 10 Productivity impacts 15 Social and other impacts 29 Barriers to Cloud adoption and use 38 Policy actions to support Cloud adoption 42 Appendix 1. Country Sections 48 Appendix 2. Methodology 105 This final report (the “Final Report”) has been prepared by Deloitte Financial Advisory, S.L.U. (“Deloitte”) for Google in accordance with the contract with them dated 23rd February 2018 (“the Contract”) and on the basis of the scope and limitations set out below. The Final Report has been prepared solely for the purposes of assessment of the economic and social impacts of Google Cloud as set out in the Contract. It should not be used for any other purposes or in any other context, and Deloitte accepts no responsibility for its use in either regard. The Final Report is provided exclusively for Google’s use under the terms of the Contract. No party other than Google is entitled to rely on the Final Report for any purpose whatsoever and Deloitte accepts no responsibility or liability or duty of care to any party other than Google in respect of the Final Report and any of its contents. As set out in the Contract, the scope of our work has been limited by the time, information and explanations made available to us. The information contained in the Final Report has been obtained from Google and third party sources that are clearly referenced in the appropriate sections of the Final Report.
    [Show full text]
  • Google Cloud Dataflow – an Insight
    International Journal of Science and Research (IJSR) ISSN (Online): 2319-7064 Index Copernicus Value (2015): 78.96 | Impact Factor (2015): 6.391 Google Cloud Dataflow – An Insight Eashani Deorukhkar1 Department of Information Technology, RamraoAdik Institute of Technology, Mumbai, India Abstract: The massive explosion of data and the need for its processing has become an area of interest with many companies vying to occupy a significant share of this market. The processing of streaming data i.e. data generated by multiple sources simultaneously and continuously is an ability that some widely used data processing technologies lack. This paper discusses the “Cloud Data Flow” technology offered by Google as a possible solution to this problem and also a few of its overall pros and cons. Keywords: Data processing, streaming data, Google, Cloud Dataflow 1. Introduction To examine a real-time stream of events for significant [2] patterns and activities The processing of large datasets to generate new insights and To implement advanced, multi-step processing pipelines to relationships from it is quite a common activity in numerous extract deep insight from datasets of any size[2] companies today. However, this process is quite difficult and resource intensive even for experts[1]. Traditionally, this 3. Technical aspects processing has been performed on static datasets wherein the information is stored for a while before being processed. 3.1 Overview This method was not scalable when it came to processing of data streams. The huge amounts of data being generated by This model is specifically intended to make data processing streams like real-time data from sensors or from social on a large scale easier.
    [Show full text]
  • Google Cloud Dataflow a Unified Model for Batch and Streaming Data Processing Jelena Pjesivac-Grbovic
    Google Cloud Dataflow A Unified Model for Batch and Streaming Data Processing Jelena Pjesivac-Grbovic STREAM 2015 Agenda 1 Data Shapes 2 Data Processing Tradeoffs 3 Google’s Data Processing Story 4 Google Cloud Dataflow Score points Form teams Geographically distributed Online and Offline mode User and Team statistics in real time Accounting / Reporting Abuse detection https://commons.wikimedia.org/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg 1 Data Shapes Data... ...can be big... ...really, really big... Thursday Wednesday Tuesday ...maybe even infinitely big... 8:00 1:00 9:00 2:00 10:00 3:00 11:00 4:00 12:00 5:00 13:00 6:00 14:00 7:00 … with unknown delays. 8:00 8:00 8:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 2 Data Processing Tradeoffs Data Processing Tradeoffs 1 + 1 = 2 $$$ Completeness Latency Cost Requirements: Billing Pipeline Important Not Important Completeness Low Latency Low Cost Requirements: Live Cost Estimate Pipeline Important Not Important Completeness Low Latency Low Cost Requirements: Abuse Detection Pipeline Important Not Important Completeness Low Latency Low Cost Requirements: Abuse Detection Backfill Pipeline Important Not Important Completeness Low Latency Low Cost Google’s 3 Data Processing Story Data Processing @ Google Dataflow MapReduce FlumeJava Dremel Spanner GFS Big Table Pregel Colossus MillWheel 2002 2004 2006 2008 2010 2012 2014 2016 Data Processing @ Google Dataflow MapReduce FlumeJava Dremel Spanner GFS Big Table Pregel Colossus MillWheel 2002 2004 2006 2008 2010
    [Show full text]
  • Create a Created Date Schema in Bigquery
    Create A Created Date Schema In Bigquery Denominative Elmore canoodles funereally. Auxiliary Rudy acquiesce: he unbar his aril adequately and contagiously. Overreaching Randolf excused some scrabble and shacks his syneresis so midnight! Wait polls with exponential backoff. The schema in tests are many cases. Perform various levels of dates in schema definition file somewhere on create date column when you created in an identical data will lead at. You'd may that avoiding two scans of SALES would improve performance However remember most circumstances the draw BY version takes about twice as suite as the lot BY version. You run some SQL queries against that data. The script then creates a partition function, if configured. The order in which clustering is done matters. Developer relations lead at SKB Kontur. We create schema in the dates and their very wide variety of bigquery operation creates a working directory as a few seconds of model you. By using the client libraries. Database services to migrate, please leave it empty. What are partitions in SQL? In our case the join condition is matching dates from the pedestrian table and the bike table. The mother source records need is have a machine image updates to merge correctly. Python community version selection works in schema of bigquery operation on create date column can be created in a new custom schema tab. Once per store the data needs to simplify etl to any data are also a bigquery? Google Analytics 360 users that have that up the automatic BigQuery export will rejoice by this. Whether to automatically infer options and schema for CSV and JSON sources.
    [Show full text]
  • Data Processing with Apache Beam and Google Cloud Dataflow
    Data Processing with Apache Beam (incubating) and Google Cloud Dataflow Jelena Pjesivac-Grbovic Staff software engineer Cloud Big Data In collaboration with Frances Perry, Tayler Akidau, and Dataflow team XLDB’16 - May 2016 Agenda 1 Infinite, Out-of-Order Data Sets 2 What, Where, When, How 3 Apache Beam (incubating) 4 Google Cloud Dataflow 1 Infinite, Out-of-Order Data Sets Data... ...can be big... ...really, really big... Thursday Wednesday Tuesday … maybe infinitely big... 8:00 9:00 10:00 11:00 12:00 13:00 14:00 … with unknown delays. 8:00 8:00 8:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 Element-wise transformations Processing 8:00 9:00 10:00 11:00 12:00 13:00 14:00 Time Aggregating via Processing-Time Windows Processing 8:00 9:00 10:00 11:00 12:00 13:00 14:00 Time Aggregating via Event-Time Windows Input Processing Time 10:00 11:00 12:00 13:00 14:00 15:00 Output Event Time 10:00 11:00 12:00 13:00 14:00 15:00 Formalizing Event-Time Skew Skew Reality Ideal Processing Time Event Time Formalizing Event-Time Skew Watermarks describe event time progress. Skew "No timestamp earlier than the ~Watermark watermark will be seen" Ideal Often heuristic-based. Processing Time Too Slow? Results are delayed. Too Fast? Some data is late. Event Time 2 What, Where, When, How What are you computing? Where in event time? When in processing time? How do refinements relate? What are you computing? Element-Wise Aggregating Composite What Where When How What: Computing Integer Sums // Collection of raw log lines PCollection<String> raw = IO.read(...); // Element-wise transformation into team/score pairs PCollection<KV<String, Integer>> input = raw.apply(ParDo.of(new ParseFn()); // Composite transformation containing an aggregation PCollection<KV<String, Integer>> scores = input.apply(Sum.integersPerKey()); What Where When How What: Computing Integer Sums What Where When How What: Computing Integer Sums What Where When How Where in event time? Windowing divides data into event-time-based finite chunks.
    [Show full text]
  • Tamr on Google Cloud Platform: Walkthrough
    Tamr on Google Cloud Platform: Walkthrough Tamr on Google Cloud Platform: Walkthrough Overview Tamr on Google Cloud Platform empowers users to manage and publish data without learning a new SDK or coding in Java. This preview version of Tamr on Google Cloud Platform allows users to move data from Google Cloud Storage to BigQuery via a visual interface for selection and transformation of data. The preview of Tamr on Google Cloud Platform covers: + Attribute selection from CSV files + Joining CSV sources + Transformation of missing values, and + Publishing a table in BigQuery, Google’s fully managed, NoOps, data analytics service Signing Into Tamr & Google Cloud Platform To get started, register with Tamr and sign into Google Cloud Platform (using a Gmail account) by going to gcp-preview.tamr.com + If you don’t have an account with Google Cloud Platform, you can go through the Tamr portion of the offering, but will not be able to push your dataset to BigQuery. + If you don’t have a Google Cloud Platform account but would like to register for one, select the “Free Trial” option at the top of the Google Cloud Platform sign-in page. Selecting Sources Once you have signed in: + Select the project and bucket on Google Cloud Platform from which you would like to pull data into Tamr. Tamr on Google Cloud Platform: Walkthrough Adding / Subtracting Attributes Now that a data source has been added, attributes related to that data source should now appear on the left side of the screen. At this point, you have the option to add all of the attributes to a preview (via ‘Add All’ button) or add some of the attributes of interest to the preview (via click-and-drag functionality).
    [Show full text]
  • For Cloud Professionals, Part of the on Cloud Podcast
    Architecting the Cloud at Google Cloud Next 2018_The past, present, and future of BigQuery For Cloud professionals, part of the On Cloud Podcast Mike Kavis, Chief Cloud Architect, Deloitte Consulting LLP File Duration: 0:18:04 Operator: Welcome to Architecting the Cloud, part of the On Cloud Podcast, where we get real about Cloud Technology what works, what doesn't and why. Now here is your host Mike Kavis. Mike Kavis: Welcome to Deloitte’s Architecting the Cloud Podcast, I'm your host Mike Kavis and I am here in San Francisco on day three of the exciting Google Next Conference, and I'm here with Google's Product Manager for BigQuery, Tino Tereshko. And Tino, nice to meet you, I‘ve been following you on Twitter for a while, you may not remember but we chatted back and forth on a couple topics over the years, it was nice to meet you in person here. Tino Tereshko: Yes, nice to meet you as well. Mike Kavis: Yes. 1 Architecting the Cloud at Google Cloud Next 2018_The past, present, and future of BigQuery Tino Tereshko: Thank you for not butchering my last name, you got it. Mike Kavis: Well I practiced it a few times, and I still worry. Tell us a little bit about your background and then tell you what keeps you busy, Google these days? Tino Tereshko: It’s a great question, so my background is in mathematics, spent my time in finance, in my time (0:01:00) had a number of start-up’s before coming to Google.
    [Show full text]
  • Composite Event Recognition for Maritime Monitoring
    NATIONAL AND KAPODISTRIAN UNIVERSITY OF ATHENS SCHOOL OF SCIENCE DEPARTMENT OF INFORMATICS AND TELECOMMUNICATIONS BSc THESIS Composite event recognition for maritime monitoring Manolis N. Pitsikalis Supervisors: Alexander Artikis Researcher, NCSR “Demokritos” Assistant Professor, UNIPI Panagiotis Stamatopoulos Assistant Professor, NKUA ATHENS JUNE 2018 ΕΘΝΙΚΟ ΚΑΙ ΚΑΠΟΔΙΣΤΡΙΑΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ ΣΧΟΛΗ ΘΕΤΙΚΩΝ ΕΠΙΣΤΗΜΩΝ ΤΜΗΜΑ ΠΛΗΡΟΦΟΡΙΚΗΣ ΚΑΙ ΤΗΛΕΠΙΚΟΙΝΩΝΙΩΝ ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ Αναγνώριση σύνθετων γεγονότων για την παρακολούθηση της ναυτιλιακής δραστηριότητας Μανώλης Ν. Πιτσικάλης Επιβλέποντες: Αλέξανδρος Αρτίκης, Ερευνητής, ΕΚΕΦΕ «∆ημόκριτος» Επίκουρος Καθηγητής, ΠΑΠΕΙ Παναγιώτης Σταματόπουλος, Επίκουρος Καθηγητής, ΕΚΠΑ ΑΘΗΝΑ ΙΟΥΝΙΟΣ 2018 BSc THESIS Composite event recognition for maritime monitoring Manolis N. Pitsikalis R.N.: 1115201300143 SUPERVISORS: Alexander Artikis Researcher, NCSR “Demokritos” Assistant Professor, UNIPI Panagiotis Stamatopoulos Assistant Professor, NKUA ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ Αναγνώριση σύνθετων γεγονότων για την παρακολούθηση της ναυτιλιακής δραστηριότητας Μανώλης Ν. Πιτσικάλης Α.Μ.: 1115201300143 ΕΠΙΒΛΕΠΟΝΤΕΣ: Αλέξανδρος Αρτίκης, Ερευνητής, ΕΚΕΦΕ «∆ημόκριτος» Επίκουρος Καθηγητής, ΠΑΠΕΙ Παναγιώτης Σταματόπουλος, Επίκουρος Καθηγητής, ΕΚΠΑ ABSTRACT Maritime monitoring systems support safe shipping as they allow for the real-time detection of dangerous, suspicious and illegal vessel activities. The intent of this thesis was the development of a composite event recognition engine for maritime monitoring and the construction
    [Show full text]
  • Serverless Through Cloud Native Architecture
    Published by : International Journal of Engineering Research & Technology (IJERT) http://www.ijert.org ISSN: 2278-0181 Vol. 10 Issue 02, February-2021 Serverless Through Cloud Native Architecture M. V. L. N. Venugopal Dr. C. R. K. Reddy Research Scholar, Professor & Head of the Department computer science, Osmania University, Hyderabad, Mahatma Gandhi Institute of Technology Hyderabad Telangana, India. Telangana, India Abstract:- Many Enterprises prefer technologies that meet their business needs and requirements. Change of market dynamics continuously demands proliferation of latest technologies. Microservices, Serverless computing, Artificial intelligence, machine learning, and IoT are few latest technologies. Ongoing technology changes insist software companies to bring quick new products with latest features as expected by client. Global modern software legends stress the need of adoption of cutting edge technologies, frameworks, tools, and infrastructure. So that applications can take the advantages like faster-delivery, lightweight scalable and lower developing and maintenance cost. Digital transformation has become very essential for serverless computing. Digital transformation has accelerated business models. Some of them are favorites of serverless technologies, micro services, and native cloud applications. These initiated a new transformation in the technological areas. Stateless functions impact on the maintainability of the software. Enterprises will now concentrate on their products and services to sustain global software market. Hence, an implementer can run an application on a third- party server and this will in-turn decrease deployment time. Serverless [2], is a hot topic and, in software world Serverless computing has evolved as new important technology for deploying cloud based applications. Serverless is latest execution model of cloud. Provider of cloud will allocate resources at run time.
    [Show full text]