DOCSLIB.ORG

Evaluierung Der Qualität Von Open Source Stream Processing Frameworks

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Evaluierung Der Qualität Von Open Source Stream Processing Frameworks Hochschule Darmstadt -Fachbereich Informatik- Evaluierung der Qualität von Open Source Stream Processing Frameworks zur Erlangung des akademischen Grades Master of Science (M. Sc.) vorgelegt von Simon Kern 737708 Referentin: Prof. Dr. Uta Störl Korreferentin: Prof. Dr. Inge Schestag Ausgabedatum: 16.03.2015 Abgabedatum: 16.09.2015 Erklärung Ich versichere hiermit, dass ich die vorliegende Arbeit selbständig verfasst und keine anderen als die im Literaturverzeichnis angegebenen Quellen benutzt habe. Alle Stellen, die wörtlich oder sinngemäß aus veröffentlichten oder noch nicht veröffent- lichten Quellen entnommen sind, sind als solche kenntlich gemacht. Die Zeichnungen oder Abbildungen in dieser Arbeit sind von mir selbst erstellt worden oder mit einem entsprechenden Quellennachweis versehen. Diese Arbeit ist in gleicher oder ähnlicher Form noch bei keiner anderen Prüfungsbehörde eingereicht worden. Darmstadt, den 16.09.2015 Simon Kern ii Kurzfassung Evaluierung der Qualität von Open Source Stream Processing Frameworks Big Data ist im Bereich der Informationstechnologie aktuell eines der großen Schlagworte. Technologisch bezieht sich der Begriff meist auf die Stapelverarbeitung von gesammelten Daten. Im Gegensatz dazu verarbeiten Stream Processing Frameworks Datenströme direkt. Innerhalb der Open-Source-Community sind mehrere konkurrierende Systeme für diesen Aufgabenbereich entstanden. Bei der Neuentwicklung einer Anwendung muss jedoch ein Framework als Basis ausgewählt werden. Ziel der Arbeit ist es daher, aktuelle Open Source Stream Processing Frameworks hinsichtlich ihrer Qualität zu evaluieren, um diese Auswahl zu unterstützen. Hierzu stellt diese Arbeit eine Referenzarchitektur vor, anhand derer die Komponenten eines Stream Processing Frameworks zur Verbesserung der Vergleichbarkeit zugeordnet werden können. Mithilfe des Evaluierungsprozesses nach ISO 25000 und insbesondere des Softwarequalitätsmodells nach ISO 25010 werden Kriterien zur Evaluierung von Stream Processing Frameworks entwickelt. Zur Erhöhung der praktischen Relevanz werden die Methoden Cognitive Walkthrough und Cognitive Dimensions zur Feststellung der Benut- zerfreundlichkeit genutzt. Dafür werden Aufgaben definiert, die mithilfe der ausgewählten Frameworks implementiert werden. Sie werden zusätzlich zur Durchsatz- und Latenzmes- sung herangezogen. Als Rahmen der Aufgaben dient die Beobachtung von Wertpapieren, die an der New York Stock Exchange gelistet sind. Für die Evaluierung werden Apache Flink, Apache Samza, Apache Spark und Apache Storm als Kandidaten identifiziert. Sie werden anhand der entwickelten Kriterien ver- glichen. Dabei zeigt sich, dass kein Framework alle Anforderungen vollständig abdeckt. Vielmehr müssen die Ergebnisse dieser Arbeit nach den konkreten Anforderungen einer Neuentwicklung gewichtet werden, um eine Auswahlentscheidung zu treffen. iii Abstract Evaluation of the Quality of Open Source Stream Processing Frameworks Big Data is a huge trend in current information technology. This term mostly refers to software that does batch processing of stored data. But there is also software, namely stream processing frameworks, which operates directly on data streams. Several competing projects have emerged in the open source community, with the goal to provide software that addresses this issue. When developing a new application, one has to choose between these alternatives. Therefore, the goal of this thesis is to evaluate the quality of open source stream processing frameworks, in order to support decision making. For this purpose, a reference architecture is proposed. Stream processing frameworks are composed of components that can be mapped onto this reference architecture. This facilitates an easy comparison of multiple frameworks and establishes a common under- standing of terminology. The evaluation criteria are developed according to the process described in ISO 25000 and especially derived from the software quality model specified in ISO 25010. To increase the practical relevance of this evaluation, the usability inspection methods cognitive walkthrough and cognitive dimensions are used. These methods are based on predefined tasks that are implemented utilizing the frameworks under test. This implementation is later used to benchmark the frameworks in terms of throughput and latency. The tasks are built around the observation of stock pricing at the New York Stock Exchange. The candidates for the evaluation are Apache Flink, Apache Samza, Apache Spark and Apache Storm. They are evaluated according to the developed criteria. The results show that no candidate meets all requirements. In fact, the results of this thesis have to be weighted to match the specific requirements of the application to be implemented, to support the decision making. iv Inhaltsverzeichnis Abbildungsverzeichnis viii Tabellenverzeichnis x Quelltextverzeichnis xi 1 Einführung 1 1.1 Motivation . .2 1.2 Zielsetzung . .2 1.3 Aufbau der Arbeit . .3 2 Vorstellung des Unternehmens 4 3 Grundlagen 5 3.1 Evaluierung . .5 3.2 ISO 25000 . .6 3.3 Information Flow Processing . .8 3.3.1 Complex Event Processing . .9 3.3.2 Data Stream Processing . 10 3.4 Framework . 11 4 Stream Processing Frameworks 12 4.1 Stream Processing Modell . 12 4.2 Anforderungen an Stream Processing Frameworks . 14 4.2.1 Verteilte Architektur . 15 4.2.2 Fehlertoleranz . 17 4.3 Referenzarchitektur . 19 5 Konzept 23 5.1 Evaluator . 23 5.2 Gegenstand . 23 5.2.1 Apache Flink . 24 5.2.2 Apache Samza . 26 5.2.3 Apache Spark . 27 5.2.4 Apache Storm . 29 5.3 Methoden . 30 5.3.1 Gewinnen der Informationen . 31 v Inhaltsverzeichnis 5.3.2 Auswahl der Vergleichskriterien . 32 5.4 Kriterien . 33 5.4.1 Functional Suitability . 33 5.4.2 Performance Efficiency . 35 5.4.3 Compatibility . 35 5.4.4 Usability . 36 5.4.5 Reliability . 38 5.4.6 Security . 39 5.4.7 Maintainability . 40 5.4.8 Portability . 41 5.4.9 Assigned Property . 42 5.5 Aufgaben . 42 5.5.1 Installation des Frameworks . 43 5.5.2 Gleitende Statistik . 43 5.5.3 Meldung bei großen Kursabweichungen . 43 5.5.4 Bewertung von Wertpapierdepots . 44 6 Praktische Umsetzung 45 6.1 Entwicklungs- und Betriebsumgebung . 45 6.2 Architektur . 47 6.2.1 Datenquelle . 48 6.2.2 Zielsysteme . 51 6.2.3 Unterstützende Systeme . 53 6.2.4 Aufbau der Anwendung . 54 6.3 Implementierung der Aufgaben . 55 6.3.1 Apache Flink . 56 6.3.2 Apache Samza . 59 6.3.3 Apache Spark . 63 6.3.4 Apache Storm . 65 6.3.5 Apache Storm Trident . 69 7 Ergebnisse 73 7.1 Functional Suitability . 73 7.1.1 Unterstützte Programmiersprachen . 73 7.1.2 Mächtigkeit der angebotenen API . 75 7.1.3 Unterstützung von zustandsbehafteten Operationen . 80 7.2 Performance Efficiency . 83 7.2.1 Mittlere Verarbeitungszeit von Tupeln . 84 7.2.2 Skalierung auf mehrere Worker . 85 7.3 Compatibility . 91 7.3.1 Integration mit bestehenden Clusterwerkzeugen . 91 7.3.2 Anbindung von Ein- und Ausgabesystemen . 92 7.4 Usability . 96 7.4.1 Dokumentation des Frameworks . 96 vi Inhaltsverzeichnis 7.4.2 Ersteinrichtung des Frameworks . 99 7.4.3 Bereitstellung von Anwendungen . 99 7.4.4 Benutzerfreundlichkeit der API . 103 7.5 Reliability . 106 7.5.1 Aktive Community . 106 7.5.2 Verarbeitungsgarantien und Fehlertoleranz . 107 7.6 Security . 114 7.6.1 Nutzerbasierte Zugriffskontrolle . 114 7.6.2 Verschlüsselung des Datenverkehrs . 115 7.7 Maintainability . 117 7.7.1 Aktualisieren des Frameworks . 117 7.7.2 Erfassung von Metriken . 118 7.7.3 Aussagekräftige Logdaten . 122 7.7.4 Testen auf dem Entwicklersystem . 124 7.8 Assigned Property . 126 7.8.1 Verfügbarkeit externer Unterstützung . 126 8 Zusammenfassung 128 8.1 Ergebnis der Evaluation . 128 8.2 Diskussion des Konzepts . 129 8.3 Ausblick . 130 Abkürzungsverzeichnis xii Literaturverzeichnis xiii vii Abbildungsverzeichnis 4.1 Grundlegendes Modell von Stream Processing nach [Boc14, S. 8] . 12 4.2 Logisches Modell einer Stream Processing Anwendung mit einer Quelle und drei Verarbeitungsknoten . 13 4.3 Physisches Modell des logischen Modells aus Abbildung 4.2 . 16 4.4 Referenzarchitektur von Stream Processing Frameworks . 20 5.1 Architektur von Apache Flink . 25 5.2 Architektur von Apache Samza . 27 5.3 Architektur von Apache Spark . 28 5.4 Architektur von Apache Storm . 30 6.1 Physische Verteilung der Anwendung mit Zuordnung der Hostnamen und installierten Komponenten . 47 6.2 Logische Architektur der Anwendung . 48 6.3 Geschwindigkeitsverlauf beim Abspielen der Tupel . 50 6.4 Architektur von Apache Kafka nach [Apa15x] . 51 7.1 Checkpointed-Interface von Flink . 80 7.2 StorageEngine-Interface von Samza . 81 7.3 KeyValueStore-Interface von Samza . 81 7.4 Funktion zum Aktualisieren des Zustands in Spark . 82 7.5 State-Interface von Storm Trident . 82 7.6 StateUpdater-Interface von Storm Trident . 83 7.7 QueryFunction-Interface von Storm Trident ..
Load more

Recommended publications
  • Large-Scale Learning from Data Streams with Apache SAMOA
    Large-Scale Learning from Data Streams with Apache SAMOA Nicolas Kourtellis1, Gianmarco De Francisci Morales2, and Albert Bifet3 1 Telefonica Research, Spain, [email protected] 2 Qatar Computing Research Institute, Qatar, [email protected] 3 LTCI, Télécom ParisTech, France, [email protected] Abstract. Apache SAMOA (Scalable Advanced Massive Online Anal- ysis) is an open-source platform for mining big data streams. Big data is defined as datasets whose size is beyond the ability of typical soft- ware tools to capture, store, manage, and analyze, due to the time and memory complexity. Apache SAMOA provides a collection of dis- tributed streaming algorithms for the most common data mining and machine learning tasks such as classification, clustering, and regression, as well as programming abstractions to develop new algorithms. It fea- tures a pluggable architecture that allows it to run on several distributed stream processing engines such as Apache Flink, Apache Storm, and Apache Samza. Apache SAMOA is written in Java and is available at https://samoa.incubator.apache.org under the Apache Software Li- cense version 2.0. 1 Introduction Big data are “data whose characteristics force us to look beyond the traditional methods that are prevalent at the time” [18]. For instance, social media are one of the largest and most dynamic sources of data. These data are not only very large due to their fine grain, but also being produced continuously. Furthermore, such data are nowadays produced by users in different environments and via a multitude of devices. For these reasons, data from social media and ubiquitous environments are perfect examples of the challenges posed by big data.
    [Show full text]
  • DSP Frameworks DSP Frameworks We Consider
    Università degli Studi di Roma “Tor Vergata” Dipartimento di Ingegneria Civile e Ingegneria Informatica DSP Frameworks Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini DSP frameworks we consider • Apache Storm (with lab) • Twitter Heron – From Twitter as Storm and compatible with Storm • Apache Spark Streaming (lab) – Reduce the size of each stream and process streams of data (micro-batch processing) • Apache Flink • Apache Samza • Cloud-based frameworks – Google Cloud Dataflow – Amazon Kinesis Streams Valeria Cardellini - SABD 2017/18 1 Apache Storm • Apache Storm – Open-source, real-time, scalable streaming system – Provides an abstraction layer to execute DSP applications – Initially developed by Twitter • Topology – DAG of spouts (sources of streams) and bolts (operators and data sinks) Valeria Cardellini - SABD 2017/18 2 Stream grouping in Storm • Data parallelism in Storm: how are streams partitioned among multiple tasks (threads of execution)? • Shuffle grouping – Randomly partitions the tuples • Field grouping – Hashes on a subset of the tuple attributes Valeria Cardellini - SABD 2017/18 3 Stream grouping in Storm • All grouping (i.e., broadcast) – Replicates the entire stream to all the consumer tasks • Global grouping – Sends the entire stream to a single task of a bolt • Direct grouping – The producer of the tuple decides which task of the consumer will receive this tuple Valeria Cardellini - SABD 2017/18 4 Storm architecture • Master-worker architecture Valeria Cardellini - SABD 2017/18 5 Storm
    [Show full text]
  • Comparative Analysis of Data Stream Processing Systems
    Shah Zeb Mian Comparative Analysis of Data Stream Processing Systems Master’s Thesis in Information Technology February 23, 2020 University of Jyväskylä Faculty of Information Technology Author: Shah Zeb Mian Contact information: [email protected] Supervisors: Oleksiy Khriyenko, and Vagan Terziyan Title: Comparative Analysis of Data Stream Processing Systems Työn nimi: Vertaileva analyysi Data Stream-käsittelyjärjestelmistä Project: Master’s Thesis Study line: All study lines Page count: 48+0 Abstract: Big data processing systems are evolving to be more stream oriented where data is processed continuously by processing it as soon as it arrives. Earlier data was often stored in a database, a file system or other form of data storage system. Applications would query the data as needed. Stram processing is the processing of data in motion. It works on continuous data retrieved from different resources. Instead of periodically collecting huge static data, streaming frameworks process data as soon as it becomes available, hence reducing latency. This thesis aims to conduct a comparative analysis of different streaming processors based on selected features. Research focuses on Apache Samza, Apache Flink, Apache Storm and Apache Spark Structured Streaming. Also, this thesis explains Apache Kafka which is a log-based data storage widely used in streaming frameworks. Keywords: Big Data, Stream Processing,Batch Processing,Streaming Engines, Apache Kafka, Apache Samza Suomenkielinen tiivistelmä: Big data-käsittelyjärjestelmät ovat tällä hetkellä kehittymässä stream-orientoituneiksi, eli data käsitellään heti saapuessaan. Perinteisemmin data säilöt- tiin tietokantaan, tiedostopohjaisesti tai muuhun tiedonsäilytysjärjestelmään, ja applikaatiot hakivat datan tarvittaessa. Stream-pohjainen järjestelmä käsittelee liikkuvaa dataa, jatkuva- aikaista dataa useasta lähteestä. Sen sijaan, että haetaan ajoittain dataa, stream-pohjaiset frameworkit pystyvät käsittelemään i dataa heti kun se on saatavilla, täten vähentäen viivettä.
    [Show full text]
  • Network Traffic Profiling and Anomaly Detection for Cyber Security
    Network traffic profiling and anomaly detection for cyber security Laurens D’hooge Student number: 01309688 Supervisors: Prof. dr. ir. Filip De Turck, dr. ir. Tim Wauters Counselors: Prof. dr. Bruno Volckaert, dr. ir. Tim Wauters A dissertation submitted to Ghent University in partial fulfilment of the requirements for the degree of Master of Science in Information Engineering Technology Academic year: 2017-2018 Acknowledgements This thesis is the result of 4 months work and I would like to express my gratitude towards the people who have guided me throughout this process. First and foremost I’d like to thank my thesis advisors prof. dr. Bruno Volckaert and dr. ir. Tim Wauters. By virtue of their knowledge and clear communication, I was able to maintain a clear target. Secondly I would like to thank prof. dr. ir. Filip De Turck for providing me the opportunity to conduct research in this field with the IDLab research group. Special thanks to Andres Felipe Ocampo Palacio and dr. Marleen Denert are in order as well. Mr. Ocampo’s Phd research into big data processing for network traffic and the resulting framework are an integral part of this thesis. Ms. Denert has been the go-to member of the faculty staff for general advice and administrative dealings. The final token of gratitude I’d like to extend to my family and friends for their continued support during this process. Laurens D’hooge Network traffic profiling and anomaly detection for cyber security Laurens D’hooge Supervisor(s): prof. dr. ir. Filip De Turck, dr. ir. Tim Wauters Abstract— This article is a short summary of the research findings of a creation of APT2.
    [Show full text]
  • Storage and Ingestion Systems in Support of Stream Processing
    Storage and Ingestion Systems in Support of Stream Processing: A Survey Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, María Pérez-Hernández, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae To cite this version: Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, María Pérez-Hernández, Radu Tudoran, et al.. Storage and Ingestion Systems in Support of Stream Processing: A Survey. [Technical Report] RT-0501, INRIA Rennes - Bretagne Atlantique and University of Rennes 1, France. 2018, pp.1-33. hal-01939280v2 HAL Id: hal-01939280 https://hal.inria.fr/hal-01939280v2 Submitted on 14 Dec 2018 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Storage and Ingestion Systems in Support of Stream Processing: A Survey Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, María S. Pérez-Hernández, Radu Tudoran, Stefano Bortoli, Bogdan Nicolae TECHNICAL REPORT N° 0501 November 2018 Project-Team KerData ISSN 0249-0803 ISRN INRIA/RT--0501--FR+ENG Storage and Ingestion Systems in Support of Stream Processing: A Survey Ovidiu-Cristian Marcu∗, Alexandru
    [Show full text]
  • A Study of Incremental Checkpointing in Distributed Stream Processing Systems
    A Study of Incremental Checkpointing in Distributed Stream Processing Systems A Thesis submitted to the designated by the General Assembly of Special Composition of the Department of Computer Science and Engineering Examination Committee by Aristidis Chronarakis in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN COMPUTER SCIENCE WITH SPECIALIZATION IN COMPUTER SYSTEMS University of Ioannina 2019 Examining Committee: • Kostas Magoutis, Assistant Professor, Department of Computer Science and Engineering, University of Ioannina (Supervisor) • Vassilios V. Dimakopoulos, Associate Professor, Department of Computer Sci- ence and Engineering, University of Ioannina • Evaggelia Pitoura, Professor, Department of Computer Science and Engineer- ing, University of Ioannina Dedication Dedicated to my family. Acknowledgements I would like to thank my advisor Prof. Kostas Magoutis for his guidance and support throughout my studies on the department, from the undergraduate level till the graduate. Special thanks to Prof. Vassilios Dimakopoulos and Prof. Evaggelia Pitoura for their participation as members of the examination committee. Finally, I would like to thank my family for the support and my friends for all the good moments we spent. Table of Contents List of Figures iii Abstract v Εκτεταμένη Περίληψη vi 1 Introduction 1 1.1 Objectives ................................... 2 1.2 Structure of this dissertation ......................... 3 2 Background 4 2.1 General concepts ............................... 4 2.2 Checkpoint-rollback methodology ..................... 7 2.3 Continuous eventual checkpointing (CEC) ................. 8 2.4 Apache Samza ................................ 9 2.4.1 Streams ................................ 9 2.4.2 Applications, Tasks, Containers ................... 10 2.4.3 State .................................. 11 2.4.4 Fault tolerance of stateful applications ............... 12 2.4.5 Message (tuple) replay and semantics ..............
    [Show full text]
  • Apache Samza
    Apache Samza Martin Kleppmann Definition vehicles, or the writes of records to a database. Apache Samza is an open source frame- Stream processing jobs are long- work for distributed processing of high- running processes that continuously volume event streams. Its primary design consume one or more event streams, goal is to support high throughput for a invoking some application logic on wide range of processing patterns, while every event, producing derived output providing operational robustness at the streams, and potentially writing output massive scale required by Internet com- to databases for subsequent querying. panies. Samza achieves this goal through While a batch process or a database a small number of carefully designed ab- query typically reads the state of a stractions: partitioned logs for messag- dataset at one point in time, and then ing, fault-tolerant local state, and cluster- finishes, a stream processor is never based task scheduling. finished: it continually awaits the arrival of new events, and it only shuts down when terminated by an administrator. Many tasks can be naturally ex- Overview pressed as stream processing jobs, for example: Stream processing is playing an increas- • aggregating occurrences of events, ingly important part of the data man- e.g., counting how many times a agement needs of many organizations. particular item has been viewed; Event streams can represent many kinds • computing the rate of certain events, of data, for example, the activity of users e.g., for system diagnostics, report- on a website, the movement of goods or ing, and abuse prevention; 1 2 Martin Kleppmann • enriching events with information the scalability of Samza is directly at- from a database, e.g., extending user tributable to the choice of these founda- click events with information about tional abstractions.
    [Show full text]
  • An Evaluation of Real-Time Processing of Call Detail Records Using Stream Processing
    UNIVERSITY OF NAIROBI COLLEGE OF BIOLOGICAL AND PHYSICAL SCIENCES SCHOOL OF COMPUTING AND INFORMATICS An Evaluation of Real-Time Processing of Call Detail Records Using Stream Processing CATHERINE KITHUSI WAMBUA P53/73389/2014 A research project report submitted to the School of Computing and Informatics in partial fulfillment of the requirements for the award of the Degree of Masters of Science in Distributed Computing Technology at the University of Nairobi December 2017. DECLARATION I certify that this research project report to the best of my knowledge, is my original authorial work except as acknowledged therein and has not been submitted for any other degree or professional qualification award in this or any other University. Signature: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Date: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Catherine Kithusi Wambua (P53/73389/2014) This research report has been submitted in partial fulfillment of the requirements for the Degree of Master of Science in Distributed Computing Technology at the University of Nairobi with my approval as the University supervisor. Signature: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Date: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Dr. Christopher Chepken i | P a g e DEDICATION To my beloved parents, for their unrelenting dedication to ensuring that my siblings and I acquired the best education despite all odds.
    [Show full text]
  • Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs"
    Appendix to the paper "Code smell prediction employing machine learning meets emerging Java language constructs" Hanna Grodzicka, Michał Kawa, Zofia Łakomiak, Arkadiusz Ziobrowski, Lech Madeyski (B) The Appendix includes two tables containing the dataset used in the paper "Code smell prediction employing machine learning meets emerging Java lan- guage constructs". The first table contains information about 792 projects selected for R package reproducer [Madeyski and Kitchenham(2019)]. Projects were the base dataset for cre- ating the dataset used in the study (Table I). The second table contains information about 281 projects filtered by Java version from build tool Maven (Table II) which were directly used in the paper. TABLE I: Base projects used to create the new dataset # Orgasation Project name GitHub link Commit hash Build tool Java version 1 adobe aem-core-wcm- www.github.com/adobe/ 1d1f1d70844c9e07cd694f028e87f85d926aba94 other or lack of unknown components aem-core-wcm-components 2 adobe S3Mock www.github.com/adobe/ 5aa299c2b6d0f0fd00f8d03fda560502270afb82 MAVEN 8 S3Mock 3 alexa alexa-skills- www.github.com/alexa/ bf1e9ccc50d1f3f8408f887f70197ee288fd4bd9 MAVEN 8 kit-sdk-for- alexa-skills-kit-sdk- java for-java 4 alibaba ARouter www.github.com/alibaba/ 93b328569bbdbf75e4aa87f0ecf48c69600591b2 GRADLE unknown ARouter 5 alibaba atlas www.github.com/alibaba/ e8c7b3f1ff14b2a1df64321c6992b796cae7d732 GRADLE unknown atlas 6 alibaba canal www.github.com/alibaba/ 08167c95c767fd3c9879584c0230820a8476a7a7 MAVEN 7 canal 7 alibaba cobar www.github.com/alibaba/
    [Show full text]
  • Parte I Studio Delle Tecnologie Utili Per L'analisi, L'elaborazione E L'interrogazione Di Big Data
    UNIVERSITÀ DEGLI STUDI DI MODENA E REGGIO EMILIA Dipartimento di Scienze Fisiche, Informatiche e Matematiche Corso di Laurea in Informatica Titolo Tesi Progettazione e sviluppo di un’applicazione Big Data per l’analisi e l’elaborazione di tweet in real-time RELATORE CANDIDATO Chiar.mo Professore Alessandro Pillo Riccardo Martoglia MATR. 111759 Anno Accademico 2018/2019 Indice Introduzione ………………………………….………………………………… pag. 1 Parte I Studio delle tecnologie utili per l’analisi, l’elaborazione e l’interrogazione di Big Data Capitolo I - “Ecosistema Hadoop” 1.1 Big Data ….………………………………….……………….………… pag. 3 1.2 Analisi derivate dalla figura ………………….…………….…….…… pag. 5 1.3 Processing layer …………………….……………………….………….. pag. 6 1.4 Distributed data processing & programming .…………….….…………. pag. 7 1.5 Sistemi analizzati in tabella .…………….……………..…….……..…… pag. 11 1.5.1 Apache Hadoop ……….….……………..………………………. pag. 11 1.5.2 Apache Apex ……….……….…………..………..……….….….. pag. 17 1.5.3 Apache Beam ……….…………………..…………………….…. pag. 20 1.5.4 Apache Flink……….…………………………………………..… pag. 24 1.5.5 Apache Samza …….……………..……………..……………….. pag. 30 1.5.6 Apache Spark .…….……………..……………………………… pag. 34 1.5.7 Apache Storm…….……………..……………………………….. pag. 38 1.5.8 Apache Tez .………….……………..……………………….…… pag. 40 1.5.9 Google MillWheel….……………..…………………………….. pag. 41 1.5.10 Google Cloud Dataflow…………..……………………………… pag. 41 1.5.11 IBM InfoSphere Streams ..………..………………………..…… pag. 43 1.5.12 Twitter Heron ………..………………..………………………… pag. 44 !I Capitolo II - “Machine Learning” 2.1 Introduzione al ML ………………………….………………………….. pag. 46 2.2 Algoritmi di machine learning …….…………………………………….. pag. 46 2.2.1 Training e Test Dataset ……….………………………………….. pag. 46 2.2.2 Fitting del modello: underfitting e overfitting ……………….. pag. 47 2.2.3 Apprendimento ………………………………………………….. pag. 49 2.3 Machine learning: “tradizionale” e “online” ………………………….… pag. 50 Parte II Studio di un’applicazione reale Capitolo III - “Applicazione reale: progetto” 3.1 Descrizione della realtà da analizzare ………………………………..….
    [Show full text]
  • Top Big Data Technologies for Data Ingestion
    International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 07 Issue: 05 | May 2020 www.irjet.net p-ISSN: 2395-0072 Top Big Data Technologies for Data Ingestion Kannan Sobti1, Prof. Deepika Dash2 1Dept. of Computer Science and Engineering, RV College of Engineering 2Dept. of Computer Science and Engineering, RV College of Engineering ---------------------------------------------------------------------***--------------------------------------------------------------------- Abstract - Businesses use data to make decisions. If must first be cleansed and transformed for analysis along analysed effectively, data enables forecasting of trends in with data from other sources. the market, helps businesses understand what customers Data can either be ingested in batches, in real time, or need and prefer, and enables a company to create strategy. in a combination of the two methods (the Lambda But the massive volume of data being generated from Architecture). Data is ingested in batches and is then numerous sources is a challenge. Businesses can only take imported at scheduled intervals. This is useful for full advantage of Big Data if all the data being generated is processes that run on a schedule, e.g., daily reports at a used to its potential. Today availability of data is not a specified time. Real-time ingestion is used when problem; the challenge is to effectively handle the incoming information collected is time-sensitive and has to be data, and do so swiftly enough so that data retains its monitored and acted upon from moment-to-moment, e.g., importance, and remains meaningful. This era of Big Data is power grid data. The Lambda Architecture balances the about efficient collection, storage and analyses of data, benefits of batch and real-time modes.
    [Show full text]
  • Complex Event Processing As a Service in Multi-Cloud Environments
    Western University Scholarship@Western Electronic Thesis and Dissertation Repository August 2016 Complex Event Processing as a Service in Multi-Cloud Environments Wilson A. Higashino The University of Western Ontario Supervisor Dr. Miriam A. M. Capretz The University of Western Ontario Graduate Program in Electrical and Computer Engineering A thesis submitted in partial fulfillment of the equirr ements for the degree in Doctor of Philosophy © Wilson A. Higashino 2016 Follow this and additional works at: https://ir.lib.uwo.ca/etd Part of the Databases and Information Systems Commons Recommended Citation Higashino, Wilson A., "Complex Event Processing as a Service in Multi-Cloud Environments" (2016). Electronic Thesis and Dissertation Repository. 4016. https://ir.lib.uwo.ca/etd/4016 This Dissertation/Thesis is brought to you for free and open access by Scholarship@Western. It has been accepted for inclusion in Electronic Thesis and Dissertation Repository by an authorized administrator of Scholarship@Western. For more information, please contact [email protected]. Abstract The rise of mobile technologies and the Internet of Things, combined with advances in Web technologies, have created a new Big Data world in which the volume and velocity of data gen- eration have achieved an unprecedented scale. As a technology created to process continuous streams of data, Complex Event Processing (CEP) has been often related to Big Data and used as a tool to obtain real-time insights. However, despite this recent surge of interest, the CEP market is still dominated by solutions that are costly and inflexible or too low-level and hard to operate. To address these problems, this research proposes the creation of a CEP system that can be o↵ered as a service and used over the Internet.
    [Show full text]