DOCSLIB.ORG

Big Data Mining Technologies by Kushagra Trivedi Contents

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Big Data Mining Technologies by Kushagra Trivedi Contents Big Data Mining Technologies By Kushagra Trivedi Contents • Introduction to big data and big data mining • Apache Hadoop for big data mining • Apache S4 for big data mining • Apache Mahout for machine learning • Some other tools of machine learning and data mining • Comparison of big data mining technologies • Conclusion • References I Introduction To Big Data And Big Data Mining • Data with large amount and greater complexity • Definition of big data • Sources of data expansion • Definition of data mining • Why data mining is necessary • Some of the technologies are used for data mining 1 Apache Hadoop • Data intensive distributes architecture • Centralized server vs. distributed server • MapReduce and Hadoop Distributed File System • HDFS divides data blocks among • Writing application that rapidly process large amount of data in parallel on large clusters of compute nodes • Applications - Yahoo, Facebook and other Fortune 50 companies are using apache Hadoop 2 Hadoop Distributed File System Name Node DataNode DataNode DataNode DataNode 1 2 3 4 b b b b b b b b … … … … 1 2 2 3 1 3 1 2 Cont… 3 Cont… • NameNode maintains all meta information about DataNodes • DataNodes contains actual data blocks • HDFS distributes and replicates data blocks among data nodes • Clients executes a query goes to NameNode and search actual data by looking at meta infomation 4 MapReduce Algorithm Austin 1 REDUCE Powers 1 defeated 1 GROUP Austin 2 the 1 Powers 2 MAP league 1 defeated 2 Austin Powers defeated of 1 the 2 1 the league of evil villains evil the 1 league 2 REDUCE villains 1 league 1 of 2 of 1 evil 2 If 1 MAP evil 1 villains 2 Austin 4 The league of evil villains villains 1 Powers 4 Austin Power defeated Austin 1 GROUP defeated4 Powers 1 the 4 if 1 defeated 1 league 4 austin 1 REDUCE of 4 powers 1 evil 4 MAP If 2 If Austin Power defeated defeated 1 GROUP villains 4 Austin 2 the 1 Where’s 1 the Powers 2 league 1 League of evil villains defeated2 of 1 the 2 evil 1 league 2 villains 1 Where’s 1 The 1 of 2 MAP evil 2 Where’s the league of evil League 1 Of 1 villains 2 villains Austin Powers Evil 1 Where’s 1 defeated Villains 1 Austin 1 Powers 1 Figure 2 MapReduce defeated 1 Cont…. 5 distribution [2] Continue…. • Uses two functions: map and reduce • Data are fed into map function in order to produce intermediate key and value pair • Intermediate result is then given to reduce function in order to produce final result • Task tracker- do work that is assigned by job tracker • Job tracker- if task tracker fails then reallocation of task tracker is done 6 Apache S4 • S4 stands for simple scalable streaming system • Uses MapReduce and Actor model for computation • Data processing is done through processing elements • S4 framework provide a way to route and create processing elements according to necessary • Applications - Yahoo, LinkedIn, A9 and Quantbench are several companies use Apache S4 for big data mining 7 Continue… Figure 3 S4 word count sample [6] 8 Continue… • Processing elements are basic computational units • Processing elements only executes those events for which key it was created • A special processing element is keyless element and it is created for accepting any type of input • Processing nodes are logical hosts of processing elements • S4 routes events to processing nodes based on hash value of keyed attributes in those events 9 Apache Mahout • Open source project of Apache foundation which allows programmer to write machine learning algorithm • Works on three different algorithms those are clustering, classification and collaborative filtering • Includes several distributed clustering algorithm such as k- Means, Fuzzy k-Means, Dirchlet, Mean-Shift and Canopy • Applications- Products you want to buy, people you might want to connect with, potential life partners and recommending songs you might like 10 Continue…. 1) Building a recommendation engine •) Currently provides “Taste Library” in order to build recommendation engine •) Library comes up with user based and item based recommendations •) Five preliminary components- DataMode, UserSimilarity, ItemSimilarity, Recommender, UserNeighborhood •) User can develop application that can give online and offline recommendations using these components 11 Continue…. 2) Clustering with Apache Mahout • Clustering algorithm written using MapReduce algorithm • Canopy, k-Means, Mean-Shift, and Dirichlet are clustering algorithms • Select the data and convert it into numerical presentation • Select particular algorithm any of above • Evacuate the result 12 Continue…. 3) Categorizing content with Mahout • Two approaches for categorizing - Naïve Bayes classifier and complementary naïve Bayes classifier • One part of Naive Bayes classifier process that deal with keeping track of the words associated with a particular document and category • Second deal with information prediction using part one • Complementary Naïve Bayes classifier is similar to naïve Bayes approach with simplicity 13 Some Other Tools of Machine Learning and Data Mining • Big data R is used for statistical computing using high performance statistical computing on big data • Machine Online Analysis is machine learning algorithm that is used for data stream mining • Massive Online Analysis uses classification, regression, clustering and frequent item set mining and frequent graph mining • Vowpal Wabbit is able to handle terabytes of data • Vowpal Wabbit can give better throughput using single machine network • Pegasus is big graph mining tool that finds patterns and anomalies from large massive graphs • GraphLab is High level parallel data mining system built without using MapReduce 14 Comparison • Apache Hadoop is used for batch processing • Data is divided into large size of blocks that makes it easy to handle • Put extra overhead of segmentation • Apache S4 is used for streaming data • No need of segmentation of data • Cannot add or remove nodes from running clusters • Apache Mahout is used for writing machine learning algorithm • No lack in community and documentation and examples are provided 15 Conclusion • Big data is crucial concern as data is going to increase in future • Different techniques are needed for mining this big data • Apache Mahout gives recommendations to users according to their past experience • Hadoop is used for data mining using MapReduce and HDFS • Apache S4 for mining streams of data • All techniques have their own significance for different types of companies 16 References [1] Apache Hadoop Fundamentals – HDFS and MapReduce Explained with a Diagram By RAMESH NATARAJAN on JANUARY 4, 2012 [2] Pros and Cons of Hadoop By Guruzon.com on June 01, 2013 [3] HDFS: Facebook has the world's largest Hadoop cluster! [4] S4 distributed stream of computing platform- overview [5] S4 distributed stream computing Platform By Aleksandar Bradic, Sr. Director, Engineering and R&D [6] Streaming Big Data By William Zhou in William Zhou's Blog on Sep 24, 2012 [7] Introducing Apache Mahout -Scalable, commercial-friendly machine learning for building intelligent applications by Grant Ingersoll on 08 September 2009 [8] Introduction to scalable machine learning with apache mahout Grant Ingersoll on 15 September 2010 [9] A. Bifet, G. Holmes, R. Kirkby, and B. Pfahringer. MOA: Massive Online Analysis [10] J. Langford. Vowpal Wabbit, 2011. [11]U. Kang, D. H. Chau, and C. Faloutsos. PEGASUS: Mining Billion-Scale Graphs in the Cloud. 2012. 17 [12]R. Smolan and J. Erwitt. The Human Face of Big Data. Sterling Publishing Company Incorporated, 2012. Any Queries :.
Load more

Recommended publications
  • Data Mining – Intro
    Data warehouse& Data Mining UNIT-3 Syllabus • UNIT 3 • Classification: Introduction, decision tree, tree induction algorithm – split algorithm based on information theory, split algorithm based on Gini index; naïve Bayes method; estimating predictive accuracy of classification method; classification software, software for association rule mining; case study; KDD Insurance Risk Assessment What is Data Mining? • Data Mining is: (1) The efficient discovery of previously unknown, valid, potentially useful, understandable patterns in large datasets (2) Data mining is the analysis step of the "knowledge discovery in databases" process, or KDD (2) The analysis of (often large) observational data sets to find unsuspected relationships and to summarize the data in novel ways that are both understandable and useful to the data owner Knowledge Discovery Examples of Large Datasets • Government: IRS, NGA, … • Large corporations • WALMART: 20M transactions per day • MOBIL: 100 TB geological databases • AT&T 300 M calls per day • Credit card companies • Scientific • NASA, EOS project: 50 GB per hour • Environmental datasets KDD The Knowledge Discovery in Databases (KDD) process is commonly defined with the stages: (1) Selection (2) Pre-processing (3) Transformation (4) Data Mining (5) Interpretation/Evaluation Data Mining Methods 1. Decision Tree Classifiers: Used for modeling, classification 2. Association Rules: Used to find associations between sets of attributes 3. Sequential patterns: Used to find temporal associations in time series 4. Hierarchical
    [Show full text]
  • Frequent Item Set Mining Using INC MINE in Massive Online Analysis Frame Work
    Available online at www.sciencedirect.com ScienceDirect Procedia Computer Science 45 ( 2015 ) 133 – 142 International Conference on Advanced Computing Technologies and Applications (ICACTA- 2015) Frequent Item set Mining using INC_MINE in Massive Online Analysis Frame work Prof.Dr.P.K.Srimania, Mrs. Malini M. Patilb* aFormer Chairman and Director, R & D, Bangalore University, Karnataka, India bAssistant Professor , Dept of ISE , J.S.S. Academy of Technical Education, Bangalore-560060, Karnataka, India Research Scholar, Bharthiar University, Coimbatore, Tamilnadu Abstract Frequent Pattern Mining is one of the major data mining techniques, which is exhaustively studied in the past decade. The technological advancements have resulted in huge data generation, having increased rate of data distribution. The generated data is called as a 'data stream'. Data streams can be mined only by using sophisticated techniques. The paper aims at carrying out frequent pattern mining on data streams. Stream mining has great challenges due to high memory usage and computational costs. Massive online analysis frame work is a software environment used to perform frequent pattern mining using INC_MINE algorithm. The algorithm uses the method of closed frequent mining. The data sets used in the analysis are Electricity data set and Airline data set. The authors also generated their own data set, OUR-GENERATOR for the purpose of analysis and the results are found interesting. In the experiments five samples of instance sizes (10000, 15000, 25000, 35000, 50000) are used with varying minimum support and window sizes for determining frequent closed itemsets and semi frequent closed itemsets respectively. The present work establishes that association rule mining could be performed even in the case of data stream mining by INC_MINE algorithm by generating closed frequent itemsets which is first of its kind in the literature.
    [Show full text]
  • Performance Analysis of Hoeffding Trees in Data Streams by Using Massive Online Analysis Framework
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by ePrints@Bangalore University Int. J. Data Mining, Modelling and Management, Vol. 7, No. 4, 2015 293 Performance analysis of Hoeffding trees in data streams by using massive online analysis framework P.K. Srimani R & D Division, Bangalore University Jnana Bharathi, Mysore Road, Bangalore-560056, Karnataka, India Email: [email protected] Malini M. Patil* Department of Information Science and Engineering, J.S.S. Academy of Technical Education, Uttaralli-Kengeri Main Road, Mylasandra, Bangalore-560060, Karnataka, India Email: [email protected] *Corresponding author Abstract: Present work is mainly concerned with the understanding of the problem of classification from the data stream perspective on evolving streams using massive online analysis framework with regard to different Hoeffding trees. Advancement of the technology both in the area of hardware and software has led to the rapid storage of data in huge volumes. Such data is referred to as a data stream. Traditional data mining methods are not capable of handling data streams because of the ubiquitous nature of data streams. The challenging task is how to store, analyse and visualise such large volumes of data. Massive data mining is a solution for these challenges. In the present analysis five different Hoeffding trees are used on the available eight dataset generators of massive online analysis framework and the results predict that stagger generator happens to be the best performer for different classifiers. Keywords: data mining; data streams; static streams; evolving streams; Hoeffding trees; classification; supervised learning; massive online analysis; MOA; framework; massive data mining; MDM; dataset generators.
    [Show full text]
  • Massive Online Analysis, a Framework for Stream Classification and Clustering
    MOA: Massive Online Analysis, a Framework for Stream Classification and Clustering. Albert Bifet1, Geoff Holmes1, Bernhard Pfahringer1, Philipp Kranen2, Hardy Kremer2, Timm Jansen2, and Thomas Seidl2 1 Department of Computer Science, University of Waikato, Hamilton, New Zealand fabifet, geoff, [email protected] 2 Data Management and Exploration Group, RWTH Aachen University, Germany fkranen, kremer, jansen, [email protected] Abstract. In today's applications, massive, evolving data streams are ubiquitous. Massive Online Analysis (MOA) is a software environment for implementing algorithms and running experiments for online learn- ing from evolving data streams. MOA is designed to deal with the chal- lenging problems of scaling up the implementation of state of the art algorithms to real world dataset sizes and of making algorithms compa- rable in benchmark streaming settings. It contains a collection of offline and online algorithms for both classification and clustering as well as tools for evaluation. Researchers benefit from MOA by getting insights into workings and problems of different approaches, practitioners can easily compare several algorithms and apply them to real world data sets and settings. MOA supports bi-directional interaction with WEKA, the Waikato Environment for Knowledge Analysis, and is released under the GNU GPL license. Besides providing algorithms and measures for evaluation and comparison, MOA is easily extensible with new contri- butions and allows the creation of benchmark scenarios through storing and sharing setting files. 1 Introduction Nowadays data is generated at an increasing rate from sensor applications, mea- surements in network monitoring and traffic management, log records or click- streams in web exploring, manufacturing processes, call detail records, email, blogging, twitter posts and others.
    [Show full text]
  • Adaptive Learning and Mining for Data Streams and Frequent Patterns
    Adaptive Learning and Mining for Data Streams and Frequent Patterns Doctoral Thesis presented to the Departament de Llenguatges i Sistemes Informatics` Universitat Politecnica` de Catalunya by Albert Bifet April 2009 Revised version with minor revisions. Advisors: Ricard Gavalda` and Jose´ L. Balcazar´ Abstract This thesis is devoted to the design of data mining algorithms for evolving data streams and for the extraction of closed frequent trees. First, we deal with each of these tasks separately, and then we deal with them together, developing classification methods for data streams containing items that are trees. In the data stream model, data arrive at high speed, and the algorithms that must process them have very strict constraints of space and time. In the first part of this thesis we propose and illustrate a framework for devel- oping algorithms that can adaptively learn from data streams that change over time. Our methods are based on using change detectors and estima- tor modules at the right places. We propose an adaptive sliding window algorithm ADWIN for detecting change and keeping updated statistics from a data stream, and use it as a black-box in place or counters or accumula- tors in algorithms initially not designed for drifting data. Since ADWIN has rigorous performance guarantees, this opens the possibility of extending such guarantees to learning and mining algorithms. We test our method- ology with several learning methods as Na¨ıve Bayes, clustering, decision trees and ensemble methods. We build an experimental framework for data stream mining with concept drift, based on the MOA framework, similar to WEKA, so that it will be easy for researchers to run experimental data stream benchmarks.
    [Show full text]
  • Online Learning for Big Data Analytics
    Online Learning for Big Data Analytics Irwin King, Michael R. Lyu and Haiqin Yang Department of Computer Science & Engineering The Chinese University of Hong Kong Tutorial presentation at IEEE Big Data, Santa Clara, CA, 2013 1 Outline • Introduction (60 min.) – Big data and big data analytics (30 min.) – Online learning and its applications (30 min.) • Online Learning Algorithms (60 min.) – Perceptron (10 min.) – Online non-sparse learning (10 min.) – Online sparse learning (20 min.) – Online unsupervised learning (20. min.) • Discussions + Q & A (5 min.) 2 Outline • Introduction (60 min.) – Big data and big data analytics (30 min.) – Online learning and its applications (30 min.) • Online Learning Algorithms (60 min.) – Perceptron (10 min.) – Online non-sparse learning (10 min.) – Online sparse learning (20 min.) – Online unsupervised learning (20. min.) • Discussions + Q & A (5 min.) 3 What is Big Data? • There is not a consensus as to how to define Big Data “A collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.” - wikii “Big data exceeds the reach of commonly used hardware environments and software tools to capture, manage, and process it with in a tolerable elapsed time for its user population.” - Tera- data magazine article, 2011 “Big data refers to data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” - The McKinsey Global Institute, 2011i 4 What is Big Data? Activity: Activity: IOPS File/Object Size, Content Volume Big Data refers to datasets grow so large and complex that it is difficult to capture, store, manage, share, analyze and visualize within current computational architecture.
    [Show full text]
  • An Extensible Framework for Data Stream Clustering Research with R
    JSS Journal of Statistical Software February 2017, Volume 76, Issue 14. doi: 10.18637/jss.v076.i14 Introduction to stream: An Extensible Framework for Data Stream Clustering Research with R Michael Hahsler Matthew Bolaños John Forrest Southern Methodist University Microsoft Corporation Microsoft Corporation Abstract In recent years, data streams have become an increasingly important area of research for the computer science, database and statistics communities. Data streams are ordered and potentially unbounded sequences of data points created by a typically non-stationary data generating process. Common data mining tasks associated with data streams include clustering, classification and frequent pattern mining. New algorithms for these types of data are proposed regularly and it is important to evaluate them thoroughly under standardized conditions. In this paper we introduce stream, a research tool that includes modeling and simu- lating data streams as well as an extensible framework for implementing, interfacing and experimenting with algorithms for various data stream mining tasks. The main advantage of stream is that it seamlessly integrates with the large existing infrastructure provided by R. In addition to data handling, plotting and easy scripting capabilities, R also provides many existing algorithms and enables users to interface code written in many program- ming languages popular among data mining researchers (e.g., C/C++, Java and Python). In this paper we describe the architecture of stream and focus on its use for data stream clustering research. stream was implemented with extensibility in mind and will be ex- tended in the future to cover additional data stream mining tasks like classification and frequent pattern mining.
    [Show full text]
  • MOA: a Real-Time Analytics Open Source Framework
    MOA: A Real-Time Analytics Open Source Framework Albert Bifet1,GeoffHolmes1, Bernhard Pfahringer1, Jesse Read1, Philipp Kranen2, Hardy Kremer2,TimmJansen2, and Thomas Seidl2 1 Department of Computer Science, University of Waikato, Hamilton, New Zealand {abifet,geoff,bernhard,jmr30}@cs.waikato.ac.nz 2 Data Management and Exploration Group, RWTH Aachen University, Germany {kranen,kremer,jansen,seidl}@cs.rwth-aachen.de Abstract. Massive Online Analysis (MOA) is a software environment for implementing algorithms and running experiments for online learning from evolving data streams. MOA is designed to deal with the challeng- ing problems of scaling up the implementation of state of the art algo- rithms to real world dataset sizes and of making algorithms comparable in benchmark streaming settings. It contains a collection of offline and online algorithms for classification, clustering and graph mining as well as tools for evaluation. For researchers the framework yields insights into advantages and disadvantages of different approaches and allows for the creation of benchmark streaming data sets through stored, shared and repeatable settings for the data feeds. Practitioners can use the frame- work to easily compare algorithms and apply them to real world data sets and settings. MOA supports bi-directional interaction with WEKA, the Waikato Environment for Knowledge Analysis. Besides providing al- gorithms and measures for evaluation and comparison, MOA is easily extensible with new contributions and allows for the creation of bench- mark scenarios. 1 Introduction In data stream scenarios data arrives at high speed strictly constraining pro- cessing algorithms in space and time. To adhere to these constraints, specific requirements have to be fulfilled by the stream processing algorithms, that are different from traditional batch processing settings.
    [Show full text]
  • Estra: an Easy Streaming Data Analysis Tool
    ESTRA: AN EASY STREAMING DATA ANALYSIS TOOL A THESIS SUBMITTED TO THE GRADUATE SCHOOL OF NATURAL AND APPLIED SCIENCES OF MIDDLE EAST TECHNICAL UNIVERSITY BY ECEHAN SAVA¸SBA¸SAK IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN COMPUTER ENGINEERING FEBRUARY 2021 Approval of the thesis: ESTRA: AN EASY STREAMING DATA ANALYSIS TOOL submitted by ECEHAN SAVA¸SBA¸SAK in partial fulfillment of the requirements for the degree of Master of Science in Computer Engineering Department, Middle East Technical University by, Prof. Dr. Halil Kalıpçılar Dean, Graduate School of Natural and Applied Sciences Prof. Dr. Halit Oguztüzün˘ Head of Department, Computer Engineering Prof. Dr. Mehmet Volkan Atalay Supervisor, Computer Engineering, METU Examining Committee Members: Prof. Dr. Ferda Nur Alpaslan Computer Engineering, METU Prof. Dr. Mehmet Volkan Atalay Computer Engineering, METU Prof. Dr. Fazlı Can Computer Engineering, Bilkent University Date: I hereby declare that all information in this document has been obtained and presented in accordance with academic rules and ethical conduct. I also declare that, as required by these rules and conduct, I have fully cited and referenced all material and results that are not original to this work. Name, Surname: Ecehan Sava¸sBa¸sak Signature : iv ABSTRACT ESTRA: AN EASY STREAMING DATA ANALYSIS TOOL Sava¸sBa¸sak,Ecehan M.S., Department of Computer Engineering Supervisor: Prof. Dr. Mehmet Volkan Atalay February 2021, 63 pages Easy Streaming Data Analysis Tool (ESTRA) is designed with the aim of creating an easy-to-use data stream analysis platform that serves the purpose of a quick and efficient tool to explore and prototype machine learning solutions on various datasets.
    [Show full text]
  • Interactive Event-Driven Knowledge Discovery from Data Streams
    UC Irvine UC Irvine Electronic Theses and Dissertations Title Interactive Event-driven Knowledge Discovery from Data Streams Permalink https://escholarship.org/uc/item/8bc5k0j3 Author Jalali, Laleh Publication Date 2016 License https://creativecommons.org/licenses/by/4.0/ 4.0 Peer reviewed|Thesis/dissertation eScholarship.org Powered by the California Digital Library University of California UNIVERSITY OF CALIFORNIA, IRVINE Interactive Event-driven Knowledge Discovery from Data Streams DISSERTATION submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Science by Laleh Jalali Dissertation Committee: Professor Ramesh Jain, Chair Professor Gopi Meenakshisundaram Professor Nalini Venkatasubramanian 2016 © 2016 Laleh Jalali TABLE OF CONTENTS Page LIST OF FIGURES v LIST OF TABLES viii LIST OF ALGORITHMS ix ACKNOWLEDGMENTS x CURRICULUM VITAE xi ABSTRACT OF THE DISSERTATION xiv 1 Introduction 1 1.1 Data-driven vs. Hypothesis-driven . .3 1.2 Knowledge Discovery from Temporal Data . .5 1.3 Contributions . .7 1.4 Thesis Outline . .9 2 Understanding Knowledge Discovery Process 10 2.1 Knowledge Discovery Definitions . 11 2.2 New Paradigm: Actionable Knowledge Extraction . 13 2.3 Data Mining and Knowledge Discovery Software Tools . 15 2.4 Knowledge Discovery in Healthcare Applications . 20 2.5 Design Requirements for an Interactive Knowledge Discovery Framework . 23 3 Literature Review 26 3.1 Temporal Data Mining . 27 3.1.1 Definitions and Concepts . 27 3.1.2 Pattern Discovery . 29 3.1.3 Temporal Association Rules . 35 3.1.4 Time Series Data Mining . 36 3.1.5 Temporal Classification and Clustering . 37 3.2 Temporal Reasoning . 38 3.2.1 Interval-based Temporal Logic .
    [Show full text]
  • A Framework and Algorithm for Data Stream Cluster Analysis
    (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 2, No. 11, 2011 CluSandra: A Framework and Algorithm for Data Stream Cluster Analysis Jose R. Fernandez Eman M. El-Sheikh Department of Computer Science Department of Computer Science University of West Florida University of West Florida Pensacola, FL, USA Pensacola, FL, USA Abstract—The clustering or partitioning of a dataset’s records to implement a clustering algorithm that is specifically into groups of similar records is an important aspect of designed to leverage the distributed framework. This paper knowledge discovery from datasets. A considerable amount of describes that clustering algorithm and the distributed research has been applied to the identification of clusters in very framework, which is entirely composed of off-the-shelf open large multi-dimensional and static datasets. However, the source components. The framework is referred to simply as traditional clustering and/or pattern recognition algorithms that CluSandra, while the algorithm, which is deployed onto the have resulted from this research are inefficient for clustering data streams. A data stream is a dynamic dataset that is framework, is referred to as the CluSandra algorithm. characterized by a sequence of data records that evolves over CluSandra‟s primary pillars are a database system called time, has extremely fast arrival rates and is unbounded. Today, Cassandra [9][15] and a message queuing system (MQS). the world abounds with processes that generate high-speed Cassandra, which is maintained by the Apache Software evolving data streams. Examples include click streams, credit Foundation (ASF), is a new breed of database system that is card transactions and sensor networks.
    [Show full text]
  • Package 'Streammoa'
    Package ‘streamMOA’ December 3, 2020 Version 1.2-3 Date 2020-12-03 Encoding UTF-8 Title Interface for MOA Stream Clustering Algorithms Description Interface for data stream clustering algorithms implemented in the MOA (Massive On- line Analysis) framework (Albert Bifet, Geoff Holmes, Richard Kirkby, Bern- hard Pfahringer (2010). MOA: Massive Online Analysis, Journal of Machine Learning Re- search 11: 1601-1604). Depends stream (>= 1.4-0), rJava (>= 0.9-0) Imports graphics, stats, methods SystemRequirements Java (>= 8) BugReports https://github.com/mhahsler/streamMOA License GPL-3 Copyright MOA code in inst/java/moa.jar is Copyright (C) The University of Waikato and distributed under the Apache License, version 2.0. All other code is Copyright (C) Matthew Bolanos, John Forrest and Michael Hahsler NeedsCompilation no Author Michael Hahsler [aut, cre, cph], John Forrest [aut, cph], Matthew Bolanos [ctb], Matthias Carnein [ctb], Dalibor Krleža [ctb] Maintainer Michael Hahsler <[email protected]> Repository CRAN Date/Publication 2020-12-03 18:20:03 UTC R topics documented: DSC_BICO_MOA . .2 1 2 DSC_BICO_MOA DSC_CluStream_MOA . .3 DSC_ClusTree_MOA . .4 DSC_DenStream_MOA . .6 DSC_DStream_MOA . .8 DSC_MCOD_MOA . .9 DSC_MOA . 10 DSC_StreamKM_MOA . 11 DSD_RandomRBFGeneratorEvents . 12 Index 14 DSC_BICO_MOA BICO - Fast computation of k-means coresets in a data stream Description This is an interface to the MOA implementation of BICO. The original BICO implementation by Fichtenberger et al is also available as DSC_BICO. Usage DSC_BICO_MOA(Cluster = 5, Dimensions, MaxClusterFeatures = 1000, Projections = 10, k = NULL, space = NULL, p = NULL) Arguments Cluster, k Number of desired centers Dimensions The number of the dimensions of the input points (stream) need to be specified in advance MaxClusterFeatures, space Maximum size of the coreset Projections, p Number of random projections used for the nearest neighbour search Details BICO maintains a tree which is inspired by the clustering tree of BIRCH, a SIGMOD Test of Time award-winning clustering algorithm.
    [Show full text]