DOCSLIB.ORG

Available Online

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Available Online ANNUAL REPORT 2018 Table of contents 2. Member logos 17. FET-HPC 2015 projects: A Wealth of Results! 3. Editorial 20. Snapshot: The technology stacks of High Performance 4. Our association Computing and Big Data Computing 4. Our Organisation 20. What they can learn from each other 5. The ETP4HPC Steering board 20. Stack overview and profiles 5. Office 23. What Big Data Computing can Learn from HPC 6. Our 96 members represent 23 countries 23. What HPC can Learn from Big Data Computing 6. Breakdown of our members by statute and profile 7. News from our Working Groups 24. Horizon Europe HPC Vision - - Working group on technological road mapping: 2018 Workshops and Future Plans - Energy Efficiency working group - Industrial User Working Group 26. New Centres of Excellence in computing applications w8. Ho will EuroHPC benefit from European HPC technology? 28. Our main events 10. New members Contact ETP4HPC [email protected] www.etp4hpc.eu @etp4h Text ETP4HPC ETP4HPC Chairman Jean-Pierre Panziera Editorial Team Pascale Bernier-Bruna (Coordinator) - Maike Gilliot - Michael Malms - Marcin Ostasz External contributions ANNUAL • Article on EuroHPC: Gustav Kalbe • New members: CoolIT, Constellcom Ltd, Infineon, Inspur, LINKS Foundation, Mellanox, Neovia, Saiver SRL, Submer, TS-JFL, UCit, University of A Coruňa • Article on FET-HPC 2015 projects: Jean-François Lavignon • New Centres of Excellence: Cheese, Excellerat, Hidalgo Graphic design and layout Antoine Maiffret (www.maiffret.net) Printed and bound by Snel Grafics – 4041 Vottem, Belgium REPORT 2018 2 • ETP4HPC • ANNUAL REPORT 2018 MEMBERS LOGOS ANNUAL REPORT 2018 • ETP4HPC • 3 Software for guided parallelization Editorial International Centre for Numerical Methods in Engineering JEAN-PIERRE PANZIERA, ETP4HPC CHAIR ear ETP4HPC Members and Partners, Things and Artificial Intelligence. The EXDCI2 project constitutes an excellent mechanism to DIt may be a fact known to only a few of you but develop those collaborations. I am an avid mountaineer. I am one of a team of old friends and each year we try to do at As we speak, we are finishing our Vision docu- least one challenging Alpine route – ‘challen- ment and we are embarking on the preparation ging’ according to our abilities, with some of us of our next Strategic Research Agenda (SRA). finding ‘the challenge’ part of it more deman- This SRA will be the source of priorities for the ding with the passing of time. When you scale research projects driven by EuroHPC. I encou- a mountain such as Dôme de Neige des Écrins rage you all to take part in the writing of the or Pic Coolidge, the most beautiful moment SRA. We will need to work very closely with all is not when you reach the summit – the most stakeholders and projects, in particular with poignant moment is when you realise that the the European Processor Initiative, in order to summit is within your reach, that you’re going deliver a vision that will allow EuroHPC to reach to make it to the top, sound and in one pie- its main objective: independent access to HPC ce. It usually happens when you pass the crux, resources for European users. the most difficult part, and you still feel strong enough to keep going and have the end of the We also need to think about the way back from journey in full view. the summit – to use another mountaineering analogy. We should focus on the areas where If you are part of the European HPC ecosys- Europe can achieve long-term leadership. tem, you should consider yourself very lucky, I encourage our members to reflect on the new because we are past that inflexion point. The challenges emerging from the discussions as- efforts made since the beginning of our jour- sociated with the SRA, e.g. end-to-end compu- ney, i.e. since the establishment of ETP4HPC ting, complex workflows, etc. and the beginnings of the construction of the European HPC eco-system, are paying off. We The rest of our journey will take place in close have a vibrant ecosystem in place meeting the cooperation with EuroHPC, and we should fo- needs of all users and we have over 150 tech- cus on ensuring that we combine efforts for the nology project results to build on. Our ins- best benefit of the European HPC value chain tinct was right when we began working with and its stakeholders. Please participate ac- BDVA – we are now working with our Big Data tively in this complex process as such a chance counterparts on the main advisory body of will not happen again. This is the time to make EuroHPC and we are extending this collabora- it happen, to shape the future of European HPC. ting pattern onto other areas: the Internet of Thank you for all your support to date. SABRI PLLANA LEADERS IN HIGH PERFORMANCE COMPUTING 2 • ETP4HPC • ANNUAL REPORT 2018 MEMBERS LOGOS ANNUAL REPORT 2018 • ETP4HPC • 3 Software for guided parallelization Editorial International Centre for Numerical Methods in Engineering JEAN-PIERRE PANZIERA, ETP4HPC CHAIR ear ETP4HPC Members and Partners, Things and Artificial Intelligence. The EXDCI2 project constitutes an excellent mechanism to DIt may be a fact known to only a few of you but develop those collaborations. I am an avid mountaineer. I am one of a team of old friends and each year we try to do at As we speak, we are finishing our Vision docu- least one challenging Alpine route – ‘challen- ment and we are embarking on the preparation ging’ according to our abilities, with some of us of our next Strategic Research Agenda (SRA). finding ‘the challenge’ part of it more deman- This SRA will be the source of priorities for the ding with the passing of time. When you scale research projects driven by EuroHPC. I encou- a mountain such as Dôme de Neige des Écrins rage you all to take part in the writing of the or Pic Coolidge, the most beautiful moment SRA. We will need to work very closely with all is not when you reach the summit – the most stakeholders and projects, in particular with poignant moment is when you realise that the the European Processor Initiative, in order to summit is within your reach, that you’re going deliver a vision that will allow EuroHPC to reach to make it to the top, sound and in one pie- its main objective: independent access to HPC ce. It usually happens when you pass the crux, resources for European users. the most difficult part, and you still feel strong enough to keep going and have the end of the We also need to think about the way back from journey in full view. the summit – to use another mountaineering analogy. We should focus on the areas where If you are part of the European HPC ecosys- Europe can achieve long-term leadership. tem, you should consider yourself very lucky, I encourage our members to reflect on the new because we are past that inflexion point. The challenges emerging from the discussions as- efforts made since the beginning of our jour- sociated with the SRA, e.g. end-to-end compu- ney, i.e. since the establishment of ETP4HPC ting, complex workflows, etc. and the beginnings of the construction of the European HPC eco-system, are paying off. We The rest of our journey will take place in close have a vibrant ecosystem in place meeting the cooperation with EuroHPC, and we should fo- needs of all users and we have over 150 tech- cus on ensuring that we combine efforts for the nology project results to build on. Our ins- best benefit of the European HPC value chain tinct was right when we began working with and its stakeholders. Please participate ac- BDVA – we are now working with our Big Data tively in this complex process as such a chance counterparts on the main advisory body of will not happen again. This is the time to make EuroHPC and we are extending this collabora- it happen, to shape the future of European HPC. ting pattern onto other areas: the Internet of Thank you for all your support to date. SABRI PLLANA LEADERS IN HIGH PERFORMANCE COMPUTING 4 • ETP4HPC • ANNUAL REPORT 2018 OUR ASSOCIATION OUR ASSOCIATION ANNUAL REPORT 2018 • ETP4HPC • 5 Our association The ETP4HPC Steering board TP4HPC, the European Technology Platform OUR ORGANISATION in the area of High-Performance Computing E Eternal (HPC), was incorporated as a Dutch association contributors in June 2012. We are an industry-led think tank gathering European HPC technology stakeholders: technology vendors, ISVs, service providers, re- search centres, and end users. ETP4HPCs Working Members Groups Our main objective is to increase the global market share of the European HPC technology vendors. Our main deliverable is our Strategic Research Agenda (SRA) which defines the priorities of the European HPC technology research programme managed by the European Commission. Full Members Until 2018, the ETP4HPC association represented the European HPC industry in the dialogue with the European Commission within the HPC contrac- tual Public-Private Partnership (cPPP). As of 2019, Office the role of the HPC cPPP is taken over by the EuroHPC Joint Undertaking (JU), in particular re- Steering Board 5 members garding HPC R&I funding and steering. In this new context, ETP4HPC will continue to help shape Europe’s HPC Research Programme. ETP4HPC is a The current Steering Board was The Steering Board appointed private member of the EuroHPC JU, and well re- elected at the Annual General the following Steering Committee: Office presented in its Research and Innovation Advisory Assembly on 13 March 2018, Chairman: for 2 years. Its current members Group.
Load more

Recommended publications
  • Horn: a System for Parallel Training and Regularizing of Large-Scale Neural Networks
    Horn: A System for Parallel Training and Regularizing of Large-Scale Neural Networks Edward J. Yoon [email protected] I Am ● Edward J. Yoon ● Member and Vice President of Apache Software Foundation ● Committer, PMC, Mentor of ○ Apache Hama ○ Apache Bigtop ○ Apache Rya ○ Apache Horn ○ Apache MRQL ● Keywords: big data, cloud, machine learning, database What is Apache Software Foundation? The Apache Software Foundation is an Non-profit foundation that is dedicated to open source software development 1) What Apache Software Foundation is, 2) Which projects are being developed, 3) What’s HORN? 4) and How to contribute them. Apache HTTP Server (NCSA HTTPd) powers nearly 500+ million websites (There are 644 million websites on the Internet) And Now! 161 Top Level Projects, 108 SubProjects, 39 Incubating Podlings, 4700+ Committers, 550 ASF Members Unknown number of developers and users Domain Diversity Programming Language Diversity Which projects are being developed? What’s HORN? ● Oct 2015, accepted as Apache Incubator Project ● Was born from Apache Hama ● A System for Deep Neural Networks ○ A neuron-level abstraction framework ○ Written in Java :/ ○ Works on distributed environments Apache Hama 1. K-means clustering Hama is 1,000x faster than Apache Mahout At UT Arlington & Oracle 2013 2. PageRank on 10 Billion edges Graph Hama is 3x faster than Facebook’s Giraph At Samsung Electronics (Yoon & Kim) 2015 3. Top-k Set Similarity Joins on Flickr Hama is clearly faster than Apache Spark At IEEE 2015 (University of Melbourne) Why we do this? 1. How to parallelize the training of large models? 2. How to avoid overfitting due to large size of the network, even with large datasets? JonathanNet Distributed Training Parameter Server Parameter Server Parameter Swapping Task 5 Each group performs Task 2 Task 4 Task 3 ..
    [Show full text]
  • Return of Organization Exempt from Income
    OMB No. 1545-0047 Return of Organization Exempt From Income Tax Form 990 Under section 501(c), 527, or 4947(a)(1) of the Internal Revenue Code (except black lung benefit trust or private foundation) Open to Public Department of the Treasury Internal Revenue Service The organization may have to use a copy of this return to satisfy state reporting requirements. Inspection A For the 2011 calendar year, or tax year beginning 5/1/2011 , and ending 4/30/2012 B Check if applicable: C Name of organization The Apache Software Foundation D Employer identification number Address change Doing Business As 47-0825376 Name change Number and street (or P.O. box if mail is not delivered to street address) Room/suite E Telephone number Initial return 1901 Munsey Drive (909) 374-9776 Terminated City or town, state or country, and ZIP + 4 Amended return Forest Hill MD 21050-2747 G Gross receipts $ 554,439 Application pending F Name and address of principal officer: H(a) Is this a group return for affiliates? Yes X No Jim Jagielski 1901 Munsey Drive, Forest Hill, MD 21050-2747 H(b) Are all affiliates included? Yes No I Tax-exempt status: X 501(c)(3) 501(c) ( ) (insert no.) 4947(a)(1) or 527 If "No," attach a list. (see instructions) J Website: http://www.apache.org/ H(c) Group exemption number K Form of organization: X Corporation Trust Association Other L Year of formation: 1999 M State of legal domicile: MD Part I Summary 1 Briefly describe the organization's mission or most significant activities: to provide open source software to the public that we sponsor free of charge 2 Check this box if the organization discontinued its operations or disposed of more than 25% of its net assets.
    [Show full text]
  • Projects – Other Than Hadoop! Created By:-Samarjit Mahapatra [email protected]
    Projects – other than Hadoop! Created By:-Samarjit Mahapatra [email protected] Mostly compatible with Hadoop/HDFS Apache Drill - provides low latency ad-hoc queries to many different data sources, including nested data. Inspired by Google's Dremel, Drill is designed to scale to 10,000 servers and query petabytes of data in seconds. Apache Hama - is a pure BSP (Bulk Synchronous Parallel) computing framework on top of HDFS for massive scientific computations such as matrix, graph and network algorithms. Akka - a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications on the JVM. ML-Hadoop - Hadoop implementation of Machine learning algorithms Shark - is a large-scale data warehouse system for Spark designed to be compatible with Apache Hive. It can execute Hive QL queries up to 100 times faster than Hive without any modification to the existing data or queries. Shark supports Hive's query language, metastore, serialization formats, and user-defined functions, providing seamless integration with existing Hive deployments and a familiar, more powerful option for new ones. Apache Crunch - Java library provides a framework for writing, testing, and running MapReduce pipelines. Its goal is to make pipelines that are composed of many user-defined functions simple to write, easy to test, and efficient to run Azkaban - batch workflow job scheduler created at LinkedIn to run their Hadoop Jobs Apache Mesos - is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.
    [Show full text]
  • Optimizing Resource Utilization in Distributed Computing Systems For
    THESE` DE DOCTORAT DE L’ETABLISSEMENT´ UNIVERSITE´ BOURGOGNE FRANCHE-COMTE´ PREPAR´ EE´ A` L’UNIVERSITE´ DE FRANCHE-COMTE´ Ecole´ doctorale n°37 Sciences Pour l’Ingenieur´ et Microtechniques Doctorat d’Informatique par ANTHONY NASSAR Optimizing Resource Utilization in Distributed Computing Systems for Automotive Applications Optimisation de l’utilisation des ressources dans les systemes` informatiques distribues´ pour les applications automobiles These` present´ ee´ et soutenue publiquement le 04-02-2021 a` Belfort, devant le Jury compose´ de : MR CERIN CHRISTOPHE Professeur a` l’Universite´ Sorbonne Paris Nord President´ MR CHBEIR RICHARD Professeur a` l’Universite´ de Pau et des Pays de l’Adour Rapporteur MME BENBERNOU SALIMA Professeur a` l’Universite´ Paris-Descartes Rapporteur MR MOSTEFAOUI AHMED Maˆıtre de conferences´ a` l’Universite´ de Franche-Comte´ Directeur de these` MR DESSABLES FRANC¸ OIS Ingenieur´ chez Groupe PSA Codirecteur de these` DOCTORAL THESIS OF THE UNIVERSITY BOURGOGNE FRANCHE-COMTE´ INSTITUTION PREPARED AT UNIVERSITE´ DE FRANCHE-COMTE´ Doctoral school n°37 Engineering Sciences and Microtechnologies Computer Science Doctorate by ANTHONY NASSAR Optimizing Resource Utilization in Distributed Computing Systems for Automotive Applications Optimisation de l’utilisation des ressources dans les systemes` informatiques distribues´ pour les applications automobiles Thesis presented and publicly defended in Belfort, on 04-02-2021 Composition of the Jury : CERIN CHRISTOPHE Professor at Universite´ Sorbonne Paris Nord President
    [Show full text]
  • Graft: a Debugging Tool for Apache Giraph
    Graft: A Debugging Tool For Apache Giraph Semih Salihoglu, Jaeho Shin, Vikesh Khanna, Ba Quan Truong, Jennifer Widom Stanford University {semih, jaeho.shin, vikesh, bqtruong, widom}@cs.stanford.edu ABSTRACT optional master.compute() function is executed by the Master We address the problem of debugging programs written for Pregel- task between supersteps. like systems. After interviewing Giraph and GPS users, we devel- We have tackled the challenge of debugging programs written oped Graft. Graft supports the debugging cycle that users typically for Pregel-like systems. Despite being a core component of pro- go through: (1) Users describe programmatically the set of vertices grammers’ development cycles, very little work has been done on they are interested in inspecting. During execution, Graft captures debugging in these systems. We interviewed several Giraph and the context information of these vertices across supersteps. (2) Us- GPS programmers (hereafter referred to as “users”) and studied vertex.compute() ing Graft’s GUI, users visualize how the values and messages of the how they currently debug their functions. captured vertices change from superstep to superstep,narrowing in We found that the following three steps were common across users: suspicious vertices and supersteps. (3) Users replay the exact lines (1) Users add print statements to their code to capture information of the vertex.compute() function that executed for the sus- about a select set of potentially “buggy” vertices, e.g., vertices that picious vertices and supersteps, by copying code that Graft gener- are assigned incorrect values, send incorrect messages, or throw ates into their development environments’ line-by-line debuggers.
    [Show full text]
  • A Review on Big Data Analytics in the Field of Agriculture
    International Journal of Latest Transactions in Engineering and Science A Review on Big Data Analytics in the field of Agriculture Harish Kumar M Department. of Computer Science and Engineering Adhiyamaan College of Engineering,Hosur, Tamilnadu, India Dr. T Menakadevi Dept. of Electronics and Communication Engineering Adhiyamaan College of Engineering,Hosur, Tamilnadu, India Abstract- Big Data Analytics is a Data-Driven technology useful in generating significant productivity improvement in various industries by collecting, storing, managing, processing and analyzing various kinds of structured and unstructured data. The role of big data in Agriculture provides an opportunity to increase economic gain of the farmers by undergoing digital revolution in this aspect we examine through precision agriculture schemas equipped in many countries. This paper reviews the applications of big data to support agriculture. In addition it attempts to identify the tools that support the implementation of big data applications for agriculture services. The review reveals that several opportunities are available for utilizing big data in agriculture; however, there are still many issues and challenges to be addressed to achieve better utilization of this technology. Keywords—Agriculture, Big data Analytics, Hadoop, HDFS, Farmers I. INTRODUCTION The technologies employed are exciting, involve analysis of mind-numbing amounts of data and require fundamental rethinking as to what constitutes data. Big data is a collecting raw data which undergoes various phases like Classification, Processing and organizing into meaningful information. Raw information cannot be consumed directly for any form of analysis. It’s a process of examining uncover patterns, finding unknown correlation and finding useful information which are adopted for decision making analysis.
    [Show full text]
  • Full-Graph-Limited-Mvn-Deps.Pdf
    org.jboss.cl.jboss-cl-2.0.9.GA org.jboss.cl.jboss-cl-parent-2.2.1.GA org.jboss.cl.jboss-classloader-N/A org.jboss.cl.jboss-classloading-vfs-N/A org.jboss.cl.jboss-classloading-N/A org.primefaces.extensions.master-pom-1.0.0 org.sonatype.mercury.mercury-mp3-1.0-alpha-1 org.primefaces.themes.overcast-${primefaces.theme.version} org.primefaces.themes.dark-hive-${primefaces.theme.version}org.primefaces.themes.humanity-${primefaces.theme.version}org.primefaces.themes.le-frog-${primefaces.theme.version} org.primefaces.themes.south-street-${primefaces.theme.version}org.primefaces.themes.sunny-${primefaces.theme.version}org.primefaces.themes.hot-sneaks-${primefaces.theme.version}org.primefaces.themes.cupertino-${primefaces.theme.version} org.primefaces.themes.trontastic-${primefaces.theme.version}org.primefaces.themes.excite-bike-${primefaces.theme.version} org.apache.maven.mercury.mercury-external-N/A org.primefaces.themes.redmond-${primefaces.theme.version}org.primefaces.themes.afterwork-${primefaces.theme.version}org.primefaces.themes.glass-x-${primefaces.theme.version}org.primefaces.themes.home-${primefaces.theme.version} org.primefaces.themes.black-tie-${primefaces.theme.version}org.primefaces.themes.eggplant-${primefaces.theme.version} org.apache.maven.mercury.mercury-repo-remote-m2-N/Aorg.apache.maven.mercury.mercury-md-sat-N/A org.primefaces.themes.ui-lightness-${primefaces.theme.version}org.primefaces.themes.midnight-${primefaces.theme.version}org.primefaces.themes.mint-choc-${primefaces.theme.version}org.primefaces.themes.afternoon-${primefaces.theme.version}org.primefaces.themes.dot-luv-${primefaces.theme.version}org.primefaces.themes.smoothness-${primefaces.theme.version}org.primefaces.themes.swanky-purse-${primefaces.theme.version}
    [Show full text]
  • Classifying, Evaluating and Advancing Big Data Benchmarks
    Classifying, Evaluating and Advancing Big Data Benchmarks Dissertation zur Erlangung des Doktorgrades der Naturwissenschaften vorgelegt beim Fachbereich 12 Informatik der Johann Wolfgang Goethe-Universität in Frankfurt am Main von Todor Ivanov aus Stara Zagora Frankfurt am Main 2019 (D 30) vom Fachbereich 12 Informatik der Johann Wolfgang Goethe-Universität als Dissertation angenommen. Dekan: Prof. Dr. Andreas Bernig Gutachter: Prof. Dott. -Ing. Roberto V. Zicari Prof. Dr. Carsten Binnig Datum der Disputation: 23.07.2019 Abstract The main contribution of the thesis is in helping to understand which software system parameters mostly affect the performance of Big Data Platforms under realistic workloads. In detail, the main research contributions of the thesis are: 1. Definition of the new concept of heterogeneity for Big Data Architectures (Chapter 2); 2. Investigation of the performance of Big Data systems (e.g. Hadoop) in virtual- ized environments (Section 3.1); 3. Investigation of the performance of NoSQL databases versus Hadoop distribu- tions (Section 3.2); 4. Execution and evaluation of the TPCx-HS benchmark (Section 3.3); 5. Evaluation and comparison of Hive and Spark SQL engines using benchmark queries (Section 3.4); 6. Evaluation of the impact of compression techniques on SQL-on-Hadoop engine performance (Section 3.5); 7. Extensions of the standardized Big Data benchmark BigBench (TPCx-BB) (Section 4.1 and 4.3); 8. Definition of a new benchmark, called ABench (Big Data Architecture Stack Benchmark), that takes into account the heterogeneity of Big Data architectures (Section 4.5). The thesis is an attempt to re-define system benchmarking taking into account the new requirements posed by the Big Data applications.
    [Show full text]
  • Harp: Collective Communication on Hadoop
    HHarp:arp: CollectiveCollective CommunicationCommunication onon HadooHadoopp Judy Qiu, Indiana University SA OutlineOutline • Machine Learning on Big Data • Big Data Tools • Iterative MapReduce model • MDS Demo • Harp SA MachineMachine LearningLearning onon BigBig DataData • Mahout on Hadoop • https://mahout.apache.org/ • MLlib on Spark • http://spark.apache.org/mllib/ • GraphLab Toolkits • http://graphlab.org/projects/toolkits.html • GraphLab Computer Vision Toolkit SA World Data DAG Model MapReduce Model Graph Model BSP/Collective Mode Hadoop MPI HaLoop Giraph Twister Hama or GraphLab ions/ ions/ Spark GraphX ning Harp Stratosphere Dryad/ Reef DryadLIN Q Pig/PigLatin Hive Query Tez Drill Shark MRQL S4 Storm or ming Samza Spark Streaming BigBig DataData ToolsTools forfor HPCHPC andand SupercomputingSupercomputing • MPI(Message Passing Interface, 1992) • Provide standardized function interfaces for communication between parallel processes. • Collective communication operations • Broadcast, Scatter, Gather, Reduce, Allgather, Allreduce, Reduce‐scatter. • Popular implementations • MPICH (2001) • OpenMPI (2004) • http://www.open‐mpi.org/ SA MapReduceMapReduce ModelModel Google MapReduce (2004) • Jeffrey Dean et al. MapReduce: Simplified Data Processing on Large Clusters. OSDI 2004. Apache Hadoop (2005) • http://hadoop.apache.org/ • http://developer.yahoo.com/hadoop/tutorial/ Apache Hadoop 2.0 (2012) • Vinod Kumar Vavilapalli et al. Apache Hadoop YARN: Yet Another Resource Negotiator, SO 2013. • Separation between resource management and computation
    [Show full text]
  • 20-Graph-Computing-S
    Distributed Systems 20. Other parallel frameworks Paul Krzyzanowski Rutgers University Fall 2017 November 20, 2017 © 2014-2017 Paul Krzyzanowski 1 Can we make MapReduce easier? 2 Apache Pig • Why? – Make it easy to use MapReduce via scripting instead of Java – Make it easy to use multiple MapReduce stages – Built-in common operations for join, group, filter, etc. • How to use? – Use Grunt – the pig shell – Submit a script directly to pig – Use the PigServer Java class – PigPen – Eclipse plugin • Pig compiles to several Hadoop MapReduce jobs 3 Apache Pig Count Job Pig Framework Hadoop (in Pig Latin) Execution • Parse A = LOAD ‘myfile’ AS (x, y, z); • Check • Map: Filter B = FILTER A by x>0; • Optimize • Reduce: Counter C = GROUP B by x; • Plan Execution D = FOREACH A GENERATE • Submit jar to Hadoop x, COUNT(B); • Monitor progress STORE D into ’output’; 4 Pig: Loading Data Load/store relations in the following formats: • PigStorage: field-delimited text • BinStorage: binary files • BinaryStorage: single-field tuples with a value of bytearray • TextLoader: plain-text • PigDump: stores using toString() on tuples, one per line 5 Example log = LOAD ‘test.log’ AS (user, timestamp, query); grpd = GROUP log by user; cntd = FOREACH grpd GENERATE group, COUNT(log); fltrd = FILTER cntd BY cnt > 50; srtd = ORDER fltrd BY cnt; STORE srtd INTO ‘output’; • Each statement defines a new dataset – Datasets can be given aliases to be used later • FOREACH iterates over the members of a ”bag” – Input is grpd: list of log entries grouped by user – Output is
    [Show full text]
  • Big Data Processing on Arbitrarily Distributed Dataset
    Big Data Processing on Arbitrarily Distributed Dataset by Dongyao Wu A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY SCHOOL OF COMPUTER SCIENCE AND ENGINEERING FACULTY OF ENGINEERING Thursday, 1st June, 2017 All rights reserved. This work may not be reproduced in whole or in part, by photocopy or other means, without the permission of the author. c 2017 by Dongyao Wu PLEASE TYPE THE UNIVERSITY OF NEW SOUTH WALES Thesis/Dissertation Sheet Surname or Family name: Wu First name: Dongyao Other name/s: Abbreviation for degree as given in the University calendar: PhD School: School of Computer Science and Engineering Faculty: Faculty of Engineering Title: Big Data Processing on Arbitrarily Distributed Dataset Abstract 350 words maximum: (PLEASE TYPE) Over the past years, frameworks such as MapReduce and Spark have been Introduced to ease the task of developing big data programs and applications. These frameworks significantly reduce the complexity of developing big data programs and applications. However, in reality, many real-world scenarios require pipelining and Integration of multiple big data jobs. As the big data pipelines and applications become more and more complicated, It Is almost Impossible to manually optimize the performance for each component not to mention the whole pipeline/application. At the same time, there are also increasing requirements to facilitate interaction, composition and Integration for big data analytics applications In continuously evolving, Integrating and delivering scenarios. In addition, with the emergence and development of cloud computing, mobile computing and the Internet of Things, data are Increasingly collected and stored In highly distributed infrastructures (e.g.
    [Show full text]
  • Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs"
    Appendix to the paper "Code smell prediction employing machine learning meets emerging Java language constructs" Hanna Grodzicka, Michał Kawa, Zofia Łakomiak, Arkadiusz Ziobrowski, Lech Madeyski (B) The Appendix includes two tables containing the dataset used in the paper "Code smell prediction employing machine learning meets emerging Java lan- guage constructs". The first table contains information about 792 projects selected for R package reproducer [Madeyski and Kitchenham(2019)]. Projects were the base dataset for cre- ating the dataset used in the study (Table I). The second table contains information about 281 projects filtered by Java version from build tool Maven (Table II) which were directly used in the paper. TABLE I: Base projects used to create the new dataset # Orgasation Project name GitHub link Commit hash Build tool Java version 1 adobe aem-core-wcm- www.github.com/adobe/ 1d1f1d70844c9e07cd694f028e87f85d926aba94 other or lack of unknown components aem-core-wcm-components 2 adobe S3Mock www.github.com/adobe/ 5aa299c2b6d0f0fd00f8d03fda560502270afb82 MAVEN 8 S3Mock 3 alexa alexa-skills- www.github.com/alexa/ bf1e9ccc50d1f3f8408f887f70197ee288fd4bd9 MAVEN 8 kit-sdk-for- alexa-skills-kit-sdk- java for-java 4 alibaba ARouter www.github.com/alibaba/ 93b328569bbdbf75e4aa87f0ecf48c69600591b2 GRADLE unknown ARouter 5 alibaba atlas www.github.com/alibaba/ e8c7b3f1ff14b2a1df64321c6992b796cae7d732 GRADLE unknown atlas 6 alibaba canal www.github.com/alibaba/ 08167c95c767fd3c9879584c0230820a8476a7a7 MAVEN 7 canal 7 alibaba cobar www.github.com/alibaba/
    [Show full text]