DOCSLIB.ORG

Cloudera® Cca175: Hadoop and Spark Developer Faq & Study Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Cloudera® Cca175: Hadoop and Spark Developer Faq & Study Guide 1. Apache Spark Professional Training with Hands On Lab Sessions 2. Oreilly Databricks Apache Spark Developer Certification Simulator 3. Hadoop Professional Training 4. Apache OOZie HandsOn Professional Training 5. NiFi Professional Training with Hands-on session CLOUDERA® CCA175: HADOOP AND SPARK DEVELOPER FAQ & STUDY GUIDE By http://www.HadoopExam.com Cloudera CCA175 (Hadoop and Spark Developer Hands-on Certification available with total 90 solved problem scenarios. Click for More Detail) Cloudera CCPDE575 (Hadoop BigData Data Engineer Professional Hands-on Certification available with total 79 solved problem scenarios. Click for More Detail) Cloudera CCA159 Data Analyst Certification Practice Questions (Total 73 HandsOn Practice Questions) Question 1: How frequently CCA175 questions or syllabus updates? Answer: Generally, we don’t see any fixed timeline for syllabus updates, CCA175 exam syllabus are certainly revised time to time. We have seen many times (almost every year) in last few years not only syllabus but questions pattern have changed. Cloudera is pioneer for the Hadoop/BigData framework and we have been following it since 2012 to make our CCA175 Certification product in-line with their syllabus. Regarding updates, you need to continuously check for the syllabus on Cloudera certification page. However, 1000’s of learners use our certification preparation material, so we get update frequently and it is easy to update http://www.HadoopExam.com CCA175 certification simulator. Once the Cloudera update the syllabus, you should always prepare according to that new syllabus. Frequency of syllabus update is once in a year, we have seen till now. Question 2: If I found any issue, with scheduling of CCA175 exam, whom should I contact? Answer: You can use the email id provided at their certification page [email protected] . We have seen many learner face problem for scheduling the exam and they need to contact Cloudera for the same. Currently exam is conducted on AWS platform and some unique AWS ID is assigned/used for each certification aspirants. Once an ID is used on AWS, then Cloudera will not be able to register that particular email again. If you face problem than use above email id, this is one of the best way to resolve this issue. Question 3: I am using Scala for my CCA175 certification, but as I know for submitting application in Scala/Java, I need to first create a Jar out of my code. It will take lot of time and I am not sure that tools will be available to create Jar file, during exam? Answer: Hmm, this is very valid question. Learners who use Scala for CCA175 exam, they always face this concern. However, Cloudera is not going to test, that you have knowledge of building jar or not. Hence, you should not worry about building the jar file for your Scala code, instead use spark-shell to submit your Scala code line by line and produce the desired result. Not creating Jar file, will save your lot of time. You need to consider these questions for preparing for CCA175 certification, because more than 50 Spark practice questions are given, in total you will find 111 questions for CCA175 exam these are always updated based on new syllabus. You can run all the example line by line as well, and same practice you need to follow in real exam. If you are using Python instead of Scala, then you don’t have to build a Jar kind of package to submit your application. Go through this 111 updated questions, you will become very well comfortable with the real exam. Make sure, you practice all questions with valid version of Spark framework, otherwise your practice go wasted, you may have learn to run code with the latest API, but that might not work with Cloudera real exam environment. Based on or learners feedback, we have recently updated these questions and added 16 new questions. Our learners average scoring is 9/10 and even many learners scored 10/10. By completing this 111 questions, you will not only able to clear your exam, but you can use same practices in your daily regular routing work for Big Data Engineering using Spark, Scala, Python, Sqoop, Flume, HDFS, Shell Script, Spark SQL and much more. These CCA175 questions are regularly updated based on our learner’s feedback, hence once you appear in exam, please provide feedback about your experience in real exam on [email protected] , and anything new you find regarding the exam, if we are missing this will help all the future aspirants. You will have to save the code, somewhere which you have used in spark-shell, so that Cloudera can verify your approach. They will provide you the detail, where to submit/save the code. Question 4: Which version of Spark is used in CCA175 certification? Answer: Let me first tell you that Cloudera is not only testing your Spark programming skill, they are testing that you are able to work with currently available CDH (Cloudera Distribution of Hadoop) version. Which is bundled with many eco-system of BigData Hadoop, Spark is one of them, and they wanted to test many of the available components for the developer. Even you see most of the organization use Spark well integrated with the Hadoop distribution either HDP, CDH or MapR Hadoop. Hence, whatever Spark version is available in that particular CDH version, will be tested. Hence, currently you should be able to write code using Spark 1.6 programming API. That is why it is very important that you practice questions based on specific version of Spark and the best approach to follow these 111 questions of CCA175, which has detailed problem statement as well as its step by step solution, which has been executed on that specific Spark platform. Not only you should get acquainted with the Spark 1.6 programming model, but also you should become comfortable working with the CDH platform, how to use path for file stored in HDFS , how to use JDBC connection, specially connection MySQL database using Spark 1.6 only (Don’t use older or newer version, be specific here). Question 5: I am quite comfortable using Sqoop and I practiced all the 111 questions of CCA175, given here. But I cannot remember all the options used in Sqoop import and export command, what to do in this case? Answer: Generally programmer don’t remember all the APIs and options of the commands, but they are aware this particular feature is available and to know the exact syntax they need to use documentation. Cloudera provides the documentation for Sqoop in CCA175 real exam. As you are quite comfortable after practicing these 111 questions, you may not need to go to documentation, to save time. Rather go to terminal and type sqoop help import or sqoop help export etc. This would be quite faster approach and save lot of time. This is very important you are well managed during the exam and no all the possible ways of saving your time. But don’t get panic, and calmly follow the documentation if you forgot to use particular command syntax. Cloudera will provide all the documentation for the products used in their CDH platform. Question 6: Cloudera introduce another exam for Data Analyst CCA159, does it make any difference with the existing CCA175 exam? Answer: Cloudera Hadoop Distribution are used by many different profile of professional, like if you are a data analyst then you should not be writing complex programming in Spark, but rather prefer SQL based solution. If you are a developer than you will be using programming to work with data. Even for the people/learners who are more comfortable with the advance feature of Cloudera Hadoop Distribution most of the Big Data architect and experienced developer use CCP:DE575 exam (this is Cloudera Certified Professional exam). So the main impact of introducing Data Analyst exam is that many type of questions, will be moved out of CCA175 exam like DDL (Data Definition Question) defining tables etc. However, we suggest being a developer you should be able to do this. More complex DML syntax are not expected from you in CCA175 exam, like write very complex Hive and Impala queries, this is expected in CCA159 exam instead. You should be able to use Hive meta-store from Spark, Sqoop etc. and able to write some queries using Spark SQL (But very complex queries, specially analytical functions based queries are not expected). You should be able to filter, format, join the data in CCA175 exam. So the best way to be specific preparation use this 111 questions for practicing your real exam. Average scoring by learners is 9/10 after practicing this CCA175, total 111 questions, which come along with the required data as well as solution and also complimentary videos are available in which solution and problem are explained in detail. Whenever syllabus changed, feedback from previous learners are very helpful and we at http://www.HadoopExam.com integrate that feedback as soon as possible after receiving it from the learners. We have launched this CCA175 certification material with just 50 practice questions and now we have in total 111 questions, which are updated and corrected based on feedback of successful candidates. Question 7: What exactly is the CCA175, 111 question pattern and HadoopExam.com problem scenario follow the same pattern? Answer: Each CCA question requires you to solve a particular scenario. In some cases, a tool such as Impala or Hive may be used. In other cases, coding is required. In order to speed up development time of Spark questions, a template is often provided that contains a skeleton of the solution, asking the candidate to fill in the missing lines with functional code.
Load more

Recommended publications
  • Apache Oozie Apache Oozie Get a Solid Grounding in Apache Oozie, the Workflow Scheduler System for “In This Book, the Managing Hadoop Jobs
    Apache Oozie Apache Oozie Apache Get a solid grounding in Apache Oozie, the workflow scheduler system for “In this book, the managing Hadoop jobs. In this hands-on guide, two experienced Hadoop authors have striven for practitioners walk you through the intricacies of this powerful and flexible platform, with numerous examples and real-world use cases. practicality, focusing on Once you set up your Oozie server, you’ll dive into techniques for writing the concepts, principles, and coordinating workflows, and learn how to write complex data pipelines. tips, and tricks that Advanced topics show you how to handle shared libraries in Oozie, as well developers need to get as how to implement and manage Oozie’s security capabilities. the most out of Oozie. ■ Install and confgure an Oozie server, and get an overview of A volume such as this is basic concepts long overdue. Developers ■ Journey through the world of writing and confguring will get a lot more out of workfows the Hadoop ecosystem ■ Learn how the Oozie coordinator schedules and executes by reading it.” workfows based on triggers —Raymie Stata ■ Understand how Oozie manages data dependencies CEO, Altiscale ■ Use Oozie bundles to package several coordinator apps into Oozie simplifies a data pipeline “ the managing and ■ Learn about security features and shared library management automating of complex ■ Implement custom extensions and write your own EL functions and actions Hadoop workloads. ■ Debug workfows and manage Oozie’s operational details This greatly benefits Apache both developers and Mohammad Kamrul Islam works as a Staff Software Engineer in the data operators alike.” engineering team at Uber.
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Persisting Big-Data the Nosql Landscape
    Information Systems 63 (2017) 1–23 Contents lists available at ScienceDirect Information Systems journal homepage: www.elsevier.com/locate/infosys Persisting big-data: The NoSQL landscape Alejandro Corbellini n, Cristian Mateos, Alejandro Zunino, Daniela Godoy, Silvia Schiaffino ISISTAN (CONICET-UNCPBA) Research Institute1, UNICEN University, Campus Universitario, Tandil B7001BBO, Argentina article info abstract Article history: The growing popularity of massively accessed Web applications that store and analyze Received 11 March 2014 large amounts of data, being Facebook, Twitter and Google Search some prominent Accepted 21 July 2016 examples of such applications, have posed new requirements that greatly challenge tra- Recommended by: G. Vossen ditional RDBMS. In response to this reality, a new way of creating and manipulating data Available online 30 July 2016 stores, known as NoSQL databases, has arisen. This paper reviews implementations of Keywords: NoSQL databases in order to provide an understanding of current tools and their uses. NoSQL databases First, NoSQL databases are compared with traditional RDBMS and important concepts are Relational databases explained. Only databases allowing to persist data and distribute them along different Distributed systems computing nodes are within the scope of this review. Moreover, NoSQL databases are Database persistence divided into different types: Key-Value, Wide-Column, Document-oriented and Graph- Database distribution Big data oriented. In each case, a comparison of available databases
    [Show full text]
  • HDP 3.1.4 Release Notes Date of Publish: 2019-08-26
    Release Notes 3 HDP 3.1.4 Release Notes Date of Publish: 2019-08-26 https://docs.hortonworks.com Release Notes | Contents | ii Contents HDP 3.1.4 Release Notes..........................................................................................4 Component Versions.................................................................................................4 Descriptions of New Features..................................................................................5 Deprecation Notices.................................................................................................. 6 Terminology.......................................................................................................................................................... 6 Removed Components and Product Capabilities.................................................................................................6 Testing Unsupported Features................................................................................ 6 Descriptions of the Latest Technical Preview Features.......................................................................................7 Upgrading to HDP 3.1.4...........................................................................................7 Behavioral Changes.................................................................................................. 7 Apache Patch Information.....................................................................................11 Accumulo...........................................................................................................................................................
    [Show full text]
  • SAS 9.4 Hadoop Configuration Guide for Base SAS And
    SAS® 9.4 Hadoop Configuration Guide for Base SAS® and SAS/ACCESS® Second Edition SAS® Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2015. SAS® 9.4 Hadoop Configuration Guide for Base SAS® and SAS/ACCESS®, Second Edition. Cary, NC: SAS Institute Inc. SAS® 9.4 Hadoop Configuration Guide for Base SAS® and SAS/ACCESS®, Second Edition Copyright © 2015, SAS Institute Inc., Cary, NC, USA All rights reserved. Produced in the United States of America. For a hard-copy book: No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, or otherwise, without the prior written permission of the publisher, SAS Institute Inc. For a web download or e-book: Your use of this publication shall be governed by the terms established by the vendor at the time you acquire this publication. The scanning, uploading, and distribution of this book via the Internet or any other means without the permission of the publisher is illegal and punishable by law. Please purchase only authorized electronic editions and do not participate in or encourage electronic piracy of copyrighted materials. Your support of others' rights is appreciated. U.S. Government License Rights; Restricted Rights: The Software and its documentation is commercial computer software developed at private expense and is provided with RESTRICTED RIGHTS to the United States Government. Use, duplication or disclosure of the Software by the United States Government is subject to the license terms of this Agreement pursuant to, as applicable, FAR 12.212, DFAR 227.7202-1(a), DFAR 227.7202-3(a) and DFAR 227.7202-4 and, to the extent required under U.S.
    [Show full text]
  • Hortonworks Data Platform Date of Publish: 2018-09-21
    Release Notes 3 Hortonworks Data Platform Date of Publish: 2018-09-21 http://docs.hortonworks.com Contents HDP 3.0.1 Release Notes..........................................................................................3 Component Versions.............................................................................................................................................3 New Features........................................................................................................................................................ 3 Deprecation Notices..............................................................................................................................................4 Terminology.............................................................................................................................................. 4 Removed Components and Product Capabilities.....................................................................................4 Unsupported Features........................................................................................................................................... 4 Technical Preview Features......................................................................................................................4 Upgrading to HDP 3.0.1...................................................................................................................................... 5 Before you begin.....................................................................................................................................
    [Show full text]
  • CLUSTER CONTINUOUS DELIVERY with OOZIE Clay Baenziger – Bloomberg Hadoop Infrastructure
    CLUSTER CONTINUOUS DELIVERY WITH OOZIE Clay Baenziger – Bloomberg Hadoop Infrastructure ApacheCon Big Data – 5/18/2017 ABOUT BLOOMBERG 2 BIG DATA AT BLOOMBERG Bloomberg quickly and accurately delivers business and financial information, news and insight around the world. A sense of scale: ● 550 exchange feeds and over 100 billion market data messages a day ● 400 million emails and 17 million IM’s daily across the Bloomberg Professional Service ● More than 2,400 journalists in over 120 countries ─ Produce more than 5,000 stories a day ─ Reaching over 360 million homes world wide 3 BLOOMBERG BIG DATA APACHE OPEN SOURCE Solr: 3 committers – commits in every Solr release since 4.6 Project JIRAs Project JIRAs Project JIRAs Phoenix 24 HBase 20 Spark 9 Zookeeper 8 HDFS 6 Bigtop 3 Oozie 4 Storm 2 Hive 2 Hadoop 2 YARN 2 Kafka 2 Flume 1 HAWQ 1 Total* 86 * Reporter or assignee from our Foundational Services group and affiliated projects 4 APACHE OOZIE What is Oozie: ● Oozie is a workflow scheduler system to manage Apache Hadoop jobs. ● Oozie workflow jobs are Directed Acyclical Graphs (DAGs) of actions. ● Oozie coordinator jobs are recurrent Oozie workflow jobs triggerd by time and data availability. ● Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs as well as system specific jobs out of the box. ● Oozie is a scalable, reliable and extensible system. Paraphrased from: Actions: http://oozie.apache.org/ ● Map/Reduce ● HDFS ● Spark ● Decision ● Hive ● Java ● Sub-Workflow ● Fork ● Pig ● Shell ● E-Mail ● Join
    [Show full text]
  • Hortonworks Data Platform Release Notes (October 30, 2017)
    Hortonworks Data Platform Release Notes (October 30, 2017) docs.cloudera.com Hortonworks Data Platform October 30, 2017 Hortonworks Data Platform: Release Notes Copyright © 2012-2017 Hortonworks, Inc. Some rights reserved. The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Software Foundation projects that focus on the storage and processing of Big Data, along with operations, security, and governance for the resulting system. This includes Apache Hadoop -- which includes MapReduce, Hadoop Distributed File System (HDFS), and Yet Another Resource Negotiator (YARN) -- along with Ambari, Falcon, Flume, HBase, Hive, Kafka, Knox, Oozie, Phoenix, Pig, Ranger, Slider, Spark, Sqoop, Storm, Tez, and ZooKeeper. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included. Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain, free and open source. Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page.
    [Show full text]
  • Big Data Security Analysis and Secure Hadoop Server
    Big Data Security Analysis And Secure Hadoop Server Kripa Shanker Master’s thesis January 2018 Master's Degree in Information Technology 2 ABSTRACT Tampereen Ammattikorkeakoulu Tampere University of Applied Sciences Master's Degree in Information Technology Kripa Shanker Big data security analysis and secure hadoop server Master's thesis 62 pages, appendices 4 pages January 2018 Hadoop is a so influential technology that’s let us to do incredible things but major thing to secure informative data and environment is a big challenge as there are many bad guys (crackers, hackers) are there to harm the society using this data. Hadoop is now used in retail, banking, and healthcare applications; it has attracted the attention of thieves as well. While storing sensitive huge data, security plays an important role to keep it safe. Security was not that much considered when Hadoop was initially designed. Security is an important topic in Hadoop cluster. Plenty of examples are available in open media on data breaches and most recently was RANSOMEWARE which get access in server level which is more dangerous for an organizations. This is best time to only focus on security at any cost and time needed to secure data and platform. Hadoop is designed to run code on a distributed cluster of machines so without proper authentication anyone could submit code and it would be executed. Different projects have started to improve the security of Hadoop. In this thesis, the security of the system in Hadoop version 1, Hadoop version 2 and Hadoop version 3 is evaluated and different security enhancements are proposed, considering security improvements made by the two mentioned projects, Project Apache Knox Gateway, Project Apache Ranger and Apache Sentry, in terms of encryption, authentication, and authorization.
    [Show full text]
  • Pentaho EMR46 SHIM 7.1.0.0 Open Source Software Packages
    Pentaho EMR46 SHIM 7.1.0.0 Open Source Software Packages Contact Information: Project Manager Pentaho EMR46 SHIM Hitachi Vantara Corporation 2535 Augustine Drive Santa Clara, California 95054 Name of Product/Product Version License Component An open source Java toolkit for 0.9.0 Apache License Version 2.0 Amazon S3 AOP Alliance (Java/J2EE AOP 1.0 Public Domain standard) Apache Commons BeanUtils 1.9.3 Apache License Version 2.0 Apache Commons CLI 1.2 Apache License Version 2.0 Apache Commons Daemon 1.0.13 Apache License Version 2.0 Apache Commons Exec 1.2 Apache License Version 2.0 Apache Commons Lang 2.6 Apache License Version 2.0 Apache Directory API ASN.1 API 1.0.0-M20 Apache License Version 2.0 Apache Directory LDAP API Utilities 1.0.0-M20 Apache License Version 2.0 Apache Hadoop Amazon Web 2.7.2 Apache License Version 2.0 Services support Apache Hadoop Annotations 2.7.2 Apache License Version 2.0 Name of Product/Product Version License Component Apache Hadoop Auth 2.7.2 Apache License Version 2.0 Apache Hadoop Common - 2.7.2 Apache License Version 2.0 org.apache.hadoop:hadoop-common Apache Hadoop HDFS 2.7.2 Apache License Version 2.0 Apache HBase - Client 1.2.0 Apache License Version 2.0 Apache HBase - Common 1.2.0 Apache License Version 2.0 Apache HBase - Hadoop 1.2.0 Apache License Version 2.0 Compatibility Apache HBase - Protocol 1.2.0 Apache License Version 2.0 Apache HBase - Server 1.2.0 Apache License Version 2.0 Apache HBase - Thrift - 1.2.0 Apache License Version 2.0 org.apache.hbase:hbase-thrift Apache HttpComponents Core
    [Show full text]
  • Technology Overview
    Big Data Technology Overview Term Description See Also Big Data - the 5 Vs Everyone Must Volume, velocity and variety. And some expand the definition further to include veracity 3 Vs Know and value as well. 5 Vs of Big Data From Wikipedia, “Agile software development is a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. Agile The Agile Manifesto It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change. It is a conceptual framework that promotes foreseen tight iterations throughout the development cycle.” A data serialization system. From Wikepedia, Avro Apache Avro “It is a remote procedure call and serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.” BigInsights Enterprise Edition provides a spreadsheet-like data analysis tool to help Big Insights IBM Infosphere Biginsights organizations store, manage, and analyze big data. A scalable multi-master database with no single points of failure. Cassandra Apache Cassandra It provides scalability and high availability without compromising performance. Cloudera Inc. is an American-based software company that provides Apache Hadoop- Cloudera Cloudera based software, support and services, and training to business customers. Wikipedia - Data Science Data science The study of the generalizable extraction of knowledge from data IBM - Data Scientist Coursera Big Data Technology Overview Term Description See Also Distributed system developed at Google for interactively querying large datasets. Dremel Dremel It empowers business analysts and makes it easy for business users to access the data Google Research rather than having to rely on data engineers.
    [Show full text]
  • HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack
    HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack Geoffrey C. Fox, Judy Qiu, Supun Kamburugamuve Shantenu Jha, Andre Luckow School of Informatics and Computing RADICAL Indiana University Rutgers University Bloomington, IN 47408, USA Piscataway, NJ 08854, USA fgcf, xqiu, [email protected] [email protected], [email protected] Abstract—We review the High Performance Computing En- systems as they illustrate key capabilities and often motivate hanced Apache Big Data Stack HPC-ABDS and summarize open source equivalents. the capabilities in 21 identified architecture layers. These The software is broken up into layers so that one can dis- cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, cuss software systems in smaller groups. The layers where DevOps, Interoperability, File Systems, Cluster & Resource there is especial opportunity to integrate HPC are colored management, Data Transport, File management, NoSQL, SQL green in figure. We note that data systems that we construct (NewSQL), Extraction Tools, Object-relational mapping, In- from this software can run interoperably on virtualized or memory caching and databases, Inter-process Communication, non-virtualized environments aimed at key scientific data Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Li- analysis problems. Most of ABDS emphasizes scalability braries and Applications, Workflow and Orchestration. We but not performance and one of our goals is to produce summarize status of these layers focusing on issues of impor- high performance environments. Here there is clear need tance for data analytics. We highlight areas where HPC and for better node performance and support of accelerators like ABDS have good opportunities for integration.
    [Show full text]