DOCSLIB.ORG

IBM Open Platform with Apache Hadoop and Biginsights 4.2 Technical Preview

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
IBM Open Platform with Apache Hadoop and Biginsights 4.2 Technical Preview BigInsights IBM IBM Open Platform with Apache Hadoop and BigInsights 4.2 Technical Preview Version 4 Release 2 BigInsights IBM IBM Open Platform with Apache Hadoop and BigInsights 4.2 Technical Preview Version 4 Release 2 Edition notice - early release documentation This document contains proprietary information. All information contained herein shall be kept in confidence. None of this information shall be divulged to persons other than (a) IBM employees authorized by the nature of their duties to receive such information, or (b) individuals with a need to know in organizations authorized by IBM to receive this document in accordance with the terms (including confidentiality) of an agreement under which it is provided. This information might include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements or changes in the product or the programs described in this publication at any time without notice. © Copyright IBM Corporation 2013, 2016. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Chapter 1. Introduction to 4.2 ..... 1 Preparing to install the BigInsights value-add Introduction .............. 1 services................ 63 Obtaining the BigInsights value-add services ... 68 Chapter 2. What's New in 4.2...... 3 Installing the BigInsights value-add packages ... 69 Installing BigInsights Home ........ 72 What's new for Version 4.2 ......... 3 Installing the BigInsights - Big SQL service ... 74 Open source technologies .......... 9 Installing the Text Analytics service ..... 84 Enabling Knox for value-add services .... 88 Chapter 3. Installing IBM Open Platform Removing BigInsights value-add services ... 90 with Apache Hadoop ......... 11 Get ready to install ............ 11 Chapter 5. Some new or enhanced Preparing your environment......... 16 features for 4.2 ........... 95 Configuring LDAP server authentication on Red Impersonation in Big SQL ......... 95 Hat Enterprise Linux 6.7 and 7.2 ...... 25 ANALYZE command ........... 99 Creating a mirror repository for the IBM Open Auto-analyze ............ 108 Platform with Apache Hadoop software ..... 27 HCAT_SYNC_OBJECTS stored procedure .... 112 Running the installation package ....... 28 Big SQL integration with Apache Spark .... 119 Upgrading the Java (JDK) version ....... 36 EXECSPARK table function........ 122 Installing and configuring Ranger in the Ambari web interface .............. 37 Configuring MySQL for Ranger ...... 41 Chapter 6. Known problems ..... 125 Installing Ranger plugins ......... 42 Set up user sync from LDAP/AD/Unix to Index ............... 127 Ranger ............... 45 Installing Ranger authentication ...... 48 Ranger KMS set up and usage ........ 53 Cleaning up nodes before reinstalling software .. 55 HostCleanup.ini file .......... 57 HostCleanup_Custom_Actions.ini file..... 59 Chapter 4. Installing the IBM BigInsights value-added services on IBM Open Platform with Apache Hadoop .............. 61 Users, groups, and ports for BigInsights value-add services................ 61 © Copyright IBM Corp. 2013, 2016 iii iv BigInsights: IBM Open Platform with Apache Hadoop and BigInsights 4.2 Technical Preview Chapter 1. Introduction to 4.2 Introduction Welcome to the Technical Preview of the IBM® Open Platform with Apache Hadoop and IBM BigInsights 4.2. This README contains information to ensure the successful installation and operation of the IOP and the BigInsights Value add services. The information contained in this Technical Preview documentation might not describe completely the functionality that is available in the 4.2 release. The information represents a snapshot of the full 4.2 release. It describes how to install the product and some of the highlights of the 4.2 release. Because the product documentation is still being refined, you might find links that are not valid. Contact your IBM representative for help in those cases. Description IBM Open Platform with Apache Hadoop and IBM BigInsights Version 4.2 deliver enterprise Hadoop capabilities with easy-to-use analytic tools and visualization for business analysts and data scientists, rich developer tools, powerful analytic functions, complete administration and management capabilities, and the latest versions of Apache Hadoop and associated projects. This 4.2 release provides full function SQL query capability, with security and performance benefits, to data that is stored in Hadoop. Obtaining the Technical Preview for 4.2 TECHNICAL PREVIEW DOWNLOAD ONLY Accept the IBM BigInsights Early Release license agreement: http://www14.software.ibm.com/cgi-bin/weblap/lap.pl? popup=Y&li_formnum=L-MLOY-9YB5S9&accepted_url= http://ibm-open-platform.ibm.com/repos/beta/4.2.0.0/&title= IBM+BigInsights+Beta+License&declined_url= http://www-01.ibm.com/software/data/infosphere/hadoop/trials.html Then select the appropriate repository file for your environment: RHEL6 https://ibm-open-platform.ibm.com/repos/beta/4.2.0.0/rhel6/ Use the following TAR files: BIPremium-4.2.0.0-beta1.el6.x86_64.tar.gz ambari-2.2.0.0-beta1.el6.x86_64.tar.gz iop-4.2.0.0-beta1-el6.x86_64.tar.gz iop-utils-4.2.0.0-beta1.el6.x86_64.tar.gz RHEL7 https://ibm-open-platform.ibm.com/repos/beta/4.2.0.0/rhel7/ Use the following TAR files: BIPremium-4.2.0.0-beta1.el7.x86_64.tar.gz ambari-2.2.0.0-beta1.el7.x86_64.tar.gz iop-4.2.0.0-beta1-el7.x86_64.tar.gz iop-utils-4.2.0.0-beta1.el7.x86_64.tar.gz © Copyright IBM Corp. 2013, 2016 1 2 BigInsights: IBM Open Platform with Apache Hadoop and BigInsights 4.2 Technical Preview Chapter 2. What's New in 4.2 What's new for Version 4.2 New features for Version 4.2 Note: There is no UPGRADE path to or from the IBM Open Platform with Apache Hadoop and BigInsights Version 4.2 technical preview. Major milestones v ODPi compliant. v Express upgrade is available. You can quickly upgrade the entire cluster while it is shut down. v Apache Spark ecosystem. v Apache Hadoop ecosystem. Operating Systems Refer to the System Requirements for the most up-to-date information on operating system support: v RHEL 6.7+ v RHEL 7.2 Open Source The following open source technologies are now supported: v Ranger 0.5.2 v Phoenix 4.6.1 v Titan 1.0.0 (Titan server and OLAP are not integrated in IBM Open Platform with Apache Hadoop 4.2) The following open source technologies are updated: v Ambari 2.2.0 v Flume 1.6.0 v Hadoop 2.7.2 v HBase 1.2.0 v Kafka 0.9.0.1 v Knox 0.7.0 v Slider 0.90.2 v Solr 5.5 v Spark 1.6.1 BigInsights Big SQL updates 1. BigInsights - Big SQL is now packaged as part of the IBM BigInsights Premium package. 2. Big SQL and Spark Integration is now available as a technical preview. You can invoke Spark jobs from Big SQL by using a table UDF abstraction. The following example calls the SYSHADOOP.EXECSPARK user-defined function to kick off a Spark job that reads a JSON file stored on HDFS: © Copyright IBM Corp. 2013, 2016 3 SELECT * FROM TABLE(SYSHADOOP.EXECSPARK( language => ’scala’, class => ’com.ibm.biginsights.bigsql.examples.ReadJsonFile’, uri => ’hdfs://host.port.com:8020/user/bigsql/demo.json’, card => 100000)) AS doc, products WHERE doc.country IS NOT NULL AND doc.language = products.language; 3. Support for update and delete on Big SQL HBase tables. 4. Impersonation feature, which allows a service user to securely access data in Hadoop on behalf of another user. 5. An auto-analyze feature that runs the ANALYZE command automatically under certain conditions. In addition, ANALYZE command now has a FOR ALL COLUMNS clause. 6. ANALYZE command improvements, some of which are listed in the following table: Table 1. Big SQL ANALYZE improvements Enhancement Description Analyze v2 There are major performance and memory improvements due to the removal of all dependencies from Hive and Map/Reduce. The ANALYZE command with no Map/Reduce dependency is called Analyze v2, which is the default for BigInsights 4.2. You can use Analyze v1 by setting the biginsights.stats.use.v2 property to false. However, Analyze v1 is deprecated and will be removed in future releases of Big SQL. Cumulative statistics When you run the ANALYZE command against a table on a set of columns, and then later run ANALYZE on a second set of columns, the statistics that are gathered from the first ANALYZE command are merged with the statistics that are gathered from the second ANALYZE command. SYSTEM sampling Instead of scanning an entire table, you can specify a percentage of the splits that ANALYZE can run against. Big SQL extrapolates the statistics for the whole table based on the sample of the table that it gathered statistics on. The SYSTEM option can reduce the time to run the ANALYZE command with minor impact on query performance. FOR ALL COLUMNS By using this option, you can collect statistics on all of the columns of the table. 7. Some maintenance improvements: 4 BigInsights: IBM Open Platform with Apache Hadoop and BigInsights 4.2 Technical Preview Table 2. Big SQL maintenance enhancements Enhancement Description Automatic analyze By default, ANALYZE is run automatically after a successful LOAD or HCAT_SYNC_OBJECTS call, which automatically gathers statistics on the table to improve query performance. Also, Big SQL determines whether a table has changed significantly and automatically schedules an ANALYZE, if necessary. Automatic HCAT_SYNC_OBJECTS By default, Big SQL automatically synchronizes the Big SQL and Hive catalogs so that when data is added to Hive it can be assessed automatically by Big SQL. In addition, table metadata is preserved through the support of ALTER column. You can customize Big SQL metadata synchronization with Hive with configuration options. 8. Some performance improvements: Table 3. Big SQL performance improvements Enhancement Description Concurrency improvements Improved performance for high concurrency Hadoop, HBase, and Hybrid workloads which allows for greater throughput and improved CPU utilization by default on Big SQL clusters.
Load more

Recommended publications
  • Extended Version
    Sina Sheikholeslami C u rriculum V it a e ( Last U pdated N ovember 2 0 18) Website: http://sinash.ir Present Address : https://www.kth.se/profile/sinash EIT Digital Stockholm CLC , https://linkedin.com/in/sinasheikholeslami Isafjordsgatan 26, Email: si [email protected] 164 40 Kista (Stockholm), [email protected] Sweden [email protected] Educational Background: • M.Sc. Student of Data Science, Eindhoven University of Technology & KTH Royal Institute of Technology, Under EIT-Digital Master School. 2017-Present. • B.Sc. in Computer Software Engineering, Department of Computer Engineering and Information Technology, Amirkabir University of Technology (Tehran Polytechnic). 2011-2016. • Mirza Koochak Khan Pre-College in Mathematics and Physics, Rasht, National Organization for Development of Exceptional Talents (NODET). Overall GPA: 19.61/20. 2010-2011. • Mirza Koochak Khan Highschool in Mathematics and Physics, Rasht, National Organization for Development of Exceptional Talents (NODET). Overall GPA: 19.17/20, Final Year's GPA: 19.66/20. 2007-2010. Research Fields of Interest: • Distributed Deep Learning, Hyperparameter Optimization, AutoML, Data Intensive Computing Bachelor's Thesis: • “SDMiner: A Tool for Mining Data Streams on Top of Apache Spark”, Under supervision of Dr. Amir H. Payberah (Defended on June 29th 2016). Computer Skills: • Programming Languages & Markups: o F luent in Java, Python, Scala, JavaS cript, C/C++, A ndroid Pr ogram Develop ment o Familia r wit h R, SAS, SQL , Nod e.js, An gula rJS, HTM L, JSP •
    [Show full text]
  • Poweredge R640 Apache Hadoop
    A Principled Technologies report: Hands-on testing. Real-world results. The science behind the report: Run compute-intensive Apache Hadoop big data workloads faster with Dell EMC PowerEdge R640 servers This document describes what we tested, how we tested, and what we found. To learn how these facts translate into real-world benefits, read the report Run compute-intensive Apache Hadoop big data workloads faster with Dell EMC PowerEdge R640 servers. We concluded our hands-on testing on October 27, 2019. During testing, we determined the appropriate hardware and software configurations and applied updates as they became available. The results in this report reflect configurations that we finalized on October 15, 2019 or earlier. Unavoidably, these configurations may not represent the latest versions available when this report appears. Our results The table below presents the throughput each solution delivered when running the HiBench workloads. Dell EMC™ PowerEdge™ R640 Dell EMC PowerEdge R630 Percentage more throughput solution solution Latent Dirichlet Allocation 4.13 1.94 112% (MB/sec) Random Forest (MB/sec) 100.66 94.43 6% WordCount (GB/sec) 5.10 3.45 47% The table below presents the minutes each solution needed to complete the HiBench workloads. Dell EMC PowerEdge R640 Dell EMC PowerEdge R630 Percentage less time solution solution Latent Dirichlet Allocation 17.11 36.25 52% Random Forest 5.55 5.92 6% WordCount 4.95 7.32 32% Run compute-intensive Apache Hadoop big data workloads faster with Dell EMC PowerEdge R640 servers November 2019 System configuration information The table below presents detailed information on the systems we tested.
    [Show full text]
  • TR-4744: Secure Hadoop Using Apache Ranger with Netapp In
    Technical Report Secure Hadoop using Apache Ranger with NetApp In-Place Analytics Module Deployment Guide Karthikeyan Nagalingam, NetApp February 2019 | TR-4744 Abstract This document introduces the NetApp® In-Place Analytics Module for Apache Hadoop and Spark with Ranger. The topics covered in this report include the Ranger configuration, underlying architecture, integration with Hadoop, and benefits of Ranger with NetApp In-Place Analytics Module using Hadoop with NetApp ONTAP® data management software. TABLE OF CONTENTS 1 Introduction ........................................................................................................................................... 4 1.1 Overview .........................................................................................................................................................4 1.2 Deployment Options .......................................................................................................................................5 1.3 NetApp In-Place Analytics Module 3.0.1 Features ..........................................................................................5 2 Ranger ................................................................................................................................................... 6 2.1 Components Validated with Ranger ................................................................................................................6 3 NetApp In-Place Analytics Module Design with Ranger..................................................................
    [Show full text]
  • Hortonworks Data Platform
    Hortonworks Data Platform Apache Ambari Installation for IBM Power Systems (November 15, 2018) docs.cloudera.com Hortonworks Data Platform November 15, 2018 Hortonworks Data Platform: Apache Ambari Installation for IBM Power Systems Copyright © 2012-2018 Hortonworks, Inc. Some rights reserved. The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included. Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain free and open source. Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to Contact Us directly to discuss your specific needs. Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License.
    [Show full text]
  • Professional Summary Technical Skills
    PROF ESSIONAL SUMMARY • Over 7.5 years of professional IT experience in analysis, design, development and implementation, support of Enterprise Application Integration. • Experience in installation, configuration, deployment and troubleshooting of TIBCO Active Enterprise Suite of applications TIBCO BusinessWorks, TIBCO Designer, TIBCO Rendezvous, TIBCO EMS, TIBCO Administrator and TIBCO Spotfire. • Good hands on experience in Web Services using SOAP, WSDL and Schemas XSD with strong skills in understanding and implementing Service Oriented Architecture (SOA). • Knowledge in using TIBCO Administrator for User Management, Resource Management and Application Management. • Well conversant in using TIBCO messaging standards including EMS and Rendezvous. • Deployed BusinessWorks interfaces for Fault Tolerance and Load balancing modes. • Knowledge and Experience in handling and supporting (L3) ESB architecture. • Having experience in Application development, deployment, debugging and troubleshooting. • Extensively handled overall responsibility Project deployment to SIT, UAT and PROD Environment using UNIX and TIBCO Admin. • Experience of analysing and defect fixing in SIT and UAT Environment. • Well conversant in Handling Technical Interview Sessions. • Experience of Project planning, Tracking, Risk analysis, counter action deployment. • Good Interpersonal, Analytic, Leadership, Verbal and Written skills. • Highly intuitive, self-motivated and energetic team player and adaptable to challenging environments. TECHNICAL SKILLS • Tibco Skills : TIBCO Business works, TIBCO iProcess, TIBCO AMX BPM suites, TIBCO Business Events, TIBCO Spotfire, TIBCO Active Space, TIBCO Administrator, TIBCO EMS, TIBCO RV, TIBCO Adapters, TIBCO Hawk, ESB and SOA frameworks. • Language: Java, J2EE (JDBC, RMI, Servlets, JSP, EJB, JMS), C, C++, C#, SQL, PL/SQL, XML, shell script. • DataBase: Oracle 8, 8i, 9i, 10g,SQL, PostgreSQL • Open Source: Apache Kafka, Apache Nifi, Apache Ambari, HDP Administration.
    [Show full text]
  • HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack
    HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack Geoffrey C. Fox, Judy Qiu, Supun Kamburugamuve Shantenu Jha, Andre Luckow School of Informatics and Computing RADICAL Indiana University Rutgers University Bloomington, IN 47408, USA Piscataway, NJ 08854, USA fgcf, xqiu, [email protected] [email protected], [email protected] Abstract—We review the High Performance Computing En- systems as they illustrate key capabilities and often motivate hanced Apache Big Data Stack HPC-ABDS and summarize open source equivalents. the capabilities in 21 identified architecture layers. These The software is broken up into layers so that one can dis- cover Message and Data Protocols, Distributed Coordination, Security & Privacy, Monitoring, Infrastructure Management, cuss software systems in smaller groups. The layers where DevOps, Interoperability, File Systems, Cluster & Resource there is especial opportunity to integrate HPC are colored management, Data Transport, File management, NoSQL, SQL green in figure. We note that data systems that we construct (NewSQL), Extraction Tools, Object-relational mapping, In- from this software can run interoperably on virtualized or memory caching and databases, Inter-process Communication, non-virtualized environments aimed at key scientific data Batch Programming model and Runtime, Stream Processing, High-level Programming, Application Hosting and PaaS, Li- analysis problems. Most of ABDS emphasizes scalability braries and Applications, Workflow and Orchestration. We but not performance and one of our goals is to produce summarize status of these layers focusing on issues of impor- high performance environments. Here there is clear need tance for data analytics. We highlight areas where HPC and for better node performance and support of accelerators like ABDS have good opportunities for integration.
    [Show full text]
  • Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs"
    Appendix to the paper "Code smell prediction employing machine learning meets emerging Java language constructs" Hanna Grodzicka, Michał Kawa, Zofia Łakomiak, Arkadiusz Ziobrowski, Lech Madeyski (B) The Appendix includes two tables containing the dataset used in the paper "Code smell prediction employing machine learning meets emerging Java lan- guage constructs". The first table contains information about 792 projects selected for R package reproducer [Madeyski and Kitchenham(2019)]. Projects were the base dataset for cre- ating the dataset used in the study (Table I). The second table contains information about 281 projects filtered by Java version from build tool Maven (Table II) which were directly used in the paper. TABLE I: Base projects used to create the new dataset # Orgasation Project name GitHub link Commit hash Build tool Java version 1 adobe aem-core-wcm- www.github.com/adobe/ 1d1f1d70844c9e07cd694f028e87f85d926aba94 other or lack of unknown components aem-core-wcm-components 2 adobe S3Mock www.github.com/adobe/ 5aa299c2b6d0f0fd00f8d03fda560502270afb82 MAVEN 8 S3Mock 3 alexa alexa-skills- www.github.com/alexa/ bf1e9ccc50d1f3f8408f887f70197ee288fd4bd9 MAVEN 8 kit-sdk-for- alexa-skills-kit-sdk- java for-java 4 alibaba ARouter www.github.com/alibaba/ 93b328569bbdbf75e4aa87f0ecf48c69600591b2 GRADLE unknown ARouter 5 alibaba atlas www.github.com/alibaba/ e8c7b3f1ff14b2a1df64321c6992b796cae7d732 GRADLE unknown atlas 6 alibaba canal www.github.com/alibaba/ 08167c95c767fd3c9879584c0230820a8476a7a7 MAVEN 7 canal 7 alibaba cobar www.github.com/alibaba/
    [Show full text]
  • Kylo Data Lakes Configuration Deployed in Public Cloud Environments in Single Node Mode
    Master of Science in Computer Science September 2019 Kylo Data Lakes Configuration deployed in Public Cloud environments in Single Node Mode Rong Peng Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full time studies. The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree. Contact Information: Author(s): Rong Peng E-mail: [email protected] University advisor: Dr. Emiliano Casalicchio Department of Computer Science Industrial advisors: Kim Hindart Daniel Gustafsson City Network Hosting AB Faculty of Computing Internet : www.bth.se Blekinge Institute of Technology Phone : +46 455 38 50 00 SE-371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 2 ABSTRACT Data Lake is a platform for centralized storage of massive, multiple sources and multiple types of data, and can quickly process and analyze data. It is essentially an advanced enterprise data architecture. The high demand for big data storage and analytics leads data lake emerging nowadays. Organizations hope to generate business profit from their data by building a data lake. However, it is worth noting that the promising concept of data lake is still evolving today.
    [Show full text]
  • Apache Ambari Operations (May 17, 2018)
    Hortonworks Data Platform Apache Ambari Operations (May 17, 2018) docs.cloudera.com Hortonworks Data Platform May 17, 2018 Hortonworks Data Platform: Apache Ambari Operations Copyright © 2012-2018 Hortonworks, Inc. Some rights reserved. The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included. Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain free and open source. Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to Contact Us directly to discuss your specific needs. Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License. http://creativecommons.org/licenses/by-sa/4.0/legalcode ii Hortonworks Data Platform May 17, 2018 Table of Contents 1.
    [Show full text]
  • Apache Ambari Troubleshooting (May 19, 2017)
    Hortonworks Data Platform Apache Ambari Troubleshooting (May 19, 2017) docs.cloudera.com hdp-ambari-troubleshooting May 19, 2017 Hortonworks Data Platform: Apache Ambari Troubleshooting Copyright © 2012-2016 Hortonworks, Inc. All rights reserved. The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included. Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain free and open source. Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. Feel free to Contact Us directly to discuss your specific needs. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
    [Show full text]
  • Koverse Release 2.7
    Koverse Release 2.7 Sep 12, 2018 Contents 1 Quick Start Guide 1 2 Usage Guide 17 3 Administration Guide 75 4 Developer Documentation 113 5 Access Control 169 i ii CHAPTER 1 Quick Start Guide This Quick Start guide for Koverse is intended for users who want to get up and running quickly with Koverse. It steps through the installation of Koverse, ingesting data and executing queries. Check out the Koverse User Guide for complete documentation of all features and installation instructions. 1.1 Recommendations The recommended Operating System is RHEL 6.x or Centos 6.x. Recommended Hadoop Release is Cloudera Manager 5.5 with Accumulo 1.7 Parcel and Service installed. See http: //www.cloudera.com/documentation/other/accumulo/latest/PDF/Apache-Accumulo-Installation-Guide.pdf for more details. Recommended Koverse release can be found at http://repo.koverse.com/latest/csd 1.1.1 Infrastructure and Software Koverse and the open source software it leverages must be run on a system with no less than 10 GB of memory. For workloads beyond simple examples and testing we recommend a properly provisioned Hadoop cluster with five or more nodes. Using the Cloudera QuickStart VM is not recommended. See http://www.koverse.com/question/ using-the-cloudera-quick-start-vim-and-the-koverse-parcel for more information. 1 Koverse, Release 2.7 1.2 Installation 1.2.1 Amazon Web Services Installation Using Koverse with AWS Marketplace The paid AMI available in the AWS marketplace is an easy way to get a Koverse instance up and running if you do not need to install on existing infrastructure.
    [Show full text]
  • Testsmelldescriber Enabling Developers’ Awareness on Test Quality with Test Smell Summaries
    Bachelor Thesis January 31, 2018 TestSmellDescriber Enabling Developers’ Awareness on Test Quality with Test Smell Summaries Ivan Taraca of Pfullendorf, Germany (13-751-896) supervised by Prof. Dr. Harald C. Gall Dr. Sebastiano Panichella software evolution & architecture lab Bachelor Thesis TestSmellDescriber Enabling Developers’ Awareness on Test Quality with Test Smell Summaries Ivan Taraca software evolution & architecture lab Bachelor Thesis Author: Ivan Taraca, [email protected] URL: http://bit.ly/2DUiZrC Project period: 20.10.2018 - 31.01.2018 Software Evolution & Architecture Lab Department of Informatics, University of Zurich Acknowledgements First of all, I like to thank Dr. Harald Gall for giving me the opportunity to write this thesis at the Software Evolution & Architecture Lab. Special thanks goes out to Dr. Sebastiano Panichella for his instructions, guidance and help during the making of this thesis, without whom this would not have been possible. I would also like to express my gratitude to Dr. Fabio Polomba, Dr. Yann-Gaël Guéhéneuc and Dr. Nikolaos Tsantalis for providing me access to their research and always being available for questions. Last, but not least, do I want to thank my parents, sisters and nephews for the support and love they’ve given all those years. Abstract With the importance of software in today’s society, malfunctioning software can not only lead to disrupting our day-to-day lives, but also large monetary damages. A lot of time and effort goes into the development of test suites to ensure the quality and accuracy of software. But how do we elevate the quality of test code? This thesis presents TestSmellDescriber, a tool with the ability to generate descriptions detailing potential problems in test cases, which are collected by conducting a Test Smell analysis.
    [Show full text]