DOCSLIB.ORG

Apache Spark Implementation on IBM Z/OS

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Apache Spark Implementation on IBM Z/OS Front cover Apache Spark Implementation on IBM z/OS Lydia Parziale Joe Bostian Ravi Kumar Ulrich Seelbach Zhong Yu Ye Redbooks International Technical Support Organization Apache Spark Implementation on IBM z/OS August 2016 SG24-8325-00 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (August 2016) This edition applies to Version 2, Release 2 of IBM z/OS (product number 5650 ZOS), Apache Spark 1.5.2 © Copyright International Business Machines Corporation 2016. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii IBM Redbooks promotions . ix Preface . xi Authors. xi Now you can become a published author, too . xii Comments welcome. xiii Stay connected to IBM Redbooks . xiii Chapter 1. Architectural overview . 1 1.1 Open source analytics on z/OS. 2 1.1.1 Benefits of Spark on z/OS. 2 1.1.2 Drawbacks of implementing off-platform analytics . 3 1.1.3 A new chapter in analytics . 4 1.2 Planning your environment . 4 1.3 Reference architecture . 6 1.3.1 Spark server architecture . 6 1.3.2 Spark environment architecture . 7 1.3.3 Implementation with Jupyter Notebooks . 9 1.3.4 Scala IDE . 12 1.4 Security . 13 Chapter 2. Components and extensions. 15 2.1 Apache Spark component overview . 16 2.1.1 Resilient Distributed Datasets and caching. 16 2.1.2 Components of a Spark cluster on z/OS. 17 2.1.3 Monitoring . 18 2.1.4 Spark and Hadoop . 21 2.2 Mainframe Data Services for IBM z/OS Platform for Apache Spark . 21 2.2.1 Virtual tables . 23 2.2.2 Virtual views . 23 2.2.3 SQL queries . 23 2.2.4 MDSS JDBC driver . 23 2.2.5 IBM z/OS Platform for Apache Spark Interface for CICS/TS . 24 2.3 Spark SQL. 25 2.3.1 Reading from z/OS data source into a DataFrame. 26 2.3.2 Writing DataFrame to a DB2 for z/OS table using saveTable method . 26 2.4 Streaming . 27 2.5 GraphX . 28 2.5.1 System G . 28 2.6 MLlib . 28 2.7 Spark R . 29 © Copyright IBM Corp. 2016. All rights reserved. iii Chapter 3. Installation and configuration . 31 3.1 Installing IBM z/OS Platform for Apache Spark . 32 3.2 The Mainframe Data Service for Apache Spark . 32 3.2.1 Installing the MDSS started task. 32 3.2.2 Configuring access to DB2 . 37 3.2.3 Configuring access to IMS databases. 38 3.2.4 The ISPF Panels. 39 3.2.5 Installing and configuring Bash . 45 3.2.6 Check for /usr/bin/env . 46 3.3 Installing workstation components . 46 3.3.1 Installing Data Service Studio . 46 3.3.2 Installing the JDBC driver on the workstation . 48 3.4 Configuring Apache Spark for z/OS . 49 3.4.1 Create log and worker directories . 49 3.4.2 Apache Spark directory structure . 50 3.4.3 Create directories and local configuration. 50 3.4.4 Installing the Data Server JDBC driver . 53 3.4.5 Modifying the log4j configuration . 54 3.4.6 Adding the Spark binaries to your PATH . 55 3.5 Verifying the installation . 55 3.6 Starting the Spark daemons . 57 Chapter 4. Spark application development on z/OS . 61 4.1 Setting up the development environment . 62 4.1.1 Installing Scala IDE. 62 4.1.2 Installing Data Server Studio plugins into Scala IDE . 62 4.1.3 Installing and using sbt . 63 4.2 Accessing VSAM data as an RDD . 65 4.2.1 Defining the data mapping . 65 4.2.2 Building and running the application . 70 4.3 Accessing sequential files and PDS members . 71 4.4 Accessing IBM DB2 data as a DataFrame . 71 4.5 Joining DB2 data with VSAM . 72 4.6 IBM IMS data to DataFrames . ..
Load more

Recommended publications
  • Analysis of Big Data Storage Tools for Data Lakes Based on Apache Hadoop Platform
    (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 8, 2021 Analysis of Big Data Storage Tools for Data Lakes based on Apache Hadoop Platform Vladimir Belov, Evgeny Nikulchev MIREA—Russian Technological University, Moscow, Russia Abstract—When developing large data processing systems, determined the emergence and use of various data storage the question of data storage arises. One of the modern tools for formats in HDFS. solving this problem is the so-called data lakes. Many implementations of data lakes use Apache Hadoop as a basic Among the most widely known formats used in the Hadoop platform. Hadoop does not have a default data storage format, system are JSON [12], CSV [13], SequenceFile [14], Apache which leads to the task of choosing a data format when designing Parquet [15], ORC [16], Apache Avro [17], PBF [18]. a data processing system. To solve this problem, it is necessary to However, this list is not exhaustive. Recently, new formats of proceed from the results of the assessment according to several data storage are gaining popularity, such as Apache Hudi [19], criteria. In turn, experimental evaluation does not always give a Apache Iceberg [20], Delta Lake [21]. complete understanding of the possibilities for working with a particular data storage format. In this case, it is necessary to Each of these file formats has own features in file structure. study the features of the format, its internal structure, In addition, differences are observed at the level of practical recommendations for use, etc. The article describes the features application. Thus, row-oriented formats ensure high writing of both widely used data storage formats and the currently speed, but column-oriented formats are better for data reading.
    [Show full text]
  • Z/OS ISPF Services Guide COMMAND NAME
    z/OS 2.4 ISPF Services Guide IBM SC19-3626-40 Note Before using this information and the product it supports, read the information in “Notices” on page 399. This edition applies to Version 2 Release 4 of z/OS (5650-ZOS) and to all subsequent releases and modifications until otherwise indicated in new editions. Last updated: 2021-06-22 © Copyright International Business Machines Corporation 1980, 2021. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Figures................................................................................................................ xv Tables................................................................................................................xvii Preface...............................................................................................................xix Who should use this document?............................................................................................................... xix What is in this document?......................................................................................................................... xix How to read the syntax diagrams..............................................................................................................xix z/OS information...............................................................................................xxiii How to send your comments to IBM...................................................................
    [Show full text]
  • OS/390 Introduction to ISPF
    z/OS Basic Skills Information Center: ISPF Course Module Module 1: Main Features of ISPF © Copyright IBM Corp., 2005. All rights reserved. z/OS Basic Skills Information Center: ISPF Course Module Introduction This module, Main Features of ISPF, introduces you to the z/OS Interactive System Productivity Facility, or ISPF, with special emphasis on the Program Development Facility, or PDF. Time to complete: 10 – 15 minutes © Copyright IBM Corp., 2005. All rights reserved. Page 2 of 15 z/OS Basic Skills Information Center: ISPF Course Module Main Features of ISPF - Objectives Upon completion of this module, you should be able to: • Describe the purpose of ISPF and its relationship to TSO • List the four major components of ISPF • Explain the function of each of the four components © Copyright IBM Corp., 2005. All rights reserved. Page 3 of 15 z/OS Basic Skills Information Center: ISPF Course Module Main Features of ISPF – Purpose of ISPF The Interactive System Productivity Facility, or ISPF, is a development tool set for the z/OS operating system. It has been used since 1975 to increase the productivity of the development of mainframe applications, because it provides an extensive set of programmer oriented facilities. © Copyright IBM Corp., 2005. All rights reserved. Page 4 of 15 z/OS Basic Skills Information Center: ISPF Course Module Main Features of ISPF – The Time Sharing Option/Extended (TSO/E) The Time Sharing Option/Extended, or TSO/E, is a base element of IBM's mainframe z/OS operating system. TSO/E allows you to communicate interactively with the MVS operating system by typing commands (one line at a time) on a computer terminal.
    [Show full text]
  • Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions
    00 Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions MUTAZ BARIKA, University of Tasmania SAURABH GARG, University of Tasmania ALBERT Y. ZOMAYA, University of Sydney LIZHE WANG, China University of Geoscience (Wuhan) AAD VAN MOORSEL, Newcastle University RAJIV RANJAN, Chinese University of Geoscienes and Newcastle University Interest in processing big data has increased rapidly to gain insights that can transform businesses, government policies and research outcomes. This has led to advancement in communication, programming and processing technologies, including Cloud computing services and technologies such as Hadoop, Spark and Storm. This trend also affects the needs of analytical applications, which are no longer monolithic but composed of several individual analytical steps running in the form of a workflow. These Big Data Workflows are vastly different in nature from traditional workflows. Researchers arecurrently facing the challenge of how to orchestrate and manage the execution of such workflows. In this paper, we discuss in detail orchestration requirements of these workflows as well as the challenges in achieving these requirements. We alsosurvey current trends and research that supports orchestration of big data workflows and identify open research challenges to guide future developments in this area. CCS Concepts: • General and reference → Surveys and overviews; • Information systems → Data analytics; • Computer systems organization → Cloud computing; Additional Key Words and Phrases: Big Data, Cloud Computing, Workflow Orchestration, Requirements, Approaches ACM Reference format: Mutaz Barika, Saurabh Garg, Albert Y. Zomaya, Lizhe Wang, Aad van Moorsel, and Rajiv Ranjan. 2018. Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions.
    [Show full text]
  • Building Machine Learning Inference Pipelines at Scale
    Building Machine Learning inference pipelines at scale Julien Simon Global Evangelist, AI & Machine Learning @julsimon © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Problem statement • Real-life Machine Learning applications require more than a single model. • Data may need pre-processing: normalization, feature engineering, dimensionality reduction, etc. • Predictions may need post-processing: filtering, sorting, combining, etc. Our goal: build scalable ML pipelines with open source (Spark, Scikit-learn, XGBoost) and managed services (Amazon EMR, AWS Glue, Amazon SageMaker) © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Spark https://spark.apache.org/ • Open-source, distributed processing system • In-memory caching and optimized execution for fast performance (typically 100x faster than Hadoop) • Batch processing, streaming analytics, machine learning, graph databases and ad hoc queries • API for Java, Scala, Python, R, and SQL • Available in Amazon EMR and AWS Glue © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. MLlib – Machine learning library https://spark.apache.org/docs/latest/ml-guide.html • Algorithms: classification, regression, clustering, collaborative filtering. • Featurization: feature extraction, transformation, dimensionality reduction. • Tools for constructing, evaluating and tuning pipelines • Transformer – a transform function that maps a DataFrame into a new
    [Show full text]
  • Developer Tool Guide
    Informatica® 10.2.1 Developer Tool Guide Informatica Developer Tool Guide 10.2.1 May 2018 © Copyright Informatica LLC 2009, 2019 This software and documentation are provided only under a separate license agreement containing restrictions on use and disclosure. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica LLC. Informatica, the Informatica logo, PowerCenter, and PowerExchange are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at https://www.informatica.com/trademarks.html. Other company and product names may be trade names or trademarks of their respective owners. U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation is subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License. Portions of this software and/or documentation are subject to copyright held by third parties. Required third party notices are included with the product. The information in this documentation is subject to change without notice. If you find any problems in this documentation, report them to us at [email protected].
    [Show full text]
  • Evaluation of SPARQL Queries on Apache Flink
    applied sciences Article SPARQL2Flink: Evaluation of SPARQL Queries on Apache Flink Oscar Ceballos 1 , Carlos Alberto Ramírez Restrepo 2 , María Constanza Pabón 2 , Andres M. Castillo 1,* and Oscar Corcho 3 1 Escuela de Ingeniería de Sistemas y Computación, Universidad del Valle, Ciudad Universitaria Meléndez Calle 13 No. 100-00, Cali 760032, Colombia; [email protected] 2 Departamento de Electrónica y Ciencias de la Computación, Pontificia Universidad Javeriana Cali, Calle 18 No. 118-250, Cali 760031, Colombia; [email protected] (C.A.R.R.); [email protected] (M.C.P.) 3 Ontology Engineering Group, Universidad Politécnica de Madrid, Campus de Montegancedo, Boadilla del Monte, 28660 Madrid, Spain; ocorcho@fi.upm.es * Correspondence: [email protected] Abstract: Existing SPARQL query engines and triple stores are continuously improved to handle more massive datasets. Several approaches have been developed in this context proposing the storage and querying of RDF data in a distributed fashion, mainly using the MapReduce Programming Model and Hadoop-based ecosystems. New trends in Big Data technologies have also emerged (e.g., Apache Spark, Apache Flink); they use distributed in-memory processing and promise to deliver higher data processing performance. In this paper, we present a formal interpretation of some PACT transformations implemented in the Apache Flink DataSet API. We use this formalization to provide a mapping to translate a SPARQL query to a Flink program. The mapping was implemented in a prototype used to determine the correctness and performance of the solution. The source code of the Citation: Ceballos, O.; Ramírez project is available in Github under the MIT license.
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Talend Open Studio for Big Data Release Notes
    Talend Open Studio for Big Data Release Notes 6.0.0 Talend Open Studio for Big Data Adapted for v6.0.0. Supersedes previous releases. Publication date July 2, 2015 Copyleft This documentation is provided under the terms of the Creative Commons Public License (CCPL). For more information about what you can and cannot do with this documentation in accordance with the CCPL, please read: http://creativecommons.org/licenses/by-nc-sa/2.0/ Notices Talend is a trademark of Talend, Inc. All brands, product names, company names, trademarks and service marks are the properties of their respective owners. License Agreement The software described in this documentation is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.html. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This product includes software developed at AOP Alliance (Java/J2EE AOP standards), ASM, Amazon, AntlR, Apache ActiveMQ, Apache Ant, Apache Avro, Apache Axiom, Apache Axis, Apache Axis 2, Apache Batik, Apache CXF, Apache Cassandra, Apache Chemistry, Apache Common Http Client, Apache Common Http Core, Apache Commons, Apache Commons Bcel, Apache Commons JxPath, Apache
    [Show full text]
  • CA SOLVE:FTS Installation Guide
    CA SOLVE:FTS Installation Guide Release 12.1 This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the “Documentation”) is for your informational purposes only and is subject to change or withdrawal by CA at any time. This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and may not be disclosed by you or used for any purpose other than as may be permitted in (i) a separate agreement between you and CA governing your use of the CA software to which the Documentation relates; or (ii) a separate confidentiality agreement between you and CA. Notwithstanding the foregoing, if you are a licensed user of the software product(s) addressed in the Documentation, you may print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced copy. The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT.
    [Show full text]
  • Flare: Optimizing Apache Spark with Native Compilation
    Flare: Optimizing Apache Spark with Native Compilation for Scale-Up Architectures and Medium-Size Data Gregory Essertel, Ruby Tahboub, and James Decker, Purdue University; Kevin Brown and Kunle Olukotun, Stanford University; Tiark Rompf, Purdue University https://www.usenix.org/conference/osdi18/presentation/essertel This paper is included in the Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18). October 8–10, 2018 • Carlsbad, CA, USA ISBN 978-1-939133-08-3 Open access to the Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIX. Flare: Optimizing Apache Spark with Native Compilation for Scale-Up Architectures and Medium-Size Data Grégory M. Essertel1, Ruby Y. Tahboub1, James M. Decker1, Kevin J. Brown2, Kunle Olukotun2, Tiark Rompf1 1Purdue University, 2Stanford University {gesserte,rtahboub,decker31,tiark}@purdue.edu, {kjbrown,kunle}@stanford.edu Abstract cessing. Systems like Apache Spark [8] have gained enormous traction thanks to their intuitive APIs and abil- In recent years, Apache Spark has become the de facto ity to scale to very large data sizes, thereby commoditiz- standard for big data processing. Spark has enabled a ing petabyte-scale (PB) data processing for large num- wide audience of users to process petabyte-scale work- bers of users. But thanks to its attractive programming loads due to its flexibility and ease of use: users are able interface and tooling, people are also increasingly using to mix SQL-style relational queries with Scala or Python Spark for smaller workloads. Even for companies that code, and have the resultant programs distributed across also have PB-scale data, there is typically a long tail of an entire cluster, all without having to work with low- tasks of much smaller size, which make up a very impor- level parallelization or network primitives.
    [Show full text]
  • Hive, Spark, Presto for Interactive Queries on Big Data
    DEGREE PROJECT IN INFORMATION AND COMMUNICATION TECHNOLOGY, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2018 Hive, Spark, Presto for Interactive Queries on Big Data NIKITA GUREEV KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE TRITA TRITA-EECS-EX-2018:468 www.kth.se Abstract Traditional relational database systems can not be efficiently used to analyze data with large volume and different formats, i.e. big data. Apache Hadoop is one of the first open-source tools that provides a dis- tributed data storage system and resource manager. The space of big data processing has been growing fast over the past years and many tech- nologies have been introduced in the big data ecosystem to address the problem of processing large volumes of data, and some of the early tools have become widely adopted, with Apache Hive being one of them. How- ever, with the recent advances in technology, there are other tools better suited for interactive analytics of big data, such as Apache Spark and Presto. In this thesis these technologies are examined and benchmarked in or- der to determine their performance for the task of interactive business in- telligence queries. The benchmark is representative of interactive business intelligence queries, and uses a star-shaped schema. The performance Hive Tez, Hive LLAP, Spark SQL, and Presto is examined with text, ORC, Par- quet data on different volume and concurrency. A short analysis and con- clusions are presented with the reasoning about the choice of framework and data format for a system that would run interactive queries on big data.
    [Show full text]