IBM Platform Computing Integration Solutions
Total Page:16
File Type:pdf, Size:1020Kb
Front cover IBM Platform Computing Integration Solutions Learn about solutions that integrate IBM Platform Symphony and IBM InfoSphere BigInsights for analytics See how the solutions use IBM System x and IBM GPFS Understand details about IBM Platform LSF implementation Dino Quintero Ricardo Dobelin Barros Ashraf Gomaa José Higino Archana Kumar Majid Ouassir Adam Parker Joanna Wong ibm.com/redbooks International Technical Support Organization IBM Platform Computing Integration Solutions March 2013 SG24-8081-00 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (March 2013) This edition applies to IBM Platform Symphony 6.1, IBM Platform Symphony 5.2, IBM Platform LSF 8.3, IBM InfoSphere BigInsights 1.4, Red Hat Enterprise Linux (RHEL) 6.2 x86_64, and RHEL 6.3 x86_64. © Copyright International Business Machines Corporation 2013. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix The team who wrote this book . ix Now you can become a published author, too! . xi Comments welcome. xi Stay connected to IBM Redbooks . xi Chapter 1. Products and portfolio for IBM Platform Computing solutions . 1 1.1 IBM Platform LSF . 2 1.1.1 IBM Platform LSF editions . 2 1.1.2 IBM Platform LSF add-ons . 4 1.1.3 New in the IBM Platform LSF V9.1 family. 4 1.2 IBM Platform HPC. 4 1.3 IBM Platform Symphony . 5 1.3.1 IBM Platform Symphony edition features . 5 1.3.2 IBM Platform Symphony add-on tools. 6 1.3.3 New in IBM Platform Symphony V6.1. 6 1.4 IBM Platform Cluster Manager . 14 1.4.1 IBM Platform Cluster Manager features . 15 1.4.2 IBM Platform Cluster Manager supported environments . 15 Chapter 2. Integration of IBM Platform Symphony and IBM InfoSphere BigInsights. 17 2.1 IBM Platform Symphony . 18 2.1.1 How InfoSphere BigInsights works with IBM Platform Symphony . 18 2.1.2 The IBM Platform Symphony performance advantage . 20 2.2 Environment . 21 2.2.1 Preconfiguration on all nodes . 21 2.3 Configuring InfoSphere BigInsights. 26 2.4 Installing IBM Platform Symphony Advanced Edition . 29 2.5 Integrating IBM Platform Symphony and InfoSphere BigInsights . 31 2.6 Additional configuration for IBM Platform Symphony . 34 2.6.1 Identifying the management nodes in the IBM Platform Symphony configuration 34 2.6.2 TCP/IP performance tuning . 35 2.7 Benchmark tests . 36 2.7.1 Test environment and monitoring tools. 36 2.7.2 Results of the stand-alone benchmark for InfoSphere BigInsights V1.4. 37 2.7.3 Results of integration for IBM Platform Symphony V5.2 and InfoSphere BigInsights V1.4 . 38 2.7.4 Advantages of integrating IBM Platform Symphony V5.2 with InfoSphere BigInsights V1.4 . 41 2.8 Adding users . 43 2.8.1 Assumptions . 44 2.8.2 Adding a user for the integrated cluster . 44 2.9 Adding nodes . 46 2.10 Troubleshooting . 47 2.10.1 InfoSphere BigInsights . 47 2.10.2 Increasing Java virtual memory . 47 © Copyright IBM Corp. 2013. All rights reserved. iii 2.10.3 Increasing system limits . 48 Chapter 3. Implementation scenario for IBM Platform LSF. 49 3.1 Valid hardware resources to ensure that the setup can use IBM Platform LSF . 50 3.2 Sizing for I/O-intensive clusters. 50 3.3 Considerations for GPGPU intensive clusters . 51 3.4 Job queues . 51 3.4.1 Defining queues . 51 3.4.2 Default queue . 53 3.4.3 Managing queues . 54 3.5 Job scheduling . 54 3.5.1 First come, first served . 55 3.5.2 Preemption scheduling . 55 3.5.3 Fairshare scheduling. 56 3.5.4 Exclusive scheduling. 59 3.5.5 Backfill scheduling . 60 3.6 Goal-oriented scheduling . 61 3.6.1 Defining a service class . 61 3.6.2 Using a service class . 62 3.7 Job submission . 63 3.7.1 Job arrays . 65 3.7.2 Lifecycle of jobs . 67 3.8 Compute units . 69 3.8.1 Configuring compute units . 69 3.8.2 Using compute units . 70 3.9 Application profiles . 73 3.9.1 Defining application profiles . 73 3.9.2 Using application profiles . 74 3.10 Job submission prechecks and setup . 75 3.10.1 Modifying a submitted job . ..