Front cover Addressing Data Volume, Velocity, and Variety with IBM InfoSphere Streams V3.0 Performing real-time analysis on big data by using System x Developing a drag and drop Streams application Monitoring and administering Streams Mike Ebbers Ahmed Abdel-Gayed Veera Bhadran Budhi Ferdiansyah Dolot Vishwanath Kamat Ricardo Picone João Trevelin ibm.com/redbooks International Technical Support Organization Addressing Data Volume, Velocity, and Variety with IBM InfoSphere Streams V3.0 March 2013 SG24-8108-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (March 2013) This edition applies to IBM InfoSphere Streams V3.0. © Copyright International Business Machines Corporation 2013. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . .x Preface . xi The team who wrote this book . xi Now you can become a published author, too! . xiv Comments welcome. xiv Stay connected to IBM Redbooks . xiv Chapter 1. Think big with big data. 1 1.1 Executive summary. 2 1.2 What is big data? . 3 1.3 IBM big data strategy . 5 1.4 IBM big data platform . 7 1.4.1 InfoSphere BigInsights for Hadoop-based analytics . 7 1.4.2 InfoSphere Streams for low-latency analytics. 8 1.4.3 InfoSphere Information Server for Data Integration . 9 1.4.4 Netezza, InfoSphere Warehouse, and Smart Analytics System for deep analytics10 1.5 Summary. 13 Chapter 2. Exploring IBM InfoSphere Streams. 15 2.1 Stream computing . 16 2.1.1 Business landscape . 19 2.1.2 Information environment . 22 2.1.3 The evolution of analytics . 26 2.2 IBM InfoSphere Streams. 28 2.2.1 Overview of Streams. 29 2.2.2 Why use Streams . 34 2.2.3 Examples of Streams implementations. 36 Chapter 3. InfoSphere Streams architecture . 41 3.1 Streams concepts and terms . 42 3.1.1 Working with continuous data flow . 42 3.1.2 Component overview . 43 3.2 InfoSphere Streams runtime system. 45 3.2.1 SPL components. 45 3.2.2 Runtime components . 47 3.2.3 InfoSphere Streams runtime files . 49 3.3 Performance requirements . 51 3.3.1 InfoSphere Streams reference architecture . 52 3.4 High availability . 55 Chapter 4. IBM InfoSphere Streams V3.0 new features. 57 4.1 New configuration features . 58 4.1.1 Enhanced first steps after installation . 58 4.2 Development . 59 4.3 Administration . 59 4.3.1 Improved visual application monitoring. 59 4.3.2 Streams data visualization . 59 © Copyright IBM Corp. 2013. All rights reserved. iii 4.3.3 Streams console application launcher . 59 4.4 Integration . 60 4.4.1 DataStage integration . 60 4.4.2 Netezza integration . 60 4.4.3 Data Explorer integration . 60 4.4.4 SPSS integration. 60 4.4.5 XML integration. 60 4.5 Analytics and accelerators toolkits . 61 4.5.1 Geospatial toolkit . 61 4.5.2 Time series toolkit . 61 4.5.3 Complex Event Processing toolkit. 61 4.5.4 Accelerators toolkit . 61 Chapter 5. InfoSphere Streams deployment. 63 5.1 Architecture, instances, and topologies . 64 5.1.1 Runtime architecture. 64 5.1.2 Streams instances . 66 5.1.3 Deployment topologies . 69 5.2 Streams runtime deployment planning . 72 5.2.1 Streams environment . 72 5.2.2 Sizing the environment . 73 5.2.3 Deployment and installation checklists . 73 5.3 Streams instance creation and configuration . 80 5.3.1 Streams shared instance configuration. 80 5.3.2 Streams private developer instance configuration . 84 5.4 Application deployment capabilities . 88 5.4.1 Dynamic application composition . 89 5.4.2 Operator host placement . 91 5.4.3 Operator partitioning . 96 5.4.4 Parallelizing operators . ..