DOCSLIB.ORG

IBM Infosphere Streams: Assembling Continuous Insight in the Information Revolution

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
IBM Infosphere Streams: Assembling Continuous Insight in the Information Revolution Front cover IBM® Information Management Software IBM InfoSphere Streams Assembling Continuous Insight in the Information Revolution Supporting scalability and dynamic adaptability Performing real-time analytics on Big Data Enabling continuous analysis of data Chuck Ballard Kevin Foster Andy Frenkiel Senthil Nathan Bugra Gedik Deepak Rajan Roger Rea Brian Williams Mike Spicer Michael P. Koranda Vitali N. Zoubov ibm.com/redbooks International Technical Support Organization IBM InfoSphere Streams: Assembling Continuous Insight in the Information Revolution October 2011 SG24-7970-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (October 2011) This edition applies to Version 2.0.0 of InfoSphere Streams (Product Number 5724-Y95). © Copyright International Business Machines Corporation 2011. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . x Preface . xi The team who wrote this book . xii Now you can become a published author, too! . xvii Comments welcome. xvii Stay connected to IBM Redbooks . xviii Chapter 1. Introduction. 1 1.1 Stream computing . 2 1.1.1 Business landscape . 6 1.1.2 Information environment . 9 1.1.3 The evolution of analytics . 14 1.1.4 Relationship to Big Data . 17 1.2 IBM InfoSphere Streams . 17 1.2.1 Overview of Streams. 19 1.2.2 Why use Streams . 24 1.2.3 Examples of Streams implementations. 27 Chapter 2. Streams concepts and terms. 33 2.1 IBM InfoSphere Streams: Solving new problems . 34 2.2 Concepts and terms . 39 2.2.1 Streams instances, hosts, host types, and admin services. 40 2.2.2 Projects, applications, streams, and operators . 44 2.2.3 Applications, Jobs, Processing Elements, and Containers . 47 2.3 End-to-end example: Streams and the lost child. 48 2.3.1 The Lost Child application. 50 2.3.2 Example Streams Processing Language code review . 53 2.4 IBM InfoSphere Streams tools . 58 2.4.1 Creating an example application inside Streams Studio. 59 Chapter 3. Streams applications . 67 3.1 Streams application design . 69 3.1.1 Design aspects . 70 3.1.2 Data sources . 72 3.1.3 Output . 75 3.1.4 Existing analytics. 78 3.1.5 Performance requirements . 79 © Copyright IBM Corp. 2011. All rights reserved. iii 3.2 SPL design patterns . 80 3.2.1 Reducing data using a simple filter . 82 3.2.2 Reducing data using a simple schema mapper . 86 3.2.3 Data Parallel . 90 3.2.4 Data Pipeline. 97 3.2.5 Outlier detection . 101 3.2.6 Enriching streams from a database . 107 3.2.7 Tuple Pacer. 111 3.2.8 Processing multiplexed streams . 116 3.2.9 Updating import stream subscriptions. 120 3.2.10 Updating export stream properties . 127 3.2.11 Adapting to observations of runtime metrics . 134 Chapter 4. InfoSphere Streams deployment. 139 4.1 Architecture, instances, and topologies. 140 4.1.1 Runtime architecture . 140 4.1.2 Streams instances. 143 4.1.3 Deployment topologies . 147 4.2 Streams runtime deployment planning . 151 4.2.1 Streams environment . 151 4.2.2 Sizing the environment . 151 4.2.3 Deployment and installation checklists . 152 4.3 Pre- and post-installation of the Streams environment . 160 4.3.1 Installation and configuration of the Linux environment . 160 4.3.2 InfoSphere Streams installation . 163 4.3.3 Post-installation configuration . 164 4.3.4 Streams user configuration . 168 4.4 Streams instance creation and configuration . 170 4.4.1 Streams shared instance configuration. 170 4.4.2 Streams private developer instance configuration . 175 4.5 Application deployment capabilities . 179 4.5.1 Dynamic application composition . 181 4.5.2 Operator host placement. 184 4.5.3 Operator partitioning . 189 4.5.4 Parallelizing operators. 190 4.6 Failover, availability, and recovery . 194 4.6.1 Restarting and relocating processing elements . 194 4.6.2 Recovering application hosts . 197 4.6.3 Recovering management hosts . 199 Chapter 5. Streams Processing Language . 203 5.1 Language elements. 204 5.1.1 Structure of an SPL program file. 206 iv IBM InfoSphere Streams: Assembling Continuous Insight in the Information Revolution 5.1.2 Streams data types . 209 5.1.3 Stream schemas . 210 5.1.4 Streams punctuation markers . 212 5.1.5 Streams windows . 212 5.1.6 Stream bundles . 220 5.2 Streams Processing Language operators . 220 5.2.1 Operator usage language syntax . 225 5.2.2 The Source operator . 227 5.2.3 The Sink operator . 231 5.2.4 The Functor operator . 232 5.2.5 The Aggregate operator . 234 5.2.6 The Split operator . 236 5.2.7 The Punctor operator . 240 5.2.8 The Delay operator . 241 5.2.9 The Barrier operator . 241 5.2.10 The Join operator . 242 5.2.11 The Sort operator . 245 5.2.12 Additional SPL operators . 247 5.3 The preprocessor ..
Load more

Recommended publications
  • A History of Mail Classification and Its Underlying Policies and Purposes
    A HISTORY OF MAIL CLASSIFICATION AND ITS UNDERLYING POLICIES AND PURPOSES Richard B. Kielbowicz AssociateProfessor School of Commuoications, Ds-40 University of Washington Seattle, WA 98195 (206) 543-2660 &pared For the Postal Rate Commission’s Mail ReclassificationProceeding, MC95-1. July 17. 1995 -- /- CONTENTS 1. Introduction . ._. ._.__. _. _, __. _. 1 2. Rate Classesin Colonial America and the Early Republic (1690-1840) ............................................... 5 The Colonial Mail ................................................................... 5 The First Postal Services .................................................... 5 Newspapers’ Mail Status .................................................... 7 Postal Policy Under the Articles of Confederation .............................. 8 Postal Policy and Practice in the Early Republic ................................ 9 Letters and Packets .......................................................... 10 Policy Toward Newspapers ................................................ 11 Recognizing Magazines .................................................... 12 Books in the Mail ........................................................... 17 3. Toward a Classitication Scheme(1840-1870) .................................. 19 Postal Reform Act of 1845 ........................................................ 19 Letters and the First Class, l&IO-l&?70 .............................. ............ 19 Periodicals and the Second Class ................................................ 21 Business
    [Show full text]
  • Mysql Workbench Abstract
    MySQL Workbench Abstract This is the MySQL Workbench Reference Manual. It documents the MySQL Workbench Community and MySQL Workbench Commercial releases for versions 8.0 through 8.0.26. If you have not yet installed the MySQL Workbench Community release, please download your free copy from the download site. The MySQL Workbench Community release is available for Microsoft Windows, macOS, and Linux. MySQL Workbench platform support evolves over time. For the latest platform support information, see https:// www.mysql.com/support/supportedplatforms/workbench.html. For notes detailing the changes in each release, see the MySQL Workbench Release Notes. For legal information, including licensing information, see the Preface and Legal Notices. For help with using MySQL, please visit the MySQL Forums, where you can discuss your issues with other MySQL users. Document generated on: 2021-09-24 (revision: 70892) Table of Contents Preface and Legal Notices ................................................................................................................ vii 1 General Information ......................................................................................................................... 1 1.1 What Is New in MySQL Workbench ...................................................................................... 1 1.1.1 New in MySQL Workbench 8.0 Release Series ........................................................... 1 1.1.2 New in MySQL Workbench 6.0 Release Series ..........................................................
    [Show full text]
  • VSI's Open Source Strategy
    VSI's Open Source Strategy Plans and schemes for Open Source so9ware on OpenVMS Bre% Cameron / Camiel Vanderhoeven April 2016 AGENDA • Programming languages • Cloud • Integraon technologies • UNIX compability • Databases • Analy;cs • Web • Add-ons • Libraries/u;li;es • Other consideraons • SoDware development • Summary/conclusions tools • Quesons Programming languages • Scrip;ng languages – Lua – Perl (probably in reasonable shape) – Tcl – Python – Ruby – PHP – JavaScript (Node.js and friends) – Also need to consider tools and packages commonly used with these languages • Interpreted languages – Scala (JVM) – Clojure (JVM) – Erlang (poten;ally a good fit with OpenVMS; can get good support from ESL) – All the above are seeing increased adop;on 3 Programming languages • Compiled languages – Go (seeing rapid adop;on) – Rust (relavely new) – Apple Swi • Prerequisites (not all are required in all cases) – LLVM backend – Tweaks to OpenVMS C and C++ compilers – Support for latest language standards (C++) – Support for some GNU C/C++ extensions – Updates to OpenVMS C RTL and threads library 4 Programming languages 1. JavaScript 2. Java 3. PHP 4. Python 5. C# 6. C++ 7. Ruby 8. CSS 9. C 10. Objective-C 11. Perl 12. Shell 13. R 14. Scala 15. Go 16. Haskell 17. Matlab 18. Swift 19. Clojure 20. Groovy 21. Visual Basic 5 See h%p://redmonk.com/sogrady/2015/07/01/language-rankings-6-15/ Programming languages Growing programming languages, June 2015 Steve O’Grady published another edi;on of his great popularity study on programming languages: RedMonk Programming Language Rankings: June 2015. As usual, it is a very valuable piece. There are many take-away from this research.
    [Show full text]
  • Licensing Information User Manual Release 8.0 E65472-04
    Oracle® Communications Calendar Server Licensing Information User Manual Release 8.0 E65472-04 March 2021 Copyright © 2000, 2021, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are “commercial computer software” pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.
    [Show full text]
  • Migrating from Microsoft SQL Server to IBM Informix
    Front cover Migrating from Microsoft SQL Server to IBM Informix Develops a data and applications migration methodology Migrates step-by-step from SQL Server to IBM Informix Provides a variety of migration examples Whei-Jen Chen Chee Fong Koh Deen Murad Holger Kirstein Rakeshkumar Naik ibm.com/redbooks International Technical Support Organization Migrating from Microsoft SQL Server to IBM Informix July 2010 SG24-7847-00 Note: Before using this information and the product it supports, read the information in “Notices” on page xi. First Edition (July 2010) This edition applies to IBM Informix Version 11.5 and Microsoft SQL Server 2008. © Copyright International Business Machines Corporation 2010. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . xi Trademarks . xii Preface . xiii The team who wrote this book . xiii Acknowledgements . xv Become a published author . xv Comments welcome. xvi Stay connected to IBM Redbooks . xvi Chapter 1. Introduction. 1 1.1 Migration considerations . 2 1.2 Informix position . 4 1.3 IBM Informix editions. 5 1.3.1 No-charge editions . 5 1.3.2 For-purchase editions . 6 1.4 Informix functionality and features. 9 1.4.1 Replication and high availability . 9 1.4.2 Performance . 11 1.4.3 Security . 12 1.4.4 Administration . 13 1.4.5 Warehouse . 14 1.4.6 Application development . 14 1.4.7 Extensibility . 15 Chapter 2. Architecture overview . 19 2.1 Process . 20 2.1.1 SQL Server . 20 2.1.2 Informix .
    [Show full text]
  • UK CMR Charts
    Figure 1.1 Communications industry revenue – telecoms, TV, radio, post £billions Annual 5 year 80 change CAGR 61.1 61.6 60.2 59.8 59.6 59.5 Total -0.2% -0.5% 60 6.8 6.8 6.7 6.5 6.7 7.2 1.2 1.1 1.1 1.1 1.2 1.2 Post 7.0% 0.9% 11.0 11.2 11.1 11.7 12.2 12.3 40 Radio 2.7% 0.3% TV 0.8% 2.2% 20 42.1 42.5 41.3 40.4 39.5 38.8 Telecoms -1.8% -1.6% 0 2007 2008 2009 2010 2011 2012 Source: Ofcom/ operators Note: Includes licence fee allocation for radio and TV, Figures are in nominal terms 0 Figure 1.2 Digital communications service availability UK UK Platform UK 2012 England Scotland Wales N Ireland 2011 change Fixed line 100% 100% 0pp 100% 100% 100% 100% 2G mobile1 99.6% 99.7% -0.1pp 99.8% 99.3% 98.8% 98.5% 3G mobile2 99.1% 99.1% 0pp 99.5% 96.6% 97.7% 97.4% Virgin Media cable broadband3 48% - - 51% 38% 22% 28% LLU ADSL broadband4 94% 92% +3pp 95% 87% 92% 85% BT Openreach / Kcom fibre b’band5 56% n/a n/a 59% 25% 41% 93% NGA broadband6 73% 65% +8pp 76% 52% 48% 95% Digital satellite TV 98% 98% 0pp - - - - Digital terrestrial TV7 99% - - 99% 99% 98% 97% DAB BBC Network88 94.3% 92% +2.3pp 95.5% 90.9% 85.9% 85.4% DAB commercial network (Digital 85% 85% 0pp 90% 75% 60% - One)9 Sources: Ofcom and operators: 1.
    [Show full text]
  • House of Lords Official Report
    Vol. 742 Wednesday No. 95 16 January 2013 PARLIAMENTARY DEBATES (HANSARD) HOUSE OF LORDS OFFICIAL REPORT ORDER OF BUSINESS Questions Property: Leasehold Valuation Tribunal NHS: Clinical Commissioning Groups Education: School Leavers EU: UK’s National and Trade Interest Age of Criminal Responsibility Bill [HL] First Reading Growth and Infrastructure Bill Order of Consideration Motion Legislative Reform (Constitution of Veterinary Surgeons Preliminary Investigation and Disciplinary Committees) Order 2013 Motion to Approve European Union (Croatian Accession and Irish Protocol) Bill Report European Union (Approvals) Bill [HL] Report Scotland Act 1998 (Modification of Schedule 5) Order 2013 Motion to Approve Health: Medical Innovation Question for Short Debate Grand Committee Enterprise and Regulatory Reform Bill Committee (8th Day) Written Statements Written Answers For column numbers see back page £4·00 Lords wishing to be supplied with these Daily Reports should give notice to this effect to the Printed Paper Office. The bound volumes also will be sent to those Peers who similarly notify their wish to receive them. No proofs of Daily Reports are provided. Corrections for the bound volume which Lords wish to suggest to the report of their speeches should be clearly indicated in a copy of the Daily Report, which, with the column numbers concerned shown on the front cover, should be sent to the Editor of Debates, House of Lords, within 14 days of the date of the Daily Report. This issue of the Official Report is also available on the Internet at www.publications.parliament.uk/pa/ld201213/ldhansrd/index/130116.html PRICES AND SUBSCRIPTION RATES DAILY PARTS Single copies: Commons, £5; Lords £4 Annual subscriptions: Commons, £865; Lords £600 LORDS VOLUME INDEX obtainable on standing order only.
    [Show full text]
  • Learning at Home
    Learning at Home A Guide for parents and carers This guide is there to help and support, if for any reason your child has to stay at home for a period of time. This is only a guide and not a manual. You should always consider your child's needs and any medical or health conditions they have before you do any activity or make any plans. This is intended for parents/carers to use. Please consider any other current Government public health advice also. There are numerous links in this guide to agencies and organisations who provide information and resources. We are not able to validate these or guarantee they will be there forever. We accept no liability or responsibility for your use of these. Make a plan Make a timetable or daily plan Make is flexible and adaptable It is important for children to have structure Make it suit your child's needs Make it visual and adapt as necessary and routine. It is good for supporting and Set some structure and routine, put in things you all developing a healthy mind and body. enjoy and offer choices Think about some quiet times or independent reading Think about your space and resources at home Keep to good bedtime routines, sleep is very important Put in some daily physical activities Make sure you do as much movement as possible (check out the healthy body section) If you can, go out and use the garden Make sure your plan has activities for the mind, body Think about daily household tasks you can include and spirit…things that are academic and focused on such as cooking learning…things that are healthy for body muscles and Make sure you put in plenty of fun and games heart…things that bring fun, joy and laughter to lift the Think about some projects or bigger tasks or spirit challenges you can set Make sure your plan is achievable for your child, don’t plan too much If some things don’t work out change it Use this time to invest in interests, hobbies and talents Use your phone or keep a note book to jot down any ideas or plans that might come to Get creative and use what you have to make something amazing.
    [Show full text]
  • VICTORIA Reginie
    VICTORIA. mm* mm ANNO DECIMO SEPTIMO VICTORIA REGINiE. .By jffis Excellency CHARLES JOSEPH LA TIIOBE, ESQUIRE, Lieutenant Governor of the Colony of Victoria and its Dependencies with the advice and conse?it of the Legislative Council. M^tummma^tkiM No. XXX. An Act to amend the Law relating to the Post Office. [Assented to 12th April, 1854.] f ? II E R E A S it is expedient to amend the Law relating to the preamble. Post Office in Victoria Be it therefore enacted by His Excellency the Lieutenant Governor of Victoria by and with the advice and consent of the Legislative Council thereof as follows— I. From and after the passing of this Act an Act of the Lieu- l^peai of is Vie., tenant Governor and Legislative Council of Victoria passed m the No-9- fifteenth year of the Reign of Her present Majesty Queen Victoria intituled "An Act to amend the Law for the Conveyance and Postage of " Letters" shall be and the same is hereby repealed except as to any proceedings commenced or instituted under the said recited Act previously to the passing hereof. II. It shall be lawful for the Lieutenant Governor with the advice Rules for establishing of the Executive Council to make Rules and Regulations for the 2f0fSrC-Cei!If establishing and managing of the several Post Offices including iottcrs, &c. the imposing of fees for private boxes and the receiving despatching conveying and delivering of Letters Newspapers Packets and Parcels and the making custody and sale of Stamps and the receipt and payment of monies in connection with the said Post Offices and the conduct of all Postmasters and other Officers of the said department and the said Rules and Regulations from time to time to alter revoke or vary and such other Rules and Regulations to establish in their stead as with the advice aforesaid he shall deem expedient.
    [Show full text]
  • PACKET 04-20-2021.PDF Board of Commissioners Board of Commissioners 400 Colorado Ave
    1. 9:00 A.M. Board Of Commissioners Regular Meeting Documents: BOARD AGENDA 04-20-2021.PDF BOARD PACKET 04-20-2021.PDF Board of Commissioners Board of Commissioners 400 Colorado Ave. Morris, MN 56267 District 1 Bob Kopitzke District 2 Jeanne Ennen Ph 320-208-6583 District 3 Ron Staples District 4 Donny Wohlers STEVENS District 5 Neil Wiese C O U N T Y STEVENS COUNTY BOARD OF COMMISSIONERS REGULAR MEETING Tuesday, April 20, 2021 9:00 a.m. 1. 9:00 am CONVENE PLEDGE • Additions to the Agenda/Approve • Approve Minutes of the 4/6/21 Regular Meeting • Public Comment Period 2. 9:05 am Dona Greiner - Emergency Management Director • EM Update for Information 3. 9:15 am Liberty Sleiter - Human Services Director • HS Warrants for Approval • Staff Introduction 4. 9:20 am Stephanie Buss - Auditor/Treasurer • Auditor's Warrants for Review • Commissioner Warrants for Review • CD Loan Advances for Approval • Quarterly Ditch Balances for Information Only 5. 9:30 am Continued Final Hearing for CD 25 • Findings and Facts CD 25 for Approval 6. 9:45 am Jon Maras- Assistant County Engineer • Bids for Approval • 2020 Highway Annual Report for Approval 7. 10:00 am Nick Young- Facilities • Lee Center Bids for Approval 8. 10:20 am Bill Kleindl- Planning and Zoning • Chippewa River Watershed Resolution for Approval www.co.stevens.mn.us Equal Opportunity Employer Board of Commissioners Board of Commissioners 400 Colorado Ave. Morris, MN 56267 District 1 Bob Kopitzke Ph 320-208-6583 District 2 Jeanne Ennen District 3 Ron Staples District 4 Donny Wohlers STEVENS District 5 Neil Wiese C O U N T Y 9.
    [Show full text]
  • Speech & Language Therapy Resource Pack
    Speech and Language Therapy Early Years Resource Pack Part 2 – Advice Sheets SLT RESOURCE PACK: Part 2 Advice Sheets 1 of 73 Contents Early Years Resource Pack ................................................................................................................................ 1 Contents ................................................................................................................................................................. 2 Attention and Listening ........................................................................................................................................ 4 Stages of Attention Development ............................................................................................................... 5 Useful strategies for helping a child develop Attention ........................................................................... 6 Now/ Next Boards ......................................................................................................................................... 8 Visual Timetable ............................................................................................................................................ 9 Shared Attention and Anticipation Games .............................................................................................. 10 Ready Steady Go Games (RSG) ............................................................................................................. 11 Attention Bucket .........................................................................................................................................
    [Show full text]
  • No. XII. an Act to Consolidate and Amend the Law for Conveyance and Postage of Letters
    No. XII. An Act to consolidate and amend the Law for Conveyance and Postage of Letters. [22nd December, 1851.] BE it enacted by His Excellency the Governor of the Colony of New South Wales by and with the advice and consent of the Legislative Council thereof That from and after the passing of this Act the Act of the said Governor and Council made and passed in the thirteenth year of the reign of Her present Majesty intituled "An Act " to establish an uniform rate of Postage and to consolidate and amend " the Law for the conveyance and Postage of Letters" shall be repealed except in so far as the same operates to repeal certain preceding Acts therein mentioned and the same shall thenceforth cease to have any operation except as aforesaid and except as to anything done or com­ menced to be done before the taking effect of this Act. 2. And be it enacted That it shall be lawful for the said Governor with the advice of the Executive Council to make rules and regulations for the establishing and managing of the several Post Offices within the said Colony and the receiving dispatching carrying and delivering of letters packets and parcels and the making custody and sale of stamps and the receipt and payment of moneys in connection with the said Post Offices and the conduct of all Post­ masters and other officers of the Department and the said rules to alter revoke or vary and such other rules and regulations to establish in their stead as with the advice aforesaid he shall deem expedient and also to reduce the postage on letters packets or parcels under any such special circumstances as shall appear to the said Governor and Executive Council to render such reduction expedient.
    [Show full text]