(Iot) and Big Data 30.9.2016 – DOAG 2016 Big Data Days

Internet of Things (IoT) and Big Data 30.9.2016 – DOAG 2016 Big Data Days

Guido Schmutz

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Guido Schmutz

Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected] Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz

2 Internet of Things (IoT) and Big Data Agenda

1. Introduction 2. Towards an IoT Architecture 3. IoT Refererence Architecture 4. Summary

3 Internet of Things (IoT) and Big Data Introduction

4 Internet of Things (IoT) and Big Data Internet of Things (IoT) Wave

Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans

Term was first proposed by Kevin Ashton in 1999

Source: Ericsson, June 2016 Source: The Economist 5 Internet of Things (IoT) and Big Data Reasons why IoT opportunity is occurring now ?

Affordable hardware Availability of supporting tools

• Costs of actuators & sensors have been • Big data tools & cloud based infrastructure cut in half over last 10 years have become widely available

Smaller, more powerful hardware Mass market awareness

• Form factors of hardware have shrunk to • IoT has surpassed a critical tipping point millimeter or even nanometer levels • Vision of a connected world has reached such a followership that companies have Ubiquitous & cheap mobility initiated IoT developments • Commitment is irreversible • Cost for mobile devices, bandwidth and data processing has declined over last 10 years

6 Internet of Things (IoT) and Big Data The Sensing-as-a-Service Model

7 Internet of Things (IoT) and Big Data Towards an IoT Architecture

15 Internet of Things (IoT) and Big Data Key Challenges for building an IoT application

1. Connect: How to collect data from intelligent devices?

• Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise

2. Analyze: How to analyze IoT data?

• Reduce noise and detect business event at real-time • Enable historical big-data analysis

3. Integrate: How to integrate IoT data & events with enterprise infrastructure?

• Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices

18 Internet of Things (IoT) and Big Data Today) Existing Service-/API Architecture as a base

REST External Cloud SOAP Service Providers

BPM and SOA Mobile SOAP Platform REST / SOAP API Gateway Apps WS Processes SOA Suite D B Services Oracle Data Integrator Event BPM Suite REST / SOAP

Rich (Web) Enterprise Apps REST / SOAP API Gateway Various 19Client Apps D WS Business B Logic/Rules Event DB

SQL Business Intelligence Visualization Enterprise Service Bus (ESB) / Data Integration Business Activity Service Bus Analytics Monitoring

19 Internet of Things (IoT) and Big Data = one way = request/response IoT 1a) Reuse exiting Service-/API-based Architecture

REST SOAP External Cloud Service Providers Mobile REST / SOAP Apps D BPM and SOA B SOAP Platform API Gateway WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Services BPM Suite REST / SOAP Event 20Client Apps D REST / SOAP JMS B Enterprise Apps

API Gateway Various WS Business Logic/Rules Event REST JMS DB

SQL REST Service Bus Business IoT Smart Intelligence Devices Visualization JMS / REST Enterprise Service Bus (ESB) / Data Integration WebSocket Business Activity Monitoring JMS Analytics JMS Weblogic

20 Internet of Things (IoT) and Big Data = one way = request/response IoT 1a) Challenges

• Do IoT devices contain enough resources to communicate directly over the internet (HTTP or JMS) ?

• Should the device only collect data (sense) or is there also the way back necessary (actuator) ?

• Can JMS be used from external devices (firewalls allow traffic over JMS) ?

• How many IoT devices are planned short and long term? How often do they send to the backend? Are the JMS server as well as the ESB capable to deal with the resulting message volume ?

• What are the operations on IoT messages / events, only simple transformations, filter and routing operations?

21 Internet of Things (IoT) and Big Data IoT 1b) Reuse existing Service-/API-based Architecture

REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP D BPM and SOA B SOAP Platform API Gateway WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite REST / SOAP 22Client Apps D REST / SOAP JMS B Enterprise Apps API Gateway Various WS Business REST Logic/Rules Event JMS DB

HTTP REST SQL REST HTTP Service Bus Business IoT Smart Intelligence Devices JMS Visualization Enterprise Service Bus (ESB) / Data Integration Business Activity WebSocket Analytics Monitoring JMS JMS Weblogic

22 Internet of Things (IoT) and Big Data = one way = request/response IoT 2) Adding Event Hub and optional IoT Gateway

REST External Cloud SOAP Service Providers Mobile REST / SOAP Apps Service Bus D BPM and SOA B SOAP Platform API Gateway WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Services REST / SOAP Event BPM Suite 23Client Apps D REST / SOAP JMS B

ESB / Data Integration Enterprise Apps API Gateway Various WS Business REST REST Logic/Rules Event JMS DB REST Kafka IoT Smart REST Devices SQL Business Kafka / MQTT / REST REST Intelligence Visualization Business Activity IoT IoT REST MQTT WebSocket Monitoring Devices Gateways Analytics Event Hub Kafka REST Kura Kafka / MQTT / REST MQTT Kafka MQTT

23 Internet of Things (IoT) and Big Data = one way = request/response How to implement an Event Hub? Apache Kafka to the rescue Producer Producer Producer Distributed publish-subscribe messaging system Kafka Cluster Designed for processing of high-volume, real Consumer Consumer Consumer time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

does not implement JMS standard

Initially developed at LinkedIn, now part of Apache

24 Internet of Things (IoT) and Big Data Oracle’s Service Bus as a consumer of Kafka

Web Apps Service Bus 12c

Business Cloud API Cloud Service Cloud Apps Mobile Apps Proxy Pipeline Routing Kafka Service Sensor / IoT Kafka Business Backend REST Service REST Apps

Database

DB CDC Proxy Pipeline Business Backend Routing Apps WSDL Kafka Service Service WSDL Stream Processing

25 Internet of Things (IoT) and Big Data IoT 2) Solutions & Challenges

Solutions

• Event Hub solves the potential scalability issue of JMS

• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data

Challenges

• Where to do complex analytics on the events? Is it scalable?

• Can we really send all data down to backend? Network bandwidth?

26 Internet of Things (IoT) and Big Data IoT 3) Adding Stream Processing / Analytics in Backend

REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP Service Bus D BPM and SOA B API Gateway SOAP Platform WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite Client Apps REST / SOAP REST / SOAP 27 D JMS B

API Gateway Enterprise Apps Various WS Business REST REST Logic/Rules

(ESB) / Data Integration Event JMS DB

IoT Smart Kafka / MQTT / REST Devices SQL Stream Processing

Kafka Business Kafka / DB REST Kafka REST Intelligence MQTT / DB Visualization Business Activity IoT MQTT IoT REST Kafka WebSocket ESP / CEP Monitoring Event Hub Analytics Devices MQTT Gateways Kafka Stream Analytics Kafka

REST Kura REST MQTT Event Hub Kafka Kafka

27 Internet of Things (IoT) and Big Data = one way = request/response IoT 3) Solutions & Challenges

Solutions

• Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems

Challenges

• Can we really send all data down to backend? Network bandwidth?

28 Internet of Things (IoT) and Big Data Oracle’s Stream Analytics as consumer of Kafka/MQTT

Web Apps

Mobile Apps Kafka Kafka Stream Analytics Kafka Kafka Sensor / IoT MQTT

Oracle Stream Analytics Machine Data

DB CDC

29 Internet of Things (IoT) and Big Data IoT 4) Adding Industry 4.0 Data Sources (machine data)

REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP Service Bus D BPM and SOA B API Gateway SOAP Platform WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite Client Apps REST / SOAP REST / SOAP 30 D JMS B

API Gateway Enterprise Apps Various WS Business REST REST Logic/Rules

(ESB) / Data Integration Event JMS DB

IoT Smart Kafka / MQTT / REST Devices SQL Stream Processing

Kafka Business Kafka / Kafka DB REST Kafka REST Intelligence MQTT / DB Visualization MQTT Business Activity IoT IoT REST WebSocket Monitoring ESP / CEP Analytics Devices MQTT Gateways Kafka Stream Analytics Kafka

REST Kura REST Event Hub Event Hub Kafka Kafka I 4.0 Kafka / MQTT / REST Machine MQTT DB CDC GoldenGate

30 Internet of Things (IoT) and Big Data = one way = request/response Oracle’s GoldenGate for Change Data Capture of existing database

Machine Data

HDFS DB Pump Capture Delivery

Oracle GoldenGate JMS

Kafka Machine Data GoldenGate Gateway

Pump HBase Capture Delivery DB Oracle GoldenGate

31 Internet of Things (IoT) and Big Data IoT 5) Adding Stream Processing / Analytics at Edge

REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP Service Bus D BPM and SOA B API Gateway SOAP Platform WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite Client Apps REST / SOAP REST / SOAP 32 D JMS B

API Gateway Enterprise Apps Various WS Business REST REST Logic/Rules

(ESB) / Data Integration Event JMS DB

IoT Smart Kafka / MQTT / REST SQL Devices REST Stream Processing Kafka REST Business Kafka / Kafka DB Kafka Intelligence MQTT / MQTT DB Visualization Business Activity IoT IoT REST Kafka WebSocket ESP / CEP Monitoring Devices MQTT Gateways Analytics Stream Analytics Kafka ESP/CEP REST REST Event Hub Event Hub

Edge Analytics Kafka I 4.0 Kafka / MQTT / REST Machine MQTT DB CDC GoldenGate Kafka

32 Internet of Things (IoT) and Big Data = one way = request/response IoT 5) Solutions & Challenges

Solutions

• Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary

Challenges

• What if we need the raw events persisted? Where do to that?

33 Internet of Things (IoT) and Big Data IoT 6) Adding Raw Data Storage and Batch Analytics

REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP Service Bus D BPM and SOA B API Gateway SOAP Platform WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite Client Apps REST / SOAP REST / SOAP 34 D JMS B

API Gateway Enterprise Apps REST Various WS Business REST Logic/Rules Oracle Big Data Appliance Event (ESB) / Data Integration JMS DB Big Data Processing SQL IoT Smart HDFS DB Kafka / MQTT / REST SQL Devices Kafka Batch

Kafka Processing Business Kafka / Intelligence Hadoop / Spark HDFS MQTT / Kafka Visualization MQTT Business Activity IoT IoT REST Monitoring

MQTT WebSocket Analytics Devices Gateways Stream Processing ESP/CEP Kafka REST REST Event Hub DB Kafka Edge Analytics DB Kafka I 4.0 ESP/CEP Kafka / MQTT / REST Event Hub MQTT Machine Stream Analytics GoldenGate Kafka DB CDC Kafka

34 Internet of Things (IoT) and Big Data = one way = request/response IoT 6) Solutions & Challenges

Solutions

• Adding Big Data platform allows to store all raw data in the distributed file system in a scalable and reliable manner

Challenges

• How can we leverage the Big Data platform for more than just storing raw data? How does it combine with the stream processing?

35 Internet of Things (IoT) and Big Data Continuous Ingestion / Fan-In from the Edge

File Source Dataflow GW 37 Log37 Topic Event Hub DB Source Topic Log CDC Native Big Data CDC GW Dataflow GW Topic Log Dataflow DB Source CDC Topic

Log Log Topic

MQTT GW Connect

Social Topic Topic Stream Processing Native Topic REST Topic IoT Sensor REST

IoT Sensor IoT GW

IoT Sensor Queue

37 Internet of Things (IoT) and Big Data Challenges for Ingesting Sensor Data

• Multitude of sensors

• Multiple Firmware versions

• Bad Data from damaged sensors

• Data Quality

38 Internet of Things (IoT) and Big Data IoT 6a) Adding Data Mining / Machine Learning and Model execution REST SOAP External Cloud Service Providers Mobile Apps REST / SOAP Service Bus D BPM and SOA B API Gateway SOAP Platform WS Processes SOA Suite

Rich (Web) Oracle Data Integrator Event Services BPM Suite Client Apps REST / SOAP REST / SOAP 40 D JMS B

Kafka Processing Business Kafka / Intelligence Hadoop / Spark HDFS MQTT / Kafka Visualization MQTT Business Activity IoT IoT REST Monitoring

40 Internet of Things (IoT) and Big Data = one way = request/response IoT Reference Architecture

42 Internet of Things (IoT) and Big Data IoT Logical Reference Architecture Big Data / BI

Storage Storage

IoT Services Streaming Analytics IoT IoT Gateway Storage Device Streaming Event Event Bulk Analytics UI Analytics Hub Hub Sensor Storage App UI BPM and SOA Streaming Service Platform Actuator Storage Analytics Bus Event Services Processes Hub BLE Service

ZigBee API Service Bus WIFI MQTT Wired CoAP Bus Enterprise XMPP Applications DDS Endpoint Bulk REST UI AMQP Management Analytics SOAP App UI HTTP KAFKA KAFKA WIFI

43 Internet of Things (IoT) and Big Data IoT Logical Reference Architecture – Oracle Oracle Data Integrator Big Data Big Data / BI on premises Appliance Storage Storage

IoT Services Streaming Stream Analytics Analytics IoT IoT Gateway Storage Edge Analytics Device Streaming Event Event Bulk Analytics UI Analytics Hub Oracle IoT CS Hub Sensor Gateway Storage Oracle IoT CS App UI Client Library BPM and SOA Streaming Service Actuator Storage Stream Analytics SOA SuitePlatform Analytics Service BusBus Event Services Processes Hub BPM Suite BLE Service

ZigBee API Service Service Bus Bus WIFI MQTT Wired CoAP Bus Enterprise API Gateway XMPP Business ActivityApplications DDS Endpoint Bulk REST Monitoring UI AMQP Management Analytics SOAP App UI HTTP KAFKA KAFKA WIFI

44 Internet of Things (IoT) and Big Data IoT Logical Reference Architecture – Oracle Oracle Big Data CS Oracle Big Data Storage Big Data / BI Cloud Services Discovery CS Storage Oracle DataFlow ML CS Oracle IoT CS IoT Services Streaming Big Data Preparation CSAnalytics IoT IoT Gateway Storage Edge Analytics Oracle Streaming Device Streaming Event Event Analytics CSBulk Analytics UI Analytics Hub Oracle IoT CS Hub Sensor Gateway Storage Oracle BI CS Oracle IoT CS App UI Client Library Oracle SOA CSBPM and SOA Streaming Service Actuator Storage Platform Analytics Bus Oracle Process CS Event Services Processes Oracle Integration CS Hub BLE Service

ZigBee API Service Bus WIFI MQTT Oracle Messaging CS Oracle Mobile CS Wired CoAP Bus Enterprise XMPP Application Container ApplicationsCS DDS Endpoint Bulk REST UI AMQP Management Analytics SOAP App UI HTTP Container CS KAFKA KAFKA WIFI

45 Internet of Things (IoT) and Big Data IoT Logical Reference Architecture – Microsoft Azure Big Data / BI StorageHD InsightStorage

Azure Storage IoT Services Streaming Analytics IoT IoT Gateway Azure StorageStorage Azure Machine Device Streaming Event Learning Event Bulk Analytics UI Analytics Hub Hub Azure Power BI Sensor Storage Azure AppIoT Gateway UI Hub Azure Event Hub

SDK IoT BPM and SOA Azure Streaming Streaming Service Actuator Storage Platform AnalyticsAnalytics Bus Event Azure Services Processes Hub BLE Service

ZigBee API Service Bus WIFI MQTT Wired CoAP Bus Enterprise

XMPP Azure Service Bus Applications DDS Endpoint Bulk REST UI AMQP Management Analytics SOAP App UI HTTP KAFKA KAFKA WIFI

46 Internet of Things (IoT) and Big Data IoT Logical Architecture – Open Source Big Data / BI

Storage Storage

ZigBee API Service Bus WIFI MQTT Wired CoAP Bus Enterprise XMPP Applications DDS Endpoint Bulk REST UI AMQP Management Analytics SOAP App UI HTTP KAFKA KAFKA WIFI

47 Internet of Things (IoT) and Big Data Summary

48 Internet of Things (IoT) and Big Data Summary

Treat events as events! Infrastructures for handling lots of events are available!

IoT tends to make Big Data / Fast Data infrastructures necessary

Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load?

• Do I have to handle huge amount of events in “real-time”?

• Do I need to filter/aggregate data before invoking existing backend systems?

• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?

• What is the network bandwidth between device/gateway and cloud/backend?

• Centralized or Decentralized IoT solution?

49 Internet of Things (IoT) and Big Data J

50 Internet of Things (IoT) and Big Data Guido Schmutz Technology Manager [email protected]

51 Internet of Things (IoT) and Big Data