Nosql in O&G

NoSQL in O&G PPDM Q4 Data Management Luncheon – Fort Worth

Part 1 – What is the context? Part 2 – What is the challenge? Part 3 – What will cause breakthrough? Part 4 – How can NoSQL be applied? Thomas Tong CTO - Energy Part 5 – Who is MarkLogic?

 WTI prices are at a 6 year low, from $110 to $44  you are in a market share war  growth portfolio has halted; risk adverse  cashflow is very tight; operations are constrained; CAPEX is reduced  production is forecasted to reduce over the next 3 years  reserves replacement ratio is down 23%  loss of primary containment has increased 27%  13% of your leases are at risk  new GHG restrictions will reduce Bakken profits by 14%

You have some hard decisions to make…

You are now the CIO of an O&G company

 IT budgets have been rolled back to 2013 levels  IT enterprise portfolio investments in BI, DQ, MDM have not realized ROI  your relationship with the business is “limited”  cyber-attacks have increase 3000% over the past 5 years  74% of your next year budget is OPEX  most of your CAPEX is already allocated  all new projects going through IT governance are scrutinized closely  new compliance reporting must be on-time, no option  you must lay off 24% of your staff

You have some hard decisions to make…

WHAT IS THE CHALLENGE?

The “Great Divide” in O&G

 Complexity of business activity and operations has resulted in fragmented processes and “silos” of data/content, impacting leadership’s capacity to see the big picture  Silo culture is inherently reflected along business lines, geographies, and KPIs, such as “operations + production is king + basin”  O&G decentralized organizational structure supports localization, but not enterprise efforts  Relationships between business, operations and IT has historically been constrained

The “Great Divide” threatens O&G performance capacities, operational excellence, and overall commitment to HSE

The “Great Diversity” in O&G

 Complexity of business activity and operations demands specialization along the value chain of assets, producing unique data and content needs  Size and complexity of data files/formats inherently has caused application-centric mentality, resulting in a diverse IT portfolio  Data and content are historically separated align core processes and applications  Compliance needs continue to increase; from records management, GHG reporting, to reverse reporting

The “Great Diversity” threatens O&G capacity to manage across the business and gain competitive advantage

The “Great Divide and Diversity” in O&G

 Core processes segregated  Application centric per core process  Data type variance – size, format  Location variance  Tight Schema adoption  Temporal business rules  Application level security  Poor practices - hard coupling  Weak enterprise search  Broken semantics

Innovations you are experiencing

1. Three dimensional seismic 2. Directional drilling 3. Horizontal drilling 4. Secondary and tertiary recovery = Data 5. Hydraulic fracturing 6. Multistage fracturing

Data is Growing at a Staggering Rate – Web 2.0 Needs

44 ZB

8 ZB

2015 2020

Full Awareness – all sources of data

Location • regions • nodes • zones • hubs Time • hourly • daily • monthly • quarterly • yearly Types • natural gas molecules • electrons • pipeline • transmission capacity • generation capacity • storage volumes

Unprecedented Data Challenges

12% Structured 88% Unstructured

Reference Data

Warehouse OLTP

Data Marts Archives ?

Systems on Relational Databases – Not the Answer

Explosion of Inability of O&G Companies to Store, Manage, and Heterogeneous Data Search Their Data

50 44 ZB 40 Reference 30 Data 20 8 ZBs OLTP Warehouse 10 0 ? 2015 2020 Data Unstructured Structured Unstructured Archives Marts Data

Source: IDC

The Endless Cycle of Data Normalization

Take snapshot of current data Build master data model based 1 on initial view 2

4 Revise static model & Extract, transform, & 3 restart process for new data load data into data model

The Endless Cycle of Data Normalization

Take snapshot of current data Build master data model based 1 on initial view 2

2-5 years $5M++

4 Revise static model and restart Extract, transform, and 3 process for new data load data into data model

Treadmill of Adding Data

Articles Books ?

Industr y Data Reports ?

1. Design infrastructure, 3. Define queries & Service 4. Define schemas, indexes 2. Analyze Data Formats services & applications APIs and services

5. Build databases, 7. Load and normalize 8. Develop, integrate and middleware and services 6. Define & implement ETL data test infrastructure & infrastructure processes applications

Well-Known SQL Database Issues

. Badly named columns . Sparse data problem . Brittle extension capabilities . Slower query time for joins . Extract, Transform, Load (ETL) changes . Reporting tool changes

WHAT WILL CAUSE A BREAKTHROUGH?

First, a Quick History Lesson

Enterprise - NoSQL “Proven for mission critical” . Hardened & certified security . Integrated – search + app services Any Structure Era - NoSQL . Scalable + Hadoop integration “For all your data!” . ACID transactions . Massive scale . High availability & disaster recovery . Built for heterogeneous & unstructured data . Faster time-to-results Open Source - NoSQL . Commodity hardware “”Contextually right” . Fraction of the cost . Scalable . Designed for purpose Relational Era . Schema-agnostic “For all your structured data!” . Eventually consistent . Bad for unstructured . Less initial cost . Difficult for heterogeneous . Proprietary hardware . Expensive Hierarchical Era “For your application “data!" . Proprietary hardware . Very Expensive SLIDE: 19 © COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Core Differentiator: Purpose-built for the Enterprise

Open Source Relational Enterprise NoSQL NoSQL

ACID TRANSACTIONS ✔ ✔ ✗

SECURITY ✔ ✔ ✗

HIGH AVAILABILITY & DISASTER RECOVERY ✔ ✔ ✗

SCHEMA-AGNOSTIC ✗ ✔ ✔

SCALE-OUT ✗ ✔ ✔

ELASTIC ✗ ✔ ✗

TIERED STORAGE ✗ ✔ ✗

SEMANTICS ✗ ✔ ✗

A Change in DB Technology

PDF

Reduced Complexity with Faster Speed to Value

PDF

The Beauty of NoSQL

Take all of Ingest your Search and 1 2 3 your data data as-is query everything

Market Trends in the Energy Sector

. Cost to manage the complexity and volumes of data are accelerating . Services to establish data agility are profound and growing . Software will increase 6x in cost, but labor to make it work is much higher

What You Can Do With NoSQL

 Bring all your data together – regardless of type  Dynamically publish your content  Analyze all of your data  Tackle your most complex data challenges

What NoSQL Databases Offer

 Schema Flexibility  Free of Complex Joins  Horizontally Scalable  Compatible with Commodity Hardware  Self-contained  Rapid Application Development

NoSQL Flavors

1 Key-value  Model data as a search/index key and a value represented as an uninterpreted sequence of bytes. You can quickly read a record based on its key, but you can’t search value data across multiple records 2 Column-family  Like very large tables with zillions of rows and possible columns, but each row actually has a small number of columns compared with the total number possible. Programmers recognize this arrangement as a hash table or dictionary mapping a key to a set of key-value pairs 3 Document  Similar to key-value, except that the value associated with the key contains structured and semi-structured data – which is labeled a “document”. You can query against the structure, as well as elements within that structure, and return only portions of the document 4 Graph  The relationships among the various entities are the most important thing. In the graph, a node is a particular entity and the edges between the nodes are labeled a particular kind of relationship. An edge may have particular attributes

newSQL Live Analytics Wide-Column Key/Value Document

Hospital Graph/RDF Hospital Number performed at likes Hospital Name Surgeon Operation Complex Simple performed Surgeon Operation on schedules JSON Surgeon Number performed by Hospital Number Operation Number friend of Surgeon First Name performs Person Surgeon Last Name Surgeon Number at classifies Operation Code classified by works at

Operation Codes Key Key at Velocity Operation Code has poor fails at success at Hospital Operation Name

Cassandra Redis MongoDB Oracle Operational GemFire Oracle Exalytics Oracle x10 SAP HANA Hbase Memcached CouchDB Neo4j VoltDB memsql Accumulo DynamoDB Couchbase MarkLogic Aerospike Riak MarkLogic Titan Oracle NoSQL RavenDB OrientDB Cloudant Virtuoso Low Latency Low GemFire Jena

SQL Data Warehouse Doc Warehouse Big Data

Hospital Dimension Drug Dimension Hospital Key Drug Dose Facts Drug Key Hospital Attributes... Drug Attributes... Hospital Key Surgeon Key Operation Key Surgeon Dimension Drug Key Operation Dimension Surgeon Key Drug Dose Operation Key Surgeon Attributes... Operation Attributes...

Volume XML

Oracle DB Hospital Name: John Hopkins Operation Number: 13 Raw Operation Type: Heart Transplant Surgeon Name: Dorothy Oz

Drug Drug Dose Dose Name Manufacturer Size UOM MySQL Minicillan Drugs R Us 200 mg Maxicillan Canada4Less Drugs 400 mg Minicillan Drug USA 150 mg SQL Server PostgreSQL Oracle Exadata

Analytical DB2 Teradata SQLite Hive Solr SAP AS Netezza ElasticSearch Hadoop Informix Vertica MarkLogic Splunk SAP HANA Greenplum Sphinx MariaDB SAP IQ Dimensional Wide-column/Key-value High Bandwidth Bandwidth High Relational Document Graph Raw More structure (schema) Less structure (schemaless)

newSQL Live Analytics Wide-Column Key/Value Document

Operation Codes Key Key at Velocity Operation Code has poor fails at success at Hospital Operation Name

Cassandra Redis MongoDB Oracle Operational GemFire Oracle Exalytics Oracle x10 SAP HANA Hbase Memcached CouchDB Neo4j VoltDB memsql Accumulo DynamoDB Couchbase MarkLogic AerospikeExample Riak MarkLogic Titan Oracle NoSQL RavenDB OrientDB of Multi- Cloudant Virtuoso Low Latency Low GemFire Jena

model SQL Data Warehouse Doc Warehouse Big Data

Hospital Dimension Drug Dimension Hospital Key Drug Dose Facts Drug Key Hospital Attributes... Drug Attributes... Hospital Key NoSQL Surgeon Key Operation Key Surgeon Dimension Drug Key Operation Dimension Surgeon Key Drug Dose Operation Key Surgeon Attributes... Operation Attributes...

Volume XML

Oracle DB Database Hospital Name: John Hopkins Operation Number: 13 Raw Operation Type: Heart Transplant Surgeon Name: Dorothy Oz

Drug Drug Dose Dose Name Manufacturer Size UOM MySQL Minicillan Drugs R Us 200 mg Maxicillan Canada4Less Drugs 400 mg Minicillan Drug USA 150 mg SQL Server PostgreSQL Oracle Exadata

newSQL Live Analytics Wide-Column Key/Value Document

Operation Codes Key Key at Velocity Operation Code has poor fails at success at Hospital Operation Name

SQL Data Warehouse Doc Warehouse Big Data

Volume XML

Oracle DB Hospital Name: John Hopkins Operation Number: 13 Raw Operation Type: Heart Transplant Surgeon Name: Dorothy Oz

Drug Drug Dose Dose Name Manufacturer Size UOM MySQL Minicillan Drugs R Us 200 mg of MultiMaxicillan Canada4Less Drugs 400 -mg Minicillan Drug USA 150 mg SQL Server PostgreSQL Oracle Exadata model

Analytical DB2 Teradata SQLite Hive SQL Solr SAP AS Netezza Database ElasticSearch Hadoop Informix Vertica MarkLogic Splunk SAP HANA Greenplum Sphinx MariaDB SAP IQ Dimensional Wide-column/Key-value High Bandwidth Bandwidth High Relational Document Graph Raw More structure (schema) Less structure (schemaless)

Flexible Data Model Schema-agnostic, structure-aware

. No need to define up front . Matched to complex content and metadata data modeling . Data is managed in its most accessible, natural form . XML, JSON, RDF, geospatial

Result: All data from multiple sources available to provide a full story around “X”

Hierarchical Data Model

. NoSQL “document-centric” database . Supports any-structured data via hierarchical data model . Stores compressed trees Document Pipe

Title Metadata Location Features Author Section Equipments Repair Protection Last First Start Section Cross Pumps Tubes Section Section Section End Meters

NoSQL is Schema-Agnostic JSON is self-describing: { "title": "Well Report", "date": "2005-07-23Z", , "type": "Well Header", "UWI": : 05889945699657, "survey:" { "operator": "ConocoPhillips", "status": "Completed" , }, "total depth (ft)": 7789 "location": [ 37.49705, -122.363319 ,] "description""description": "A"Well blue Header van…" Report of a well in Brazos County in Eagleford, play" "meta": [ { "triple": {"subject ": "Well" , "predicate" : "is-a," "object": "Gas Well" }}, { "triple": {"subject ": “Well" " ,"predicate": “drilled, -by" ,"object": "Ocean Drilling & Exploration"}} ] }

How can NoSQL help? “Schema Agnostic”

360˚ View Notes: corrosion Asset Asset on battery Name Model terminals Unstructured