March 30th 2021 - Starts at 3PM
CEF Online Webinar Context Broker A deeper look into Linked Data, Data Models and Performance
1 Instructions for the live webinar:
This is an interactive session. Please ask your questions Please note that this webinar will be recorded. in the chat unanimously
2 Welcome to the live CB webinar Agenda for today
CEF Welcome and Introduction Erbil Kop – Onboarding Manager DG DIGIT Context Broker in a nutshell Stefano de Panfilis – COO FIWARE Foundation
Linked Data Jason Fox – Technical Evangelist FIWARE Foundation Data Models Alberto Abella – Data Modelling Expert FIWARE Foundation Performance Stefan Wiedemann – Kubernetes and Performance expert 5 FIWARE Foundation
Q&A Ken Zangelin – Senior Technical Expert and Evangelist 6 FIWARE Foundation 3 A deeper look into Linked Data, Data Models and Performance 01 CEF Welcome and Introduction
4 How can CEF Digital help you Our mission is to increase the interconnection between trans-European networks, by helping public administrations, businesses and citizens fully benefit from what digital has to offer.
We promote the adoption of common digital standards by:
• Providing technical support to Public Administrations and businesses in their digital transition;
• Helping them develop secure, interoperable digital services;
• Providing funding to projects that can contribute to a more connected Europe;
This support package comes in the form of Building Blocks.
5 Meet the Building Blocks Find the one you need
Big Data Test Infrastructure Context Broker eArchiving European Blockchain Services eInvoicing Explore and experiment Gather, manage and share Facilitates the preservation, Infrastructure Promote the implementation of the with big data for improved context data, in real-time, migration, reuse and trust Harness the power of a Europeanwide European standard for electronic performance and decision making. throughout Europe of your data. network of blockchain services, invoicing across border increasing trust through data security, privacy and transparency..
eID eDelivery eSignature Once Only Principle eTranslation Allow citizens to prove who they are Exchange online data and Create and verify electronic Re-use data held by Public Use machine translation to across borders, making it easier to documents reliably and signatures between businesses Administrations translate your documents and web access online services in another securely and EU citizens content into any official EU EU Member State language, and many more
6 How to use a Building Block? There are 3 options: buy, build or reuse and you can always co-develop your solution with other parties.
Buy Build Reuse
Buy a compliant, Build an EU-compliant solution from Reuse a sample soſtware interoperable solution scratch based on Building Block available through from the market. standards. We’ll help you test it for CEF Digital. compliance and interoperability.
Whatever you choose, the relevant CEF Digital team will support you in implementing the Building Block into your project.
7 Why choose CEF Digital Here are some of the benefits of using a Building Block in your digital project.
Faster Safer Future-proof Funding • The Building Blocks are • The Building Blocks are • Access a larger market • You can visit INEA’s website mature, ready-to-deploy based on open European thanks to interconnectivity for more information on digital solutions; standards, so you avoid how to apply, or you can vendor lockin; • Building Blocks can be contact the Building Block’s • This saves you time used in any digital onboarding manager. and money; • Our Building Block European project; team ensures your • Up to 75% of your costs • The CEF Digital website project is EU-compliant • Building Blocks are based are eligible for funding. has details on a range of and interoperable. on standards and ensure services available to you comply with EU help your Building Block \regulation. implementation. 8 A deeper look into Linked Data, Data Models and performance Stefano de Panfilis
COO 02 Fiware Foundation Context Broker in a nutshell
9 The new Digital Life gravitates around context information. Context information describes what is going on, where, when, why …
… i.e. implementing Smart Solutions means gathering and managing data from many different heterogeneous sources! 10 Today data is often organized in “silos”
Sensor Data Smart e-Government from Cities Smart Smart Smart Home Smart Mobility Smart Energy Parking Light Waste
Source: GovDelivery 11 What is the Context Broker Enable Organizations – from public administration to business - to collect, manage and share context information
A system able to inform in right Context information support the time what is currently happening adoption of smart decision
Objective 1 Managing Real time Data gathered from the different vertical systems within an organization generating a holistic view on what is currently going on within the organization.
Objective 2 Processing and analysis of data, referred as context information bringing support to take smart decisions or make smart automation of certain processes.
Objective 3 Core component for Open platform standards easing the development of smart solutions for collecting, managing and sharing context information. 12 Existence of a Context Information Management standard brings the opportunity to share information across domains/systems
Sensor Data Smart Smart e-Government from Cities Smart Smart Smart Smart Mobility Home Light Waste Energy Parking
Standard API Standard Context Information Management layer DE4Bios: A Covid-19 solution • Developed in 5 days • Adopted by various PAs in Europe • Leveraging existing data
14 DE4Bios: A Covid-19 solution • Developed in 5 days • Adopted by various PAs in Europe • Leveraging existing data
15 What is Context Broker?
Data Broker Leverage Investments Assemble information from different Enable the creation in the Digital Single systems, eventually belonging to different Market (of portable and interoperable organisations, instead of having them solutions) a data marketplace. perform in separate silos.
Context broker Gather, analyse and share context Easy Development data from multiple sources in real- Reduce costs and time in the development of time to make smart, data-driven context information based platforms and decisions. solutions. Experience has shown that it is possible to create meaningful dashboards in a few days!
16 A deeper look into Linked Data, Data Models and performance
Jason Fox
Technical Evangelist 03 FIWARE Foundation Linked Data in context-driven Smart systems
Why do we need it?
How do we use it? 17 My data is useful to me, but is more powerful shared with others … and what about Conway's law?
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.
— Melvin E. Conway
… how can I share data and benefit from other organizations if their organization “communicates” differently?
18 Two illustrative Linked Data Examples What is the context data? What are the barriers to data exchange?
Car Cross Border Parking Tourism 19 Linked Data: JSON to JSON-LD From: https://json-ld.org/
▪ JSON-LD is a lightweight Linked Data format. It is easy for humans to read and write. It is based on the already successful JSON format and provides a way to help JSON data interoperate at Web- scale.
▪ Linked Data empowers people that publish and use information on the Web. It is a way to create a network of standards-based, machine-readable data across Web sites. It allows an application to start at one piece of Linked Data and follow embedded links to other pieces of Linked Data that are hosted on different sites across the Web.
{ "@context": "https://json-ld.org/contexts/person.jsonld", "@id": "http://dbpedia.org/resource/John_Lennon", "name": "John Lennon", "born": "1940-10-09", "spouse": "http://dbpedia.org/resource/Cynthia_Lennon" }
20 Who is Harpreet Singh, the Blockchain in Smart Cities Expert?
21 URL: https://www.linkedin.com/in/singhhp1069/
22 Linked Data: NGSI v2 to NGSI-LD From: https://fiware-datamodels.readthedocs.io/en/latest/ngsi-ld_faq/index.html
▪ NGSI-LD is an evolution of the NGSI-9, NGSI-10, NGSI v2 information models and has been updated/improved to support linked data (entity relationships), property graphs and semantics (exploiting the capabilities offered by JSON-LD). This work has been conducted under the ETSI ISG Context Information Management initiative.
{ "@context": [ "https://fiware.github.io/data-models/context.jsonld", "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld" ], "id": "http://dbpedia.org/resource/John_Lennon", "type": "Person", "name": {"type": "Property", "value": "John Lennon"}, "born": {"type": "Property", "value": "1940-10-09"}, "spouse": {"type": "Relationship", "object": "http://dbpedia.org/resource/Cynthia_Lennon" } }
▪ Creating proper machine-readable Linked Data is fundamental to NGSI-LD. ▪ NGSI-LD Payloads are valid JSON-LD 23 NGSI-LD Properties: Creating an Entity
NGSI v2 NGSI-LD
curl -iX POST 'http://localhost:1026/v2/entities' \ curl -iX POST http://localhost:1026/ngsi-ld/v1/entities \ -H 'Content-Type: application/json' \ -H 'Content-Type: application/ld+json' \ -d '{ -d '{ "type": "Store", "id": "store001", "type": "Building", "id": "urn:ngsi-ld:Building:store001", "category": { "type": "Array", "value": ["commercial"]}, "category": { "type": "Property", "value": ["commercial"]}, "address": { "type": "PostalAddress", "value": { "address": { "type": "Property"," value": { "streetAddress": "Bornholmer Straße 65", "streetAddress": "Bornholmer Straße 65", "addressRegion": "Berlin", "addressRegion": "Berlin", "addressLocality": "Prenzlauer Berg", "addressLocality": "Prenzlauer Berg", "postalCode": "10439" "postalCode": "10439" }, }, "metadata": { "verified": { "type": "Property", "value": true } "verified": { "type": "Boolean","value": true} }, } "location": { "type": "GeoProperty", }, "value": { "type": "Point", "coordinates": [13.3986, 52.5547]} "location": {"type": "geo:json", }, "value": {"type": "Point", "coordinates": [13.3986, 52.5547]} "name": { "type": "Property", "value": "Bösebrücke Einkauf" }, }, "@context": [ "name": {"type": "Text", "value": "Bösebrücke Einkauf"} "https://fiware.github.io/data-models/context.jsonld", }' "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld" ] }'
24 NGSI-LD Properties: Data Model
The NGSI LD data model is more complex than just the definitions of use are more rigid which lead to a navigable knowledge graph.
• Entities • Properties of Properties • Properties of Properties of Properties • Relationships of Properties of Properties • Properties of Relationships • Properties • Properties of Properties of Relationships
• • • Relationships Relationships of Properties Relationships of Properties of Relationships • Relationships of Relationships • Properties of Relationships of Properties • Values • Relationships of Relationships of Properties
• Properties of Relationships of Relationships 25 • Relationships of Relationships of Relationships NGSI-LD Properties: Data Model
The Entity Example Notes
Has an id urn:ngsi-ld:Building:store001 URI/URN. id must be unique.
Has a type. https://uri.fiware.org/ns/ ● Fully qualified URI of a well defined data model data-models#Building ● Short-hand strings for types, mapped to fully qualified URIs through the JSON-LD @context.
Has a series of properties name, address, category etc. This can be expanded into http://schema.org/address, which is known as a fully qualified name (FQN).
Has a series of a verified field for the address This is the equivalent of NGSI v2 metadata properties-of-properties
Has a series of relationships managedBy The object corresponds to the URI/URN of another data entity. Equivalent of NGSI v2 refXXX
Has a series of managedBy.since Holds additional information about a relationship. properties-of-relationships This is the equivalent of metadata about a refXXX property
Has a series of managedBy.subordinateTo holds the URI/URN of another relationship. relationships-of-relationships
26 NGSI-LD Properties: Reading Entity Data
curl -G -X GET \ 'http://localhost:1026/ngsi-ld/v1/entities' \ [ -H 'Link:
27 NGSI-LD Properties: What to call a location?
▪ location NGSI-LD core @context "@context": { ▪ "ngsi-ld": "https://uri.etsi.org/ngsi-ld/", locatedAt "id": "@id", "type": "@type", ▪ geocoordinate "value": "https://uri.etsi.org/ngsi-ld/hasValue", "object": { ▪ geocoordinates "@id": "https://uri.etsi.org/ngsi-ld/hasObject", "@type": "@id" ▪ place }, "Property": "https://uri.etsi.org/ngsi-ld/Property" "Relationship": "https://uri.etsi.org/ngsi-ld/Relationship", ▪ ubicación ... etc. "unitCode": "https://uri.etsi.org/ngsi-ld/unitCode", ▪ standort "location": "https://uri.etsi.org/ngsi-ld/location", ... etc. ▪ 置き場所 }
With NGSI-LD core @context a location is always https://uri.etsi.org/ngsi-ld/location
28 NGSI-LD Properties: @vocab and Enumerated Values With NGSI-LD Data Models, attributes and enums are well-defined in a computer-readable fashion
My Application’s public @context ▪ An “address” is: "@context": { https://schema.org/address "type": "@type", "id": "@id", ▪ A “category” is: "schema": "https://schema.org/", https://uri.fiware.org/ns/data-models#category "fiware": "https://uri.fiware.org/ns/data-models#", ... etc. "address": "schema:address", ▪ A category is an enum which takes a "category": { set of values such as: "@id" :"fiware:category", https://uri.fiware.org/ns/data-models#commercial "@type": "@vocab" https://uri.fiware.org/ns/data-models#office }, https://uri.fiware.org/ns/data-models#retail "commercial": "fiware:commercial", https://uri.fiware.org/ns/data-models#residential "office": "fiware:office", "retail": "fiware:retail", "residential": "fiware:residential", enum is in progress. ... etc. }
29 Useful links
JSON-LD ▪ Website: https://json-ld.org/ ▪ Linked Data Video: https://www.youtube.com/watch?v=vioCbTo3C-4 ▪ JSON-LD Video: https://www.youtube.com/watch?v=4x_xzT5eF5Q NGSI-LD ▪ ETSI Specification: https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.03.01_60/gs_cim009v010301p.pdf ▪ NGSI-LD Video: https://www.youtube.com/watch?v=rZ13IyLpAtA ▪ Tutorials: https://ngsi-ld-tutorials.readthedocs.io/ Smart Data Models ▪ Website: http://smartdatamodels.org/ ▪ Smart Cities Data Models Video: https://www.youtube.com/watch?v=dfMo0HnaIUQ
30 Thank You !
Jason Fox Technical Evangelist FIWARE Foundation [email protected] https://www.linkedin.com/in/jason-fox-8a79563/
31 31 A deeper look into Linked Data, Data Models and performance
Alberto Abella Data Model Expert 04 FIWARE Foundation Data Models
32 Agenda for today
Smart Data Models Initiative Description of the initiative
Using the data models Services for users of the data models
Contributing to the data models Services for contributors to new or updated data models Next steps Extension of the data models and its supporting organizations Smart Data Models initiative
▪ For free, for all, forever
▪ AGILE standardization initiative by TMForum +IUDX + FIWARE Foundation
▪ A community site with detailed data models
− Single source of truth + examples
− Automatic generation and translation of specifications and other docs
▪ Structured into subjects and domains
▪ Coherent and consistent across different domains Smart Data Models initiative
▪ To provide extended usefulness to FIWARE platform users
▪ Integrated with other platforms (csv, soon SQL, etc)
▪ Using open licensing to allow extensive use and adoption
▪ Used in real case scenarios
▪ Based on git platform and github as development frontend
▪ Based on open and widely adopted standards, ontologies and international schemas (i.e. schema.org) Smart Data Models initiative DATA-MODELS - Official list of data models, subjects and domains - Guides for coding new data models - Template for new data models and examples data-models Umbrella repo - Directory for scripting tools to check data models - @context for json-ld - Commons elements DOMAINS REPOSITORIES Readme pointing to the Smart Smart Smart Smart Smart Smart Smart Smart Smart list of subjects Water Cities Environment Sensoring Destination Agrifood Robotics Manufacturing Energy General info or shared Smart Aeronautics Smart Health resources SUBJECTS’ REPOSITORIES Readme pointing to the list of data models for the objects Subject 1 Subject 2 Subject 3 Subject 4 Contributors.md (sewage) (parking) (weather) (...) subject-schema.json LIFECYCLE MANAGEMENT REPOSITORIES DATA MODELS README.md Incubated Harmonization /doc/spec.md /examples schema.json Current Adopters Using the data models
1. Search by property name, subject, data Model, data type, etc (12600 elements) http://bit.ly/searchSDM 2. Create a NGSI example payload for a data model. http://bit.ly/device_example 3. Last news https://smartdatamodels.org 4. @smartdatamodels 5. Subscribe to the newsletter
6. Disseminate your project using a data model Using the data models
1. Documents to be contributed 1. Schema (json schema) 2. Examples of payloads 3. Contributors.yaml (voluntary) 4. Adopers.yaml (voluntary) 2. Automatically generated contents 1. README.md subject and data model 2. Model.yaml 3. Swagger.yaml 4. Csv exports 5. LICENSE.md 6. spec.md, spec-ES.md, spec-FR.md Contributing to data models
1. Validate the examples against a data model
2. Check the documentation of a new data Model
3. Credit the authors of a data model.
4. Contribution manual: https://bit.ly/contribution_manual
5. Check your schema before contributing it
6. Create a schema out of a google spreadsheet
7. Harmonization data models in adaptation
8. Incubated data models in development Next steps
● Promote the interoperability by extending the exports
● Extending the steering board and governing bodies
● Keep translating regulations (i.e. Energy CIM, schema.org)
● Simplify contribution and use (survey users)
● Increase over the 12.600 searchable elements and 37 subjects and growing (Energy, Water,
Mobility, Agrifood, Smart Cities, Health, Buildings, etc) Thank You !
Alberto Abella Data Modelling Expert FIWARE Foundation [email protected] @smartdatamodels
FIWARE Foundation
41 A deeper look into Linked Data, Data Models and performance
Stefan Wiedemann Kubernetes and Performance expert 05 FIWARE Foundation Performance How to dimension your framework to serve the performance you need
42 Know your use-case Decide about the core measurements to optimize your broker for
Read vs. write Update size Subscriptions • Focus on high • Single entity/attribute • Required at all? frequency updates updates • Multiple Subscribers • Many parallel retrieval • Batch updates of varying • Single/few subscribers requests sizes • Complex subscriptions queries • Mixture of both • Few high-frequency clients • Full data updates • Many parallel clients
43 Be aware of your infrastructure Setup the infrastructure to enable the required performance
• Underlying infrastructure should provide flexibility for up and down-scaling • Automate configuration and deployment • See: https://github.com/FIWARE/helm-charts/tree/main/charts/orion • Proper operational tooling • Logging • Monitoring • Alerting
44 Images: https://github.com/cncf/artwork, https://commons.wikimedia.org Test the assumptions Assert your setup, before releasing it to the public
• Load-tests for running against the NGSI-LD/V2 context broker • https://github.com/FIWARE/orion-loadtest • Gatling framework, multiple scenarios implemented • Reports for multiple sizes available, including all configurations • Execute tests from local (mvn, gatling reporter) or distributed in the cluster (via kubernetes jobs) • Can be used to evaluate different configurations for specific use-cases
45 The Test-framework Walk through the git repo
https://github.com/FIWARE/orion-loadtest
46 Q&A time
15 min 47 Ready to get started?
Reach out to us to learn more! Visit us at https://ec.europa.eu/cefdigital/
Follow us on social media
@Connecting_EU Connecting Europe
Connecting Europe
48