W3C Video on the Web Workshop 12-13 December 2007, San Jose, California

Video on the Semantic Sensor Web

Amit Sheth with Cory Henson, Prateek Jain, Josh Pschorr and Terry Rapoch

LexisNexis Ohio Eminent Scholar Kno.e.sis Center, Wright State University http://knoesis.wright.edu Outline

1. Background on the Sensor Web

2. Introducing the Semantic Sensor Web

3. Prototyping the Semantic Sensor Web Background on the Sensor Web Open Geospatial Consortium

• Consortium of 330+ companies, government agencies, and academic institutes • Open Standards development by consensus process OGC Mission • Interoperability Programs provide end-to-end implementation and testing before spec approval To lead in the • Standard encodings, e.g. development, – GeographyML, SensorML, Observations & Measurements, TransducerML, etc. promotion and • Standard Web Service interfaces, e.g. harmonization of – – open spatial – standards – Catalog Service – Sensor Web Enablement Services (Sensor Observation Service, Sensor Alert Service, Sensor Process Service, etc.)

http://www.opengeospatial.org/projects/groups/sensorweb Sensor Web Enablement

Constellations of heterogeneous sensors Vast set of users and applications

Satellite

Airborne

Sensor Web Enablement

Weather Surveillance

• Distributed self-describing sensors and related servicesNetwork Services • Link sensors to network and network- centric services Biological Chemical • Common XML encodings, information Detectors Detectors models, and metadata for sensors and observations • Access observation data for value added processing and decision support applications • Users on exploitation workstations, web Sea State browsers, and mobile devices

http://www.opengeospatial.org/projects/groups/sensorweb SWE Languages and Encodings

Information Model Sensor and Processing for Observations and Description Language Sensing

Observations & Measurements SensorML (O&M) (SML)

GeographyML TransducerML (GML) (TML)

Common Model for Multiplexed, Real Geography Systems Time Streaming and Features Protocol

Sam Bacharach, “GML by OGC to AIXM 5 UGM,” OGC, Feb. 27, 2007. SWE Components – Web Services

Access Sensor Description and Command and Task Data Sensor Systems

Discover Services, SOS SPS Sensors, Providers, Data

SAS Dispatch Sensor Alerts to registered Catalog Users Service

Accessible from various types of clients from PDAs and Cell Phones to high end Clients Workstations

Sam Bacharach, “GML by OGC to AIXM 5 UGM,” OGC, Feb. 27, 2007. Introducing the Semantic Sensor Web Semantic Sensor Web

What is the Semantic Sensor Web?

• Adding semantic annotations to existing standard Sensor Web languages in order to provide semantic descriptions and enhanced access to sensor data

• This is accomplished with model-references to ontology concepts that provide more expressive concept descriptions

• By using model-references to link SML annotated sensor data with concepts within an OWL-Time ontology we attain a representation of temporal semantics on sensor data Model Reference

XLink • Used for describing links between resources in XML documents. • Several important attributes within XLink include: – type: describes the element type of the link (i.e., simple, extended) – role: semantic attribute that describes the meaning of resources within the context of a link – href: locator attribute that supplies the URI needed to find a remote resource

Other Model Reference Languages • SAWSDL: Defines mechanisms to add semantic annotations to WSDL and XML-Schema components (W3C Recommendation) • SA-REST: Defines mechanisms to add semantic annotations to REST-based Web services.

W3C, XML Linking Language, http://www.w3.org/TR/xlink Model Reference (SensorML)

Instant

Interval OWL-Time Ontology

12 Semantic Query

Semantic Temporal Query

• Model-references from SML to OWL-Time ontology concepts provides the ability to perform semantic temporal queries • Supported semantic query operators include: – contains: user-specified interval falls wholly within a sensor reading interval (also called inside) – within: sensor reading interval falls wholly within the user-specified interval (inverse of contains or inside) – overlaps: user-specified interval overlaps the sensor reading interval • Example SPARQL query defining the temporal operator ‘within’ Prototyping the Semantic Sensor Web Architecture

Data Collection

YouTube

Extraction & Metadata Creation Storage Query UI

Video Converted Conversion Videos

AVI SML SML Interface Google Maps (Xindice) Filtering & OCR

OWL-Time Time & Date OWL-Time JSP information (Jena) Interface SML OWL-Time Annotation Annotation Generation Generation

15 Temporal Data Extraction

Channel Minimal Suppression 1 8-neighbor median for ‘bad’ pixels 1

Temporal Minimal Suppression 2

Binarization via adaptive threshold 1

Tesseract OCR engine

Regular Expression parsing SensorML output

1. https://research.microsoft.com/~xshua/publications/pdf/2002_ISCAS_TimeStampOCR.pdf 16 2. http://www.informedia.cs.cmu.edu/documents/vocr_ieee98.pdf Prototype Application

Search

Keywords C- _J Sensor Type ~o

Túne

Start Date ~0 2--01-01 Start Tune @::00:00 End Date ~0 8--0 1-01 EndTune @Iõo:OO Query Type lwithin Order [ãS"Cã"nding

Space

Latitude ê7B1 547~ Longitude [§6401395í Radius @:'.1so --=i Query Type lwithin 1

r'. ~ Semantic W3v ~ ll"' Web

!Illifttílillllll ! WRIGHT STATE http://130.108.28.48:8080/sensorweb/prototype.jsp 17 UNIVERSITY Conclusion

Future Work

• Incorporation of spatial ontology in order to include spatial analytics and query (perhaps with OGC GML Ontology or ontology developed by W3C Geospatial Incubator Group - GeoXG)

• Thematic query with keyword search through YouTube API and model reference to domain ontology

• Extension of SPARQL with enhanced spatiotemporal analytics and query

• Integration of framework with emergent applications, including video on mobile devices running Android OS

W3C, XML Linking Language, http://www.w3.org/TR/xlink References

• Kno.e.sis project on Semantic Sensor Web: http://knoesis.wright.edu/projects/sensorweb/

• Open Geospatial Consortium, Sensor Web Enablement WG, http://www.opengeospatial.org/projects/groups/sensorweb

• W3C, Time Ontology in OWL, http://www.w3.org/TR/owl-time/

• W3C, Geospatial Incubator Group, http://www.w3.org/2005/Incubator/geo/

• Amit Sheth et al., SA-REST: Semantically Interoperable and Easier-to-Use Services and Mashups, IEEE Internet Computing, November/December 2007 (Vol.11, No.6) pp.91- 94. DOI: http://doi.ieeecomputersociety.org/10.1109/MIC.2007.133

• W3C, Semantic Annotations for WSDL and XML Schema, http://www.w3.org/TR/sawsdl/

• W3C, XML Linking Language, http://www.w3.org/TR/xlink/

• Google Code, Tesseract, http://code.google.com/p/tesseract-ocr/