DOCSLIB.ORG

And Template-Based Data Compression Approach to Improve Resource-Constrained Iot Systems Interoperability

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
And Template-Based Data Compression Approach to Improve Resource-Constrained Iot Systems Interoperability Department OF Computer Architecture AND TECHNOLOGY ACONTEXT- AND TEMPLATE-BASED DATA COMPRESSION APPROACH TO IMPROVE RESOURCE-CONSTRAINED IOTSYSTEMS INTEROPERABILITY Dissertation FOR THE DEGREE OF Doctor OF PhilosophY Candidate Jorge Berzosa Supervisors Roberto Cortiñas - Luis Gardeazabal 2019 (cc)2019 JORGE BERZOSA MACHO (cc by-nc 4.0) AbstrACT The Internet OF Things (IoT) HAS EMERGED AS A POWERFUL PARADIGM WITH A HUGE RANGE OF possibilities, PROMOTING ITS ADOPTION ACROSS MULTIPLE TECHNOLOGICAL domains. Roughly speaking, THE IoT ABSTRACTION AIMS TO INTERCONNECT EVERY KIND OF THINGS, LIKE SIMPLE de- VICES (a LIGHT BULB OR A thermostat) OR MORE COMPLEX AND ABSTRACT systems, e.g., FACILITY management. Behind THESE THINGS, THERE ARE PHYSICAL DEVICES TASKED WITH SPECIfiC SENSING OR ACTUATION roles. Similarly TO THE THINGS themselves, THESE DEVICES OFTEN HAVE signif- ICANT DIFFERENCES AMONG THEM IN TERMS OF CAPABILITIES AND THE SET OF COMMUNICATION TECHNOLOGIES THEY use. This HETEROGENEITY LEADS TO AN INTEGRATION CHALLENGE REGARDING INTEROPERABILITY AT THE CONNECTIVITY level, INCLUDING DATA representation. A COMMON APPROACH TO DEAL WITH INTEROPERABILITY IN IoT SYSTEMS IS TO re-use Internet MATURE TECHNOLOGIES AND approaches. At DATA level, THIS CAN BE IMPLEMENTED BY structur- ING DATA FOLLOWING A STANDARD DATA MODEL AND USING text-based DATA formats, e.g., XML. However, THE TYPE OF DEVICES USUALLY DEPLOYED IN IoT SYSTEMS HAS LIMITED CAPABILITIES AS WELL AS SCARCE PROCESSING AND COMMUNICATION resources. Due TO THESE restrictions, text- BASED DATA FORMATS CANNOT BE INTEGRATED INTO resource-constrAINED DEVICES AND NETWORKS IN AN EASY AND EffiCIENT WAY. Consequently, THESE LIMITATIONS ALSO APPLY TO INTEROPERABLE TECHNOLOGIES THAT RELY ON text-based DATA FORMATS SUCH AS WEB Services. In THIS Thesis, WE PRESENT A NOVEL DATA COMPRESSION APPROACH FOR text-based DATA for- mats, NAMELY Context- AND Template-based Compression (CTC), WHICH IS SPECIALLY DESIGNED TAKING INTO ACCOUNT THE LIMITATIONS OF resource-constrAINED DEVICES AND networks. CTC ENHANCES data-level INTEROPERABILITY IN IoT SYSTEMS WHILE KEEPING VERY LOW RESOURCE re- QUIREMENTS (in TERMS OF COMMUNICATION bandwidth, MEMORY SIZE AND PROCESSING power). This Thesis ALSO INCLUDES THE SPECIfiCATION OF A SET OF COMPLEMENTARY SOLUTIONS WHICH FACILITATE THE DEPLOYMENT OF CTC IN IoT NETWORKS AND ITS INTEGRATION INTO APPLICATIONS TARGETED TO resource-constrAINED devices. Interestingly, CTC IS DESIGNED WITH INTEROPERABILITY AND EXTENSIBILITY IN MIND SO THAT IT CAN BE APPLIED TO DIFFERENT DATA formats. This WORK ALSO INCLUDES THE EVALUATION OF THE PROPOSED SOLUTION FOR TWO POPULAR DATA formats, XML AND JSON, BOTH IN REAL AND SIMULATED scenarios. ADDITIONALLY, THE RESULTS OF THE EVALUATIONS HAVE BEEN COMPARED WITH SOME OTHER CURRENT DATA COMPRESSION approaches, SHOWING THAT CTC IS A SUITABLE CANDIDATE FOR DATA COMPRESSION IN resource-constrAINED IoT DEPLOyments. II Laburpena Gauzen Interneta (THE Internet OF Things, IoT) AUKERA ANITZ ESKAINTZEN DITUEN PARADIGMA BIHURTU da, BERE ERABILERA DOMEINU TEKNOLOGIKO DESBERDINETAN sustatuz. IoT-ren HELBURU NAGUSIA MOTA ASKOTAKO GAUZAK ELKAR KONEKTATZEA da: BONBILA EDO TERMOSTATOA BEZALAKO GAILU SINPLEETATIK hasita, ETXEA EDO MAKINA MODUKO ELEMENTU ABSTRAKTU ETA Konplexuago- RAINO IRITSIZ . Gauza HAUEN atzean, SENTSORE ETA AKTUAZIO ARDURA ESPEZIfiKOK DITUZTEN GAILU fiSIKOAK daude. Gauzekin GERTATZEN DEN bezala, GAILU HAUEK ELKARRENGAN OSO EZBERDINAK DIRa, BEREZIKI GAITASUNEI ETA ERABILTZEN DITUZTEN TEKNOLOGIEI dagokienez. Heterogeneota- SUN HONEK KONEKTAGARRITASUN MAILAKO ELKARRERAGINGARRITASUNEAN INTEGRAZIOA ERRONKA HANDIA AURKEZTEN du, DATUEN ERREPRESENTAZIOA barne. IoT sistemetan, ELKARRERagingarritasuna, Interneten DENBORA LUZEZ ERABILI DIREN TEKNOLOGIEKIN JORRATZEA OHIKO JOERA da. Planteamendu honek, DATUEN ERREPRESENTAZIO mailan, DATUAK EREDU ESTANDAR BAT JARRAITUZ EGITURATZEA ESKATZEN du, BAITA TESTUAN oinar- RITUTAKO DATU FORMATUAK ERABILTZEA ERE (esaterAKo, XML). IoT SISTEMETAN ERABILI OHI DIREN gailuak, aldiz, GAITASUN MUGATUAK ETA PROZESATZEKO ETA KOMUNIKATZEKO BALIABIDE URRIKOAK IZAN OHI DIRa. MurrizkETA HAUEK medio, TESTUAN OINARRITUTAKO DATU formatuak, BALIABIDE MURRIZTUKO GAILU ETA SAREETAN ERA ERRAZ ETA ERAGINKOR BATEAN INTEGRATZEA EZ DA posible. Are gehiago, MURRIZKETA hauek, TESTUAN OINARRITUTAKO DATU FORMATUAK ERABILTZEN DITUZTEN BESTELAKO TEKNOLOGIA ELKARRERAGILEETARA ERE HEDATZEN da, BESTEAK beste, WEB Zerbitzuak. TESI honetan, TESTUAN OINARRITUTAKO DATU FORMATUEN KONPRESIORAKO SOLUZIO BAT au- RKEZTEN da, BALIABIDE MURRIZTUKO GAILU ETA SAREEN MUGAK AINTZAT HARTUTA DISEINATUA IZAN dena. Soluzio HONEK Context- AND Template-based Compression, CTC, DU izena. CTC-ek IoT SISTEMEN DATU MAILAKO ELKARRERAGINGARRITASUNA HOBETZEA DU helburu, BEHARREZKO KOMUNIKAZIO BANDA ZABALERa, MEMORIA TAMAINA ETA PROZESAMENDU AHALMEN ERABILERA minimizatuz. Are gehiago, TESI honetan, CTC SOLUZIOA IoT SAREETAN ETA BALIABIDE murriz- TUKO GAILUETAN ERRAZAGO INTEGRATZEKO bidean, HAINBAT SOLUZIO OSAGARRI ZEHAZTU DIRa. CTC ELKARRERAGINGARRITASUNA ETA HEDAGARRITASUNA ARDATZ IZANIK DISEINATU da, DATU FORMATU EZBERDINETAN APLIKAGARRIA izanik. ProposatutakO SOLUZIOA BI DATU FORMATUREKIN EBALUATU da, XML ETA JSON, INGURUNE SIMULATU ETA errealetan. Ebaluazio HAUEN emaitzak, DATU KONPRESIOA JORRATZEN DUTEN GAUR EGUNGO BESTE SOLUZIO BATZUEN EMAITZEKIN KONPARATU DIRa. KONPARAKETA honetan, CTC BALIABIDE MURRIZTUKO IoT SISTEMEN DATU KONPRESIORAKO HAUTAGAI ONA DELA IKUSI da. IV Resumen El Internet DE LAS Cosas (THE Internet OF Things, IoT) HA SURGIDO COMO UN PARADIGMA CON UN AMPLIO número DE posibilidades, PROMOVIENDO así SU adopción EN múltiples DOMINIOS tecnológicos. El OBJETIVO DE IoT ES EL DE INTERCONECTAR TODO TIPO DE COSAS, DESDE DISPOSITIVOS simples, COMO UNA BOMBILLA O UN termostato, A ELEMENTOS más COMPLEJOS Y ABSTRACTOS COMO UNA máquina O UNA casa. Detrás DE ESTAS COSAS SE ENCUENTRAN DISPOSITIVOS físicos QUE desempeñan ROLES específiCOS DE SENSOR O activador. De MANERA SIMILAR A LAS COSAS, ESTOS DISPOSITIVOS varían ENORMEMENTE ENTRE sí, ESPECIALMENTE EN LAS CAPACIDADES QUE POSEEN Y EL TIPO DE tecnologías QUE utilizan. Esta HETEROGENEIDAD GENERA UNA GRAN COMPLEJIDAD EN LOS PROCESOS INTEGRación EN LO QUE A LA INTEROPERABILIDAD SE REfiere, INCLUYENDO LA representación DE LOS datos. Un ENFOQUE HABITUAL PARA ABORDAR LA INTEROPERABILIDAD EN SISTEMAS IoT ES EL DE REUTILIZAR tecnologías DE Internet YA consolidadas. Más concretamente, A NIVEL DE repre- sentación DE LOS datos, LOS DATOS SE PUEDEN REPRESENTAR SIGUIENDO UN MODELO DE DATOS estándar, así COMO FORMATOS DE DATOS BASADOS EN texto, e.g., XML. Sin embargo, nor- MALMENTE EL TIPO DE DISPOSITIVOS QUE SE SE ENCUENTRA EN SISTEMAS IoT TIENE CAPACIDADES limitadas, así COMO RECURSOS DE PROCESAMIENTO Y DE comunicación escasos. Debido A ESTAS LIMITACIONES NO ES POSIBLE INTEGRAR FORMATOS DE DATOS BASADOS EN TEXTO DE MANERA SENCILLA YEfiCIENTE EN DISPOSITIVOS Y REDES CON RECURSOS restringidos. Además, ESTA situación también APLICA A tecnologías INTEROPERABLES QUE DEPENDEN DE FORMATOS DE DATOS BASADOS EN TEXTO como, POR ejemplo, LOS Servicios Web. En ESTA Tesis, PRESENTAMOS UNA NOVEDOSA solución QUE PERMITE UNA gestión EfiCIENTE DE LOS DATOS MEDIANTE LA compresión DE DATOS PARA FORMATOS DE DATOS BASADOS EN TEXTO Y QUE está ESPECIALMENTE diseñada TENIENDO EN CUENTA LAS LIMITACIONES DE DISPOSITIVOS Y REDES CON RECURSOS restringidos. Denominamos A ESTA solución Context- AND Template- BASED Compression (CTC). CTC MEJORA LA INTEROPERABILIDAD A NIVEL DE LOS DATOS DE LOS SISTEMAS IoT A LA VEZ QUE REQUIERE MUY POCOS RECURSOS EN CUANTO A ANCHO DE BANDA DE LAS comunicaciones, tamaño DE MEMORIA Y CAPACIDAD DE procesamiento. Esta TESIS también ESPECIfiCA UNA SERIE DE SOLUCIONES COMPLEMENTARIAS QUE FACILITAN EL DESPLIEGUE DE CTC EN REDES IoT, así COMO SU INTEGRación EN APLICACIONES ORIENTADAS A DISPOSITIVOS CON RECURSOS restringidos. Como ASPECTO destacable, CTC está diseñado CON LA INTEROPERABILIDAD Y EXTENSIBILIDAD EN mente, POR LO QUE PUEDE SER APLICADO A DIFERENTES FORMATOS DE datos. En ESTE TRABAJO SE PRESENTA LA evaluación DE LA solución PROPUESTA PARA DOS FORMATOS DE DATOS populares, XML Y JSON, TANTO EN ENTORNOS REALES COMO EN simulados. Además, LOS RESULTADOS DE ESTAS EVALUACIONES SE HAN CONTRASTADO CON SOLUCIONES DE compresión DE DATOS actuales. Los RESULTADOS MUESTRAN QUE CTC ES UN CANDIDATO válido PARA LA compresión DE DATOS EN DESPLIEGUES IoT CON RECURSOS restringidos. VI ACKNOWLEDGEMENTS A Maite, POR SU PACIENCIA Y APOYO CUANDO más LO necesitaba. Además, POR DECIRME QUE DESCONECTARA DE LA TESIS DE VEZ EN cuando. A MIS padres, PORQUE ME DIERON EL PRIMER empujón QUE ME HA TRaído HASTA aquí Y AHORa. A MIS HERMANOS Y hermana, POR CREER EN mí AUN SABIENDO COMO SOY. Nire zuzendarientzat, DOKUMENTU HONI FORMA EMAN DIOTEN fiNKetak, hobekuntzak, eta, bereziki, ZUZENKETA guztiengatik. VIII ACRONYMS 6LowPAN Internet Protocol (IPv6) AND Low-power Wireless Personal Area Networks ACK ACKNOWLEDGEMENT API Application ProgrAMMING Interface CBOR Concise Binary Object Representation CoAP ConstrAINED Application Protocol CoRE ConstrAINED RESTful Environments CPS Cyber PhYSICAL SYSTEM CPU CentrAL PROCESSING UNIT CTC Context- AND Template-based Compression DNS Domain Name SYSTEM DOM Document Object Model DPWS Devices ProfiLE FOR WEB Services DTD Document Type DefiNITION DTLS DatagrAM TrANSPORT LaYER Security EXI EffiCIENT XML Interchange EXIP EXIProcessor HTML Hyper TEXT Markup Language HTTP HyperTEXT TrANSFER Protocol IANA Internet Assigned Numbers AUTHORITY
Load more

Recommended publications
  • Document Databases, JSON, Mongodb 17
    MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-MIE-PDB/ Lecture 13: Document Databases, JSON, MongoDB 17. 5. 2016 Lecturer: Martin Svoboda [email protected] Authors: Irena Holubová, Martin Svoboda Faculty of Mathematics and Physics, Charles University in Prague Course NDBI040: Big Data Management and NoSQL Databases Document Databases Basic Characteristics Documents are the main concept Stored and retrieved XML, JSON, … Documents are Self-describing Hierarchical tree data structures Can consist of maps, collections, scalar values, nested documents, … Documents in a collection are expected to be similar Their schema can differ Document databases store documents in the value part of the key-value store Key-value stores where the value is examinable Document Databases Suitable Use Cases Event Logging Many different applications want to log events Type of data being captured keeps changing Events can be sharded by the name of the application or type of event Content Management Systems, Blogging Platforms Managing user comments, user registrations, profiles, web-facing documents, … Web Analytics or Real-Time Analytics Parts of the document can be updated New metrics can be easily added without schema changes E-Commerce Applications Flexible schema for products and orders Evolving data models without expensive data migration Document Databases When Not to Use Complex Transactions Spanning Different Operations Atomic cross-document operations Some document databases do support (e.g., RavenDB) Queries against Varying Aggregate Structure Design of aggregate is constantly changing → we need to save the aggregates at the lowest level of granularity i.e., to normalize the data Document Databases Representatives Lotus Notes Storage Facility JSON JavaScript Object Notation Introduction • JSON = JavaScript Object Notation .
    [Show full text]
  • Smart Grid Serialization Comparison
    Downloaded from orbit.dtu.dk on: Sep 28, 2021 Smart Grid Serialization Comparison Petersen, Bo Søborg; Bindner, Henrik W.; You, Shi; Poulsen, Bjarne Published in: Computing Conference 2017 Link to article, DOI: 10.1109/SAI.2017.8252264 Publication date: 2017 Document Version Peer reviewed version Link back to DTU Orbit Citation (APA): Petersen, B. S., Bindner, H. W., You, S., & Poulsen, B. (2017). Smart Grid Serialization Comparison. In Computing Conference 2017 (pp. 1339-1346). IEEE. https://doi.org/10.1109/SAI.2017.8252264 General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain You may freely distribute the URL identifying the publication in the public portal If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Computing Conference 2017 18-20 July 2017 | London, UK Smart Grid Serialization Comparision Comparision of serialization for distributed control in the context of the Internet of Things Bo Petersen, Henrik Bindner, Shi You Bjarne Poulsen DTU Electrical Engineering DTU Compute Technical University of Denmark Technical University of Denmark Lyngby, Denmark Lyngby, Denmark [email protected], [email protected], [email protected] [email protected] Abstract—Communication between DERs and System to ensure that the control messages are received within a given Operators is required to provide Demand Response and solve timeframe, depending on the needs of the power grid.
    [Show full text]
  • Arxiv:2103.14485V2 [Physics.Med-Ph] 12 Apr 2021
    ADWI-BIDS: AN EXTENSION TO THE BRAIN IMAGING DATA STRUCTURE FOR ADVANCED DIFFUSION WEIGHTED IMAGING James Gholam1,2, Filip Szczepankiewicz3, Chantal M.W. Tax1,2,4, Lars Mueller2,5, Emre Kopanoglu2,5, Markus Nilsson3, Santiago Aja-Fernandez6, Matt Griffin1, Derek K. Jones2,5, and Leandro Beltrachini1,2 1School of Physics and Astronomy, Cardiff University, Cardiff, United Kingdom 2Cardiff Univeristy Brain Research Imaging Centre (CUBRIC), Cardiff, United Kingdom 3Department of Diagnostic Radiology, Lund University, Lund, Sweden 4Image Sciences Institute, University Medical Center Utrecht, Utrecht, Netherlands 5School of Psychology, Cardiff University, Cardiff, United Kingdom 6Universidad de Valladolid, Valladolid, Spain ABSTRACT Diffusion weighted imaging techniques permit us to infer microstructural detail in biological tissue in vivo and noninvasively. Modern sequences are based on advanced diffusion encoding schemes, allowing probing of more revealing measures of tissue microstructure than the standard apparent diffusion coefficient or fractional anisotropy. Though these methods may result in faster or more revealing acquisitions, they generally demand prior knowledge of sequence-specific parameters for which there is no accepted sharing standard. Here, we present a metadata labelling scheme suitable for the needs of developers and users within the diffusion neuroimaging community alike: a lightweight, unambiguous parametric map relaying acqusition parameters. This extensible scheme supports a wide spectrum of diffusion encoding methods, from single diffusion encoding to highly complex sequences involving arbitrary gradient waveforms. Built under the brain imaging data structure (BIDS), it allows storage of advanced diffusion MRI data comprehensively alongside any other neuroimaging information, facilitating processing pipelines and multimodal analyses. We illustrate the usefulness of this BIDS-extension with a range of example data, and discuss the extension’s impact on pre- and post-processing software.
    [Show full text]
  • Spindle Documentation Release 2.0.0
    spindle Documentation Release 2.0.0 Jorge Ortiz, Jason Liszka June 08, 2016 Contents 1 Thrift 3 1.1 Data model................................................3 1.2 Interface definition language (IDL)...................................4 1.3 Serialization formats...........................................4 2 Records 5 2.1 Creating a record.............................................5 2.2 Reading/writing records.........................................6 2.3 Record interface methods........................................6 2.4 Other methods..............................................7 2.5 Mutable trait...............................................7 2.6 Raw class.................................................7 2.7 Priming..................................................7 2.8 Proxies..................................................8 2.9 Reflection.................................................8 2.10 Field descriptors.............................................8 3 Custom types 9 3.1 Enhanced types..............................................9 3.2 Bitfields..................................................9 3.3 Type-safe IDs............................................... 10 4 Enums 13 4.1 Enum value methods........................................... 13 4.2 Companion object methods....................................... 13 4.3 Matching and unknown values...................................... 14 4.4 Serializing to string............................................ 14 4.5 Examples................................................. 14 5 Working
    [Show full text]
  • Rdf Repository Replacing Relational Database
    RDF REPOSITORY REPLACING RELATIONAL DATABASE 1B.Srinivasa Rao, 2Dr.G.Appa Rao 1,2Department of CSE, GITAM University Email:[email protected],[email protected] Abstract-- This study is to propose a flexible enable it. One such technology is RDF (Resource information storage mechanism based on the Description Framework)[2]. RDF is a directed, principles of Semantic Web that enables labelled graph for representing information in the information to be searched rather than queried. Web. This can be perceived as a repository In this study, a prototype is developed where without any predefined structure the focus is on the information rather than the The information stored in the traditional structure. Here information is stored in a RDBMS’s requires structure to be defined structure that is constructed on the fly. Entities upfront. On the contrary, information could be in the system are connected and form a graph, very complex to structure upfront despite the similar to the web of data in the Internet. This tremendous potential offered by the existing data is persisted in a peculiar way to optimize database systems. In the ever changing world, querying on this graph of data. All information another important characteristic of information relating to a subject is persisted closely so that in a system that impacts its structure is the reqeusting any information of a subject could be modification/enhancement to the system. This is handled in one call. Also, the information is a big concern with many software systems that maintained in triples so that the entire exist today and there is no tidy approach to deal relationship from subject to object via the with the problem.
    [Show full text]
  • CBOR (RFC 7049) Concise Binary Object Representation
    CBOR (RFC 7049) Concise Binary Object Representation Carsten Bormann, 2015-11-01 1 CBOR: Agenda • What is it, and when might I want it? • How does it work? • How do I work with it? 2 CBOR: Agenda • What is it, and when might I want it? • How does it work? • How do I work with it? 3 Slide stolen from Douglas Crockford History of Data Formats • Ad Hoc • Database Model • Document Model • Programming Language Model Box notation TLV 5 XML XSD 6 Slide stolen from Douglas Crockford JSON • JavaScript Object Notation • Minimal • Textual • Subset of JavaScript Values • Strings • Numbers • Booleans • Objects • Arrays • null Array ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"] [ [0, -1, 0], [1, 0, 0], [0, 0, 1] ] Object { "name": "Jack B. Nimble", "at large": true, "grade": "A", "format": { "type": "rect", "width": 1920, "height": 1080, "interlace": false, "framerate": 24 } } Object Map { "name": "Jack B. Nimble", "at large": true, "grade": "A", "format": { "type": "rect", "width": 1920, "height": 1080, "interlace": false, "framerate": 24 } } JSON limitations • No binary data (byte strings) • Numbers are in decimal, some parsing required • Format requires copying: • Escaping for strings • Base64 for binary • No extensibility (e.g., date format?) • Interoperability issues • I-JSON further reduces functionality (RFC 7493) 12 BSON and friends • Lots of “binary JSON” proposals • Often optimized for data at rest, not protocol use (BSON ➔ MongoDB) • Most are more complex than JSON 13 Why a new binary object format? • Different design goals from current formats – stated up front in the document • Extremely small code size – for work on constrained node networks • Reasonably compact data size – but no compression or even bit-fiddling • Useful to any protocol or application that likes the design goals 14 Concise Binary Object Representation (CBOR) 15 “Sea Boar” “Sea Boar” 16 Design goals (1 of 2) 1.
    [Show full text]
  • Msgpack Documentation Release 1.0
    msgpack Documentation Release 1.0 Author 2021-03-18 Contents 1 API reference 3 2 Advanced usage 9 Python Module Index 11 Index 13 i ii msgpack Documentation, Release 1.0 MessagePack is a efficient format for inter language data exchange. Contents 1 msgpack Documentation, Release 1.0 2 Contents CHAPTER 1 API reference msgpack.pack(o, stream, **kwargs) Pack object o and write it to stream See Packer for options. dump() is alias for pack() msgpack.packb(o, **kwargs) Pack object o and return packed bytes See Packer for options. dumps() is alias for packb() msgpack.unpack(stream, **kwargs) Unpack an object from stream. Raises ExtraData when stream contains extra bytes. See Unpacker for options. load() is alias for unpack() msgpack.unpackb(packed, *, object_hook=None, list_hook=None, bool use_list=True, bool raw=False, int timestamp=0, bool strict_map_key=True, unicode_errors=None, object_pairs_hook=None, ext_hook=ExtType, Py_ssize_t max_str_len=-1, Py_ssize_t max_bin_len=-1, Py_ssize_t max_array_len=-1, Py_ssize_t max_map_len=-1, Py_ssize_t max_ext_len=-1) Unpack packed_bytes to object. Returns an unpacked object. Raises ExtraData when packed contains extra bytes. Raises ValueError when packed is incomplete. Raises FormatError when packed is not valid msgpack. Raises StackError when packed contains too nested. Other exceptions can be raised during unpacking. See Unpacker for options. max_xxx_len options are configured automatically from len(packed). loads() is alias for unpackb() 3 msgpack Documentation, Release 1.0 class msgpack.Packer(default=None, *, bool use_single_float=False, bool autoreset=True, bool use_bin_type=True, bool strict_types=False, bool datetime=False, uni- code_errors=None) MessagePack Packer Usage: packer= Packer() astream.write(packer.pack(a)) astream.write(packer.pack(b)) Packer’s constructor has some keyword arguments: Parameters • default (callable) – Convert user type to builtin type that Packer supports.
    [Show full text]
  • Advanced JSON Handling in Go 19:40 05 Mar 2020 Jonathan Hall Devops Evangelist / Go Developer / Clean Coder / Salsa Dancer About Me
    Advanced JSON handling in Go 19:40 05 Mar 2020 Jonathan Hall DevOps Evangelist / Go Developer / Clean Coder / Salsa Dancer About me Open Source contributor; CouchDB PMC, author of Kivik Core Tech Lead for Lana Former eCommerce Dev Manager at Bugaboo Former backend developer at Teamwork.com Former backend developer at Booking.com Former tech lead at eFolder/DoubleCheck 2 Show of hands Who has... ...used JSON in a Go program? ...been frustrated by Go's strict typing when dealing with JSON? ...felt limited by Go's standard JSON handling? What have been your biggest frustrations? 3 Today's Topics Very brief intro to JSON in Go Basic use of maps and structs Handling inputs of unknown type Handling data with some unknown fields 4 A brief intro to JSON JavaScript Object Notation, defined by RFC 8259 Human-readable, textual representation of arbitrary data Limted types: null, Number, String, Boolean, Array, Object Broad applications: Config files, data interchange, simple messaging 5 Alternatives to JSON YAML, TOML, INI BSON, MessagePack, CBOR, Smile XML ProtoBuf Custom/proprietary formats Many principles discussed in this presentation apply to any of the above formats. 6 Marshaling JSON Creating JSON from a Go object is (usually) very straight forward: func main() { x := map[string]string{ "foo": "bar", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 7 Marshaling JSON, #2 Creating JSON from a Go object is (usually) very straight forward: func main() { type person struct { Name string `json:"name"` Age int `json:"age"` Description string `json:"descr,omitempty"` secret string // Unexported fields are never (un)marshaled } x := person{ Name: "Bob", Age: 32, secret: "Shhh!", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 8 Unmarshaling JSON Unmarshaling JSON is often a bit trickier.
    [Show full text]
  • A Coap Server with a Rack Interface for Use of Web Frameworks Such As Ruby on Rails in the Internet of Things
    A CoAP Server with a Rack Interface for Use of Web Frameworks such as Ruby on Rails in the Internet of Things Diploma Thesis Henning Muller¨ Matriculation No. 2198830 March 10, 2015 Supervisor Prof. Dr.-Ing. Carsten Bormann Reviewer Dr.-Ing. Olaf Bergmann Adviser Dipl.-Inf. Florian Junge Faculty 3: Computer Science and Mathematics 2afc1e5 cbna This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License. http://creativecommons.org/licenses/by-nc-sa/4.0/ Henning Muller¨ [email protected] Abstract We present a Constrained Application Protocol (CoAP) server with a Rack interface to enable application development for the Internet of Things (or Wireless Embedded Internet) using frameworks such as Ruby on Rails. Those frameworks avoid the need for reinvention of the wheel, and simplify the use of Test-driven Development (TDD) and other agile software development methods. They are especially beneficial on less constrained devices such as infrastructure devices or application servers. Our solution supports development of applications almost without paradigm change compared to HTTP and provides performant handling of numerous concurrent clients. The server translates transparently between the protocols and also supports specifics of CoAP such as service and resource discovery, block-wise transfers and observing resources. It also offers the possibility of transparent transcoding between JSON and CBOR payloads. The Resource Directory draft was implemented by us as a Rails application running on our server software. Wir stellen einen Constrained Application Protocol (CoAP) Server mit einem Rack In- terface vor, der Anwendungsentwicklung fur¨ das Internet der Dinge (bzw. das Wireless Embedded Internet) mit Frameworks wie Ruby on Rails ermoglicht.¨ Solche Framworks verhindern die Notwendigkeits, das Rad neu zu erfinden und vereinfachen die Anwen- dung testgetriebener Entwicklung (TDD) und anderer agiler Methoden der Softwareen- twicklung.
    [Show full text]
  • Decentralized Identifier WG F2F Sessions
    Decentralized Identifier WG F2F Sessions Day 1: January 29, 2020 Chairs: Brent Zundel, Dan Burnett Location: Microsoft Schiphol 1 Welcome! ● Logistics ● W3C WG IPR Policy ● Agenda ● IRC and Scribes ● Introductions & Dinner 2 Logistics ● Location: “Spaces”, 6th floor of Microsoft Schiphol ● WiFi: SSID Publiek_theOutlook, pwd Hello2020 ● Dial-in information: +1-617-324-0000, Meeting ID ● Restrooms: End of the hall, turn right ● Meeting time: 8 am - 5 pm, Jan. 29-31 ● Breaks: 10:30-11 am, 12:30-1:30 pm, 2:30-3 pm ● DID WG Agenda: https://tinyurl.com/didwg-ams2020-agenda (HTML) ● Live slides: https://tinyurl.com/didwg-ams2020-slides (Google Slides) ● Dinner Details: See the “Dinner Tonight” slide at the end of each day 3 W3C WG IPR Policy ● This group abides by the W3C patent policy https://www.w3.org/Consortium/Patent-Policy-20040205 ● Only people and companies listed at https://www.w3.org/2004/01/pp-impl/117488/status are allowed to make substantive contributions to the specs ● Code of Conduct https://www.w3.org/Consortium/cepc/ 4 Today’s agenda 8:00 Breakfast 8:30 Welcome, Introductions, and Logistics Chairs 9:00 Level setting Chairs 9:30 Security issues Brent 10:15 DID and IoT Sam Smith 10:45 Break 11:00 Multiple Encodings/Different Syntaxes: what might we want to support Markus 11:30 Different encodings: model incompatibilities Manu 12:00 Abstract data modeling options Dan Burnett 12:30 Lunch (brief “Why Are We Here?” presentation) Christopher Allen 13:30 DID Doc Extensibility via Registries Mike 14:00 DID Doc Extensibility via JSON-LD Manu
    [Show full text]
  • Interoperability for the Semantic Web: a Loosely Coupled Mediation Approach Antoine Zimmermann
    Interoperability for the Semantic Web: A Loosely Coupled Mediation Approach Antoine Zimmermann To cite this version: Antoine Zimmermann. Interoperability for the Semantic Web: A Loosely Coupled Mediation Ap- proach. Modeling and Simulation. Université Jean Monnet, 2021. tel-03163932 HAL Id: tel-03163932 https://tel.archives-ouvertes.fr/tel-03163932 Submitted on 9 Mar 2021 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. These` d'habilitation Universite´ Jean Monnet Ecole´ doctorale 488 Sciences, Ing´enierieet Sant´e(SIS) Interop´erabilit´epour le Web s´emantique : une approche par m´ediationfaiblement coupl´ee Soutenue publiquement par : Antoine Zimmermann MINES Saint-Etienne´ Membres du jury : Olivier Boissier Professeur, MINES Saint-Etienne´ Tuteur de recherche Pierre-Antoine Champin Ma^ıtrede conf´erenceHDR, Universit´eClaude-Bernard Examinateur J´er^ome Euzenat Directeur de recherche, Inria Examinateur Fabien Gandon Directeur de recherche, Inria Rapporteur Andreas Harth Professeur, Friedrich-Alexander-Universit¨atErlangen-N¨urnberg Rapporteur
    [Show full text]
  • Healing the Fragmentation to Realise the Full Potential of The
    HealingHealing thethe fragmentationfragmentation toto realiserealise thethe fullfull potentialpotential of thethe IoTIoT Dr.Dr. DaveDave Raggett,Raggett, W3C/ERCIMW3C/ERCIM email:email: [email protected]@w3.org 2525 JuneJune 20202020 ThisThis talktalk isis supportedsupported byby thethe CreateCreate--IoTIoT CoordinationCoordination andand SupportSupport ActionAction withwith fundingfunding fromfrom thethe EuropeanEuropean CommissionCommission Investments in IoT are at risk! Eric Siow, Director, Open Web Platform Standards and Ecosystem Strategies at Intel: • IoT is a little over 10 years old • Hype has been much greater than present reality • IoT is “biting off more that it can chew” • Trying to address too many markets • Involves too many and mostly uncoordinated SDOs and SIGs 2/14 Key IoT Challenges Facing Smart Cities • Lack of coalescence around a set of complementary standards • Hinders scalability, interoperability and evolution • Need to simplify: prioritise and define requirements • Regional regulatory differences adding to the confusion • Diverse requirements impede scalability of the market • Need regulatory agencies to participate and help with standardisation requirements • Lack of interoperability wastes up to 40% of IoT value1 • Cities and technology partners may waste up to $321 billion by 20252 1. https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world 2. https://machinaresearch.com/news/smart-cities-could-waste-usd341-billion-by-2025-on-non-standardized-iot-deployments/
    [Show full text]