DOCSLIB.ORG

Serialization

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Serialization Serialization Philip Rodrigues University of Oxford November 25, 2020 1 Introduction I Proposal for an object serialization/deserialization interface and implementation, for comments I Implementation builds on existing DAQ components: IPM and moo object schema I Code currently lives in https://github.com/philiprodrigues/networkqueue: needs a better home 2 Use of moo schema local fsd = { count : s.number("Count", "i4", doc="A count of not too many things"), fakedata: s.record("FakeData", [ I Given a moo schema, can produce C++ classes and s.field("fake_count", self.count, -4, doc="A fake count of something"), functions to convert to nlohmann::json objects (in ], Nljs.hpp) doc="Fake Serializable data"), }; I nlohmann::json objects can be dumped to JSON text, and also to binary formats MsgPack, CBOR, BSON, and UBJSON ⇓ https://nlohmann.github.io/json/features/binary_formats // @brief A count of not too many things using Count= int32_t; I (And of course, the reverse) // @brief Fake Serializable data struct FakeData { // @brief A fake count of something Count fake_count; }; https://github.com/philiprodrigues/networkqueue/blob/main/schema/networkqueue-fsd-schema.jsonnet https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Structs.hpp https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Nljs.hpp 3 Serialization steps FakeData fd; // Some class created via moo schema fd.fake_count=1; nlohmann::json j=fd; // Uses to_json(nlohmann::json& j, const FakeData& obj) in Nljs.hpp nlohmann::json::string_t s=j.dump(); // Dump to JSON string std::vector<uint8_t> bytes(s.begin(), s.end()); // Convert to vector of bytes // or... std::vector<uint8_t> byes=nlohmann::json::to_cbor(j); I And the reverse to deserialize 4 Serialization interface I Implemented in https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/Serialization.hpp enum SerializationType { JSON, MsgPack, CBOR // could add more here }; template<classT > std::vector<uint8_t> serialize(const T& obj, SerializationType stype); template<classT > T deserialize(const std::vector<char>& v, SerializationType stype); I These work for any T that has to_json() and from_json() methods (which is true for all moo-generated classes) I Here, user would deal with handing the vector of bytes to IPM (or getting it from IPM) 5 Object sending/receiving interface I Some uses might find it convenient to have the serialization and IPM parts wrapped up together in one call I NetworkObjectSender<T> and NetworkObjectReceiver<T> provide this // Sender process: NetworkObjectSender<FakeData> sender(sender_conf); FakeData fd; fd.fake_count=25; sender.send(fd, std::chrono::milliseconds(2)); // Receiver process: NetworkObjectReceiver<FakeData> receiver(receiver_conf); FakeData fd_recv=receiver.recv(std::chrono::milliseconds(2)); // Now fd_recv.fake_count==25 https://github.com/philiprodrigues/networkqueue/blob/main/test/apps/network_object_send_receive.cxx I sender_conf and receiver_conf contain serialization type, IPM plugin type (pub/sub or direct send/receive) and send/receive address 6 Open questions; next steps I Is the general idea what we want? I Where to put it? New repo “daq-serialization”? I Needs more comprehensive tests I Performance testing? But need some idea of what constitutes “Good Enough” I Could attempt more type safety (eg add class name/ID/checksum to message) 7.
Load more

Recommended publications
  • CBOR (RFC 7049) Concise Binary Object Representation
    CBOR (RFC 7049) Concise Binary Object Representation Carsten Bormann, 2015-11-01 1 CBOR: Agenda • What is it, and when might I want it? • How does it work? • How do I work with it? 2 CBOR: Agenda • What is it, and when might I want it? • How does it work? • How do I work with it? 3 Slide stolen from Douglas Crockford History of Data Formats • Ad Hoc • Database Model • Document Model • Programming Language Model Box notation TLV 5 XML XSD 6 Slide stolen from Douglas Crockford JSON • JavaScript Object Notation • Minimal • Textual • Subset of JavaScript Values • Strings • Numbers • Booleans • Objects • Arrays • null Array ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"] [ [0, -1, 0], [1, 0, 0], [0, 0, 1] ] Object { "name": "Jack B. Nimble", "at large": true, "grade": "A", "format": { "type": "rect", "width": 1920, "height": 1080, "interlace": false, "framerate": 24 } } Object Map { "name": "Jack B. Nimble", "at large": true, "grade": "A", "format": { "type": "rect", "width": 1920, "height": 1080, "interlace": false, "framerate": 24 } } JSON limitations • No binary data (byte strings) • Numbers are in decimal, some parsing required • Format requires copying: • Escaping for strings • Base64 for binary • No extensibility (e.g., date format?) • Interoperability issues • I-JSON further reduces functionality (RFC 7493) 12 BSON and friends • Lots of “binary JSON” proposals • Often optimized for data at rest, not protocol use (BSON ➔ MongoDB) • Most are more complex than JSON 13 Why a new binary object format? • Different design goals from current formats – stated up front in the document • Extremely small code size – for work on constrained node networks • Reasonably compact data size – but no compression or even bit-fiddling • Useful to any protocol or application that likes the design goals 14 Concise Binary Object Representation (CBOR) 15 “Sea Boar” “Sea Boar” 16 Design goals (1 of 2) 1.
    [Show full text]
  • Decentralized Identifier WG F2F Sessions
    Decentralized Identifier WG F2F Sessions Day 1: January 29, 2020 Chairs: Brent Zundel, Dan Burnett Location: Microsoft Schiphol 1 Welcome! ● Logistics ● W3C WG IPR Policy ● Agenda ● IRC and Scribes ● Introductions & Dinner 2 Logistics ● Location: “Spaces”, 6th floor of Microsoft Schiphol ● WiFi: SSID Publiek_theOutlook, pwd Hello2020 ● Dial-in information: +1-617-324-0000, Meeting ID ● Restrooms: End of the hall, turn right ● Meeting time: 8 am - 5 pm, Jan. 29-31 ● Breaks: 10:30-11 am, 12:30-1:30 pm, 2:30-3 pm ● DID WG Agenda: https://tinyurl.com/didwg-ams2020-agenda (HTML) ● Live slides: https://tinyurl.com/didwg-ams2020-slides (Google Slides) ● Dinner Details: See the “Dinner Tonight” slide at the end of each day 3 W3C WG IPR Policy ● This group abides by the W3C patent policy https://www.w3.org/Consortium/Patent-Policy-20040205 ● Only people and companies listed at https://www.w3.org/2004/01/pp-impl/117488/status are allowed to make substantive contributions to the specs ● Code of Conduct https://www.w3.org/Consortium/cepc/ 4 Today’s agenda 8:00 Breakfast 8:30 Welcome, Introductions, and Logistics Chairs 9:00 Level setting Chairs 9:30 Security issues Brent 10:15 DID and IoT Sam Smith 10:45 Break 11:00 Multiple Encodings/Different Syntaxes: what might we want to support Markus 11:30 Different encodings: model incompatibilities Manu 12:00 Abstract data modeling options Dan Burnett 12:30 Lunch (brief “Why Are We Here?” presentation) Christopher Allen 13:30 DID Doc Extensibility via Registries Mike 14:00 DID Doc Extensibility via JSON-LD Manu
    [Show full text]
  • Interoperability for the Semantic Web: a Loosely Coupled Mediation Approach Antoine Zimmermann
    Interoperability for the Semantic Web: A Loosely Coupled Mediation Approach Antoine Zimmermann To cite this version: Antoine Zimmermann. Interoperability for the Semantic Web: A Loosely Coupled Mediation Ap- proach. Modeling and Simulation. Université Jean Monnet, 2021. tel-03163932 HAL Id: tel-03163932 https://tel.archives-ouvertes.fr/tel-03163932 Submitted on 9 Mar 2021 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. These` d'habilitation Universite´ Jean Monnet Ecole´ doctorale 488 Sciences, Ing´enierieet Sant´e(SIS) Interop´erabilit´epour le Web s´emantique : une approche par m´ediationfaiblement coupl´ee Soutenue publiquement par : Antoine Zimmermann MINES Saint-Etienne´ Membres du jury : Olivier Boissier Professeur, MINES Saint-Etienne´ Tuteur de recherche Pierre-Antoine Champin Ma^ıtrede conf´erenceHDR, Universit´eClaude-Bernard Examinateur J´er^ome Euzenat Directeur de recherche, Inria Examinateur Fabien Gandon Directeur de recherche, Inria Rapporteur Andreas Harth Professeur, Friedrich-Alexander-Universit¨atErlangen-N¨urnberg Rapporteur
    [Show full text]
  • Healing the Fragmentation to Realise the Full Potential of The
    HealingHealing thethe fragmentationfragmentation toto realiserealise thethe fullfull potentialpotential of thethe IoTIoT Dr.Dr. DaveDave Raggett,Raggett, W3C/ERCIMW3C/ERCIM email:email: [email protected]@w3.org 2525 JuneJune 20202020 ThisThis talktalk isis supportedsupported byby thethe CreateCreate--IoTIoT CoordinationCoordination andand SupportSupport ActionAction withwith fundingfunding fromfrom thethe EuropeanEuropean CommissionCommission Investments in IoT are at risk! Eric Siow, Director, Open Web Platform Standards and Ecosystem Strategies at Intel: • IoT is a little over 10 years old • Hype has been much greater than present reality • IoT is “biting off more that it can chew” • Trying to address too many markets • Involves too many and mostly uncoordinated SDOs and SIGs 2/14 Key IoT Challenges Facing Smart Cities • Lack of coalescence around a set of complementary standards • Hinders scalability, interoperability and evolution • Need to simplify: prioritise and define requirements • Regional regulatory differences adding to the confusion • Diverse requirements impede scalability of the market • Need regulatory agencies to participate and help with standardisation requirements • Lack of interoperability wastes up to 40% of IoT value1 • Cities and technology partners may waste up to $321 billion by 20252 1. https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world 2. https://machinaresearch.com/news/smart-cities-could-waste-usd341-billion-by-2025-on-non-standardized-iot-deployments/
    [Show full text]
  • Weave TLV White Paper
    Nest Weave TLV White Paper Revision 4 2020-03-04 Status: Approved / Active ​ ​ Copyright © 2020 Google LLC Google LLC Public Information APPROVED / ACTIVE Revision History Revision Date Modified By Description 1 2014-11-24 Grant Erickson Initial revision. 2 2015-02-25 Grant Erickson Final draft. 3 2015-06-03 Grant Erickson Updated quantitative analysis to include Flatbuffers. Reran all quantitative analysis using GCC 4.8.2 targeted to an ARM Cortex A9 with a hard floating point ABI. 4 2020-03-04 Grant Erickson Updated for CBOR. Rev. 4 Copyright © 2020 Google LLC 2 2020-03-04 Google LLC Public Information APPROVED / ACTIVE Table of Contents Revision History Table of Contents Summary Introduction Motivation and Rationale Target System Resources Core Message Format Application Data Representation Requirements Capable of Representing Basic Machine Types Capable of Representing Arrays Capable of Representing Structures Capable of Forward- and Backward-Compatibility Capable of Representing Optional Content Partitioned and Controlled Tag Space System Neutrality Resource Overhead Trivial In-place Access of Basic Machine Types Over-the-wire Compactness Lossless Translation to JSON Licensing and Seat Costs Market Penetration Proliferation and Quality of Infrastructure and Tools Competitive Analysis Summary JSON memcpy Google Protocol Buffers Trivial In-place Access of Basic Machine Types ASN.1 EXI Other Thrift CBOR Partitioned and Controlled Tag Space Rev. 4 Copyright © 2020 Google LLC 3 2020-03-04 Google LLC Public Information APPROVED / ACTIVE Trivial In-place Access of Basic Machine Types Flatbuffers Conclusion Appendix A: Size Analysis JSON memcpy Google Protocol Buffers libprotobuf Size Speed libprotobuf-lite nanopb Thrift Flatbuffers Weave CBOR References Rev.
    [Show full text]
  • Efficient XML Interchange for Afloat Networks
    LT Bruce Hill MAKING BIG FILES SMALL AND SMALL FILES TINY 1 XML and JSON ● JavaScript Object Notation (JSON) is a common alternative to XML in web applications ● JSON is a plaintext data-interchange format based on JavaScript code ● JSON has compact binary encodings analogous to EXI: ○ CBOR ○ BSON ● Research Question: Is EXI more compact than CBOR and BSON? 2 EXI for Large XML Files ● W3C and previous NPS research measured EXI performance on XML up to 100MB ● Large data dumps can easily exceed that ● Research Question: How does EXI (but not CBOR/BSON) perform on files from 100MB - 4GB? 3 Methods Use Case Focus Configuration Focus ● Compression results ● EXI has many across multiple use configuration options cases look different from that affect results for multiple files ● Compactness within a single use case ● Processing speed ● Select a few use cases ● Memory footprint ● Fidelity and study them in-depth ● XML Schema affects EXI compression as well 4 Encodings Compared Small Files Large Files When in doubt, try every possible combination of options 5 Small-file Use Cases (B to KB) ● OpenWeatherMap ● Automated Identification ● Global Position System System (AIS) XML (GPX) 6 AIS Use Case EXI smaller than CBOR/BSON, aggregating data helps 7 AIS Use Case Well-designed XML Schema improves performance 8 Large-file Use Cases (KB to GB) ● Digital Forensics XML ● Packet Description (DFXML) Markup Language ● OpenStreetMap (PDML) 9 PDML Use Case EXI performs well on large files, aggregation benefits plateau 10 EXI and MS Office ● Microsoft Office is ubiquitous in Navy/DoD ● Since 2003, the file format has been a Zipped archive of many small XML files ● Since 2006, the file format has been an open standard ● Since 2013, MS Office 365 can save in compliant format ● Tools such as NXPowerLite target excess image resolution and metadata to shrink them ● EXI can target the remainder..
    [Show full text]
  • Second Screen CG Berlin F2F
    Second Screen CG Berlin F2F May 17-18, 2018 Mark A. Foltz [email protected] Brandon Tolsch [email protected] Outline: Day 1 ● Agenda review ● Open Screen overview ● Discovery ○ Review of Chrome data ○ Mandatory vs. optional mechanisms ● Transport ○ QUIC Data Channels ● Authentication ○ J-PAKE ○ Public key based Outline: Day 2 ● Control protocol, serialization ○ CBOR vs. Protocol Messages ● HbbTV/ATSC compatibility ● Open Screen Protocol Library ● Future use cases and APIs ● Planning Open Screen Protocol Background & Status Second Screen CG History ● Nov 2013: Initial charter ● Nov 2013 - Dec 2014: Incubation of Presentation API ● Dec 2014: Presentation API transitioned to Second Screen Working Group ● Sept 2016: CG rechartered to focus on interoperability ● 2016-2017: Requirements, protocol alternatives, benchmarking plan ● Sept 2017: F2F at TPAC ● Jan-Feb 2018: SSWG rechartered. Phone conference, work plan ● May 2018: This meeting :-) Presentation API 1. Controlling page (in a browser) requests presentation of a URL on a receiver device (on a connected display). 2. Browser lists displays compatible with the URL; the user selects one to start the presentation. 3. Controlling and receiving pages each receive a presentation connection. 4. The connection can be used to exchange messages between the two pages. 5. Either side may close the connection or terminate the presentation. Presentation API: 2-UA Mode https://www.youtube.com/tv PLAY PAUSE ... Presentation API: 1-UA Mode https://www.youtube.com/tv Remote Playback API <audio> or <video>
    [Show full text]
  • JSON-LD 1.1 – a JSON-Based Serialization for Linked Data Gregg Kellogg, Pierre-Antoine Champin, Dave Longley
    JSON-LD 1.1 – A JSON-based Serialization for Linked Data Gregg Kellogg, Pierre-Antoine Champin, Dave Longley To cite this version: Gregg Kellogg, Pierre-Antoine Champin, Dave Longley. JSON-LD 1.1 – A JSON-based Serialization for Linked Data. [Technical Report] W3C. 2019. hal-02141614v2 HAL Id: hal-02141614 https://hal.archives-ouvertes.fr/hal-02141614v2 Submitted on 28 Jan 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. https://www.w3.org/TR/json-ld11/ JSON-LD 1.1 A JSON-based Serialization for Linked Data W3C Candidate Recommendation 12 December 2019 This version: https://www.w3.org/TR/2019/CR-json-ld11-20191212/ Latest published version: https://www.w3.org/TR/json-ld11/ Latest editor's draft: https://w3c.github.io/json-ld-syntax/ Test suite: https://w3c.github.io/json-ld-api/tests/ Implementation report: https://w3c.github.io/json-ld-api/reports/ Previous version: https://www.w3.org/TR/2019/WD-json-ld11-20191112/ Latest Recommendation: https://www.w3.org/TR/2014/REC-json-ld-20140116/ Editors: Gregg Kellogg (v1.0 and v1.1) Pierre-Antoine Champin
    [Show full text]
  • Author(S) Hill, Bruce W. Title Evaluation of Efficient XML
    Author(s) Hill, Bruce W. Evaluation of efficient XML interchange (EXI) for large datasets and as an alternative Title to binary JSON encodings Publisher Monterey, California: Naval Postgraduate School Issue Date 2015-03 URL http://hdl.handle.net/10945/45196 This document was downloaded on June 12, 2015 at 11:53:11 NAVAL POSTGRADUATE SCHOOL MONTEREY, CALIFORNIA THESIS EVALUATION OF EFFICIENT XML INTERCHANGE (EXI) FOR LARGE DATASETS AND AS AN ALTERNATIVE TO BINARY JSON ENCODINGS by Bruce W. Hill March 2015 Thesis Advisor: Don Brutzman Co-Advisor: Don McGregor Approved for public release; distribution is unlimited THIS PAGE INTENTIONALLY LEFT BLANK REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704–0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington, DC 20503. 1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED March 2015 Master’s Thesis 4. TITLE AND SUBTITLE 5. FUNDING NUMBERS EVALUATION OF EFFICIENT XML INTERCHANGE (EXI) FOR LARGE W4V02 DATASETS AND AS AN ALTERNATIVE TO BINARY JSON ENCODINGS 6. AUTHOR Bruce W. Hill 7. PERFORMING ORGANIZATION NAME AND ADDRESS 8.
    [Show full text]
  • Smart Cities – Standardization Proposed Strategy Outline
    Next Steps for The Web of Things: Moving from Innovation to Mainstream Eric Siow – Part 1 – Crossing the Chasm Michael McCool – Part 2 – Standards Development Prioritization PART 1 Crossing the Chasm 2 Outline Part 1 ● State of the Union - Internet of Things (IoT) ● What can the W3C and WoT Community Do? Part 2 ● Outline of plan to converge on data framework and standards ● Discussion and ideas for collaboration 3 State of the Union (IoT) • IoT is about 10 years old • Hype has been much greater than present reality • IoT is “biting off more than it can chew”: − Trying to address too many markets − Involves too many and mostly uncoordinated SDOs and SIGs Investments in IoT are at risk 4 Crossing the Chasm (IOT) We are here $ $ Focus on a vertical & address the needs of the Early Majority 1. Simplify technical complexity 2. Lower deployment risk and cost 3. Create customer peer references 8 Illustration: A look at Smart Cities 9 Key Challenges Facing Smart Cities • Lack of coalescence around a set of complementary standards − Hinders scalability, interoperability and evolution − Need to simplify: prioritize and define requirements − Increases cost of deployment • Regional regulatory differences adding to confusion − Diverse requirements impede the scalability of the market − Need regulatory agencies to participate and help with standardization requirement • Lack of interoperability wastes up to 40% of IoT value (1) • Cities and technology partners may waste up to $321 billion by 2025 (2) 10 What Can W3C and THE WoT Community Do? 1. Align, unite and cross the chasm together • Focus on an Application: Vertical Market Segment − Difficult to align given different business priorities & interests − May increase fragmentation rather than reduce it • Focus on a Platform: Data Interoperability − Easier to align: Most pressing shared problem − Enable different devices and platforms to interoperable − Plays to W3C’s and WoT’s Core competences 11 What Can W3C and the WoT Community Do? 2.
    [Show full text]
  • Syntactic Translation of Message Payloads Between at Least Partially Equivalent Encodings
    Syntactic Translation of Message Payloads Between At Least Partially Equivalent Encodings Emanuel Palm, Cristina Paniagua, Ulf Bodin, Olov Schelen´ Lulea˚ University of Technology, Lulea,˚ Sweden E-mail: ffirstname:[email protected] Abstract—Recent years have seen a surge of interest in using A significant ambition of this work is to provide mes- IoT systems for an increasingly diverse set of applications, with sage payload translation capabilities to systems such as the use cases ranging from medicine to mining. Due to the disparate Arrowhead Framework [4], a SOA-based IoT framework needs of these applications, vendors are adopting a growing number of messaging protocols, encodings and semantics, which that supports the creation of scalable cloud-based automation result in poor interoperability unless systems are explicitly systems. This kind of framework facilitates device service designed to work together. Key efforts, such as Industry 4.0, discovery and orchestration at runtime,without any need for put heavy emphasis on being able to compose arbitrary IoT human intervention. A would-be Arrowhead translator service systems to create emergent applications, which makes mitigating could be thought of as an intermediary situated between two this barrier to interoperability a significant objective. In this paper, we present a theoretical method for translating message communicating systems, as is shown in Figure 1. As work payloads in transit between endpoints, complementing previous has already been done on creating a multiprotocol translation work on protocol translation. The method involves representing system [5] for Arrowhead, the translation method we describe and analyzing encoding syntaxes with the aim of identifying in this paper could be thought of as a complement to existing the concrete translations that can be performed without risk systems only supporting protocol translation.
    [Show full text]
  • AWS Nitro Enclaves User Guide AWS AWS Nitro Enclaves User Guide
    AWS AWS Nitro Enclaves User Guide AWS AWS Nitro Enclaves User Guide AWS: AWS Nitro Enclaves User Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. AWS AWS Nitro Enclaves User Guide Table of Contents What is Nitro Enclaves? ....................................................................................................................... 1 Learn more ............................................................................................................................... 1 Requirements ............................................................................................................................ 2 Considerations ........................................................................................................................... 2 Pricing ...................................................................................................................................... 2 Related services ......................................................................................................................... 2 Nitro Enclaves concepts .....................................................................................................................
    [Show full text]