Serialization

Philip Rodrigues

University of Oxford

November 25, 2020

1 Introduction

I Proposal for an object serialization/deserialization interface and implementation, for comments I Implementation builds on existing DAQ components: IPM and moo object schema I Code currently lives in https://github.com/philiprodrigues/networkqueue: needs a better home

2 Use of moo schema

local fsd = { count : s.number("Count", "i4", doc="A count of not too many things"),

fakedata: s.record("FakeData", [ I Given a moo schema, can produce C++ classes and s.field("fake_count", self.count, -4, doc="A fake count of something"), functions to convert to nlohmann::json objects (in ], Nljs.hpp) doc="Fake Serializable data"), }; I nlohmann::json objects can be dumped to JSON text, and also to binary formats MsgPack, CBOR, BSON, and UBJSON ⇓

https://nlohmann.github.io/json/features/binary_formats // @brief A count of not too many things using Count= int32_t; I (And of course, the reverse) // @brief Fake Serializable data struct FakeData { // @brief A fake count of something Count fake_count; }; https://github.com/philiprodrigues/networkqueue/blob/main/schema/networkqueue-fsd-schema.jsonnet https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Structs.hpp https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Nljs.hpp

3 Serialization steps

FakeData fd; // Some class created via moo schema fd.fake_count=1; nlohmann::json j=fd; // Uses to_json(nlohmann::json& j, const FakeData& obj) in Nljs.hpp nlohmann::json::string_t s=j.dump(); // Dump to JSON string std::vector bytes(s.begin(), s.end()); // Convert to vector of bytes // or... std::vector byes=nlohmann::json::to_cbor(j);

I And the reverse to deserialize

4 Serialization interface

I Implemented in https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/Serialization.hpp enum SerializationType { JSON, MsgPack, CBOR // could add more here };

template std::vector serialize(const T& obj, SerializationType stype);

template T deserialize(const std::vector& v, SerializationType stype);

I These work for any T that has to_json() and from_json() methods (which is true for all moo-generated classes) I Here, user would deal with handing the vector of bytes to IPM (or getting it from IPM)

5 Object sending/receiving interface

I Some uses might ﬁnd it convenient to have the serialization and IPM parts wrapped up together in one call I NetworkObjectSender and NetworkObjectReceiver provide this // Sender process: NetworkObjectSender sender(sender_conf);

FakeData fd; fd.fake_count=25;

sender.send(fd, std::chrono::milliseconds(2));

// Receiver process: NetworkObjectReceiver receiver(receiver_conf); FakeData fd_recv=receiver.recv(std::chrono::milliseconds(2)); // Now fd_recv.fake_count==25 https://github.com/philiprodrigues/networkqueue/blob/main/test/apps/network_object_send_receive.cxx I sender_conf and receiver_conf contain serialization type, IPM plugin type (pub/sub or direct send/receive) and send/receive address

6 Open questions; next steps

I Is the general idea what we want? I Where to put it? New repo “daq-serialization”? I Needs more comprehensive tests I Performance testing? But need some idea of what constitutes “Good Enough” I Could attempt more type safety (eg add class name/ID/checksum to message)