Serialization Philip Rodrigues University of Oxford November 25, 2020 1 Introduction I Proposal for an object serialization/deserialization interface and implementation, for comments I Implementation builds on existing DAQ components: IPM and moo object schema I Code currently lives in https://github.com/philiprodrigues/networkqueue: needs a better home 2 Use of moo schema local fsd = { count : s.number("Count", "i4", doc="A count of not too many things"), fakedata: s.record("FakeData", [ I Given a moo schema, can produce C++ classes and s.field("fake_count", self.count, -4, doc="A fake count of something"), functions to convert to nlohmann::json objects (in ], Nljs.hpp) doc="Fake Serializable data"), }; I nlohmann::json objects can be dumped to JSON text, and also to binary formats MsgPack, CBOR, BSON, and UBJSON ⇓ https://nlohmann.github.io/json/features/binary_formats // @brief A count of not too many things using Count= int32_t; I (And of course, the reverse) // @brief Fake Serializable data struct FakeData { // @brief A fake count of something Count fake_count; }; https://github.com/philiprodrigues/networkqueue/blob/main/schema/networkqueue-fsd-schema.jsonnet https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Structs.hpp https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/fsd/Nljs.hpp 3 Serialization steps FakeData fd; // Some class created via moo schema fd.fake_count=1; nlohmann::json j=fd; // Uses to_json(nlohmann::json& j, const FakeData& obj) in Nljs.hpp nlohmann::json::string_t s=j.dump(); // Dump to JSON string std::vector<uint8_t> bytes(s.begin(), s.end()); // Convert to vector of bytes // or... std::vector<uint8_t> byes=nlohmann::json::to_cbor(j); I And the reverse to deserialize 4 Serialization interface I Implemented in https://github.com/philiprodrigues/networkqueue/blob/main/include/networkqueue/Serialization.hpp enum SerializationType { JSON, MsgPack, CBOR // could add more here }; template<classT > std::vector<uint8_t> serialize(const T& obj, SerializationType stype); template<classT > T deserialize(const std::vector<char>& v, SerializationType stype); I These work for any T that has to_json() and from_json() methods (which is true for all moo-generated classes) I Here, user would deal with handing the vector of bytes to IPM (or getting it from IPM) 5 Object sending/receiving interface I Some uses might find it convenient to have the serialization and IPM parts wrapped up together in one call I NetworkObjectSender<T> and NetworkObjectReceiver<T> provide this // Sender process: NetworkObjectSender<FakeData> sender(sender_conf); FakeData fd; fd.fake_count=25; sender.send(fd, std::chrono::milliseconds(2)); // Receiver process: NetworkObjectReceiver<FakeData> receiver(receiver_conf); FakeData fd_recv=receiver.recv(std::chrono::milliseconds(2)); // Now fd_recv.fake_count==25 https://github.com/philiprodrigues/networkqueue/blob/main/test/apps/network_object_send_receive.cxx I sender_conf and receiver_conf contain serialization type, IPM plugin type (pub/sub or direct send/receive) and send/receive address 6 Open questions; next steps I Is the general idea what we want? I Where to put it? New repo “daq-serialization”? I Needs more comprehensive tests I Performance testing? But need some idea of what constitutes “Good Enough” I Could attempt more type safety (eg add class name/ID/checksum to message) 7.