DOCSLIB.ORG

Design of a Stream Based Software Radar Architecture

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Design of a Stream Based Software Radar Architecture Design of a Stream Based Software Radar Architecture Karl Cronburg (Bucknell University), Mentors: Frank Lind & Robert Schaefer (MIT Haystack) Goals: YAML Object Definition: Stream data directly from instruments to signal processing and This project primarily focused on the design and analysis systems implementation of a YAML-based Interface Definition Dynamic scaling of computation and storage resources Language (IDL) used for defining composite objects with named attributes. Event-based control of processing and analysis systems Plug-and-play message distribution YAML object definition provides… Automatic system configuration for various tasks A human readable data format Robust data / object formats for: Easily listen to / debug data streams Radar frequency voltages Non-programmer understandable object composition Metadata Implementation agnostic object definitions Status, logging, and debugging events Use any YAML-aware programming language A highly capable yet robust object format Capable of default attribute values Handles all basic data types (strings, floats, ints, etc) Tools: Numpy aware Figure 1 – Example object definition for RF voltage data and Figure 2 – Example usage of the dynamic object instantiation, serialization, and Redhat Enterprise Linux Operating Environment Automatic construction of composite data types corresponding metadata. streaming capabilities of various python modules written for this project. Python Implementation GNU C Based Instruments Object Serialization: Runtime Results: ZeroMQ Sockets Library (python binding) Gevent – deterministic python event-based threading Formats: YAML (Yet Another Markup Language) YAML – slow but human-readable serialization for message stream debugging MessagePack Object Serialization MessagePack – fast & compact serialization for on-the-wire data transmission PyCUDA GPU Acceleration HDF5 – object storage in a consistent scientific data format Numpy & Scipy Scientific Analysis Tools YAML !Person: {name : ‘Bob’, age : 22} Matplotlib Plotting Person HDF5 Data Storage name = Bob Serialize msgpack '\x83\xa3age\x16\xa6__type\xa6Person\xa4name\xa3Bob' Figure 3 – Average message rate using MessagePack serialization (with and without compression), age = 22 broken up by task. Small (126 – 512 byte messages) were used. HDF5 (binary) Features: Passive Radar Prototype: On-the-fly deserialization – dynamic loading of object definitions from revision control ‘Pickle’ like syntax for loading and dumping objects Optional gzip compression Example serialization usage shown in Fig. 2. ZeroMQ Object Transport: This Project: The ZeroMQ sockets library provides elastic / scalable plug-and-play style messaging. Messaging Patterns: The primary objective of this project is to collect, analyze, and Push / Pull Request / Reply – one-to-one connection process Radar data with live streaming and viewing capabilities using one-to-many connection, one-to-one and messaging existing and future processing and analysis components. messaging Puller 1 REQ 1 The work I undertook towards this goal involved setting up the Replier REQ/REP 2 architecture for defining data objects apart from Radar processing Pusher msg 2 REQ 2 Puller 2 implementations, and further serializing these objects for transport over networks. Puller 3 REQ 3 Publish / Subscribe A similar architecture being designed is Ocean Observatories one-to-many connection and messaging Initiative (OOI) Cyberinfrastructure, for dynamically collecting and distributing Ocean-related data. SUB 1 Features Implemented: Automatic object (de)serialization Thanks to PUB msg 1,2,3 SUB 2 Debugging / re-routing message filters Frank Lind for giving clear specifications and assistance Optional hash-based integrity verification Robert Schaefer for implementation guidance Clean messaging interface (see Fig. 2) Phil Erickson, Vincent Fish, and KT Paul for organizing the REU SUB 3 The MIT Haystack staff for support throughout the summer .
Load more

Recommended publications
  • Deserialization Vulnerability by Abdelazim Mohammed(@Intx0x80)
    Deserialization vulnerability By Abdelazim Mohammed(@intx0x80) Thanks to: Mazin Ahmed (@mazen160) Asim Jaweesh(@Jaw33sh) 1 | P a g e Table of Contents Serialization (marshaling): ............................................................................................................................ 4 Deserialization (unmarshaling): .................................................................................................................... 4 Programming language support serialization: ............................................................................................... 4 Risk for using serialization: .......................................................................................................................... 5 Serialization in Java ...................................................................................................................................... 6 Deserialization vulnerability in Java: ............................................................................................................ 6 Code flow work........................................................................................................................................... 11 Vulnerability Detection: .............................................................................................................................. 12 CVE: ........................................................................................................................................................... 17 Tools: .........................................................................................................................................................
    [Show full text]
  • Serializing C Intermediate Representations for Efficient And
    Serializing C intermediate representations for efficient and portable parsing Jeffrey A. Meister1, Jeffrey S. Foster2,∗, and Michael Hicks2 1 Department of Computer Science and Engineering, University of California, San Diego, CA 92093, USA 2 Department of Computer Science, University of Maryland, College Park, MD 20742, USA SUMMARY C static analysis tools often use intermediate representations (IRs) that organize program data in a simple, well-structured manner. However, the C parsers that create IRs are slow, and because they are difficult to write, only a few implementations exist, limiting the languages in which a C static analysis can be written. To solve these problems, we investigate two language-independent, on-disk representations of C IRs: one using XML, and the other using an Internet standard binary encoding called XDR. We benchmark the parsing speeds of both options, finding the XML to be about a factor of two slower than parsing C and the XDR over six times faster. Furthermore, we show that the XML files are far too large at 19 times the size of C source code, while XDR is only 2.2 times the C size. We also demonstrate the portability of our XDR system by presenting a C source code querying tool in Ruby. Our solution and the insights we gained from building it will be useful to analysis authors and other clients of C IRs. We have made our software freely available for download at http://www.cs.umd.edu/projects/PL/scil/. key words: C, static analysis, intermediate representations, parsing, XML, XDR 1. Introduction There is significant interest in writing static analysis tools for C programs.
    [Show full text]
  • The Programmer's Guide to Apache Thrift MEAP
    MEAP Edition Manning Early Access Program The Programmer’s Guide to Apache Thrift Version 5 Copyright 2013 Manning Publications For more information on this and other Manning titles go to www.manning.com ©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders. http://www.manning-sandbox.com/forum.jspa?forumID=873 Licensed to Daniel Gavrila <[email protected]> Welcome Hello and welcome to the third MEAP update for The Programmer’s Guide to Apache Thrift. This update adds Chapter 7, Designing and Serializing User Defined Types. This latest chapter is the first of the application layer chapters in Part 2. Chapters 3, 4 and 5 cover transports, error handling and protocols respectively. These chapters describe the foundational elements of Apache Thrift. Chapter 6 describes Apache Thrift IDL in depth, introducing the tools which enable us to describe data types and services in IDL. Chapters 7 through 9 bring these concepts into action, covering the three key applications areas of Apache Thrift in turn: User Defined Types (UDTs), Services and Servers. Chapter 7 introduces Apache Thrift IDL UDTs and provides insight into the critical role played by interface evolution in quality type design. Using IDL to effectively describe cross language types greatly simplifies the transmission of common data structures over messaging systems and other generic communications interfaces. Chapter 7 demonstrates the process of serializing types for use with external interfaces, disk I/O and in combination with Apache Thrift transport layer compression.
    [Show full text]
  • Msgpack Documentation Release 1.0
    msgpack Documentation Release 1.0 Author 2021-03-18 Contents 1 API reference 3 2 Advanced usage 9 Python Module Index 11 Index 13 i ii msgpack Documentation, Release 1.0 MessagePack is a efficient format for inter language data exchange. Contents 1 msgpack Documentation, Release 1.0 2 Contents CHAPTER 1 API reference msgpack.pack(o, stream, **kwargs) Pack object o and write it to stream See Packer for options. dump() is alias for pack() msgpack.packb(o, **kwargs) Pack object o and return packed bytes See Packer for options. dumps() is alias for packb() msgpack.unpack(stream, **kwargs) Unpack an object from stream. Raises ExtraData when stream contains extra bytes. See Unpacker for options. load() is alias for unpack() msgpack.unpackb(packed, *, object_hook=None, list_hook=None, bool use_list=True, bool raw=False, int timestamp=0, bool strict_map_key=True, unicode_errors=None, object_pairs_hook=None, ext_hook=ExtType, Py_ssize_t max_str_len=-1, Py_ssize_t max_bin_len=-1, Py_ssize_t max_array_len=-1, Py_ssize_t max_map_len=-1, Py_ssize_t max_ext_len=-1) Unpack packed_bytes to object. Returns an unpacked object. Raises ExtraData when packed contains extra bytes. Raises ValueError when packed is incomplete. Raises FormatError when packed is not valid msgpack. Raises StackError when packed contains too nested. Other exceptions can be raised during unpacking. See Unpacker for options. max_xxx_len options are configured automatically from len(packed). loads() is alias for unpackb() 3 msgpack Documentation, Release 1.0 class msgpack.Packer(default=None, *, bool use_single_float=False, bool autoreset=True, bool use_bin_type=True, bool strict_types=False, bool datetime=False, uni- code_errors=None) MessagePack Packer Usage: packer= Packer() astream.write(packer.pack(a)) astream.write(packer.pack(b)) Packer’s constructor has some keyword arguments: Parameters • default (callable) – Convert user type to builtin type that Packer supports.
    [Show full text]
  • Implementation of a Serializer to Represent PHP Objects in the Extensible Markup Language Lidia N
    World Academy of Science, Engineering and Technology International Journal of Computer and Information Engineering Vol:13, No:7, 2019 Implementation of a Serializer to Represent PHP Objects in the Extensible Markup Language Lidia N. Hernández-Piña, Carlos R. Jaimez-González context of data storage and transmission, serialization is the Abstract—Interoperability in distributed systems is an important process of rendering an object into a state that can be saved feature that refers to the communication of two applications written persistently into a storage medium, such as a file, database, or in different programming languages. This paper presents a serializer a stream to be transmitted through the network. De- and a de-serializer of PHP objects to and from XML, which is an serialization is the opposite process, which puts the serialized independent library written in the PHP programming language. The XML generated by this serializer is independent of the programming version of the object into a live object in memory" [1]. language, and can be used by other existing Web Objects in XML This paper presents a serializer and a de-serializer of PHP (WOX) serializers and de-serializers, which allow interoperability objects to XML, called PHP Web Objects in XML with other object-oriented programming languages. (PHPWOX) [2]. The XML generated by PHPWOX is independent of the programming language, and can be used by Keywords—Interoperability, PHP object serialization, PHP to other existing serializers and de-serializers WOX [3], which XML, web objects in XML, WOX. allow interoperability between applications written in PHP and applications written in the programming languages supported I.
    [Show full text]
  • The Phpserialize Package
    The phpSerialize Package March 9, 2005 Title Serialize R to PHP associative array Version 0.8-01 Author Dieter Menne <[email protected]> Description Serializes R objects for import by PHP into an associative array. Can be used to build interactive web pages with R. Maintainer Dieter Menne<[email protected]> License GPL version 2 or newer R topics documented: phpSerialize . 1 Index 4 phpSerialize R to PHP Serialization Description Serializes R objects for PHP import into an associative array. Main use is for building web pages with R-support. Usage phpSerialize(x, file = NULL, append = FALSE, associative = "1D", simplifyMono=TRUE, phpTestCode = FALSE) phpSerializeAll(include=NULL, exclude=NULL, file = NULL, append = FALSE, associative = "1D", simplifyMono=TRUE, phpTestCode = FALSE) 1 2 phpSerialize Arguments x an object file a file name or a connection, or NULL to return the output as a string. If the connection is not open it will be opened and then closed on exit. append append or overwrite the file? associative a character string of "1D" (default), "2D" or "no". For "1D", only scalars and vectors are serialized as associative arrays which can be retrieved by name, while arrays are exported with numeric indexes. For "2D", arrays are also exported by name. For "no", objects are serialized with numerical indexes only, and factors are serialized as integers. simplifyMono if TRUE (default), unnamed vectors of length 1 are reduced to scalars, which is easier to understand in PHP. For simplyMono=FALSE, these vectors are serialized as arrays with length 1. We need another level of indexing ([1]) in PHP, but we are closer to the way R ticks.
    [Show full text]
  • File IO Serialization
    File IO serialization HOM HVD Streams and Java-I/O Streams in Java File IO serialization java.nio.file Object Streams and serialization Pieter van den Hombergh Serialisation formats Richard van den Ham WARNING Javascript Object Notation Fontys Hogeschool voor Techniek en Logistiek March 13, 2018 HOMHVD/FHTenL File IO serialization March 13, 2018 1/23 File IO Topics serialization HOM HVD Streams and Streams and Java-I/O Java-I/O Streams in Java Streams in Java java.nio.file java.nio.file Object Streams and serialization Serialisation formats WARNING Object Streams and serialization Javascript Object Notation Serialisation formats WARNING Javascript Object Notation HOMHVD/FHTenL File IO serialization March 13, 2018 2/23 File IO Streams in Java serialization HOM HVD Streams and Java-I/O Streams in Java java.nio.file Object Streams and serialization Serialisation formats WARNING Javascript Object Notation Figure: Taken from the Oracle/Sun Java tutorial Read or write information from different sources and types. sources: network, files, devices types: text, picture, sound Streams are FIFOs, uni-directional HOMHVD/FHTenL File IO serialization March 13, 2018 3/23 File IO Two basic stream types serialization HOM Byte Streams HVD java.io.InputStream, java.io.OutputStream Streams and Java-I/O read and write pdf, mp3, raw... Streams in Java java.nio.file Character Streams (16-bit Unicode) Object Streams java.io.Reader, java.io.Writer and serialization simplifies reading and writing characters, character-arrays or Strings Serialisation formats WARNING Javascript
    [Show full text]
  • Advanced JSON Handling in Go 19:40 05 Mar 2020 Jonathan Hall Devops Evangelist / Go Developer / Clean Coder / Salsa Dancer About Me
    Advanced JSON handling in Go 19:40 05 Mar 2020 Jonathan Hall DevOps Evangelist / Go Developer / Clean Coder / Salsa Dancer About me Open Source contributor; CouchDB PMC, author of Kivik Core Tech Lead for Lana Former eCommerce Dev Manager at Bugaboo Former backend developer at Teamwork.com Former backend developer at Booking.com Former tech lead at eFolder/DoubleCheck 2 Show of hands Who has... ...used JSON in a Go program? ...been frustrated by Go's strict typing when dealing with JSON? ...felt limited by Go's standard JSON handling? What have been your biggest frustrations? 3 Today's Topics Very brief intro to JSON in Go Basic use of maps and structs Handling inputs of unknown type Handling data with some unknown fields 4 A brief intro to JSON JavaScript Object Notation, defined by RFC 8259 Human-readable, textual representation of arbitrary data Limted types: null, Number, String, Boolean, Array, Object Broad applications: Config files, data interchange, simple messaging 5 Alternatives to JSON YAML, TOML, INI BSON, MessagePack, CBOR, Smile XML ProtoBuf Custom/proprietary formats Many principles discussed in this presentation apply to any of the above formats. 6 Marshaling JSON Creating JSON from a Go object is (usually) very straight forward: func main() { x := map[string]string{ "foo": "bar", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 7 Marshaling JSON, #2 Creating JSON from a Go object is (usually) very straight forward: func main() { type person struct { Name string `json:"name"` Age int `json:"age"` Description string `json:"descr,omitempty"` secret string // Unexported fields are never (un)marshaled } x := person{ Name: "Bob", Age: 32, secret: "Shhh!", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 8 Unmarshaling JSON Unmarshaling JSON is often a bit trickier.
    [Show full text]
  • Pharmaceutical Serialization Track and Trace
    PHARMACEUTICAL SERIALIZATION TRACK & TRACE EASY GUIDE TO COUNTRY- WISE MANDATES External Document © 2018 Infosys Limited External Document © 2018 Infosys Limited Table of Contents Introduction 05 US Federal 06 California 08 Argentina 10 Brazil 12 South Korea 14 India 16 China 18 EU 20 France 22 Turkey 24 References 26 External Document © 2018 Infosys Limited External Document © 2018 Infosys Limited External Document © 2018 Infosys Limited Introduction Pharmaceutical companies have to contend with challenges stemming from supply chain security lapses (resulting in theft, diversion and product recalls), counterfeiting and stringent regulations. In addition to alarming safety concerns, these challenges also impair the health of the industry by adversely impacting profits, brand credibility and research initiatives. With both industry and governments around the world realizing the significance of implementing product serialization, it becomes mandatory for all entities within the supply chain to comply with federal and/or state legislations pertaining to the locations in which they operate. Typically, drug distribution systems consist of entities such as manufacturers, wholesale distributors and pharmacies before products reach the end consumer. Ensuring secure product track and trace capabilities across various touch points throughout the supply chain - through product serialization implementation - is crucial to address the challenges faced by the industry. Apart from providing visibility and full traceability within the supply chain, successful serialization programs will prove to be a key differentiator and a clear competitive advantage for pharmaceutical companies. This guide, compiled by the Legal and Research team at the Infosys Life Sciences Center of Excellence provides a summary of the legal and regulatory framework proposed by countries including Argentina, Brazil, China, EU, France, India, South Korea, Turkey, and USA (Federal and California) to maintain supply chain integrity and ensure patient safety.
    [Show full text]
  • Rprotobuf: Efficient Cross-Language Data Serialization in R
    RProtoBuf: Efficient Cross-Language Data Serialization in R Dirk Eddelbuettel Murray Stokely Jeroen Ooms Debian Project Google, Inc UCLA Abstract Modern data collection and analysis pipelines often involve a sophisticated mix of applications written in general purpose and specialized programming languages. Many formats commonly used to import and export data between different programs or systems, such as CSV or JSON, are verbose, inefficient, not type-safe, or tied to a specific program- ming language. Protocol Buffers are a popular method of serializing structured data between applications|while remaining independent of programming languages or oper- ating systems. They offer a unique combination of features, performance, and maturity that seems particulary well suited for data-driven applications and numerical comput- ing. The RProtoBuf package provides a complete interface to Protocol Buffers from the R environment for statistical computing. This paper outlines the general class of data serialization requirements for statistical computing, describes the implementation of the RProtoBuf package, and illustrates its use with example applications in large-scale data collection pipelines and web services. Keywords: R, Rcpp, Protocol Buffers, serialization, cross-platform. 1. Introduction Modern data collection and analysis pipelines increasingly involve collections of decoupled components in order to better manage software complexity through reusability, modularity, and fault isolation (Wegiel and Krintz 2010). These pipelines are frequently built using dif- ferent programming languages for the different phases of data analysis | collection, cleaning, modeling, analysis, post-processing, and presentation | in order to take advantage of the unique combination of performance, speed of development, and library support offered by different environments and languages. Each stage of such a data analysis pipeline may pro- arXiv:1401.7372v1 [stat.CO] 28 Jan 2014 duce intermediate results that need to be stored in a file, or sent over the network for further processing.
    [Show full text]
  • Performance Comparison of Data Serialization Schemes for ETSI ITS Car-To-X Communication Systems
    International Journal on Advances in Telecommunications, vol 8 no 1 & 2, year 2015, http://www.iariajournals.org/telecommunications/ 48 Performance Comparison of Data Serialization Schemes for ETSI ITS Car-to-X Communication Systems Sebastian Bittl, Arturo A. Gonzalez, Michael Spahn,¨ and Wolf A. Heidrich Fraunhofer ESK Munich, Germany Email: fsebastian.bittl, arturo.gonzalez, michael.spaehn, [email protected] Abstract—Wireless Car-to-X communication is about to enter sense collision avoidance mechanism (CSMA-CA) for multi- the mass market in upcoming years. The non-licensed, but user channel access, longer packets imply longer transmission reserved bandwidth for such communications is relatively narrow times and hence a higher channel busy ratio. This means that in terms of the few usable channels and the expected number nodes near a transmitter, which are willing to transmit as well, of communicating entities. Among other communication aspects, must wait longer until the channel is free to use. Moreover, data serialization schemes that allow compact encoding as well as the wireless channel has a limited capacity allowing only a fast encoding and decoding are required. Compact representation of data helps in keeping a minimal bandwidth overhead in limited number of transmitted bits per unit of time. The lower the wireless channel. Moreover, since delay is an important the number of transmitted messages, the higher the number of performance parameter in Car-to-X communication, the pro- users that could use the given frequency spectrum in parallel. cessing time of the serialization/deserialization schemes shall be kept to a minimum. So far, no detailed analysis of different The data serialization schemes also influence processing data serialization schemes for Car-to-X communication regarding power requirements in both ETSI Intelligent Transport Sys- important properties such as runtime, memory consumption tems (ITS) in Europe [2] and Wireless Access in Vehicular and encoded output length has been published.
    [Show full text]
  • A Lightweight and Efficient Document Representation
    docrep: A lightweight and efficient document representation framework Tim Dawborn and James R. Curran -lab,e School of Information Technologies University of Sydney NSW 2006, Australia tim.dawborn,james.r.curran @sydney.edu.au { } Abstract Modelling linguistic phenomena requires highly structured and complex data representations. Document representation frameworks (DRFs) provide an interface to store and retrieve multiple annotation layers over a document. Researchers face a difficult choice: using a heavy-weight DRF or implement a custom DRF. The cost is substantial, either learning a new complex system, or continually adding features to a home-grown system that risks overrunning its original scope. We introduce DOCREP, a lightweight and efficient DRF, and compare it against existing DRFs. We discuss our design goals and implementations in C++, Python, and Java. We transform the OntoNotes 5 corpus using DOCREP and UIMA, providing a quantitative comparison, as well as discussing modelling trade-offs. We conclude with qualitative feedback from researchers who have used DOCREP for their own projects. Ultimately, we hope DOCREP is useful for the busy researcher who wants the benefits of a DRF, but has better things to do than to write one. 1 Introduction Computational Linguistics (CL) is increasingly a data-driven research discipline with researchers us- ing diverse collections of large-scale corpora (Parker et al., 2011). Representing linguistic phenomena can require modelling intricate data structures, both flat and hierarchical, layered over the original text; e.g. tokens, sentences, parts-of-speech, named entities, coreference relations, and trees. The scale and complexity of the data demands efficient representations. A document representation framework (DRF) should support the creation, storage, and retrieval of different annotation layers over collections of hetero- geneous documents.
    [Show full text]