DOCSLIB.ORG

Parsing Gigabytes of JSON Per Second

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Parsing Gigabytes of JSON Per Second Parsing Gigabytes of JSON per Second Geoff Langdale · Daniel Lemire Abstract JavaScript Object Notation or JSON is a The JSON syntax can be viewed as a restricted form ubiquitous data exchange format on the Web. Ingesting of JavaScript, but it is used in many programming lan- JSON documents can become a performance bottleneck guages. JSON has four primitive types or atoms (string, due to the sheer volume of data. We are thus motivated number, Boolean, null) that can be embedded within to make JSON parsing as fast as possible. composed types (arrays and objects). An object takes Despite the maturity of the problem of JSON pars- the form of a series of key-value pairs between braces, ing, we show that substantial speedups are possible. where keys are strings (e.g., f"name":"Jack","age":22g). We present the first standard-compliant JSON parser An array is a list of comma-separated values between to process gigabytes of data per second on a single brackets (e.g., [1,"abc",null]). Composed types can core, using commodity processors. We can use a quar- contain primitive types or arbitrarily deeply nested com- ter or fewer instructions than a state-of-the-art refer- posed types as values. See Fig. 1 for an example. The ence parser like RapidJSON. Unlike other validating JSON specification defines six structural characters (`[', parsers, our software (simdjson) makes extensive use of `f', `]', `g', `:', `,'): they serve to delimit the locations Single Instruction, Multiple Data (SIMD) instructions. and structure of objects and arrays. To ensure reproducibility, simdjson is freely available as To access the data contained in a JSON document open-source software under a liberal license. from software, it is typical to transform the JSON text into a tree-like logical representation, akin to the right- hand-side of Fig. 1, an operation we call JSON parsing. 1 Introduction We refer to each value, object and array as a node in the parsed tree. After parsing, the programmer can access JavaScript Object Notation (JSON) is a text format each node in turn and navigate to its siblings or its used to represent data [4]. It is commonly used for children without need for complicated and error-prone browser-server communication on the Web. It is sup- string parsing. ported by many database systems such as MySQL, Post- Parsing large JSON documents is a common task. greSQL, IBM DB2, SQL Server, Oracle, and data-science Palkar et al. state that big-data applications can spend arXiv:1902.08318v6 [cs.DB] 2 Jan 2020 frameworks such as Pandas. Many document-oriented 80{90% of their time parsing JSON documents [25]. databases are centered around JSON such as CouchDB Boncz et al. identified the acceleration of JSON parsing or RethinkDB. as a topic of interest for speeding up database process- Geoff Langdale ing [2]. branchfree.org JSON parsing implies error checking: arrays must Sydney, NSW start and end with a bracket, objects must start and end Australia E-mail: geoff[email protected] with a brace, objects must be made of comma-separated pairs of values (separated by a colon) where all keys are Daniel Lemire Universit´edu Qu´ebec (TELUQ) strings. Numbers must follow the specification and fit Montreal, Quebec within a valid range. Outside of string values, only a Canada few ASCII characters are allowed. Within string val- E-mail: [email protected] ues, several characters (like ASCII line endings) must root { "Width": 800 "Width": 800, "Height": 600, "Height": 600 "Title":"View from my room", "Title": "View from my room" "Url":"http://ex.com "Url": "http://ex.com/img.png" /img.png", "Private": false, "Private": false "Thumbnail":{ "Thumbnail" "Url":"http://ex. com/th.png", "Url": "http://ex.com/th.png" "Height": 125, "Height": 125 "Width": 100 }, "Width": 100 "array":[ "array" 116 , 943 , 116 234 ], 943 "Owner": null 234 } "Owner": null Fig. 1: JSON example be escaped. The JSON specification requires that docu- To our knowledge, publicly available JSON validat- ments use a unicode character encoding (UTF-8, UTF- ing parsers make little use of SIMD instructions. Due 16, or UTF-32), with UTF-8 being the default. Thus to its complexity, the full JSON parsing problem may we must validate the character encoding of all strings. not appear immediately amenable to vectorization. JSON parsing is therefore more onerous than merely One of our core results is that SIMD instructions locating nodes. Our contention is that a parser that combined with minimal branching can lead to new speed accepts erroneous JSON is both dangerous|in that it records for JSON parsing|often processing gigabytes will silently accept malformed JSON whether this has of data per second on a single core. We present sev- been generated accidentally or maliciously|and poorly eral specific performance-oriented strategies that are of specified—it is difficult to anticipate or widely agree on general interest. what the semantics of malformed JSON files should be. { We detect quoted strings, using solely arithmetic To accelerate processing, we should use our proces- and logical operations and a fixed number of in- sors as efficiently as possible. Commodity processors structions per input bytes, while omitting escaped (Intel, AMD, ARM, POWER) support single-instruction- quotes (x 3.1.1). multiple-data (SIMD) instructions. These SIMD instruc- { We differentiate between sets of code-point values tions operate on several words at once unlike regular in- using vectorized classification thus avoiding the bur- structions. For example, starting with the Haswell mi- den of doing N comparisons to recognize that a croarchitecture (2013), Intel and AMD processors sup- value is part of a set of size N (x 3.1.2). port the AVX2 instruction set and 256-bit vector reg- { We validate UTF-8 strings using solely SIMD in- isters. Hence, on recent x64 processors, we can com- structions (x 3.1.5). pare two strings of 32 characters in a single instruction. It is thus straightforward to use SIMD instructions to 2 Related Work locate significant characters (e.g., `"', `=') using few in- structions. We refer to the application of SIMD instruc- A common strategy to accelerate JSON parsing in the tions as vectorization. Vectorized software tends to use literature is to parse selectively. Alagiannis et al. [1] fewer instructions than conventional software. Every- presented NoDB, an approach where one queries the thing else being equal, code that generates fewer in- JSON data without first loading it in the database. It structions is faster. relies in part on selective parsing of the input. Bonetta A closely related concept to vectorization is branch- and Brantner use speculative just-in-time (JIT) com- less processing: whenever the processor must choose be- pilation and selective data access to speed up JSON tween two code paths (a branch), there is a risk of in- processing [3]. They find repeated constant structures curring several cycles of penalty due to a mispredicted and generate code targeting these structures. branch on current pipelined processors. In our experi- Li et al. present their fast parser, Mison which can ence, SIMD instructions are most likely to be beneficial jump directly to a queried field without parsing inter- in a branchless setting. mediate content [17]. Mison uses SIMD instructions to 2 quickly identify some structural characters but other- FishStore [29] parses JSON data and selects sub- wise works by processing bit-vectors in general purpose sets of interest, storing the result in a fast key-value registers with branch-heavy loops. Mison does not at- store [6]. While the original FishStore relied on Mison, tempt to validate documents; it assumes that docu- the open-source version1 uses simdjson by default for ments are pure ASCII as opposed to unicode (UTF- fast parsing. 8). We summarize the most relevant component of the Pavlopoulou et al. [26] propose a parallelized JSON Mison architecture as follows: processor that supports advanced queries and rewrite rules. It avoids the need to first load the data. 1. In a first step, the input document is compared Sparser filters quickly an unprocessed document to against each of the structural characters (`[', `f', `]', find mostly just the relevant information [25], and then `g', `:', `,') as well as the backslash (`n'). Each com- relies on a parser. We could use simdjson with Sparser. parison uses a SIMD instruction, comparing 32 pairs Systems based on selective parsing like Mison or of bytes at a time. The comparisons are then con- Sparser might be beneficial when only a small subset verted into bitmaps where the bit value 1 indicate of the data is of interest. However, if the data is ac- the presence of the corresponding structural char- cessed repeatedly, it might be preferable to load the acter. Mison omits the structural characters related data in a database engine using a standard parser. Non- to arrays (`[', `]') when they are unneeded. Mison validating parsers like Mison might be best with tightly only uses SIMD instructions during this first step; integrated systems where invalid inputs are unlikely. it also appears to be the only step that is essentially branch-free. 2. During a second step, Mison identifies the starting 2.1 XML Parsing and ending point of each string in the document. It uses the quote and backslash bitmaps. For each Before JSON, there has been a lot of similar work done quote character, Mison counts the number of pre- on parsing XML. Noga et al. [24] report that when fewer ceding backslashes using a fast instruction (popcnt): than 80% of the values need to be parsed, it is more quotes preceded by an odd number of backslashes economical to parse just the needed values. Marian et are turned off and ignored.
Load more

Recommended publications
  • JSON Hijacking for the Modern Web About Me
    JSON hijacking For the modern web About me • I’m a researcher at PortSwigger • I love hacking JavaScript let:let{let:[x=1]}=[alert(1)] • I love breaking browsers • @garethheyes History of JSON hijacking • Array constructor attack function Array(){ for(var i=0;i<this.length;i++) { alert(this[i]); } } [1,2,3] • Found by Joe Walker in 2007 • Worked against Gmail in 2007 by Jeremiah Grossman • Fixed in every browser History of JSON hijacking • Object.prototype setter attack Object.prototype.__defineSetter__('user', function(obj){ for(var i in obj) { alert(i + '=' + obj[i]); } }); [{user:{name:"test"}}] • Worked against Twitter • Fixed in every browser Journey of bug discovery James:Can you create a polyglot js/jpeg? Me:Yeah, that sounds like fun. “Polyglot is something that executes in more than one language or format” Anatomy of a jpeg FF D8 FF E0 Anatomy of a jpeg • Start of image marker: FF D8 • Application header: FF E0 00 00 Two bytes we control Anatomy of a jpeg • Guess which two bytes I chose? Rest of app header Valid JS variable • 2F 2A JS comment • /* • FF D8 FF E0 2F 2A 4A 46 49 46 00 01 01 01 00 48 00 48 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00… Padding of nulls for 0x2f2a Anatomy of a jpeg • Inject our payload inside a jpeg comment • FF FE 00 1C • */=alert("Burp rocks.")/* Anatomy of a jpeg • At the end of the image we need to modify the image data • Close our comment • Inject a single line comment after • */// • 2A 2F 2F 2F FF D9 Anatomy of a jpeg • That should work right? <script src="polyglot/uploads/xss.jpg"></script>
    [Show full text]
  • Introduction to JSON (Javascript Object Notation)
    IInnttrroodduuccttiioonn ttoo JJSSOONN ((JJaavvaaSSccrriipptt OObbjjeecctt NNoottaattiioonn)) SSaanngg SShhiinn JJaavvaa TTeecchhnonollogogyy AArrcchhiitteecctt SSunun MMiiccrrososyysstteemmss,, IIncnc.. ssaanngg..sshhiinn@@ssunun..ccoomm wwwwww..jjaavvaapapassssiioon.n.ccoomm Disclaimer & Acknowledgments ● Even though Sang Shin is a full-time employee of Sun Microsystems, the contents here are created as his own personal endeavor and thus does not reflect any official stance of Sun Microsystems 2 Topics • What is JSON? • JSON Data Structure > JSON Object > JSON text • JSON and Java Technology • How to send and receive JSON data at both client and server sides • JSON-RPC • Resources 3 WWhhaatt iiss && WWhhyy JJSSOONN?? What is JSON? • Lightweight data-interchange format > Compared to XML • Simple format > Easy for humans to read and write > Easy for machines to parse and generate • JSON is a text format > Programming language independent > Uses conventions that are familiar to programmers of the C- family of languages, including C, C++, C#, Java, JavaScript, Perl, Python 5 Why Use JSON over XML • Lighter and faster than XML as on-the-wire data format • JSON objects are typed while XML data is typeless > JSON types: string, number, array, boolean, > XML data are all string • Native data form for JavaScript code > Data is readily accessible as JSON objects in your JavaScript code vs. XML data needed to be parsed and assigned to variables through tedious DOM APIs > Retrieving values is as easy as reading from an object property in your JavaScript
    [Show full text]
  • Base64 Character Encoding and Decoding Modeling
    Base64 Character Encoding and Decoding Modeling Isnar Sumartono1, Andysah Putera Utama Siahaan2, Arpan3 Faculty of Computer Science,Universitas Pembangunan Panca Budi Jl. Jend. Gatot Subroto Km. 4,5 Sei Sikambing, 20122, Medan, Sumatera Utara, Indonesia Abstract: Security is crucial to maintaining the confidentiality of the information. Secure information is the information should not be known to the unreliable person, especially information concerning the state and the government. This information is often transmitted using a public network. If the data is not secured in advance, would be easily intercepted and the contents of the information known by the people who stole it. The method used to secure data is to use a cryptographic system by changing plaintext into ciphertext. Base64 algorithm is one of the encryption processes that is ideal for use in data transmission. Ciphertext obtained is the arrangement of the characters that have been tabulated. These tables have been designed to facilitate the delivery of data during transmission. By applying this algorithm, errors would be avoided, and security would also be ensured. Keywords: Base64, Security, Cryptography, Encoding I. INTRODUCTION Security and confidentiality is one important aspect of an information system [9][10]. The information sent is expected to be well received only by those who have the right. Information will be useless if at the time of transmission intercepted or hijacked by an unauthorized person [7]. The public network is one that is prone to be intercepted or hijacked [1][2]. From time to time the data transmission technology has developed so rapidly. Security is necessary for an organization or company as to maintain the integrity of the data and information on the company.
    [Show full text]
  • OCF Core Optional 2.1.0
    OCF Core - Optional Specification VERSION 2.1.0 | November 2019 CONTACT [email protected] Copyright Open Connectivity Foundation, Inc. © 2019 All Rights Reserved. 2 Legal Disclaimer 3 4 NOTHING CONTAINED IN THIS DOCUMENT SHALL BE DEEMED AS GRANTING YOU ANY KIND 5 OF LICENSE IN ITS CONTENT, EITHER EXPRESSLY OR IMPLIEDLY, OR TO ANY 6 INTELLECTUAL PROPERTY OWNED OR CONTROLLED BY ANY OF THE AUTHORS OR 7 DEVELOPERS OF THIS DOCUMENT. THE INFORMATION CONTAINED HEREIN IS PROVIDED 8 ON AN "AS IS" BASIS, AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, 9 THE AUTHORS AND DEVELOPERS OF THIS SPECIFICATION HEREBY DISCLAIM ALL OTHER 10 WARRANTIES AND CONDITIONS, EITHER EXPRESS OR IMPLIED, STATUTORY OR AT 11 COMMON LAW, INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF 12 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. OPEN CONNECTIVITY 13 FOUNDATION, INC. FURTHER DISCLAIMS ANY AND ALL WARRANTIES OF NON- 14 INFRINGEMENT, ACCURACY OR LACK OF VIRUSES. 15 The OCF logo is a trademark of Open Connectivity Foundation, Inc. in the United States or other 16 countries. *Other names and brands may be claimed as the property of others. 17 Copyright © 2016-2019 Open Connectivity Foundation, Inc. All rights reserved. 18 Copying or other form of reproduction and/or distribution of these works are strictly prohibited. 19 Copyright Open Connectivity Foundation, Inc. © 2016-2019. All rights Reserved 20 CONTENTS 21 1 Scope .............................................................................................................................
    [Show full text]
  • Understanding JSON Schema Release 2020-12
    Understanding JSON Schema Release 2020-12 Michael Droettboom, et al Space Telescope Science Institute Sep 14, 2021 Contents 1 Conventions used in this book3 1.1 Language-specific notes.........................................3 1.2 Draft-specific notes............................................4 1.3 Examples.................................................4 2 What is a schema? 7 3 The basics 11 3.1 Hello, World!............................................... 11 3.2 The type keyword............................................ 12 3.3 Declaring a JSON Schema........................................ 13 3.4 Declaring a unique identifier....................................... 13 4 JSON Schema Reference 15 4.1 Type-specific keywords......................................... 15 4.2 string................................................... 17 4.2.1 Length.............................................. 19 4.2.2 Regular Expressions...................................... 19 4.2.3 Format.............................................. 20 4.3 Regular Expressions........................................... 22 4.3.1 Example............................................. 23 4.4 Numeric types.............................................. 23 4.4.1 integer.............................................. 24 4.4.2 number............................................. 25 4.4.3 Multiples............................................ 26 4.4.4 Range.............................................. 26 4.5 object................................................... 29 4.5.1 Properties...........................................
    [Show full text]
  • Convert Json Into Html Table
    Convert Json Into Html Table Unsatisfying and insufficient Ephrayim run-throughs her wavemeters outbalanced or uncoils brassily. Smoke-dried Waring Tedstill serialises: criticizes her derogate throttle and precinct sea-level alkalising Percival and frog seaplane quite mighty desolately. but inures her complacency vividly. Ralline and usufruct Now should return the question is that makes it as a little bit more readable html table into an array into the table header, with intent to In out output html file because i our trying to convert their huge json data. Json-to-HTML-Table 101 NuGet Gallery. Use this tool for convert JSON into an HTML Table And thanks to the MySQL JSONTABLE function we operate always transform the JSON objects into unique virtual. Ankitaps You survive try using Dataweave to convert JSON to XML and probably use XSLT to convert XML to HTML Or simply you read try using. ResponseText convert soap response object a json object appendjsondata pass the. Productivity picks for contributing an html table into the query because checkboxes allow multiple values? My knowledge approach you I built it 4-5 years ago so to embed JSON data augment the HTML page and sludge use JavaScript to crumple that JSON data. When there was this follow users and we only work fast, indentation and beautify and read data into an html generation part goes inside html table into json html tables from an online? Convert JSON to HTML Tool Slick. DOCTYPE html 2 3 4 Convert JSON Data to HTML Table 5 6 th td p input 7 font14px Verdana.
    [Show full text]
  • Msgpack Documentation Release 1.0
    msgpack Documentation Release 1.0 Author 2021-03-18 Contents 1 API reference 3 2 Advanced usage 9 Python Module Index 11 Index 13 i ii msgpack Documentation, Release 1.0 MessagePack is a efficient format for inter language data exchange. Contents 1 msgpack Documentation, Release 1.0 2 Contents CHAPTER 1 API reference msgpack.pack(o, stream, **kwargs) Pack object o and write it to stream See Packer for options. dump() is alias for pack() msgpack.packb(o, **kwargs) Pack object o and return packed bytes See Packer for options. dumps() is alias for packb() msgpack.unpack(stream, **kwargs) Unpack an object from stream. Raises ExtraData when stream contains extra bytes. See Unpacker for options. load() is alias for unpack() msgpack.unpackb(packed, *, object_hook=None, list_hook=None, bool use_list=True, bool raw=False, int timestamp=0, bool strict_map_key=True, unicode_errors=None, object_pairs_hook=None, ext_hook=ExtType, Py_ssize_t max_str_len=-1, Py_ssize_t max_bin_len=-1, Py_ssize_t max_array_len=-1, Py_ssize_t max_map_len=-1, Py_ssize_t max_ext_len=-1) Unpack packed_bytes to object. Returns an unpacked object. Raises ExtraData when packed contains extra bytes. Raises ValueError when packed is incomplete. Raises FormatError when packed is not valid msgpack. Raises StackError when packed contains too nested. Other exceptions can be raised during unpacking. See Unpacker for options. max_xxx_len options are configured automatically from len(packed). loads() is alias for unpackb() 3 msgpack Documentation, Release 1.0 class msgpack.Packer(default=None, *, bool use_single_float=False, bool autoreset=True, bool use_bin_type=True, bool strict_types=False, bool datetime=False, uni- code_errors=None) MessagePack Packer Usage: packer= Packer() astream.write(packer.pack(a)) astream.write(packer.pack(b)) Packer’s constructor has some keyword arguments: Parameters • default (callable) – Convert user type to builtin type that Packer supports.
    [Show full text]
  • The Latest IBM Z COBOL Compiler: Enterprise COBOL V6.2!
    The latest IBM Z COBOL compiler: Enterprise COBOL V6.2! Tom Ross Captain COBOL SHARE Providence August 7,2017 1 COBOL V6.2 ? YES! • The 4 th release of the new generation of IBM Z COBOL compilers • Announced: July 17, 2017 • The same day as IBM z14 hardware…coincidence? • GA: September 8, 2017 • Compiler support for the new IBM z14 hardware and IBM z/OS V2.3 • Ability to exploit the new Vector Packed Decimal Facility of z14 • ‘OLD’ news: • COBOL V5 EOM Sept 11, 2017 (announced Dec 6, 2016) • EOS for COBOL V4 ‘might’ be earlier than 2020, still discussing 2 COBOL V6.2 ? What else does it have? • New and changed COBOL statements, such as the new JSON PARSE statement • Support of COBOL 2002/2014 standards with the addition of the COBOL Conditional Compilation language feature • New and changed COBOL options for increased flexibility • Improved compiler listings with compiler messages at the end of the listing as in previous releases of the compiler • Improved interfaces to optional products and tools such as IBM Debug for z Systems (formerly Debug Tool for z/OS) and IBM Application Discovery and Delivery Intelligence (formerly EzSource) • Compile-time and runtime performance enhancements • Improved usability of the compiler in the z/OS UNIX System Services environment 3 Vector Packed Decimal Facility of z14 • Enterprise COBOL V6.2 adds support for exploiting the new Vector Packed Decimal Facility in z14 through the ARCH(12) compiler option. • The Vector Packed Decimal Facility allows the dominant COBOL data types, packed and zoned decimal, to be handled in wide 16-byte vector registers instead of in memory.
    [Show full text]
  • The Base16, Base32, and Base64 Data Encodings
    Network Working Group S. Josefsson Request for Comments: 4648 SJD Obsoletes: 3548 October 2006 Category: Standards Track The Base16, Base32, and Base64 Data Encodings Status of This Memo This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the “Internet Official Protocol Standards” (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. Copyright Notice Copyright © The Internet Society (2006). All Rights Reserved. Abstract This document describes the commonly used base 64, base 32, and base 16 encoding schemes. It also discusses the use of line-feeds in encoded data, use of padding in encoded data, use of non-alphabet characters in encoded data, use of different encoding alphabets, and canonical encodings. RFC 4648 Base-N Encodings October 2006 Table of Contents 1 Introduction...............................................................................................................................................................3 2 Conventions Used in This Document..................................................................................................................... 4 3 Implementation Discrepancies................................................................................................................................ 5 3.1 Line Feeds in Encoded Data.................................................................................................................................5
    [Show full text]
  • Plain Text & Character Encoding
    Journal of eScience Librarianship Volume 10 Issue 3 Data Curation in Practice Article 12 2021-08-11 Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson Pennsylvania State University Let us know how access to this document benefits ou.y Follow this and additional works at: https://escholarship.umassmed.edu/jeslib Part of the Scholarly Communication Commons, and the Scholarly Publishing Commons Repository Citation Erickson S. Plain Text & Character Encoding: A Primer for Data Curators. Journal of eScience Librarianship 2021;10(3): e1211. https://doi.org/10.7191/jeslib.2021.1211. Retrieved from https://escholarship.umassmed.edu/jeslib/vol10/iss3/12 Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 License. This material is brought to you by eScholarship@UMMS. It has been accepted for inclusion in Journal of eScience Librarianship by an authorized administrator of eScholarship@UMMS. For more information, please contact [email protected]. ISSN 2161-3974 JeSLIB 2021; 10(3): e1211 https://doi.org/10.7191/jeslib.2021.1211 Full-Length Paper Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson The Pennsylvania State University, University Park, PA, USA Abstract Plain text data consists of a sequence of encoded characters or “code points” from a given standard such as the Unicode Standard. Some of the most common file formats for digital data used in eScience (CSV, XML, and JSON, for example) are built atop plain text standards. Plain text representations of digital data are often preferred because plain text formats are relatively stable, and they facilitate reuse and interoperability.
    [Show full text]
  • Advanced JSON Handling in Go 19:40 05 Mar 2020 Jonathan Hall Devops Evangelist / Go Developer / Clean Coder / Salsa Dancer About Me
    Advanced JSON handling in Go 19:40 05 Mar 2020 Jonathan Hall DevOps Evangelist / Go Developer / Clean Coder / Salsa Dancer About me Open Source contributor; CouchDB PMC, author of Kivik Core Tech Lead for Lana Former eCommerce Dev Manager at Bugaboo Former backend developer at Teamwork.com Former backend developer at Booking.com Former tech lead at eFolder/DoubleCheck 2 Show of hands Who has... ...used JSON in a Go program? ...been frustrated by Go's strict typing when dealing with JSON? ...felt limited by Go's standard JSON handling? What have been your biggest frustrations? 3 Today's Topics Very brief intro to JSON in Go Basic use of maps and structs Handling inputs of unknown type Handling data with some unknown fields 4 A brief intro to JSON JavaScript Object Notation, defined by RFC 8259 Human-readable, textual representation of arbitrary data Limted types: null, Number, String, Boolean, Array, Object Broad applications: Config files, data interchange, simple messaging 5 Alternatives to JSON YAML, TOML, INI BSON, MessagePack, CBOR, Smile XML ProtoBuf Custom/proprietary formats Many principles discussed in this presentation apply to any of the above formats. 6 Marshaling JSON Creating JSON from a Go object is (usually) very straight forward: func main() { x := map[string]string{ "foo": "bar", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 7 Marshaling JSON, #2 Creating JSON from a Go object is (usually) very straight forward: func main() { type person struct { Name string `json:"name"` Age int `json:"age"` Description string `json:"descr,omitempty"` secret string // Unexported fields are never (un)marshaled } x := person{ Name: "Bob", Age: 32, secret: "Shhh!", } data, _ := json.Marshal(x) fmt.Println(string(data)) } Run 8 Unmarshaling JSON Unmarshaling JSON is often a bit trickier.
    [Show full text]
  • IDOL RSS Connector (CFS)
    HPE RSS Connector Software Version: 10.11.0 RSS Connector (CFS) Release Notes Document Release Date: November 2015 Software Release Date: November 2015 RSS Connector (CFS) Release Notes Legal Notices Warranty The only warranties for Hewlett Packard Enterprise Development LP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HPE shall not be liable for technical or editorial errors or omissions contained herein. The information contained herein is subject to change without notice. Restricted Rights Legend Confidential computer software. Valid license from HPE required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. Copyright Notice © Copyright 2015 Hewlett Packard Enterprise Development LP Trademark Notices Adobe™ is a trademark of Adobe Systems Incorporated. Microsoft® and Windows® are U.S. registered trademarks of Microsoft Corporation. UNIX® is a registered trademark of The Open Group. This product includes an interface of the 'zlib' general purpose compression library, which is Copyright © 1995-2002 Jean-loup Gailly and Mark Adler. Documentation Updates HPE Big Data Support provides prompt and accurate support to help you quickly and effectively resolve any issue you may encounter while using HPE Big Data products. Support services include access to the Customer Support Site (CSS) for online answers, expertise-based service by HPE Big Data support engineers, and software maintenance to ensure you have the most up-to-date technology.
    [Show full text]