JSON Decoder for PIC® and AVR® Devices

Total Page:16

File Type:pdf, Size:1020Kb

JSON Decoder for PIC® and AVR® Devices TB3239 JSON Decoder for PIC® and AVR® Devices Introduction Author: Alexandru Niculae, Microchip Technology Inc. This document describes the use and implementation of a JSON subset decoder, aimed for embedded devices such as PIC® and AVR® microcontrollers. The decoder translates the string format of a JSON object into a C data structure representation. This way programmers can access the key-value pairs. Using JSON objects makes it easy to interconnect applications. The JSON Decoder code is available on GitHub. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 1 TB3239 Table of Contents Introduction.....................................................................................................................................................1 1. Overview................................................................................................................................................. 3 2. API.......................................................................................................................................................... 4 3. Implementation........................................................................................................................................7 3.1. Input Buffer...................................................................................................................................7 3.2. Decoding...................................................................................................................................... 7 3.3. Internal Representation................................................................................................................8 3.4. Encoding...................................................................................................................................... 9 4. Demo: Controlling the Four LEDs on AVR-IoT from the PC..................................................................10 4.1. Instructions.................................................................................................................................10 5. Demo: Generic Cloud Communication with an IoT Node..................................................................... 15 6. Appendix............................................................................................................................................... 16 The Microchip Website.................................................................................................................................18 Product Change Notification Service............................................................................................................18 Customer Support........................................................................................................................................ 18 Microchip Devices Code Protection Feature................................................................................................18 Legal Notice................................................................................................................................................. 18 Trademarks.................................................................................................................................................. 19 Quality Management System....................................................................................................................... 19 Worldwide Sales and Service.......................................................................................................................20 © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 2 TB3239 Overview 1. Overview JSON is the most used data serialization format on the web and on desktop applications. Since it has an implementation in virtually all modern programming language, it is the perfect tool for data interchange between applications. It has a text format that is easy for humans to read and write and easy for machines to parse and generate. A JSON object is a set of key-value pairs. While keys can only be strings, values can be a lot of things. This decoder implementation supports string, number, and object values. Note: To keep this library suited for low-memory devices, only a subset of features was selected to be implemented, therefore, array, boolean, and null values are not supported. The following block of code shows an example usage of this decoder. Section 2. API describes the API functions in detail. #define TEST_JSON "{\"main\":{\"key\" : 10,\"foo\":\"bar\"}, \"alt\":2}" ... jsonNode_t *root, *objmain; char str[64], foo[10]; int alt; memcpy(str, TEST_JSON, sizeof(TEST_JSON)); JSON_DECODER_fromString(str) JSON_DECODER_getRoot(&root); JSON_DECODER_getObject(root, "main", &objmain); JSON_DECODER_getNumber(root, "alt", &alt) JSON_DECODER_getString(objmain, "foo", sizeof(foo), foo) Note: The outermost object in a JSON is called the root. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 3 TB3239 API 2. API Refer to this section for detailed explanation of the public functions in the JSON Decoder. jsonDecoderStatus_t JSON_DECODER_fromString(char *str) Description Parses a JSON string into a C representation. It supports string, number, and object values. Parameters Name Description str Pointer to an existing string representation of a JSON object. After parsing, the string will have been altered. Return Value Name Description JSON_DECODER_OK Decoding was successful JSON_DECODER_BAD_FORMAT The input is invalid JSON_DECODER_getRoot(jsonNode_t **pNode) Description Finds the outermost object in the JSON, called the root. This function must not be called before JSON_DECODER_fromString. Parameters Name Description pNode Pointer to a jsonNode_t Return Value Name Description JSON_DECODER_OK pNode now points to the root jsonDecoderStatus_t JSON_DECODER_getObject(jsonNode_t *current, char *key, jsonNode_t **pNode) Description Finds a JSON object by key in another JSON object. This function must not be called before JSON_DECODER_fromString. Parameters Name Description current Pointer to a jsonNode_t © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 4 ...........continued Name Description key The key of the object to find pNode Pointer to the object, if found Return Value Name Description JSON_DECODER_OK Object was found and pNode now points to it JSON_DECODER_KEY_NOT_FOUND The specified key does not exist jsonDecoderStatus_t JSON_DECODER_getString(jsonNode_t *current, char *key, uint8_t size, char TB3239 API Return Value Name Description JSON_DECODER_OK Number was found and pVal now points to it JSON_DECODER_KEY_NOT_FOUND The specified key does not exist © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 6 TB3239 Implementation 3. Implementation This section describes the implementation details with the purpose of facilitating the in-depth understanding of the code. Moreover, it describes some decisions made to optimize the memory usage and implementation safety. For example, because using recursion is not recommended, a part of the algorithm was refactored. 3.1 Input Buffer The input buffer is the buffer in which the JSON string is received, by an MQTT library, a UART driver or by other communication method. This buffer will be passed to the JSON Decoder using the JSON_DECODER_parseString function. When this function returns, the buffer’s content will have been modified. To use as little memory as possible, the internal JSON object representation does not hold any strings, but holds pointers to where the strings were originally located in the input buffer. Therefore, null terminators are inserted into that buffer to isolate the strings. All the keys and the string values are inside the input buffer. Figure 3-1. String Buffer { "" f o o " : " b a r " } JSON_DECODER_parseString { " f o o \0 : " b a r \0 } jsonNode->key jsonNode->value An important consequence is that care must be taken not to rewrite the buffer before the JSON data is consumed. If the buffer is rewritten in between the JSON_DECODE_parseString and accessing the JSON structure, this will corrupt the data. 3.2 Decoding Decoding means transforming the string format of the JSON object into a C data structure representation. This is a grammar parsing problem and it is usually solved in a two-step process: lexing and parsing. Lexing splits the input text into tokens, as well as stripping out spare characters, such as white spaces. Tokens are both single charter (curly braces, commas, colons) and multicharacter (strings, numbers). This implementation uses a “lazy lexer”, meaning that instead of creating the list of all the tokens, they are retrieved one by one as needed by the parser. The parser verifies that the sequence of tokens makes sense in the context of the JSON grammar. It constructs the internal representation into a C data structure. The parser is inherently recursive in nature because it deals with nested JSON objects. To avoid recursion (and possible stack overflow errors), the algorithm was refactored to use a heap stack and a while loop, instead of the actual stack. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 7 TB3239 Implementation Figure 3-2. JSON Decoder HLD JSON string JSON Decoder Request token Nonrecursive Parser Lazy Lexer Return next token JSON object in C representation 3.3 Internal Representation After parsing the string format, the JSON object is stored in the memory as a tree-like structure. Using the API functions, the tree nodes are traversed
Recommended publications
  • JSON Hijacking for the Modern Web About Me
    JSON hijacking For the modern web About me • I’m a researcher at PortSwigger • I love hacking JavaScript let:let{let:[x=1]}=[alert(1)] • I love breaking browsers • @garethheyes History of JSON hijacking • Array constructor attack function Array(){ for(var i=0;i<this.length;i++) { alert(this[i]); } } [1,2,3] • Found by Joe Walker in 2007 • Worked against Gmail in 2007 by Jeremiah Grossman • Fixed in every browser History of JSON hijacking • Object.prototype setter attack Object.prototype.__defineSetter__('user', function(obj){ for(var i in obj) { alert(i + '=' + obj[i]); } }); [{user:{name:"test"}}] • Worked against Twitter • Fixed in every browser Journey of bug discovery James:Can you create a polyglot js/jpeg? Me:Yeah, that sounds like fun. “Polyglot is something that executes in more than one language or format” Anatomy of a jpeg FF D8 FF E0 Anatomy of a jpeg • Start of image marker: FF D8 • Application header: FF E0 00 00 Two bytes we control Anatomy of a jpeg • Guess which two bytes I chose? Rest of app header Valid JS variable • 2F 2A JS comment • /* • FF D8 FF E0 2F 2A 4A 46 49 46 00 01 01 01 00 48 00 48 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00… Padding of nulls for 0x2f2a Anatomy of a jpeg • Inject our payload inside a jpeg comment • FF FE 00 1C • */=alert("Burp rocks.")/* Anatomy of a jpeg • At the end of the image we need to modify the image data • Close our comment • Inject a single line comment after • */// • 2A 2F 2F 2F FF D9 Anatomy of a jpeg • That should work right? <script src="polyglot/uploads/xss.jpg"></script>
    [Show full text]
  • Introduction to JSON (Javascript Object Notation)
    IInnttrroodduuccttiioonn ttoo JJSSOONN ((JJaavvaaSSccrriipptt OObbjjeecctt NNoottaattiioonn)) SSaanngg SShhiinn JJaavvaa TTeecchhnonollogogyy AArrcchhiitteecctt SSunun MMiiccrrososyysstteemmss,, IIncnc.. ssaanngg..sshhiinn@@ssunun..ccoomm wwwwww..jjaavvaapapassssiioon.n.ccoomm Disclaimer & Acknowledgments ● Even though Sang Shin is a full-time employee of Sun Microsystems, the contents here are created as his own personal endeavor and thus does not reflect any official stance of Sun Microsystems 2 Topics • What is JSON? • JSON Data Structure > JSON Object > JSON text • JSON and Java Technology • How to send and receive JSON data at both client and server sides • JSON-RPC • Resources 3 WWhhaatt iiss && WWhhyy JJSSOONN?? What is JSON? • Lightweight data-interchange format > Compared to XML • Simple format > Easy for humans to read and write > Easy for machines to parse and generate • JSON is a text format > Programming language independent > Uses conventions that are familiar to programmers of the C- family of languages, including C, C++, C#, Java, JavaScript, Perl, Python 5 Why Use JSON over XML • Lighter and faster than XML as on-the-wire data format • JSON objects are typed while XML data is typeless > JSON types: string, number, array, boolean, > XML data are all string • Native data form for JavaScript code > Data is readily accessible as JSON objects in your JavaScript code vs. XML data needed to be parsed and assigned to variables through tedious DOM APIs > Retrieving values is as easy as reading from an object property in your JavaScript
    [Show full text]
  • OCF Core Optional 2.1.0
    OCF Core - Optional Specification VERSION 2.1.0 | November 2019 CONTACT [email protected] Copyright Open Connectivity Foundation, Inc. © 2019 All Rights Reserved. 2 Legal Disclaimer 3 4 NOTHING CONTAINED IN THIS DOCUMENT SHALL BE DEEMED AS GRANTING YOU ANY KIND 5 OF LICENSE IN ITS CONTENT, EITHER EXPRESSLY OR IMPLIEDLY, OR TO ANY 6 INTELLECTUAL PROPERTY OWNED OR CONTROLLED BY ANY OF THE AUTHORS OR 7 DEVELOPERS OF THIS DOCUMENT. THE INFORMATION CONTAINED HEREIN IS PROVIDED 8 ON AN "AS IS" BASIS, AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, 9 THE AUTHORS AND DEVELOPERS OF THIS SPECIFICATION HEREBY DISCLAIM ALL OTHER 10 WARRANTIES AND CONDITIONS, EITHER EXPRESS OR IMPLIED, STATUTORY OR AT 11 COMMON LAW, INCLUDING, BUT NOT LIMITED TO, IMPLIED WARRANTIES OF 12 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. OPEN CONNECTIVITY 13 FOUNDATION, INC. FURTHER DISCLAIMS ANY AND ALL WARRANTIES OF NON- 14 INFRINGEMENT, ACCURACY OR LACK OF VIRUSES. 15 The OCF logo is a trademark of Open Connectivity Foundation, Inc. in the United States or other 16 countries. *Other names and brands may be claimed as the property of others. 17 Copyright © 2016-2019 Open Connectivity Foundation, Inc. All rights reserved. 18 Copying or other form of reproduction and/or distribution of these works are strictly prohibited. 19 Copyright Open Connectivity Foundation, Inc. © 2016-2019. All rights Reserved 20 CONTENTS 21 1 Scope .............................................................................................................................
    [Show full text]
  • Understanding JSON Schema Release 2020-12
    Understanding JSON Schema Release 2020-12 Michael Droettboom, et al Space Telescope Science Institute Sep 14, 2021 Contents 1 Conventions used in this book3 1.1 Language-specific notes.........................................3 1.2 Draft-specific notes............................................4 1.3 Examples.................................................4 2 What is a schema? 7 3 The basics 11 3.1 Hello, World!............................................... 11 3.2 The type keyword............................................ 12 3.3 Declaring a JSON Schema........................................ 13 3.4 Declaring a unique identifier....................................... 13 4 JSON Schema Reference 15 4.1 Type-specific keywords......................................... 15 4.2 string................................................... 17 4.2.1 Length.............................................. 19 4.2.2 Regular Expressions...................................... 19 4.2.3 Format.............................................. 20 4.3 Regular Expressions........................................... 22 4.3.1 Example............................................. 23 4.4 Numeric types.............................................. 23 4.4.1 integer.............................................. 24 4.4.2 number............................................. 25 4.4.3 Multiples............................................ 26 4.4.4 Range.............................................. 26 4.5 object................................................... 29 4.5.1 Properties...........................................
    [Show full text]
  • Convert Json Into Html Table
    Convert Json Into Html Table Unsatisfying and insufficient Ephrayim run-throughs her wavemeters outbalanced or uncoils brassily. Smoke-dried Waring Tedstill serialises: criticizes her derogate throttle and precinct sea-level alkalising Percival and frog seaplane quite mighty desolately. but inures her complacency vividly. Ralline and usufruct Now should return the question is that makes it as a little bit more readable html table into an array into the table header, with intent to In out output html file because i our trying to convert their huge json data. Json-to-HTML-Table 101 NuGet Gallery. Use this tool for convert JSON into an HTML Table And thanks to the MySQL JSONTABLE function we operate always transform the JSON objects into unique virtual. Ankitaps You survive try using Dataweave to convert JSON to XML and probably use XSLT to convert XML to HTML Or simply you read try using. ResponseText convert soap response object a json object appendjsondata pass the. Productivity picks for contributing an html table into the query because checkboxes allow multiple values? My knowledge approach you I built it 4-5 years ago so to embed JSON data augment the HTML page and sludge use JavaScript to crumple that JSON data. When there was this follow users and we only work fast, indentation and beautify and read data into an html generation part goes inside html table into json html tables from an online? Convert JSON to HTML Tool Slick. DOCTYPE html 2 3 4 Convert JSON Data to HTML Table 5 6 th td p input 7 font14px Verdana.
    [Show full text]
  • The Latest IBM Z COBOL Compiler: Enterprise COBOL V6.2!
    The latest IBM Z COBOL compiler: Enterprise COBOL V6.2! Tom Ross Captain COBOL SHARE Providence August 7,2017 1 COBOL V6.2 ? YES! • The 4 th release of the new generation of IBM Z COBOL compilers • Announced: July 17, 2017 • The same day as IBM z14 hardware…coincidence? • GA: September 8, 2017 • Compiler support for the new IBM z14 hardware and IBM z/OS V2.3 • Ability to exploit the new Vector Packed Decimal Facility of z14 • ‘OLD’ news: • COBOL V5 EOM Sept 11, 2017 (announced Dec 6, 2016) • EOS for COBOL V4 ‘might’ be earlier than 2020, still discussing 2 COBOL V6.2 ? What else does it have? • New and changed COBOL statements, such as the new JSON PARSE statement • Support of COBOL 2002/2014 standards with the addition of the COBOL Conditional Compilation language feature • New and changed COBOL options for increased flexibility • Improved compiler listings with compiler messages at the end of the listing as in previous releases of the compiler • Improved interfaces to optional products and tools such as IBM Debug for z Systems (formerly Debug Tool for z/OS) and IBM Application Discovery and Delivery Intelligence (formerly EzSource) • Compile-time and runtime performance enhancements • Improved usability of the compiler in the z/OS UNIX System Services environment 3 Vector Packed Decimal Facility of z14 • Enterprise COBOL V6.2 adds support for exploiting the new Vector Packed Decimal Facility in z14 through the ARCH(12) compiler option. • The Vector Packed Decimal Facility allows the dominant COBOL data types, packed and zoned decimal, to be handled in wide 16-byte vector registers instead of in memory.
    [Show full text]
  • Plain Text & Character Encoding
    Journal of eScience Librarianship Volume 10 Issue 3 Data Curation in Practice Article 12 2021-08-11 Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson Pennsylvania State University Let us know how access to this document benefits ou.y Follow this and additional works at: https://escholarship.umassmed.edu/jeslib Part of the Scholarly Communication Commons, and the Scholarly Publishing Commons Repository Citation Erickson S. Plain Text & Character Encoding: A Primer for Data Curators. Journal of eScience Librarianship 2021;10(3): e1211. https://doi.org/10.7191/jeslib.2021.1211. Retrieved from https://escholarship.umassmed.edu/jeslib/vol10/iss3/12 Creative Commons License This work is licensed under a Creative Commons Attribution 4.0 License. This material is brought to you by eScholarship@UMMS. It has been accepted for inclusion in Journal of eScience Librarianship by an authorized administrator of eScholarship@UMMS. For more information, please contact [email protected]. ISSN 2161-3974 JeSLIB 2021; 10(3): e1211 https://doi.org/10.7191/jeslib.2021.1211 Full-Length Paper Plain Text & Character Encoding: A Primer for Data Curators Seth Erickson The Pennsylvania State University, University Park, PA, USA Abstract Plain text data consists of a sequence of encoded characters or “code points” from a given standard such as the Unicode Standard. Some of the most common file formats for digital data used in eScience (CSV, XML, and JSON, for example) are built atop plain text standards. Plain text representations of digital data are often preferred because plain text formats are relatively stable, and they facilitate reuse and interoperability.
    [Show full text]
  • JSON JSON (Javascript Object Notation) Ecmascript Javascript
    ECMAScript JSON ECMAScript is standardized JavaScript. The current version is the 12th edition: Péter Jeszenszky Ecma International, ECMAScript 2021 Language Specification, Standard ECMA-262, 12th ed., June 2021. https://www.ecma- international.org/publications-and-standards/standards/ecma-262/ October 8, 2021 The next version currently under development is ECMAScript 2022: ECMAScript 2022 Language Specification https://tc39.es/ecma262/ Péter Jeszenszky JSON October 8, 2021 1 / 94 Péter Jeszenszky JSON October 8, 2021 3 / 94 JSON (JavaScript Object Notation) JavaScript Lightweight, textual, and platform independent data exchange format. Used for representing structured data. The term JavaScript is used for the implementations of ECMAScript by different vendors. Can be read and written easily by humans. See also: JavaScript technologies overview Can be generated and processed easily by computer programs. https://developer.mozilla.org/en- US/docs/Web/JavaScript/JavaScript_technologies_overview Originates from the ECMAScript programming language. Website: https://www.json.org/ Péter Jeszenszky JSON October 8, 2021 2 / 94 Péter Jeszenszky JSON October 8, 2021 4 / 94 JavaScript Engines (1) Node.js (1) SpiderMonkey (written in: C/C++; license: Mozilla Public License 2.0) https://spidermonkey.dev/ A JavaScript runtime environment built on the V8 JavaScript engine The JavaScript engine of the Mozilla Project. that is designed to build scalable network applications. V8 (written in: C++; license: New BSD License) https://v8.dev/ Website: https://nodejs.org/ https://github.com/nodejs/node https://github.com/v8/v8/ License: MIT License The JavaScript engine of Chromium. Written in: C++, JavaScript JavaScriptCore (written in: C++; license: LGPLv2) https://developer.apple.com/documentation/javascriptcore https: Platform: Linux, macOS, Windows //github.com/WebKit/webkit/tree/master/Source/JavaScriptCore The JavaScript engine developed for the WebKit rendering engine.
    [Show full text]
  • International Standard Iso/Iec 9075-2
    This is a previewINTERNATIONAL - click here to buy the full publication ISO/IEC STANDARD 9075-2 Fifth edition 2016-12-15 Information technology — Database languages — SQL — Part 2: Foundation (SQL/Foundation) Technologies de l’information — Langages de base de données — SQL — Partie 2: Fondations (SQL/Fondations) Reference number ISO/IEC 9075-2:2016(E) © ISO/IEC 2016 ISO/IEC 9075-2:2016(E) This is a preview - click here to buy the full publication COPYRIGHT PROTECTED DOCUMENT © ISO/IEC 2016, Published in Switzerland All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form orthe by requester. any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of Ch. de Blandonnet 8 • CP 401 ISOCH-1214 copyright Vernier, office Geneva, Switzerland Tel. +41 22 749 01 11 Fax +41 22 749 09 47 www.iso.org [email protected] ii © ISO/IEC 2016 – All rights reserved This is a preview - click here to buy the full publication ISO/IEC 9075-2:2016(E) Contents Page Foreword..................................................................................... xxi Introduction.................................................................................. xxii 1 Scope.................................................................................... 1 2 Normative references.....................................................................
    [Show full text]
  • 2021-2022 Recommended Formats Statement
    Library of Congress Recommended Formats Statement 2021-2022 For online version, see Recommended Formats Statement - 2021-2022 (link) Introduction to the 2021-2022 revision ....................................................................... 3 I. Textual Works ...................................................................................................... 5 i. Textual Works – Print (books, etc.) ........................................................................................................... 5 ii. Textual Works – Digital ............................................................................................................................ 7 iii. Textual Works – Electronic serials .......................................................................................................... 9 II. Still Image Works ............................................................................................... 12 i. Photographs – Print ................................................................................................................................ 12 ii. Photographs – Digital............................................................................................................................. 13 iii. Other Graphic Images – Print (posters, postcards, fine prints) ............................................................ 14 iv. Other Graphic Images – Digital ............................................................................................................. 15 v. Microforms ...........................................................................................................................................
    [Show full text]
  • ISO/IEC Version of ECMA-404
    © ISO/IEC 2015 – All rights reserved Ecma/TC39/2015/025 ISO/IEC JTC 1/SC 22 First edition 2015-mm-dd ISO/IEC XXXXX:2015 (E) ISO/IEC/JTC 1/SC 22 Secretariat: ANSI Information technology — Programming languages, their environments and system software interfaces — The JSON Data Interchange Format Document type: International standard Document subtype: if applicable Document stage: (20) Preparation Document language: E ISO/IEC XXXXX:2015(E) Copyright notice This ISO document is a Draft International Standard and is copyright-protected by ISO. Except as permitted under the applicable laws of the user's country, neither this ISO draft nor any extract from it may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, photocopying, recording or otherwise, without prior written permission being secured. Requests for permission to reproduce should be addressed to either ISO at the address below or ISO's member body in the country of the requester. ISO copyright office Case postale 56 • CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail [email protected] Web www.iso.org Reproduction may be subject to royalty payments or a licensing agreement. Violators may be prosecuted. ii © ISO/IEC 2015 – All rights reserved ISO/IEC XXXXX:2015(E) Contents Page 1 Scope ....................................................................................................................................................... 1 2 Conformance .........................................................................................................................................
    [Show full text]
  • CAT Reporting Customer & Account Technical Specifications for Industry
    CAT Reporting Customer & Account Technical Specifications for Industry Members 4/21/2020 Version 1.1 Table of Contents Preface ........................................................................................................................................................ iii Executive Summary ................................................................................................................................... iv 1. Introduction ......................................................................................................................................... 1 1.1. CAT Overview ............................................................................................................................. 1 2. CAT Customer & Account Reporting Fundamentals ...................................................................... 2 2.1. Key Data Elements ..................................................................................................................... 2 Firm Identifiers in File Submissions ........................................................................... 2 2.2. Reference Data ........................................................................................................................... 2 Firm Designated ID (FDID) ........................................................................................... 2 Large Trader ID (LTID) and Unidentified Large Trader ID (ULTID)........................... 3 FDID Open Date ...........................................................................................................
    [Show full text]