TB3239 JSON Decoder for PIC® and AVR® Devices Introduction Author: Alexandru Niculae, Microchip Technology Inc. This document describes the use and implementation of a JSON subset decoder, aimed for embedded devices such as PIC® and AVR® microcontrollers. The decoder translates the string format of a JSON object into a C data structure representation. This way programmers can access the key-value pairs. Using JSON objects makes it easy to interconnect applications. The JSON Decoder code is available on GitHub. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 1 TB3239 Table of Contents Introduction.....................................................................................................................................................1 1. Overview................................................................................................................................................. 3 2. API.......................................................................................................................................................... 4 3. Implementation........................................................................................................................................7 3.1. Input Buffer...................................................................................................................................7 3.2. Decoding...................................................................................................................................... 7 3.3. Internal Representation................................................................................................................8 3.4. Encoding...................................................................................................................................... 9 4. Demo: Controlling the Four LEDs on AVR-IoT from the PC..................................................................10 4.1. Instructions.................................................................................................................................10 5. Demo: Generic Cloud Communication with an IoT Node..................................................................... 15 6. Appendix............................................................................................................................................... 16 The Microchip Website.................................................................................................................................18 Product Change Notification Service............................................................................................................18 Customer Support........................................................................................................................................ 18 Microchip Devices Code Protection Feature................................................................................................18 Legal Notice................................................................................................................................................. 18 Trademarks.................................................................................................................................................. 19 Quality Management System....................................................................................................................... 19 Worldwide Sales and Service.......................................................................................................................20 © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 2 TB3239 Overview 1. Overview JSON is the most used data serialization format on the web and on desktop applications. Since it has an implementation in virtually all modern programming language, it is the perfect tool for data interchange between applications. It has a text format that is easy for humans to read and write and easy for machines to parse and generate. A JSON object is a set of key-value pairs. While keys can only be strings, values can be a lot of things. This decoder implementation supports string, number, and object values. Note: To keep this library suited for low-memory devices, only a subset of features was selected to be implemented, therefore, array, boolean, and null values are not supported. The following block of code shows an example usage of this decoder. Section 2. API describes the API functions in detail. #define TEST_JSON "{\"main\":{\"key\" : 10,\"foo\":\"bar\"}, \"alt\":2}" ... jsonNode_t *root, *objmain; char str[64], foo[10]; int alt; memcpy(str, TEST_JSON, sizeof(TEST_JSON)); JSON_DECODER_fromString(str) JSON_DECODER_getRoot(&root); JSON_DECODER_getObject(root, "main", &objmain); JSON_DECODER_getNumber(root, "alt", &alt) JSON_DECODER_getString(objmain, "foo", sizeof(foo), foo) Note: The outermost object in a JSON is called the root. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 3 TB3239 API 2. API Refer to this section for detailed explanation of the public functions in the JSON Decoder. jsonDecoderStatus_t JSON_DECODER_fromString(char *str) Description Parses a JSON string into a C representation. It supports string, number, and object values. Parameters Name Description str Pointer to an existing string representation of a JSON object. After parsing, the string will have been altered. Return Value Name Description JSON_DECODER_OK Decoding was successful JSON_DECODER_BAD_FORMAT The input is invalid JSON_DECODER_getRoot(jsonNode_t **pNode) Description Finds the outermost object in the JSON, called the root. This function must not be called before JSON_DECODER_fromString. Parameters Name Description pNode Pointer to a jsonNode_t Return Value Name Description JSON_DECODER_OK pNode now points to the root jsonDecoderStatus_t JSON_DECODER_getObject(jsonNode_t *current, char *key, jsonNode_t **pNode) Description Finds a JSON object by key in another JSON object. This function must not be called before JSON_DECODER_fromString. Parameters Name Description current Pointer to a jsonNode_t © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 4 ...........continued Name Description key The key of the object to find pNode Pointer to the object, if found Return Value Name Description JSON_DECODER_OK Object was found and pNode now points to it JSON_DECODER_KEY_NOT_FOUND The specified key does not exist jsonDecoderStatus_t JSON_DECODER_getString(jsonNode_t *current, char *key, uint8_t size, char TB3239 API Return Value Name Description JSON_DECODER_OK Number was found and pVal now points to it JSON_DECODER_KEY_NOT_FOUND The specified key does not exist © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 6 TB3239 Implementation 3. Implementation This section describes the implementation details with the purpose of facilitating the in-depth understanding of the code. Moreover, it describes some decisions made to optimize the memory usage and implementation safety. For example, because using recursion is not recommended, a part of the algorithm was refactored. 3.1 Input Buffer The input buffer is the buffer in which the JSON string is received, by an MQTT library, a UART driver or by other communication method. This buffer will be passed to the JSON Decoder using the JSON_DECODER_parseString function. When this function returns, the buffer’s content will have been modified. To use as little memory as possible, the internal JSON object representation does not hold any strings, but holds pointers to where the strings were originally located in the input buffer. Therefore, null terminators are inserted into that buffer to isolate the strings. All the keys and the string values are inside the input buffer. Figure 3-1. String Buffer { "" f o o " : " b a r " } JSON_DECODER_parseString { " f o o \0 : " b a r \0 } jsonNode->key jsonNode->value An important consequence is that care must be taken not to rewrite the buffer before the JSON data is consumed. If the buffer is rewritten in between the JSON_DECODE_parseString and accessing the JSON structure, this will corrupt the data. 3.2 Decoding Decoding means transforming the string format of the JSON object into a C data structure representation. This is a grammar parsing problem and it is usually solved in a two-step process: lexing and parsing. Lexing splits the input text into tokens, as well as stripping out spare characters, such as white spaces. Tokens are both single charter (curly braces, commas, colons) and multicharacter (strings, numbers). This implementation uses a “lazy lexer”, meaning that instead of creating the list of all the tokens, they are retrieved one by one as needed by the parser. The parser verifies that the sequence of tokens makes sense in the context of the JSON grammar. It constructs the internal representation into a C data structure. The parser is inherently recursive in nature because it deals with nested JSON objects. To avoid recursion (and possible stack overflow errors), the algorithm was refactored to use a heap stack and a while loop, instead of the actual stack. © 2020 Microchip Technology Inc. Technical Brief DS90003239A-page 7 TB3239 Implementation Figure 3-2. JSON Decoder HLD JSON string JSON Decoder Request token Nonrecursive Parser Lazy Lexer Return next token JSON object in C representation 3.3 Internal Representation After parsing the string format, the JSON object is stored in the memory as a tree-like structure. Using the API functions, the tree nodes are traversed