JSON at Work PRACTICAL DATA INTEGRATION FOR THE WEB Tom Marrs JSON at Work Practical Data Integration for the Web Tom Marrs Beijing Boston Farnham Sebastopol Tokyo JSON at Work by Tom Marrs Copyright © 2017 Vertical Slice, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐ tutional sales department: 800-998-9938 or [email protected]. Editor: Meg Foley Indexer: Ellen Troutman-Zaig Production Editor: Nicholas Adams Interior Designer: David Futato Copyeditor: Sharon Wilkey Cover Designer: Randy Comer Proofreader: Charles Roumeliotis Illustrator: Rebecca Demarest July 2017: First Edition Revision History for the First Edition 2017-06-16: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781449358327 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. JSON at Work, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-449-35832-7 [LSI] To everyone who produces or consumes JSON data with web/mobile applications, REST APIs, and messaging systems—I hope this makes your job easier. To the unsung JSON community that produces JSON-based tools and libraries for the rest of us—thank you for all your hard work to make JSON useful and meaningful. Table of Contents Preface. xiii Part I. JSON Overview and Platforms 1. JSON Overview. 3 JSON Is a Standard 3 A Brief Sample 4 Why JSON? 6 Core JSON 8 JSON Data Types 8 JSON Value Types 11 JSON Versions 14 JSON Comments 14 JSON File and MIME Type 14 JSON Style Guidelines 15 Our Example—MyConference 17 Our Technical Stack 17 Our Architectural Style—noBackEnd 17 Model JSON Data with JSON Editor Online 18 Generate Sample JSON Data with JSON Generator 20 Create and Deploy a Stub API 20 What We Covered? 24 What’s Next? 24 2. JSON in JavaScript. 25 Node.js Setup 25 JSON Serialization/Deserialization with JSON.stringify() and JSON.parse() 26 v The JSON Stringifier/Parser Object 26 JSON Serialization with Simple JavaScript Data Types 26 JSON Serialization with an Object and toJSON() 29 JSON Deserialization Using eval() 30 JSON Deserialization with an Object and JSON.parse() 31 JavaScript Objects and JSON 32 Node REPL 33 Where to Learn More About JavaScript Objects 35 Unit Testing with a Stub API 35 Unit Test Style—TDD and BDD 35 Just Enough Unit Testing with Mocha and Chai 36 Setting Up the Unit Test 36 Unirest 36 Test Data 37 Speakers Unit Test 37 Building a Small Web Application 39 Yeoman 39 Iteration 1—Generate a Web Application with Yeoman 41 Iteration 2—Make an HTTP Call with jQuery 45 Iteration 3—Consume Speaker Data from a Stub API and Use a Template 49 How to Go Deeper with JavaScript 54 What We Covered 55 What’s Next? 55 3. JSON in Ruby on Rails. 57 Ruby on Rails Setup 57 Ruby JSON Gems 58 JSON Serialization/Deserialization with MultiJson 58 The MultiJson Object 58 JSON Serialization/Deserialization with Simple Ruby Data Types 59 JSON Deserialization with Objects and MultiJson 61 A Word on Camel Casing and JSON 63 JSON Serialization with Objects and ActiveSupport 64 JSON Deserialization with Objects and ActiveSupport 65 Unit Testing with a Stub API 66 Just Enough Unit Testing with Minitest 66 Setting Up the Unit Test 67 Test Data 68 JSON and Minitest Testing with APIs 68 Speakers Unit Test 68 Further Reading on Ruby and Minitest 72 What Is Missing in the Unit Tests? 72 vi | Table of Contents Build a Small Web API with Ruby on Rails 73 Choose a JSON Serializer 73 speakers-api-1—Create an API with Camel-Cased JSON 75 speakers-api-2—Create an API that Customizes the JSON Representation 82 Further Reading on Rails and Rails-based APIs 84 What We Covered 84 What’s Next? 84 4. JSON in Java. 85 Java and Gradle Setup 85 Gradle Overview 85 Just Enough Unit Testing with JUnit 87 Java-Based JSON Libraries 87 JSON Serialization/Deserialization with Jackson 88 Serialization/Deserialization with Simple Java Data Types 88 Serialization/Deserialization with Java Objects 91 Unit Testing with a Stub API 96 Test Data 96 JSON and JUnit Testing with APIs 96 Build a Small Web API with Spring Boot 100 Create the Model 101 Create the Controller 103 Register the Application 104 Write the Build Script 105 Deploy the API 107 Test the API with Postman 107 What We Covered 108 What’s Next? 109 Part II. The JSON Ecosystem 5. JSON Schema. 113 JSON Schema Overview 113 What Is JSON Schema? 113 Syntactic Versus Semantic Validation 114 A Simple Example 114 JSON Schema on the Web 115 Why JSON Schema? 116 My Journey with JSON Schema 117 The Current State of the JSON Schema Standard 117 JSON Schema and XML Schema 117 Table of Contents | vii Core JSON Schema—Basics and Tooling 118 JSON Schema Workflow and Tooling 118 Core Keywords 120 Basic Types 121 Numbers 125 Arrays 126 Enumerated Values 128 Objects 129 Pattern Properties 131 Regular Expressions 133 Dependent Properties 135 Internal References 136 External References 138 Choosing Validation Rules 141 How to Design and Test an API with JSON Schema 146 Our Scenario 146 Model a JSON Document 146 Generate a JSON Schema 148 Validate the JSON Document 151 Generate Sample Data 152 Deploy a Stub API with json-server 155 Final Thoughts on API Design and Testing with JSON Schema 157 Validation Using a JSON Schema Library 157 Where to Go Deeper with JSON Schema 158 What We Covered 158 What’s Next? 158 6. JSON Search. 159 Why JSON Search? 159 JSON Search Libraries and Tools 160 Honorable Mention 160 What to Look For 160 Test Data 161 Setting Up Unit Tests 162 Comparing JSON Search Libraries and Tools 163 JSONPath 163 JSON Pointer 170 jq 173 JSON Search Library and Tool Evaluations—The Bottom Line 184 What We Covered 185 What’s Next? 185 viii | Table of Contents 7. JSON Transform. 187 Types of JSON Transformation 187 What to Look For in a JSON Transform Library 188 Test Input Data 189 JSON-to-HTML Transformation 191 Target HTML Document 191 Mustache 192 Handlebars 198 JSON-to-HTML Transformation Evaluations—The Bottom Line 204 JSON-to-JSON Transform 204 The Issues 205 JSON-to-JSON Transform Libraries 205 Honorable Mention 205 Target JSON Output 206 JSON Patch 207 JSON-T 213 Mustache 217 Handlebars 219 JSON-to-JSON Transformation Evaluations—The Bottom Line 221 JSON-XML Transformation 222 JSON-XML Transformation Conventions 222 The Issues with JSON-XML Transformation Conventions 231 XML-JSON Transform—The Bottom Line 231 JSON-XML Transformation Unit Test 233 What We Covered 235 What’s Next? 235 Part III. JSON in the Enterprise 8. JSON and Hypermedia. 239 Comparing Hypermedia Formats 240 Defining Key Terms 241 My Opinion on Hypermedia 241 Siren 242 JSON-LD 244 Collection+JSON 249 json:api 250 HAL 254 Conclusions on Hypermedia 259 Recommendations for Working with Hypermedia 260 Practical Issues with Hypermedia 260 Table of Contents | ix Testing with HAL in the Speakers API 261 Test Data 261 HAL Unit Test 263 Server-Side HAL 267 Going Deeper with Hypermedia 268 What We Covered 268 What’s Next? 268 9. JSON and MongoDB. 269 What About BSON? 269 MongoDB Setup 270 MongoDB Server and Tools 270 MongoDB Server 270 Importing JSON into MongoDB 271 MongoDB Command Shell 273 Basic CRUD with mongo 274 Exporting from MongoDB to a JSON Document 277 What About Schema? 280 RESTful API Testing with MongoDB 281 Test Input Data 282 Providing a RESTful Wrapper for MongoDB 282 What We Covered 285 What’s Next? 285 10. JSON Messaging with Kafka. 287 Kafka Use Cases 288 Kafka Concepts and Terminology 288 The Kafka Ecosystem—Related Projects 289 Kafka Environment Setup 290 Why Do I Need ZooKeeper? 290 Kafka Command-Line Interface (CLI) 291 How to Publish a JSON Message with the CLI 291 Start ZooKeeper 291 Start Kafka 292 Create a Topic 292 List Topics 293 Start a Consumer 293 Publish a JSON Message 294 Consume a JSON Message 295 Clean Up and Shut Down Kafka 295 Kafka Libraries 297 End-to-End Example—Speaker Proposals at MyConference 297 x | Table of Contents Test Data 297 Architecture Components 299 Set Up the Kafka Environment 300 Set Up Fake Email Server and Client—MailCatcher 301 Set Up Node.js Project Environment 302 Speaker Proposal Producer (Send Speaker Proposals) 302 Proposal Reviewer (Consumer/Producer) 302 Speaker Notifier (Consumer) 308 Review Notification Email Messages with MailCatcher 313 What We Covered 315 A. Installation Guides. 317 B. JSON Community. 335 Index. 337 Table of Contents | xi Preface JavaScript Object Notation (JSON) has become the de facto standard for RESTful interfaces, but an ecosystem of little-known standards, tools, and technologies is available that architects and developers can start using today to build well-designed applications.