A Restful Middleware Application for Storing JSON Data

Third Year Project Report

A RESTful Middleware Application for Storing JSON Data

Author: Alexandru Aanei Supervisor: Prof. Norman Paton

A report submitted in partial fulfillment of the requirements for the degree of BSc (Hons) Computer Science with Industrial Experience

May 2016 Abstract

Over the last decade, Representational State Transfer, commonly known as REST, has become a popular style for designing the APIs of distributed applications which integrate with the World Wide Web. REST has also been adopted by data storage solutions, such as Document-oriented databases, to expose interfaces which benefit from the advantages of HTTP. Such solutions, however, do not fully respect the REST architectural style. This report presents two main issues identified in the HTTP APIs of data storage solutions, which prevent them from being entirely RESTful. Consequently, a design for a RESTful interface which aims to solve those issues is proposed. The report details how the interface was implemented in an application which uses an embedded database to allow clients to store and retrieve data. The functional evaluation of the interface reveals that the design successfully supports common data storage operations, but has limited querying functionality. Furthermore, the performance evaluation indicates that the application performs satisfactorily in most scenarios. Based on the limitations identified, directions for future work are provided in the conclusion. Acknowledgements

First of all, I would like to thank my supervisor, Prof. Norman Paton, for all his support throughout the project, especially for helping me organise my ideas and for showing me how to develop and sustain an argument. I would also like to thank my family and friends for the help, support and words of encouragement offered so far.

1 Contents

1 Context 7 1.1 JSON ...... 7 1.2 The REST Architectural Style ...... 8 1.3 REST Specifications ...... 10 1.4 REST Interfaces for Data Storage ...... 11 1.5 Report Outline ...... 11

2 Motivations 12 2.1 RPC-style Endpoints ...... 12 2.2 No Use of Hypermedia ...... 15 2.3 Conclusion ...... 20

3 Design of the Interface 21 3.1 Choice of Specification ...... 21 3.2 JSON:API Extensions For Data Storage ...... 22 3.2.1 Root Endpoint ...... 22 3.2.2 ID Generation Strategy ...... 22 3.2.3 Pagination Strategy ...... 23 3.2.4 Filtering Strategy ...... 23 3.2.5 Entity Tags ...... 23 3.3 Design Solutions to the Existing Problems ...... 24 3.3.1 Hypermedia Support ...... 25 3.3.2 Relationship Modelling ...... 28 3.3.3 RPC-like Endpoints ...... 30 3.4 Conclusion ...... 31

4 Implementation 32 4.1 Implementation Overview ...... 32 4.2 Choice of Programming Language ...... 33 4.3 Database Choice ...... 33 4.4 Architectural Overview ...... 34 4.5 Testing ...... 37 4.6 Conclusion ...... 38

5 Evaluation 40 5.1 Functional Evaluation ...... 40 5.1.1 Purpose ...... 40 5.1.2 Method ...... 40

2 5.1.3 Findings and Results ...... 41 5.2 Performance Evaluation ...... 42 5.2.1 Purpose ...... 42 5.2.2 Method ...... 43 5.2.3 Findings and Results ...... 43 5.3 Conclusion ...... 46

6 Conclusions 47 6.1 Future Work ...... 47

A Interface Documentation 56 A.1 URI Schemes ...... 56 A.2 Document Structure ...... 57 A.2.1 Top - level ...... 57 A.2.2 Resource Objects ...... 57 A.2.3 Attributes ...... 58 A.2.4 Relationships ...... 59 A.2.5 Member Names ...... 59 A.3 Errors ...... 59 A.4 Fetching Data ...... 60 A.4.1 Fetching Available Collections ...... 60 A.4.2 Fetching Collections ...... 61 A.4.3 Fetching Individual Resources ...... 63 A.4.4 Fetching Relationships ...... 64 A.5 Querying Supported ...... 65 A.5.1 Inclusion of related resources ...... 66 A.5.2 Sparse Fieldsets ...... 66 A.5.3 Sorting ...... 66 A.5.4 Paging ...... 66 A.5.5 Filtering ...... 67 A.6 Creating Resources ...... 67 A.7 Updating Resources ...... 69 A.8 Updating Relationships ...... 71 A.9 Deleting Resources ...... 74 A.10 HEAD and OPTIONS ...... 75 A.11 A Full JSON:API Document ...... 76

3 List of Figures

1.1 High-level mode of operation of the application ...... 7

4.1 The application running in a terminal ...... 33 4.2 System context view of the interface ...... 34 4.3 The main components of the implementation ...... 35 4.4 Complete architectural view of the interface ...... 36 4.5 Screenshot of an end-to-end test running in Postman ...... 39

5.1 Screenshot of the blog web application used for functional evaluation . . 41 5.2 Plot showing the average response time for resource fetching, in milliseconds 44 5.3 Plot showing the average response time for resource creation, in seconds 44 5.4 Plot showing the average response time for collection fetching, in seconds 45

4 List of Listings

1 Example of JSON document ...... 8 2 ElasticSearch search HTTP request using query parameters ...... 13 3 ElasticSearch search HTTP request using the Query DSL ...... 13 4 ElasticSearch update request for a stored document ...... 14 5 CouchDB /db/_missing_revs endpoint usage example ...... 15 6 Fetching a list of collections in Kinto ...... 16 7 Application-side joins in ElasticSearch ...... 17 8 Nested objects in ElasticSearch ...... 18 9 Inline objects in CouchDB ...... 18 10 Joins in CouchDB ...... 19 11 Fetching all available collections ...... 25 12 Fetching the articles collection ...... 26 13 Fetching a collection of tags related to an article ...... 27 14 Fetching the tags relationship ...... 28 15 Adding a tag to an article ...... 29 16 Removing tags from an article ...... 30 17 SQL query for fetching the latest posts of the blog ...... 42 18 Request used for profiling ...... 45 19 Example of a resource object ...... 58 20 Example of an error response ...... 60 21 Root endpoint GET request ...... 60 22 Fetching a Collection ...... 62 23 Fetching an individual resource ...... 63 24 A conditional GET request ...... 64 25 Fetching a relationship ...... 65 26 Creating a new resource ...... 68 27 Creating a new collection ...... 69 28 Updating an existing resource ...... 70 29 Conflicted update request ...... 71 30 Complete update request for a relationship ...... 72 31 Adding members to a relationship ...... 73 32 Removing members from a relationship ...... 74 33 Deleting an individual resource ...... 75 34 Example of a HEAD request ...... 75 35 Example of an OPTIONS request ...... 76

5 List of Tables

1.1 Example usage of HTTP methods and URI schemes in a REST API . . . 10

A.1 URI schemes of the interface ...... 56

6 Chapter 1

Context

The scope of this project is to develop an application which exposes a RESTful interface for data storage. The application itself acts as a middleware between a client and a database, therefore it is not concerned with the persistance of the data itself but with the mapping between a JSON representation of data exposed via a RESTful API and the model of an exisiting data storage solution, such a SQL or NoSQL database. The high-level mode of operation of the application is portrayed in Figure 1.1.

Figure 1.1: High-level mode of operation of the application

One of the main challenges of this project is to design the aforementioned interface such that it respects the REST architectural style and also takes advantage of existing standards or specifications readily available in the REST ecosystem. Ideally, the interface should provide solutions to common problems of other established interfaces while not deviating from the REST architectural style. Other aims for the application include com- petitive performance, simple distribution and setup, and an architecture which facilitates support for multiple databases. The remaining sections in this chapter introduce the concepts which are fundamental to this project.

1.1 JSON

The Javascript Object Notation (JSON) is a format which is “easy for humans to read and write”[1] and “easy for machines to parse and generate”[1]. Listing 1 shows an example of a JSON document which contains all the data types and data structures supported by JSON: objects (key - value pairs), arrays, strings, numbers and booleans. JSON became popular over the last decade as a format for REST APIs, databases, and other types of applications, as it fits the requirements of modern applications in

7 terms of data representation. Chapter 2 will show examples of popular data storage solutions which use JSON as a format. Because it is such a popular, widely-used format, especially in the world of REST API development, JSON is also the format used for data representation in the application developed for this project.

{ "firstName": "John", "secondName": "Doe", "height": 1.88, "married": true, "emailAddresses": { "primary": "[email protected]", "secondary": "[email protected]" }, "children": ["Luke", "Betty"] }

Listing 1: Example of JSON document

1.2 The REST Architectural Style

The Representational State Transfer (REST) architectural style was introduced by Roy Fielding in his PhD dissertation “Architectural Styles and the Design of Network-based Software Architectures”[2] as a set of constraints for designing distributed hypermedia systems. REST was the first attempt to theorise the World Wide Web and to capture the essence of it in a set of principles[3]. REST became popular when major online services, like Facebook, Twitter or Google, started offering REST APIs[4] to allow third party applications to integrate with them. The constraints of REST can be summarised as:

1. Client-server model The client-server model is probably the most popular style used for network interactions[5]. Roy Fielding describes this model as follows [5]:

A server component, offering a set of services, listens for requests upon those services. A client component, desiring that a service be performed, sends a request to the server via a connector. The server either rejects or performs the request and sends a response back to the client.

2. Stateless Each request made by the client can be understood in isolation and no client context is stored on the server between requests[2].

3. Cache

8 Responses returned by the server can be cached by clients or intermediaries in order to improve performance. The responses must indicate whether they are cache-able or not[2].

4. Layered system A client is not aware of any intermediaries between itself and the server[6]. A request might be directed through a number of components like gateways, proxies or load balancers before reaching the server, all of which are invisible to the client[6].

5. Code-on-demand (optional) Servers can return executable code in addition to data in order to extend the functionality of the client (e.g. JavaScript scripts)[6].

6. Uniform Interface This feature is fundamental to REST as it allows components (server and clients) to be decoupled and evolve independently. It is defined by four constraints[2]:

(a) Identification of resources Resources are identified in requests. In most cases, this is done using URIs[7]. Resources can be stored differently than the representation the server returns in responses[2]. (b) Manipulation of resources through representations A client can alter a resource using only the information that it holds in the representation of a resource[7]. (c) Self-descriptive messages Requests and responses contain information about how to be processed. This is usually done using an Internet Media Type[7]. (d) Hypermedia as the engine of application state (HATEOAS) A client navigates through resources on a server using only actions described in previous representations received from the server through hypermedia. A client does not make any assumptions about the actions that can be performed on a resource, besides the entry point to the application[8].

REST was designed for HTTP, therefore most of the constraints of REST are implic- itly satisfied by HTTP[9]. For this reason, this project mostly focuses on the fundamental constraint of REST, Uniform Interface, because it is the one which raises most of the problems and challenges in designing REST interfaces. HTTP provides tools which can be used to satisfy this constraint (URIs, hyperlinks, HTTP methods, etc.) but not a clear way of achieving it. Two concepts are essential in REST: resources (”A resource is anything that’s important enough to be referenced as a thing in itself.”[10]) and representations(”A representation can be any machine-readable document containing any information about a resource.”[10]). Table 1.1 shows an example of representing resources and actions on those resource using HTTP in a RESTful manner. A cell in the table describes the action performed by the HTTP method in the corresponding column on the resource identified by the URI in the corresponding row. The example given is based on common guidelines and

9 Resource GET POST PATCH DELETE http://.../users Retrieve the Add a new Modify the Delete the collection of resource to collection entire collec- resources the collection tion of this type (all users) http://.../users/1 Retrieve the (not used) Modify the Delete the user with ID user user 1 http://.../users/1/roles Retrieve all (not used) (not used) Delete the the roles user’s roles which belong to the user

Table 1.1: Example usage of HTTP methods and URI schemes in a REST API it is not the only way HTTP URIs and methods can be used to represent the resources. By sending requests to those endpoints, the client and the server can exchange resource representations in any format, such as HTML, XML, JSON, plain text, or others. It is also important to introduce the concept of collections as this point. A collection is a special type of resource which represents a group of resources and serves the purpose of providing links to those resources in its representation[11]. Optionally, it can include representations of those resources as well[11]. Collections are a widely spread design pattern for REST APIs[11] and examples of them will be shown throughout this report. More aspects of RESTful API design will be discussed in Chapters 2 and 3, where examples of interactions between the client and the server will be given in various contexts.

1.3 REST Specifications

The REST architectural style can be viewed as a generic collection of high-level guidelines for designing applications. That gives the developer the freedom to choose how to use HTTP in order to satisfy the constraints of the style. This freedom, however, can lead to implementations of REST APIs which share very little in common and need to be supported by documentation[12], thus making it difficult for clients to use shared conventions in order to use the APIs. REST specifications try to solve this problem by providing a standardised wayof implementing a REST API. In most cases, they provide solutions to problems such as URI schemes, response formatting, usage of HTTP methods, usage of HTTP status codes, etc. One area where specifications become very useful is that of hypermedia. Within the context of JSON REST APIs, negotiating of a way of representing links in resources is difficult as JSON does not have built-in support for hyperlinks[12]. For this reason, existing specifications go into great detail about hyperlinking support inJSON. By conforming to a specification, REST APIs and clients of such APIs can makeuse of shared tooling, which facilitates code reuse for implementing APIs and clients, thus improving developer productivity.

10 Popular such specifications for JSON REST APIs are Hal[13], oData[14], JSON:API[15], Siren[16] or Collection+JSON[17]. This project will make use of the JSON:API specification for reasons which will be discussed later in this report.

1.4 REST Interfaces for Data Storage

With the increased popularity of REST as a solution for distributed computing problems[18], a large number of data storage solutions, such as NoSQL databases, which expose a RESTful interface over HTTP have emerged and are gaining popularity in the industry. By exposing RESTful interfaces, data storage solutions can benefit from several advantages of the Web as an application platform[18], such as: technology support (all major operating systems and programming languages offer support for web technologies)[18], scalability and performance (HTTP is a synchronous protocol with strict semantics which facilitates optimisations such as caching)[18], support for modelling business processes (HTTP and hypermedia can be easily used to model resources and state transitions between resources)[18] or consistency and uniformity (the strict semantics of REST and HTTP enable integration with various components or reuse of such components)[18]. In the context of this project the data storage solutions which are of particular inter- est are Document-oriented databases (ElasticSearch[19], CouchDB[20] and others[21]) or applications which provide a RESTful data access layer on top of one or more database of various types (Kinto[22], Orchestrate[23], and others). Even though these solutions promote themselves as having RESTful APIs, some of them do not fully respect the REST architectural style. Chapter 2 will discuss two main issues with such interfaces and solutions for these problems will be explored in Chapter 3.

1.5 Report Outline

The next chapter will discuss issues of data storage REST APIs, as mentioned previously. Chapter 3 explains the design decisions that were made for the interface in order to attempt to solve the problems presented in Chapter 2. Chapter 4 contains an architectural and technical overview of the application which implements the interface presented in Chapter 3, while Chapter 5 shows the method and results of the evaluation of the interface. The reports ends with an overall conclusion and suggestions for future work, in Chapter 6.

11 Chapter 2

Motivations

This chapter will discuss two recurring issues found in REST data storage interfaces such as those mentioned in Section 1.4 of Chapter 1. Three popular data storage solutions will be used to exemplify these issues: ElasticSearch[19], CouchDB[20] and Kinto[22]. In order to better illustrate the problems, examples of HTTP requests and responses which highlight the nonconformities of these interfaces with the REST architectural style will be shown and references to the REST constraints in Chapter 1 will be made.

2.1 RPC-style Endpoints

A common problem with numerous REST APIs, otherwise called services, is the presence of methods or actions in URIs[24]. This is a characteristic one would expect to find in a RPC (Remote Procedure Call) service which uses HTTP as an underlying protocol but not in a RESTful interface. This is a problem which also occurs in data storage REST APIs, as this section will show. In the RPC style, endpoints represent procedures or actions which can be called or executed by a client (e.g. add_user, submit_order, etc). In the REST style, endpoints represent resources, as illustrated in the examples given in Chapter 1. The actions the client can execute on a particular resource are described using an HTTP method which is part of the HTTP request: GET - Read, POST - Create, PUT/PATCH - Update, DELETE - Remove. REST, therefore, is resource-oriented while RPC is action-oriented. APIs which mix the two styles are sometimes described as REST-RPC hybrids[24] and their main drawback is that they create confusion for a user[24] which is mostly due to the fact that they violate the fundamental REST constraint (Uniform Interface). Ultimately, this is not meant to be a criticism of RPC but an indication that mixing RPC with REST is to be avoided. In order to illustrate this issue for data-storage REST APIs, the interfaces of two popular[25] Document-Oriented databases, ElasticSearch[19] and CouchDB[20] will be used as examples. Even though both these databases seem to follow the REST style in their interfaces, RPC-like endpoints are interleaved with resource endpoints. The search endpoint (_search)[26] in ElasticSearch is a good example. It allows two types of search, one through a GET request with special parameters encoded in the URI[27] and through a POST or GET (with body) using a special Query DSL[28]. It can be applied on indices (collection of documents of multiple types) or types within an index. Listing 2 shows an example request which will execute a search on the user field

12 on all documents stored within the twitter index. The HTTP response will contain the hits the search generated.

GET /twitter/_search?q=user:kimchy HTTP/1.1

Listing 2: ElasticSearch search HTTP request using query parameters [27]

The search operation can also be performed using a POST request, with the query written in the Query DSL, as shown in Listing 3.

POST /twitter/tweet/_search HTTP/1.1

{ "query" : { "term" : { "user" : "kimchy" } } }

Listing 3: ElasticSearch search HTTP request using the Query DSL [28]

It can be seen that the _search operation is a method defined through an URI and is not a resource. Moreover, using this endpoint requires a thorough read of the provided documentation as it does not conform with common conventions or specifications. Another relevant example in ElasticSearch is the _update endpoint[29], which can be used to perform scripted updates on a document stored in ElasticSearch. This can be done by sending a POST request to this endpoint with a script describing the desired changes. For the document shown in Listing 4, the HTTP request will add the tag “blue” to the array of tags in the document.

13 Stored document:

{ "counter" : 1, "tags" : ["red"] }

Update HTTP request:

POST /test/type1/1/_update HTTP/1.1

{ "script" : { "inline": "ctx._source.tags += tag", "params" : { "tag" : "blue" } } }

Listing 4: ElasticSearch update request for a stored document [29]

Not only is the example above an RPC call, but it also transfers executable logic through HTTP. As noted previously, code transfer in REST is allowed only from server to client and not vice-versa because clients should only exchange representations of resources with the server. Equally, a client should have enough information on how to modify a resource from the representation of the resource it previously received from the server. The _update operation clearly deviates from the Uniform Interface constraint of REST. Another similar endpoint in ElasticSearch is _bulk[30]. In the case of CouchDB, the documentation explains the deviation from the REST principles[31]. The main argument offered is that REST is not suited for all problems. As a consequence some of their APIs are not RESTful. To make it simpler for the user, CouchDB has separate APIs under the same host. The Document API is the only one which is declared as being RESTful while the other ones (Server, Database and Replication APIs) are not[31]. Although CouchDB’s API seems to be cleaner and better organized than the API of ElasticSearch, it exposes various RPC-like endpoints used for either performing large operations on databases, like /db/_purge[32], to control the server like /_restart[33] or to execute views[34]. In CouchDB, views are used for querying and performing complex operations on documents using JavaScript and a MapReduce model[35]. Listing 5 shows an example of a request to the /db/_missing_revs endpoint[36] in CouchDB. The request uses the POST method to send a list of document revisions and the server will return the documents that do not exist in the database[36]. Similar to previous examples, the _missing_revs endpoint is a method, not a resource. Because the endpoint accepts POST requests, it can also be argued that the POST method is used for a different purpose than the ones defined by the HTTP semantics, asthePOST method is intended for creating new resources[37] thus being a source of confusion.

14 HTTP Request:

POST /db/_missing_revs HTTP/1.1 Accept: application/json Content-Type: application/json

{ "c6114c65e295552ab1019e2b046b10e": [ "3-b06fcd1c1c9e0ec7c480ee8aa467bf3b", "3-0e871ef78849b0c206091f1a7af6ec41" ] }

HTTP Response:

HTTP/1.1 200 OK Content-Type: application/json

{ "missed_revs":{ "c6114c65e295552ab1019e2b046b10e": [ "3-b06fcd1c1c9e0ec7c480ee8aa467bf3b" ] } }

Listing 5: CouchDB /db/_missing_revs endpoint usage example [36]

Given these points, one of the aims of this project is to try to avoid using RPC endpoints in the interface and to seek alternative solutions to problems which other interfaces have solved using RPC-like methods.

2.2 No Use of Hypermedia

Another recurring problem with popular data storage REST APIs is the absence of hypermedia in resource representations, which allows clients to navigate through resources. This is concerned with the Hypermedia as the Engine of the Application State (HA- TEOAS) constraint of REST, explained in Section 1.2. To understand why this concept is important for REST APIs, one can think of the way humans interact with the World Wide Web. Web pages are displayed in a browser and the user can click on links in order to navigate to other pages. Users are not supposed to know the URIs of specific pages on a website in order to access them, besides the address of the website (e.g. www.somewebsite.com) which takes them to the index page. Moreover, the user does not make assumptions about the structure of the links or the URI formats used by the web server on the page he or she is viewing. For example, a user (client) does not assume that http://www.bbc.co.uk/news/world-35959604 points to a news

15 article with the ID world-35959604 or that navigating to http://www.bbc.co.uk/news will return a list of news articles. The user expects to navigate from the list of news article to an individual article using a hyperlink. The idea explained above can be applied not only to Web Sites but to Web APIs as well. Clients of a REST API should be able to navigate through the data and explore it without knowing the URI scheme of the server besides the entry point to the application. From that point onwards, the client should use links provided by the server in resource representations in order to fetch, create, update, etc individual resources. In order to show how hypermedia can help improve a REST API, Kinto[22], a popular JSON storage service built by Mozilla which is similar in many ways to the application built as part of this project, will be used as an example. Kinto stores data in a hierarchy of containers: buckets contain collection which, in turn, contain records, which are individual entities[38]. A client might find it useful, for example, to navigate from a bucket toa list of records stored in a collection and so on. This is not possible, however, with the API of Kinto. The HTTP request and response in Listing 6 show how a client can retrieve the list of collections stored in the blog bucket and that no links are given to the scores, game and articles collections returned in the response. The same stands when retrieving the buckets available[39] or the records in a collection[40]. Because there are no links returned in resource representations, the client needs to be aware of the URI schemes of Kinto (one example would be /buckets/(bucket_id)/collections/(collection_id)/records), amongst other details, in order to access data stored via the interface.

HTTP Request:

GET /v1/buckets/blog/collections HTTP/1.1

HTTP Response:

HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8

{ "data": [ { "id": "scores", "last_modified": 1456496072475 }, { "id": "game", "last_modified": 1456496060675 }, { "id": "articles", "last_modified": 1456496056908 } ] }

Listing 6: Fetching a list of collections in Kinto [41]

A more significant use of hypermedia in data storage APIs is that of representing relationships between entities within a particular data set, as it is usually the case with data models of modern applications - blog posts have comments, orders have items, books have authors, etc. In such a scenario, hypermedia allows clients to explore data on the server using links - to navigate from a blog post to a comment, to change the name of the author of a book having obtained a representation of the book resource,

16 etc. Existing data storage REST APIs, however, do not expose hypermedia such that clients can easily explore relationships between resources. To show why this is important, examples of how relationships can be modelled in CouchDB and ElasticSearch will be given in the remainder of this section. ElasticSearch can model relationships in two ways, which are described in the documentation: application-side joins[42], which are similar to relationship modelling in a relational database and nested objects[42]. Listing 7 shows an example of representing a relationship between a blogpost and a user. The blogpost entity references the user with the ID 1. An example of nested objects in ElasticSearch is shown in Listing 8, where the blogpost entity contains two comments, all part of the same document.

PUT /my_index/user/1 HTTP/1.1

{ "name": "John Smith", "email": "[email protected]", "dob": "1970/10/24" }

PUT /my_index/blogpost/2 HTTP/1.1

{ "title": "Relationships", "body": "It's complicated...", "user": 1 }

Listing 7: Application-side joins in ElasticSearch [43]

17 PUT /my_index/blogpost/1 HTTP/1.1

{ "title": "Nest eggs", "body": "Making your money work...", "tags": [ "cash", "shares" ], "comments": [ { "name": "John Smith", "comment": "Great article", "age": 28, "date": "2014-09-01" }, { "name": "Alice White", "comment": "More like this please", "age": 31, "date": "2014-10-22" } ] }

Listing 8: Nested objects in ElasticSearch [44]

In CouchDB relationships can be modelled using views[35]. Similarly to Elastic- search CouchDB offers two alternatives for relationship modelling, inline objects[45] and joins[46]. Examples of both are given in Listings 9 and 10, which are similar to the examples given earlier for ElasticSearch. The JavaScript code which performs the JOIN operations for the views in the Listings has been omitted in the examples due to the large amount of background information required to explain it, but it can be read on the referenced documentation.

{ "_id": "myslug", "_rev": "123456", "author": "john", "title": "My blog post", "comments": [ {"author": "jack", "content": "…"}, {"author": "jane", "content": "…"} ] }

Listing 9: Inline objects in CouchDB [45]

18 Example post:

{ "_id": "myslug", "_rev": "123456", "type": "post", "author": "john", "title": "My blog post" }

Example comments, referencing the post

{ "_id": "ABCDEF", "_rev": "123456", "type": "comment", "post": "myslug", "author": "jack" }

{ "_id": "DEFABC", "_rev": "123456", "type": "comment", "post": "myslug", "author": "jane" }

Listing 10: Joins in CouchDB [46]

Both methods exemplified above bring a few drawbacks which make the support for hypermedia almost impossible in a REST API. Although JOIN-like operations bring specific disadvantages to each of the databases mentioned above, such as search efficiency in ElasticSearch[47] or the need for a JavaScript view function for performing the JOIN operation in CouchDB[35], they have a common drawback - the relationships between the entities are not explicitly represented in the resource obtained by the client in a way which would allow the client to navigate from one resource to another. In the example in Listing 10, the client having retrieved the blogpost resource, it cannot fetch the user resource without knowing the URI for that resource. Moreover, if the client needs both the blogpost and user resources, it might have to send two requests in order to fetch the required data, which has implications for performance. In CouchDB, this can be avoided by using a view to return the linked resources together. The client will have to access the view rather than the original resource which is processed by the view. Unlike JOIN-like operations, which imply that resources are stored in separate documents or entities in the database, nested objects are part of the same document. The main implication of this approach for both CouchDB and ElasticSearch is concurrency

19 control[48][45]. In Listing 9, for instance, adding a new comment to an article involves updating the entire document. This can result in concurrency errors for clients which are intending on updating the fields of the article[45]. In essence, because the linked resources are stored together, it is not possible to fetch or update them without changing the state, in some way, of the other resources stored in the same document. In order to support a better, hypermedia-enabled representation of relationships, this project tries to find a middle ground between the two methods shown above, which will be detailed in Chapter 3.

2.3 Conclusion

As can be seen, existing popular data-storage REST APIs have a few design characteristics which forbid them from being RESTful in the full sense of the term. Although other small issues can be identified in such APIs, the ones presented in this chapter are those which violate the REST constraints the most and which represent the main motivations behind the project. They will be addressed in the next chapter.

20 Chapter 3

Design of the Interface

In this chapter the decisions which were made on the design of the interface will be presented, starting with the reasons why the JSON:API specification was used as a starting point and how the specification was extended for this project such that dynamic data storage is supported. The following section shows how the features of the interface form solutions to the problems presented in Chapter 2. Additional design features which could improve the interface but were not implemented will be presented at the end. This chapter is associated with Appendix A, which explains and exemplifies all the features of the interface. It was included to aid the reader in understanding how the interface works. This chapter will make occasional references to sections from the appendix.

3.1 Choice of Specification

REST specifications were introduced in Section 1.3 and some of the advantages they offer for REST APIs were presented there. Although using such a specification wasnot strictly required for this project in order to design a fully RESTful API for data storage, following one brings a considerable number of advantages:

• Designing an API from scratch is more difficult to validate and defend

• The shared conventions and good practises proposed in specifications usually lead to better API designs

• Existing tooling built for a specification

In general, specifications or standards like those mentioned in Section 1.3 target REST APIs which are designed and implemented from scratch to solve a specific problem in a given domain. Resource types and actions on those resources in such APIs are strictly defined. This project, however, aims to build a REST API for data storage, which requires support for dynamic operations. Resources are created dynamically by the client and the actions available on resources should be flexible enough to accommodate common data storage scenarios. As mentioned previously, this project will use the JSON:API specification as a foundation for designing the interface. JSON:API was chosen, first and foremost, because it is popular, being widely used in a variety of projects[49] and actively developed by a

21 large and diverse community[50]. Because of its popularity, a large amount of tooling, such as libraries, has been developed for various programming languages[49]. JSON:API covers a wide range of API design aspects and was designed to account for design challenges API developers usually face. While other specifications are lighter or focused on a single aspect, such as hypermedia (Hal[13], Collection+JSON[17]) and others are too strict or comprehensive (oData[14]), JSON:API offers the right balance between constraints and extensibility[51], which allows adding functionality for supporting dynamic data storage. To sum up, the comprehensiveness, popularity and extensibility of JSON:API makes it the best choice for a base specification for the design the interface.

3.2 JSON:API Extensions For Data Storage

This section presents and motivates the additions made to the features and constraints of the JSON:API specification for the purpose of making data storage possible inthe interface. The subsections below will explain each one in detail. Support for extensions in JSON:API is under development at the time of writing [52]. The features below were designed to be compliant with the existing work done for the future support of extensions in JSON:API[53].

3.2.1 Root Endpoint The JSON:API specification does not mention any behaviour for the root endpoint(/) of an API which implements the specification. The interface, therefore, adds two features for the root endpoint. By sending a GET request to the root endpoint, a client can gain access to a list of all the collections of resources stored under the interface and to links to those collections. This is exemplified in Section A.4.1. The behaviour was designed so that it is as simpleas possible for clients to process and add additional support for hypermedia. The response contains the minimum amount of information needed for the client to navigate to a collection from the root endpoint. Without this feature, the support for HATEOAS would be incomplete, as clients have no way of exploring the available data, starting from the root endpoint. The second feature added to the root endpoint is the creation of new collections. By sending a POST request containing a new resource, the interface will initialise a new collection based on the type of resource given in the request and store the resource in the newly-created collection. The interface will respond with the created collection and a link to it. An example request is given in Section A.6. This feature is fundamental for data storage, as it allows clients to define and store new types of resources. It is similar, in some respects, to creating a new table in a relational database.

3.2.2 ID Generation Strategy When creating a new resource, the client is not allowed to supply its own ID for the resource. JSON:API allows both server-side and client-side IDs in this scenario, however the interface is strict in this respect. All the IDs for resources are generated and assigned

22 server-side in order to prevent collisions in the back-end data storage and maintain consistency. The IDs generated by the server are UUIDs[54]. They were chosen because they are generated randomly and offer a very low chance of collisions. Examples of requests for creating new resources which show how IDs are assigned to new resources are given in Section A.6.

3.2.3 Pagination Strategy Collections in JSON:API can be paginated by clients when fetched in order to retrieve a subset of the results[55]. Examples and further details on pagination in the interface are given in Section A.5.4. JSON:API does not enforce a specific pagination strategy[55] for implementations, as long as the strategy uses the assigned query parameter, page, and provides links in collection representations for navigating forward or backwards through the result set. The interface uses the offset and limit paging strategy, using the page[offset] and page[limit] parameters. This strategy was chosen as it was considered to be the most flexible one. Moreover, it imitates the pagination found in querying languages for popular databases such as MySQL[56] or PostgreSQL[57].

3.2.4 Filtering Strategy Filtering allows clients to retrieve only the resources whose attributes match a given query, from a collection. JSON:API is agnostic about filtering strategies[58] as long as the implementation uses the filter query parameter for this purpose. The filtering functionality supported by the interface is rather basic as it only supports one type of query - equality between resource attributes and given values in the filter. Examples of the filter parameter are given in Section A.5.5. There is a common reason for the limited filtering supported by the interface and the position JSON:API takes with regards to filtering. Designing a powerful querying language, like SQL, which is integrated with JSON:API and, at the same time, compatible with the REST constraints can be considered an entire topic on its own, relative to the aims of this project. This is a matter which is constantly debated by the JSON:API community[59].

3.2.5 Entity Tags Entity Tags (ETags) are a mechanism offered by HTTP in the form of an HTTP header (ETag) present in responses. Its value is a string which identifies the state of a resource on a server[60]. Any changes made to a resource will result in a new ETag to be assigned to the resource[60]. The means by which the ETag is computed in the interface will be discussed in Chapter 4, as it is a concern for the implementation. The clients are not concerned with the meaning of a string returned as ETag. ETags are used in the interface for two purposes: conditional GET requests and optimistic concurrency control.

23 Conditional GET Requests A client can send a GET request for an individual resource, which includes an ETag the client has previously obtained, in the If-None-Match header. The server can either return a quick 304 Not Modified response, if the ETag sent by the client matches the ETag of the resource on the server (the resource hasn’t changed since the client last fetched it) or the resource in its newest state if the ETag sent by the client does not match the ETag of the resource on the server[60]. Examples of this behaviour are available in Subsection A.4.3. This feature allows support for caching, which is part of the REST constraints. Clients or intermediaries can cache a resource and later check if the resource has been changed by sending a conditional GET request. Because the 304 Not Modified response is very small (it does not include a response body or any other data), it saves the client and the server time and bandwidth[60].

Optimistic Concurrency Control A common challenge in REST APIs is the avoidance of the Lost Update Problem[60]. The Lost Update Problem refers to the scenario in which two or more clients update the same resource (using a PATCH request) at roughly the same time. If there are no checks in place on the server, one of the clients might overwrite the changes made by the other[60]. In HTTP, this problem can be solved by implementing optimistic concurrency control using ETags and conditional unsafe requests[60]. In optimistic concurrency control, the clients do not use locks when updating data. Instead, the control relies on a version for each resource which is updated by the server whenever a client performs an update on a resource[61]. When a client retrieves a resource, it also receives the version for that resource (the ETag). If the client then wishes to perform an update, it must send the ETag it has previously received along its updated version of the resource. The server checks the ETag provided by the client and if it matches the ETag of the resource stored on the server, then the server applies the update. If the two ETags do not match, the server will abort the update and return an appropriate response to the client.[62] For this interface, the functionality explained above has been designed according to the HTTP semantics[63]. Therefore, when a client sends a PATCH request, which is unsafe, to update a resource, it must also provide the ETag received from the server in a previous GET request, in the If-Match conditional header of the request. If another update has been made by another client since the first client retrieved the resource, the server will respond with a 412 Precondition Failed response and the current ETag of the resource. The client must then fetch the new resource, reapply the update and send the PATCH request. If the update operation is successful, the client will receive a 200 OK response with the updated resource and its new ETag. Examples of requests showing this behaviour are given in Section A.7.

3.3 Design Solutions to the Existing Problems

In Chapter 2, two problems associated with RESTful data storage interfaces were presented. This section will discuss three relevant aspects of the design of the interface which

24 can be used to determine whether the design choices made for the interface, presented in the previous two sections, provide solutions for those problems.

3.3.1 Hypermedia Support Clients of the interface can make state transitions from any type of resource as hypermedia is provided throughout the interface. This is made possible by the JSON:API specification and the root endpoint behaviour which was explained in Section 3.2.1. In order to show this aspect, a path a client might take through the resources stored under the interface is given in the listings below. The IDs of the resources have been replaced with short integer values and other irrelevant details have been omitted for the purpose of saving space. A client can retrieve the list of available collections and links to them from the root endpoint (Listing 11).

GET / HTTP/1.1

HTTP/1.1 200 OK

{ "articles": "/articles", "people": "/authors", "tags": "/tags" }

Listing 11: Fetching all available collections

By following the link to the articles collections returned by the server, the client can fetch the collection of resources (Listing 12).

25 GET /articles HTTP/1.1 Accept: application/vnd.api+json

{ "links": { "self": "/articles" }, "data": [{ "type": "articles", "id": "1", "attributes": { "title": "JSON API paints my bikeshed!" }, "relationships": { "author": { "links": { "self": "/articles/1/relationships/author", "related": "/articles/1/author" }, "data": [ { "type": "people", "id": "6" } ] }, "tags": { "links": { "self": "/articles/1/relationships/tags", "related": "/articles/1/tags" }, "data": [ { "type": "tags", "id": "25" },{ "type": "tags", "id": "27" } ] } }, "links": { "self": "/articles/1" } }, { "type": "articles", "id": "2", "attributes": { "title": "Rails is Omakase" }, "links": { "self": "/articles/2" } }] }

Listing 12: Fetching the articles collection

26 From the representation of the collection, the client can, for example, navigate to a collection representing all the tags of the article with the ID 1 (Listing 13) by following the related link of the relationship. The client can also fetch a relationship (Listing 14) in order to change it individually by following the self link given in the representation of the relationship in the collection.

GET /articles/1/tags HTTP/1.1 Accept: application/vnd.api+json

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "links": { "self": "/articles/1/tags" }, "data": [{ "id": "25", "type": "tags", "attributes": { "name": "rest", "colour": "blue" }, "links": { "self": "/tags/25" } }, { "id": "27", "type": "tags", "attributes": { "name": "json", "colour": "green" }, "links": { "self": "/tags/27" } }] }

Listing 13: Fetching a collection of tags related to an article

27 GET /articles/1/relationships/tags HTTP/1.1 Accept: application/vnd.api+json

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "links": { "self": "/articles/1/relationships/tags", "related": "/articles/1/tags" }, "data": [ { "type": "tags", "id": "25" }, { "type": "tags", "id": "27" } ] }

Listing 14: Fetching the tags relationship

As can be seen, the interface is hypermedia driven. The advantages of supporting HATEOAS can be hopefully observed in comparison to the examples shown in Section 2.2.

3.3.2 Relationship Modelling Based on the examples of hypermedia support shown in the previous subsection, the approach the interface takes to relationship modelling, based on the JSON:API specification, will be discussed next. This subsection will compare relationships in the interface to the two models presented in Section 2.2, nested objects and application-side joins. In Listing 14, an example of how a client can fetch an individual relationship is shown. A relationship, therefore, is a type of resource in the interface. The complete details of relationship documents are given in Section A.2.4. The advantage of representing a relationship as a resource is that the client can update the relationship independently of the resources which are referenced by the relationship. All the operations allowed for relationships are given in Section A.8. The remainder of this section shows how two of those operations, POST and DELETE, make it simple for clients to modify relationships. Using POST or DELETE, a client can add or delete members from a relationship. Listing 15 shows an example of a POST request which adds more tags to the article and Listing 16 shows how a client can remove tags from an article. What is important to note about these operations is that the client does not supply an ETag, even though the operations change the state of a resource. The two operations cannot overwrite changes made by other clients. Therefore, they are not susceptible to race conditions[64].

28 HTTP request:

POST /articles/1/relationships/tags HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": [ { "type": "tags", "id": "64" } ] }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "links": { "self": "/articles/1/relationships/tags", "related": "/articles/1/tags" }, "data": [ { "type": "tags", "id": "25" }, { "type": "tags", "id": "27" }, { "type": "tags", "id": "64" } ] }

Listing 15: Adding a tag to an article

29 HTTP request:

DELETE /articles/1/relationships/tags HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": [ { "type": "tags", "id": "25" }, { "type": "tags", "id": "27" }, ] }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "links": { "self": "/articles/1/relationships/tags", "related": "/articles/1/tags" }, "data": [ { "type": "tags", "id": "64" } ] }

Listing 16: Removing tags from an article

Compared to nested objects, the approach JSON:API takes to relationships offers the possibility of updating documents which are in a relationship individually, and avoids the concurrency issues mentioned in Section 2.2. Unlike the JOIN-like approach, the way relationships are modelled in the interface allows clients to explore the related resources using hypermedia. The interface also offers support to retrieve all the related resources in a request by following the related link provided in the representation of a relationship.

3.3.3 RPC-like Endpoints The interface does not expose any RPC-like endpoints, as can be seen from the documentation provided in Appendix A and the examples given in this section. This achievement, however, does not mean that the interface manages to provide RESTful solutions to all the problems which other interfaces, such as those used as examples in Section 2.1, have solved using RPC-like endpoints. At this stage, the interface supports basic and common data storage functionality which has been provided without using any RPC-like endpoints. The evaluation of the project, in Chapter 5, and the conclusions, in Chapter 6, offer more information on features which are yet to be supported

30 by the interface in a non-RPC fashion.

3.4 Conclusion

To summarise, the JSON:API specification and the extensions presented in the sections above have been used to build an interface which is capable of providing data storage capabilities in a RESTful fashion. Although the design has a few limitations in terms of querying, it can support common data storage scenarios when implemented in an application, as the next two chapters will show.

31 Chapter 4

Implementation

This chapter details the software which was developed as part of this project for implementing the interface introduced in Chapter 3. The next sections will focus on motivating the use of development technologies, the architecture of the application and the testing approach taken during the development of the application, such that the implementation achieves the aims of the project.

4.1 Implementation Overview

This section aims to show the high-level characteristics of the implementation before delving into its complex aspects in the next sections. The application is built as an executable binary. It is written in Go, a programming language, which amongst other advantages, allows the application to be cross-compiled for all major platforms (Linux, OSX, Windows, etc). More details on the choice of the programming language are given in Section 4.2. The application exposes the interface presented in Chapter 3 through a web server. It also handles data persistence, on the filesystem, out-of-the-box, without the need of a separate data storage system. The means by which this was achieved are discussed in Section 4.3. The setup required for the application is minimal. Once the executable has been obtained, it can be executed straight away. Figure 4.1 represents a screenshot showing how the application can be started from the terminal on GNU\Linux. The program is named dyrest. It takes two command line arguments, the path to the database file and the TCP port the web server listens on. Once started, clients can begin using it. The same screenshot shows how a resource can be fetched using an HTTP command line client (httpie).

32 Figure 4.1: The application running in a terminal

4.2 Choice of Programming Language

The application has been developed in Go[65], a compiled, statically typed programming language developed by Google[66]. The following features of the language made it the best fit for this project: [67] • Modern and simple concurrency primitives which make it suitable for network applications • Compiled to machine code but has the convenience of garbage collection • Performance comparable to C and C++ • All-round complete toolchain for software development - package management, testing, benchmarking and debugging tools all part of the same package • Relatively young (~6 years) but widely used, having a large ecosystem The language also includes all the necessary tools which were required for the development of this application, like a high performance HTTP library for handling requests and flexible JSON serialisation tools. There was no need for additional frameworks or other abstraction layers to be used for implementing the interface.

4.3 Database Choice

In Section 4.1, the fact that the application provides data persistence out-of-the-box was mentioned. This is possible because the application uses an embedded database. An embedded database is, in simple terms, a library which is used by the application to persist data and is linked together with the application[68]. The embedded database used by the application is named BoltDB[69] and it is a key-value store. The database stores the key - value pairs, both arrays of bytes, under

33 multiple ”buckets”[69]. The reasons why BoltDB was chosen was the simplicity of the data storage model and of the API, its performance, and its popularity amongst big companies[69]. In spite of its simplicity and small size, BoltDB manages to be very flexible and accommodate many scenarios and use cases. The reason for choosing BoltDB over other similar, popular, embeddable, databases is the fact that it is written in Go, whereas other main-stream alternatives are written in either C or C++. Using those would have required wrappers to be written such that interoperability between the application and the library(the database) is possible. That could have added additional development time or introduced stability issues in the application. BoltDB, therefore, allowed a quicker implementation time. Although the implementation was designed to be extensible for multiple types of databases, Bolt is the only database supported by the implementation. More details on adding support for other databases will be given later in this report.

4.4 Architectural Overview

The architecture of the implementation will be presented in a top-down fashion, starting with the system context level, shown in Figure 4.2. The figure illustrates how the application operates from outside, as explained in Section 4.1. The application takes the form of an HTTP server which serves a REST API and stores data in a file.

Figure 4.2: System context view of the interface

34 The application can be broken into three components, all part of the same program: the web server, which is provided by Golang (from the net/http[70] package), the implementation of the interface, and the embedded database, Bolt. This is illustrated in Figure 4.3.

Figure 4.3: The main components of the implementation

The RESTful API can be broken further into its smaller components. Figure 4.4 illustrates the collaboration between the components and data flow within the program. The role of each of the components of the interface is presented below.

JSON:API Module The JSON:API module provides all the necessary functionality for validating and converting HTTP requests according to the JSON:API specification. It also provides data structures and helper functions for building JSON:API responses and for converting them to standard HTTP responses. The module was developed as a separate, reusable, package for the application and was designed to integrate with the net/http package of Go. For HTTP requests, it takes a standard HTTP request, validates the path, method, headers and request body and returns the request in a simple, compact, data structure which facilitates the processing of the request further in the program. For HTTP responses, it exposes data structures and methods which can be used to build a response according to the JSON:API specification and to convert them to standard HTTP responses.

Controller The controller receives a standard HTTP request from the web server, passes it to the JSON:API module for validation and conversion, then takes an action on the data store depending on the request. The data and/or any errors are used to construct a JSON:API response which is converted to an HTTP response and returned to the web server.

35 Figure 4.4: Complete architectural view of the interface

36 Each pair of HTTP path type (root, collection, etc) - HTTP method (GET, POST, etc) (Table A.1 lists them) is mapped to a particular action (a function) which executes the business logic required for satisfying the request.

Mapper This component is used to convert from a JSON:API data format and a simpler, compact format, used in the DB Interface. This conversion, along with the DB Interface, allows a database adapter to be completely decoupled from the JSON:API module and other components which deal with HTTP.

DB Interface The DB Interface is a thin abstraction layer which hides the data storage details from the rest of the application. Its role is to provide a CRUD-like interface which the database adapters must implement and to load a specific database adapter. It is used bythe controller to retrieve or store data. The main advantage of this component, is that it facilitates the creation of new database adapters, making the implementation extensible.

BoltDB Adapter The BoltDB adapter handles all the data storage operations for BoltDB and implements the DB interface explained above. The adapter performs the final data conversion which takes place in the application under the form of binary serialisation. The serialisation is performed using Mes- sagePack[71], an efficient binary serialisation format, which offers fast serialisation times and relatively compressed output. The mapping between the interface (JSON:API) representation and the key-value model of BoltDB is done by representing each collection as a bucket in the database and each resource as a key-value pair. The ID of the resource is used as key and the binary, MessagePack, representation of the resource is stored as the value of the pair. The adapter is also concerned with computing ETags for resources. ETags and their purpose were explained in Section 3.2.5. The adaptor computes a hash code for the byte representation of a resource. This results in a fast computation time for ETags and eliminates the need to store the ETags for each resource. The hash function used is SHA1[72] which was chosen because of its quick computation time and because it is used for similar purposes in systems such as Git[73]. The hash function is not used for any security features, whatsoever. This component also has a critical role in performance. The adapter aims to manage read and write transactions[69] efficiently for the purpose of obtaining better performance from BoltDB. In order to achieve that, it makes use of the concurrency primitives of Go (goroutines and channels[74]) to perform most of the computation required for serialisation, conversions, etc outside of the transactions.

4.5 Testing

The development of the application has been aided by two levels of testing performed on the implementation.

37 The first type of testing performed is unit testing. The application includes asuite of unit tests which test the functions in the program, in isolation. Most of the unit tests were written in a Test First fashion using the standard testing[75] package provided by Go, plus a helper package for assertions[76]. There were also a number of tests written for the BoltDB adaptor which perform the test operations on real databases instead of mocked ones, to ensure that the storage operations work correctly. Because of the small cost of bringing up and tearing down BoltDB databases, they run as quickly as the unit tests do and offer more feedback than standard unit tests. The second level of testing is represented by end-to-end tests. These tests take the form of an HTTP request and an HTTP response expectation. They were used to test some of features and operations of the interface, such as those explained in Appendix A. They are similar, in many ways, to acceptance tests[77]. The tests were created in Post- man[78], a popular REST API client and testing tool. Figure 4.5 represents a screenshot showing one of the end-to-end tests running in Postman. The test checks, using nine assertions made on the response returned by the server, whether a client can create a new resource in the interface. The test suites proved to be valuable for checking whether the implementation meets the requirements described in Chapter 3 and to prevent bugs.

4.6 Conclusion

As shown above, the application manages to provide, in a single, easy-to-use package, a complete data storage solution which allows clients to store and retrieve data using the RESTful interface designed in this project. Additionally, the application facilitates, through its architecture, support for multiple types of back-end databases. Given these points, the implementation fulfils the aims of the project in terms of setup simplicity and extensible design.

38 Figure 4.5: Screenshot of an end-to-end test running in Postman

39 Chapter 5

Evaluation

This chapter details how the design of the interface and the implementation were evalu- ated and the observations made during the process. The evaluation of the project tried to determine how well the implementation performs, and if the implementation, along with the interface, offer sufficient data storage functionality to suit the needs ofanex- isting web application. Therefore, the evaluation is split into functional evaluation and performance evaluation, each presented in a Purpose - Method - Findings and Results structure.

5.1 Functional Evaluation

5.1.1 Purpose This part of the evaluation was performed such that the following questions can be answered:

1. Can the interface accommodate the data model of an existing application?

2. Can the interface and the implementation satisfy the data persistence requirements of a standard application?

3. Are there any characteristics or limitations of the interface which make certain data storage scenarios impossible or difficult to implement?

5.1.2 Method The evaluation was performed by taking an existing web application, which was developed to persist data in a relational database, and changing it to persist data using the interface. The application is the Symfony Demo Application[79], a simple blog engine developed for demonstration purposes for Symfony[80], a web framework for the PHP programming language. The blog allows users to write and manage posts while guests can view posts and make comments on them. A screenshot of one of its pages is shown in Figure 5.1. The blog engine was deemed appropriate as a client application for the interface because of the following reasons:

• It contains widely-used patterns in its data model

40 Figure 5.1: Screenshot of the blog web application used for functional evaluation

• Having a simple, straightforward Model-View-Controller[81] architecture, the application was easy to change to store data using a different storage mechanism.

• It is released under an open-source licence (MIT)[82].

To make data storage possible using the RESTful interface, the data storage layer of the application, which was used for data persistence in a relational database, was removed and replaced with a custom one, built around an HTTP client (Guzzle[83]) which implemented the same interfaces as the default one, in order to make it compatible with the rest of the application and the framework. After the new storage layer was implemented, the application looked and behaved exactly as before, but it was performing all the data storage operations using the interface, without the use of any other database. The blog engine does not cover, of course, the wide range of data storage necessities of a complex, web-scale, application but it does provide the necessary requirements which can be used to evaluate the interface at a basic level, by satisfying them. The modified application is intended for demonstration and evaluation purposes only and is not part of this project.

5.1.3 Findings and Results Rather than the default storage layer, which used the Doctrine ORM (Object-relational Mapping) Framework[84], the new storage layer was relatively lightweight. The majority of the implementation deals with mapping the existing entities in the application (Post,

41 Comment, User, etc) to JSON documents in the JSON:API format and with translating between CRUD actions and HTTP requests (Create - POST, GET - Read, etc). Error handling was straightforward to implement as well, as the errors returned by the interface (Section A.3) can be easily converted into exceptions. The main limitation identified was querying. This was expected, as explained in Section 3.2.4. One of the SQL queries in the original storage layer, given in Listing 17, was not possible to translate. The query fetches the latest blog posts and sorts them by the date they were published on (:now is a parameter). This query was not possible to translate in the interface because the filter parameter does not support the less-than- or-equal relational operator and sorting by a date is not possible because the interface does not offer a special type for dates (dates are just strings). Therefore, someparts of the filtering had to be performed in the application. Other, simpler, queries were successfully translated. Possible solutions to the querying limitations will be presented later, as future work.

SELECT p FROM Post p WHERE p.publishedAt <= :now ORDER BY p.publishedAt DESC

Listing 17: SQL query for fetching the latest posts of the blog

On the other hand, the relationship modelling features of the interface (Section 3.3.2) became useful for representing the relationship between posts and comments and for simplifying some of the actions. For example, when a post is displayed, its comments are fetched as well in a single request using the include parameter(Section A.5). When a comment is made on a post, the comment is saved and then assigned to the post by sending a POST request to the link of the comments relationship of the post. All things considered, the interface was able to support the persistence requirements of the blog engine. Despite the limitations encountered, all the necessary data storage functionality for the blog engine was successfully implemented using the interface.

5.2 Performance Evaluation

5.2.1 Purpose The purpose of the evaluation was to determine whether the implementation of the interface manages to perform well and be responsive under different workloads. It would probably be expected to perform a performance comparison against a similar, competing, piece of software. Due to the nature of the application, it was difficult to find a similar solution to benchmark and compare against. More precisely, at the moment of writing there was no RESTful interface for BoltDB. Nonetheless, the performance evaluation presented in this section should offer enough information to aid a comparison with a solution which serves similar purposes, but is different in terms of design or underlying technologies.

42 5.2.2 Method The performance of the application was measured using end-to-end benchmarks and profiles. The end-to-end benchmarks were used to measure the average response time for different types of requests. They were performed using Vegeta[85], a HTTP load testing tool and library for Golang, on two virtual servers on the DigitalOcean[86] cloud platform. Both servers had 2 CPU cores, 4 GB of memory and SSD storage. They were connected to the same private network and were running Ubuntu Server 16.04. One of the cloud instances was used to run the application while the other one was used for running the benchmarking tool. The benchmarks were performed for the most common operations: fetching collections(Section A.4.2), fetching an individual resource(Section A.4.3) and creating resources(Section A.6). The measurements were taken in multiple runs, at various intervals, to account for the noise associated with the virtualized environment. Profiling, on the other hand, helped measure where the application is spending most of its time internally, between the interface and the embedded database, and to observe how big the overhead added to the database by the interface is. Profiles were taken using the standard profiling tools in Golang[87] under the form of call-graph profiles[87]. The performance profiles taken for the program did not provide a satisfactory amount of information on the performance of the program’s components. This was mostly because they included the internal functions from the runtime of Go and because of the multiple concurrent components in the implementation or in the underlying libraries. This made it difficult to follow a path of function calls through the components of the interface. In spite of that, the profiles made visible the bottlenecks in the program in some scenarios.

5.2.3 Findings and Results The end-to-end benchmarks revealed that the interface is able to perform well under small to moderate loads (data size and requests per second). Figure 5.2 shows the performance plot for the individual resource fetch operation, for an increasing number of requests. From 1 request per second, up to 1000, there were no big differences observed in response times. Individual resource fetching is, therefore, relatively quick. Resource creation is slower, as data inserts in BoltDB are expected to take longer[69]. Figure 5.3 shows the performance plot for individual resource creation. As can be seen, performance decreases significantly after 1000 requests per second. After 3000 requests per second, the application becomes unresponsive and/or starts dropping requests.

43 Figure 5.2: Plot showing the average response time for resource fetching, in milliseconds

Figure 5.3: Plot showing the average response time for resource creation, in seconds

Benchmarking collection fetching indicated that the size of the collection, in the database, has a more significant effect on performance than the number of requests per second does. Figure 5.4 shows how the response time increases with the size of the

44 collection being fetched, starting from 100 resources, at 100 requests per second. The resources in the collections were fetched in order, based on a field, which partly explains the curve observed in the plot.

Figure 5.4: Plot showing the average response time for collection fetching, in seconds

A worst case scenario was simulated using the request given in Listing 18 for a collection having a total size of 20000. The request fetches 18000 resources plus a few tens of thousands more in included resources. This resulted in a response time of 1.6 seconds for a single request per second and high memory utilisation, close to 2 GB.

GET /orders?page[limit]=20000&page[offset]=2000&sort=order-attr-2&f ⌋ ,→ ields[orders]=order-attr1,order-attr2&include=items ,→ HTTP/1.1

Listing 18: Request used for profiling

To gain a better insight into the cause of the slowness of the program for that request, the application was profiled while executing the request in Listing 18. Profiling revealed that most of the time taken by the program to return a responseis spent converting between different representations in the BoltDB adapter (Section 4.4) (about 20%) and in the JSON:API module(Section 4.4) (about 11%). This results in a large number of memory allocations and large workloads for the garbage collector, which combined, slow the program down. For requests which fetch collections greater than 100000 resources, however, the database becomes the bottleneck.

45 The profiles did not reveal much when profiling data inserts or updates, as thoseare performed relatively quickly by the application. The increase in the response time for a higher number of individual resource fetches or inserts per second is mainly due to the database which becomes slower as the number of concurrent read/insert transactions increases. The performance limitations identified are, in some ways, expected due to the necessary conversions required for mapping between the two very different data representations - JSON:API versus binary key-value pairs. A possible improvement in terms of performance for BoltDB could be a different serialisation strategy, where each resource is unfolded into a set of key - value pairs based on the attributes. The time frame allo- cated for the project, and especially for evaluation, did not allow such experiments to be performed or any other type of advanced optimisations. Despite the mediocre performance in some scenarios, the application is responsive and usable. There might be other scenarios, however, not identified by this evaluation, in which the application performs poorly. Therefore, the application can be used for data storage as long as it is not deployed in a large-scale, production environment.

5.3 Conclusion

The evaluation of the interface and of the application demonstrates that the interface and the implementation can be used to support data storage in a basic application. Despite the limitations discovered, the application could be useful for various small-scale purposes such as replacement of other JSON:API-based REST APIs in development or testing environments. The application could support data storage in a more complex, large-scale, system, if some improvements were to be made to the design of the interface and optimisations performed to the implementation. Examples of these are given in the next chapter, as future work.

46 Chapter 6

Conclusions

The project provides a design for a RESTful interface which manages to solve the problems identified in existing data storage REST APIs. The interface has been implemented in a compact application which supports data storage using an embedded database, and allows the creation of adapters for supporting other databases. The evaluation of the project was used to identify issues or limitations in the design of the interface and in the implementation. It revealed that the implementation offers satisfactory performance in most scenarios. However, it is not suitable for large-scale use. Moreover, the evaluation indicated that the design of the interface requires a more powerful filtering strategy in order to support advanced querying functionality. Nonetheless, this project has shown that the JSON:API specification can be used to design and implement a RESTful API for data storage. Other, similar, standards can probably be expanded to serve this purpose. There is, however, a lack of standardised techniques for implementing data storage features which existing solutions have managed to provide by diverging from the REST architectural style. Standards could help data storage API implementers abide by common practises which make data storage solutions, such as the databases referenced in Chapter 2, able to completely integrate with the World Wide Web. To suit the requirements of complex, real-world applications, more research is needed for the design and implementation of such an interface. The findings of this project and the solutions proposed in this report could represent the starting point for a complete specification, based on JSON:API, aimed at data storage systems which expose functionality over a REST API. The section below suggests directions for future work which could help overcome the limitations encountered during this project.

6.1 Future Work

This section presents features which were either planned for the project but not implemented due to time constraints or were too complex to fit within the project. The features are either related to the interface, building upon the design presented in Chapter 3, or related to the implementation, which was detailed in Chapter 4.

47 Schema Support The interface, at the stage it is presented in Chapter 3 and Appendix A, it is very permis- sive in terms of data structure modifications. The client can modify the attributes ofan individual resource as it wishes without being constrained in any way by the server. A collection can end up containing resources which have completely different attributes. The client is, therefore, in charge of maintaining consistency across a collection of resources in terms of attributes, the type of those attributes and the relationships resources stored in that collection have. Providing schema support for collections would solve this problem. A well designed implementation of schema support for collections is offered by Kinto[88], using JSON- Schema[89], a format for defining the structure of JSON data. A similar solution couldbe adopted for JSON:API and integrated with the existing design of the interface by using the PATCH method for submitting or modifying the schema of a collection, although the details of how this would fit within the existing JSON:API constraints have notbeen determined yet.

Advanced Querying Following the querying limitations identified in Section 3.2.4 and Section 5.1, the interface would greatly benefit from a filtering strategy as powerful as, for example, SQL.At the time of writing there was no standard specifically designed for RESTful querying, although the querying functionality from the oData protocol[14] could be adopted and integrated with JSON:API. Schema support, presented above, could greatly enhance querying by allowing the identification of multiple data types in resources in addition to the standard ones offered by JSON.

Permissions Support The interface does not have any support for user authentication or permissions. Such features would allow clients to control access to collections or individual resources. The implementation of permissions in Kinto[90] serves as a reference for this topic.

Support for Other Databases Support for other databases could be added to the interface by implementing new adapters in the interface. For popular JSON databases, such as MongoDB[91] or CouchDB[20], adapters would be relatively lightweight since the data representation format between the interface and the databases is similar. For relational databases, mapping between the two models (relational - structured versus JSON - semi-structured) could be more difficult to implement due to complexities associated with the side effects of updating tables’ schemas (locking, data consistency, etc) and efficient SQL query building. Aref- erence implementation of an automated mapping layer for storing and querying JSON data in a relational database is Argo[92].

48 References

[1] JSON. url: http://json.org (visited on Apr. 10, 2016). [2] Roy Thomas Fielding. “Architectural styles and the design of network-based software architectures”. PhD thesis. 2000. Chap. 5. url: https://www.ics.uci. edu/~fielding/pubs/dissertation/top.htm. [3] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. Appendix A, pp. 341–342. isbn: 9781449358068. [4] Programmable Web. REST API Directory. url: http://www.programmableweb. com/category/all/apis?search_id=136704&data_format=21190 (visited on Apr. 30, 2016). [5] Roy Thomas Fielding. “Architectural styles and the design of network-based software architectures”. PhD thesis. 2000. Chap. 3. url: https://www.ics.uci. edu/~fielding/pubs/dissertation/top.htm. [6] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. Appendix A, pp. 351–352. isbn: 9781449358068. [7] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. Appendix A, pp. 345–348. isbn: 9781449358068. [8] Roy T. Fielding. REST APIs must be hypertext-driven. url: http://roy.gbiv. com /untangled / 2008/ rest - apis - must - be- hypertext- driven (visited on Apr. 30, 2016). [9] Roy Thomas Fielding. “Architectural styles and the design of network-based software architectures”. PhD thesis. 2000. Chap. 6. url: https://www.ics.uci. edu/~fielding/pubs/dissertation/top.htm. [10] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. 3, p. 30. isbn: 9781449358068. [11] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. 6, pp. 91–93. isbn: 9781449358068. [12] Kevin Sookocheff. On choosing a hypermedia type for your API. url: http : //sookocheff.com/post/api/on-choosing-a-hypermedia-format/ (visited on Apr. 10, 2016). [13] Mike Kelly. HAL - Hypertext Application Language. url: http://stateless.co/ hal_specification.html (visited on Apr. 10, 2016).

49 [14] OData. OData - the best way to REST. url: http://www.odata.org/ (visited on Apr. 10, 2016). [15] JSON API - A specification for building APIs in JSON. url: http://jsonapi. org/ (visited on Apr. 10, 2016). [16] Kevin Swiber. Siren: a hypermedia specification for representing entities. url: https://github.com/kevinswiber/siren (visited on Apr. 10, 2016). [17] Collection+JSON - Hypermedia Type. url: http : / / amundsen . com / media - types/collection/ (visited on Apr. 10, 2016). [18] Jim Webber, Savas Parastatidis, and Ian Robinson. “REST in practice: Hyperme- dia and systems architecture”. In: O’Reilly Media, Inc, USA, Oct. 2010. Chap. 1, pp. 15–18. isbn: 9780596805821. [19] Elastic. Elasticsearch. url: https://www.elastic.co/products/elasticsearch (visited on Apr. 7, 2016). [20] Apache Software Foundation. Apache CouchDB. url: http://couchdb.apache. org/ (visited on Apr. 7, 2016). [21] NoSQL Databases. url: http: / / nosql - database . org/ (visited on Apr. 10, 2016). [22] Mozilla Services. Overview — Kinto 2.0.0 documentation. url: https://kinto. readthedocs.org (visited on Apr. 7, 2016). [23] Orchestrate. Orchestrate. url: https://orchestrate.io/ (visited on Apr. 7, 2016). [24] Leonard Richardson Sam Ruby and David Heinemeier Hansson. “RESTful web services: Web services for the real world”. In: United States: O’Reilly Media, Inc, USA, May 2007. Chap. 1, pp. 16–17. isbn: 9780596529260. [25] solid IT. DB-Engines Ranking. url: http : / / db - engines . com / en / ranking (visited on Apr. 7, 2016). [26] Elastic. Search API. url: https://www.elastic.co/guide/en/elasticsearch/ reference/current/search-search.html (visited on Apr. 7, 2016). [27] Elastic. URI Search. url: https://www.elastic.co/guide/en/elasticsearch/ reference/1.4/search-uri-request.html (visited on Apr. 7, 2016). [28] Elastic. Request body Search. url: https : / / www . elastic . co / guide / en / elasticsearch/reference/1.4/search-request-body.html (visited on Apr. 7, 2016). [29] Elastic. Update API. url: https://www.elastic.co/guide/en/elasticsearch/ reference/current/docs-update.html (visited on Apr. 7, 2016). [30] Elastic. Bulk API. url: https://www.elastic.co/guide/en/elasticsearch/ reference/1.4/docs-bulk.html (visited on Apr. 7, 2016). [31] Apache Software Foundation. 1.5. The core API — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/intro/api.html (visited on Apr. 7, 2016).

50 [32] Apache Software Foundation. 10.3.11. /db/_purge — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/api/database/misc. html#db-purge (visited on Apr. 7, 2016). [33] Apache Software Foundation. 10.2.1. / — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/api/server/common.html#restart (visited on Apr. 7, 2016). [34] Apache Software Foundation. 10.5. Design documents — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/api/ddoc/index. html (visited on Apr. 7, 2016). [35] Apache Software Foundation. 6.2.1. Introduction into the views — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/couchapp/ views/intro.html#what-is-a-view (visited on Apr. 7, 2016). [36] Apache Software Foundation. 10.3.11. /db/_missing_revs — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/api/database/ misc.html#db-missing-revs (visited on Apr. 7, 2016). [37] R. Fielding and J. Reschke. Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content. RFC 7231. http://www.rfc-editor.org/rfc/rfc7231.txt. RFC Editor, June 2014. url: http://www.rfc-editor.org/rfc/rfc7231.txt. [38] Mozilla Services. Concepts — Kinto 2.0.0 documentation. url: https://kinto. readthedocs.org/en/latest/concepts.html (visited on Apr. 7, 2016). [39] Mozilla Services. Buckets — Kinto 2.0.0 documentation. url: https://kinto. readthedocs.org/en/latest/api/1.x/buckets.html#retrieving- all- buckets (visited on Apr. 7, 2016). [40] Mozilla Services. Records — Kinto 2.0.0 documentation. url: https://kinto. readthedocs.org/en/latest/api/1.x/records.html#retrieving-stored- records (visited on Apr. 7, 2016). [41] Mozilla Services. Collections — Kinto 2.0.0 documentation. url: https://kinto. readthedocs.org/en/latest/api/1.x/collections.html (visited on Apr. 7, 2016). [42] Elastic. Handling Relationships. url: https://www.elastic.co/guide/en/ elasticsearch/guide/current/relations.html (visited on Apr. 5, 2016). [43] Elastic. Application-side Joins. url: https : / / www . elastic . co / guide / en / elasticsearch/guide/current/application-joins.html (visited on Apr. 7, 2016). [44] Elastic. Nested Objects. url: https://www.elastic.co/guide/en/elasticsearch/ guide/current/nested-objects.html (visited on Apr. 7, 2016). [45] Apache Software Foundation. 6.2.3. Joins with views — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/couchapp/views/ joins.html#approach-1-comments-inlined (visited on Apr. 7, 2016). [46] Apache Software Foundation. 6.2.3. Joins with views — Apache CouchDB 1.6 documentation. url: http://docs.couchdb.org/en/1.6.1/couchapp/views/ joins.html#approach-2-comments-separate (visited on Apr. 7, 2016).

51 [47] Elastic. Denormalizing your Data. url: https://www.elastic.co/guide/en/ elasticsearch / guide / current / denormalization . html (visited on Apr. 7, 2016). [48] Elastic. Solving Concurrency Issues. url: https://www.elastic.co/guide/ en/elasticsearch/guide/current/concurrency-solutions.html (visited on Apr. 7, 2016). [49] JSON API - Implementations. url: http://jsonapi.org/implementations/ (visited on Apr. 12, 2016). [50] JSON API - Monthly GitHub Pulse (specification repository activity). url: https: //github.com/json-api/json-api/pulse/monthly (visited on Apr. 12, 2016). [51] JSON API - Extensions. url: http://jsonapi.org/extensions/ (visited on Apr. 12, 2016). [52] JSON API - Extensions. url: http://jsonapi.org/extensions (visited on Apr. 18, 2016). [53] Extending JSON API. url: https://github.com/json-api/json-api/blob/ 9c7a03dbc37f80f6ca81b16d444c960e96dd7a57/extensions/index.md (visited on Apr. 18, 2016). [54] Paul J. Leach, Michael Mealling, and Rich Salz. A Universally Unique IDentifier (UUID) URN Namespace. RFC 4122. http : / / www . rfc - editor . org / rfc / rfc4122.txt. RFC Editor, July 2005. url: http://www.rfc-editor.org/rfc/ rfc4122.txt. [55] JSON API - Pagination. url: http : / / jsonapi . org / format / #fetching - pagination (visited on Apr. 17, 2016). [56] Oracle Corporation. MySQL 5.7 Reference Manual - SELECT Syntax. url: https: //dev.mysql.com/doc/refman/5.7/en/select.html (visited on Apr. 17, 2016). [57] PosgreSQL Global Development Group. PostgreSQL: Documentation: 7.3 LIMIT and OFFSET. url: http://www.postgresql.org/docs/7.3/static/queries- limit.html (visited on Apr. 17, 2016). [58] JSON API - Filtering. url: http://jsonapi.org/format/#fetching-filtering (visited on Apr. 18, 2016). [59] Share/Propose a Filtering Strategy - JSON API. url: http://discuss.jsonapi. org/t/share-propose-a-filtering-strategy/257/15 (visited on Apr. 18, 2016). [60] Leonard Richardson, Mike Amundsen, and Sam Ruby. “RESTful web APIs”. In: O’Reilly Media, Inc, USA, Sept. 2013. Chap. 11, pp. 241–249. isbn: 9781449358068. [61] Microsoft TechNet. Types of Concurrency Control. url: https : / / technet . microsoft.com/en- us/library/ms189132%28v=sql.105%29.aspx (visited on Apr. 18, 2016). [62] Jeffrey Wang. Concurrency With Rest. url: http://ternarysearch.blogspot. co.uk/2013/01/concurrency-with-rest.html (visited on Apr. 18, 2016).

52 [63] W3C. Editing the Web - Detecting the Lost Update Problem Using Unreserved Checkout. url: https://www.w3.org/1999/04/Editing/ (visited on Apr. 18, 2016). [64] JSON API - Updating Relationships. url: http://jsonapi.org/format/#crud- updating-relationships (visited on Apr. 18, 2016). [65] The Go Programming Language. url: https://golang.org/ (visited on Apr. 22, 2016). [66] The Go Programming Language - FAQ. url: https://golang.org/doc/faq# Origins (visited on Apr. 22, 2016). [67] Documentation - The Go Programming Language. url: https://golang.org/ doc/ (visited on Apr. 22, 2016). [68] Embedded Database Definition From PC Magazine Encyclopedia. url: http:// www.pcmag.com/encyclopedia/term/42544/embedded- database (visited on Apr. 22, 2016). [69] boltdb/bolt: An embedded key/value database for Go. url: https://github.com/ boltdb/bolt (visited on Apr. 22, 2016). [70] http package - The Go Programming Language. url: https://golang.org/pkg/ net/http/ (visited on Apr. 22, 2016). [71] Sadayuki Furuhashi. MessagePack. url: http : / / msgpack . org/ (visited on Apr. 22, 2016). [72] W3C. SHA1 version 1.0. url: https://www.w3.org/PICS/DSig/SHA1_1_0.html (visited on Apr. 22, 2016). [73] Git Objects. url: https://git-scm.com/book/en/v2/Git-Internals-Git- Objects (visited on Apr. 22, 2016). [74] Caleb Doxsey. Concurrency — An Introduction to Programming in Go. url: https://www.golang-book.com/books/intro/10 (visited on Apr. 22, 2016). [75] testing - The Go Programming Language. url: https : / / golang . org / pkg / testing/ (visited on Apr. 22, 2016). [76] stretchr/testify: A sacred extension to the standard go testing package. url: https: //github.com/stretchr/testify (visited on Apr. 22, 2016). [77] Acceptance Tests. url: http://www.extremeprogramming.org/rules/functionaltests. html (visited on Apr. 22, 2016). [78] Postman | Supercharge Your API Workflow. url: https://www.getpostman.com (visited on Apr. 22, 2016). [79] Symfony Blog. Introducing the Symfony Demo application. url: http://symfony. com/blog/introducing-the-symfony-demo-application (visited on Apr. 26, 2016). [80] What is Symfony. url: http://symfony.com/what- is- symfony (visited on Apr. 26, 2016). [81] Microsoft Developer Network. Model-View-Controller. url: https://msdn.microsoft. com/en-us/library/ff649643.aspx (visited on Apr. 26, 2016).

53 [82] Symfony Demo App Licence. url: https://github.com/symfony/symfony- demo/blob/master/LICENSE (visited on Apr. 26, 2016). [83] Guzzle, PHP HTTP Client. url: http://docs.guzzlephp.org/en/latest/ (visited on Apr. 26, 2016). [84] Object Relational Mapper - Doctrine Project. url: http : / / www . doctrine - project.org/projects/orm.html (visited on Apr. 26, 2016). [85] Vegeta - HTTP load testing tool and library. url: https://github.com/tsenart/ vegeta (visited on Apr. 27, 2016). [86] DigitalOcean - Simple Cloud Computing, Built for Developers. url: https:// www.digitalocean.com/ (visited on Apr. 27, 2016). [87] The Go Blog. Profiling Go Programs. url: https://blog.golang.org/profiling- go-programs (visited on Apr. 27, 2016). [88] Mozilla Services. Collection JSON Schema. url: https://kinto.readthedocs. org / en / latest / api / 1 . x / collections . html # collection - json - schema (visited on Apr. 27, 2016). [89] JSON Schema. url: http://json-schema.org/ (visited on Apr. 27, 2016). [90] Mozilla Services. Kinto Permissions. url: https://kinto.readthedocs.org/ en/latest/concepts.html#permissions (visited on Apr. 27, 2016). [91] MongoDB. url: https://www.mongodb.org/ (visited on Apr. 27, 2016). [92] Craig Chasseur, Yinan Li, and Jignesh M. Patel. “Enabling JSON Document Stores in Relational Systems”. In: Proceedings of the 16th International Workshop on the Web and Databases. 2013. [93] JSON API - Recommendations. url: http://jsonapi.org/recommendations/ #urls (visited on Apr. 12, 2016). [94] JSON API - Document Structure. url: http://jsonapi.org/format/#document- structure (visited on Apr. 12, 2016). [95] IANA (Internet Assigned Numbers Authority). JSON:API Media Type Assign- ment. url: https://www.iana.org/assignments/media-types/application/ vnd.api+json (visited on Apr. 13, 2016). [96] JSON API - Member Names. url: http://jsonapi.org/format/#document- member-names (visited on Apr. 13, 2016). [97] JSON API - Errors. url: http://jsonapi.org/format/#errors (visited on Apr. 14, 2016). [98] P. Bryan, K. Zyp, and M. Nottingham. JavaScript Object Notation (JSON) Pointer. RFC 6901. http://www.rfc-editor.org/rfc/rfc6901.txt. RFC Editor, Apr. 2013. url: http://www.rfc-editor.org/rfc/rfc6901.txt. [99] JSON API - Include. url: http://jsonapi.org/format/#fetching-includes (visited on Apr. 18, 2016). [100] JSON API - Sparse Fieldsets. url: http://jsonapi.org/format/#fetching- sparse-fieldsets (visited on Apr. 18, 2016).

54 [101] JSON API - Sorting. url: http://jsonapi.org/format/#fetching-sorting (visited on Apr. 18, 2016).

55 Appendix A

Interface Documentation

This appendix provides a concise API documentation for the interface built as part of the project. Its purpose is to aid the reader in understanding how the interface works, what its main operations are and how it supports data storage. Considering the interface adheres to the JSON:API specification, which is considerably comprehensive, some details are omitted and references to the the specification are provided instead. The sections below will also capture the features described in Section 3.2, which were added on top of the specification for supporting data storage. Most of the examples given in this Appendix are HTTP requests and responses. Because the host name is not relevant in requests, it has been omitted in order to save space.

A.1 URI Schemes

Table A.1 shows all the types of URIs available in the interface, along with the type of resource they identify and allowed HTTP methods. The URI schemes used are the ones recommended by the JSON:API specification[93]. The parameters are specified using the ”:” prefix notation in the URI structures shown in the table. The HTTP methods for each of the URI types are the ones fixed by JSON:API and the behaviour of the interface for each of them will be shown in the next sections.

Type URL Description Allowed methods Lists all the collections Root / GET, POST available and links to them Resource Identifies all the resources GET, POST, /:collection-name collection of the given type DELETE, OPTIONS Individual Identifies an GET, PATCH, /:collection-name/:resource-id resource individual resource DELETE, OPTIONS /:collection-name/:resource-id/ Identifies a relationship GET, POST, PATCH, Relationship relationships/:relationship-name of a resource DELETE, OPTIONS Identifies the resources Related /:collection-name/:resource-id/ which are related GET, OPTIONS resources :relationship-name to the given resource Table A.1: URI schemes of the interface

56 A.2 Document Structure

This section will describe the structure of the documents used by the interface for resource representation. With the exception of a few naming rules which are described at the end of this section, the interface conforms with the document structure set by JSON:API. For this reason, this section will try to extract or summarise the essential points related to document structure from the JSON:API specification[94].

A.2.1 Top - level JSON:API documents are identified in HTTP requests and responses bythe application/vnd.api+json[95] media type. A document contains, at the top level, at least one of the following:

• data - the document’s primary data, which can be either a resource object if the document is the representation of an individual resource or an array of resource objects if the document represents a collection resource. Primary data is the representation of a resource addressed by a request.

• errors - an array of error objects (error objects are described later in this section)

• meta - contains non-standard information

Additionally, a document may contain any of the following top-level members:

• links - an object containing links to the primary data

• included - an array of resource objects that are related to the primary data which can be present only if the data attribute is present at the top-level.

A top-level links objects may contain the following:

• self - the URI of the resource

• related - when the primary data represents a resource relationship

• Pagination links - when the primary data represents a collection. These are described subsection A.5

A.2.2 Resource Objects A resource object contains the representation of a resource. Listing 19 shows an example of a resource object.

57 { "type": "articles", "id": "1", "attributes": { "title": "Some title" }, "relationships": { "authors": { "links": { "self": "/articles/1/relationships/author", "related": "/articles/1/author" }, "data": [ { "type": "people", "id": "9" }, { "type": "people", "id": "20" }, ] } } }

Listing 19: Example of a resource object

Every resource object contains an id member and a type member. The values of the id and type members must be strings. A resource’s object type and id identify a single, unique resource. The type member is used to describe resource objects that share common attributes and relationships. If the document is part of a request from the client, such as one for creating a new resource, then the id member must be omitted as IDs in the interface are generated and assigned server-side. In addition, a resource object may contain any of these members:

• attributes: an attributes object representing some of the resource’s data.

• relationships: a relationships object describing relationships between the resource and other JSON API resources.

• links: a links object containing links related to the resource.

A.2.3 Attributes The value of attributes is an object containing information about the resource object. All the data types available in JSON are permitted. The members of the attributes object are referred to as fields. A resource cannot contain fields named type, id as those are used in the resource object to identify the resource nor relationships or links as those names are used for other purposes, explained below, by the interface.

58 A.2.4 Relationships The value of the relationships key is an object. Members of the relationships object represent references from the resource object in which it’s defined to other resource objects. A relationship object contains the following:

• links which contains self, a link for the relationship itself which allows a client to fetch or change the relationship without affecting the resources which are referenced by the relationship (exemplified in Section A.8) and related, a link which provides access to resource objects linked in a relationship.

• data, containing an array of resource identifier objects

A resource identifier object is an object that identifies an individual resource. It contains type and id of a resource as members. In Listing 19, the article resource has an authors relationship with two people with IDs 9 and 20. Section A.11 contains an example of a full document representing a collection of resources, which is too lengthy to include here. The document features all of the components described above.

A.2.5 Member Names In addition to the restrictions imposed by JSON:API on naming (resource fields, relationship names, resource type)[96], the interface restricts the allowed word separator to ”-”(hyphen) e.g. user-name not user_name, for reasons related to the implementation of the mapping required for the back-end database.

A.3 Errors

Whenever the server encounters one or more errors, it will stop processing the request and an HTTP response code with an appropriate status code(4xx or 5xx) will be returned. The response body will contain one or more error objects, as specified by JSON:API[97]. An error object contains the following fields:

• status - the HTTP status code associated with the error

• code - interface-specific, precise, error code

• title - human-readable short description of the error which remain fixed with every occurrence

• detail - human-readable explanation for the error, specific to the occurrence

• source - an object containing the source of the error, which can be either pointer, a JSONPointer[98] to the specific value in the JSON document given in the request which caused the error or parameter, a string indicating which parameter in the URL query string caused the error

59 Listing 20 shows an example of an error object.

HTTP/1.1 400 Bad Request

{ "errors": [ { "code": 405, "status": 400, "title": "Invalid query parameter", "detail": "'invalid-param' is not a supported parameter", "source": { "parameter": "invalid-param" } } ] }

Listing 20: Example of an error response

A.4 Fetching Data

All the types of resources available in the interface can be fetched using a GET HTTP request to an URI.

A.4.1 Fetching Available Collections Sending a GET request to the root endpoint (/) will result in a response containing a simple collection resource, which maps the names of all the collections of resources stored under the interface to links to those collections. An example is shown in Listing 21.

HTTP Request:

GET / HTTP/1.1

HTTP Response:

HTTP/1.1 200 OK

{ "articles": "/articles", "users": "/users", "books:" "/books" }

Listing 21: Root endpoint GET request

60 If there are no collection stored under the interface (no data, empty database), then the interface will respond with an empty object ({}). This behaviour is not specified by JSON:API as the specification is not concerned with the behaviour of the root endpoint. This has been explained in Section 3.2.1.

A.4.2 Fetching Collections Listing 22 shows an example of a GET request to a collection endpoint and the response returned by the interface containing the representation of the collection. The collection contains two resources of type items. Links to the individual resources are also provided. If the collection does not exist, the interface will respond with a 404 Not Found error response. If the collection exists, but there are no resources stored under it, the response will have a 200 OK status code and the data attribute will be an empty array ([]).

61 HTTP Request:

GET /items HTTP/1.1 Accept: application/vnd.api+json

HTTP Response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "links": { "self": "/items" }, "data": [ { "id": "0dd64cec-5ee3-4213-a6bc-31acd5e06f31", "type": "items", "attributes": { "available": true, "colour": "green", "name": "Product9", "price": 21.5, "stock": 2 }, "links": { "self": "/items/0dd64cec-5ee3-4213-a6bc-31acd5e06f31" } }, { "id": "3b6a482d-5f5e-496d-b3aa-a302f181603e", "type": "items", "attributes": { "available": true, "colour": "yellow", "name": "Product11", "price": 60, "stock": 10 }, "links": { "self": "/items/3b6a482d-5f5e-496d-b3aa-a302f181603e" } } ] }

Listing 22: Fetching a Collection

62 A.4.3 Fetching Individual Resources Further to the example in Listing 22, Listing 23 shows how an individual resource can be fetched by following a link given in the collection. As before, if the resource does not exist, the interface will respond with a 404 Not Found response code. The HTTP response also includes an ETag header. ETags and their purpose are described in Section 3.2.5. The client can also send a conditional GET request to check if the state of the resource has changed. The GET request has to include the ETag in the If-None-Match header. If the ETag of the resource on the server is the same as the ETag of the resource given in the request, then the server will respond with a 304 Not Modified code and an empty response body. If the ETags do not match, the server will respond with the current state of the resource, as usual. An example of a conditional GET request is shown in Listing 23.

HTTP Request:

GET /items/3b6a482d-5f5e-496d-b3aa-a302f181603e HTTP/1.1 Accept: application/vnd.api+json

HTTP Response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json ETag: efa7b5d721e2d0edd3bae1fe969ffa9877d6f725

{ "links": { "self": "/items/3b6a482d-5f5e-496d-b3aa-a302f181603e" }, "data": [ { "id": "3b6a482d-5f5e-496d-b3aa-a302f181603e", "type": "items", "attributes": { "available": true, "colour": "yellow", "name": "Product11", "price": 60, "stock": 10 } } ] }

Listing 23: Fetching an individual resource

63 HTTP Request:

GET /items/3b6a482d-5f5e-496d-b3aa-a302f181603e HTTP/1.1 Accept: application/vnd.api+json If-None-Match: efa7b5d721e2d0edd3bae1fe969ffa9877d6f725

HTTP Response:

HTTP/1.1 304 Not Modified

Listing 24: A conditional GET request

A.4.4 Fetching Relationships Similar to the examples above, relationships can be fetched by sending a GET request to a relationship URI. Links to individual relationships are given in resource representation (an example is given in Listing 19). The response codes returned by the server are the same as in the previous scenarios. Listing 25 shows an example of fetching a relationship between an order resource and two items.

64 HTTP Request:

GET /orders/361f0399-9a6b-43e3-be24-cd112eb64810/relationships/items ,→ HTTP/1.1 Accept: application/vnd.api+json

HTTP Response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "data": [ { "type": "items", "id": "64335af0-57ad-4754-b007-8c69397a1968" }, { "type": "items", "id": "76eb9503-5160-4d91-a2fa-02737adafae7" } ], "links": { "self": ,→ "/orders/361f0399-9a6b-43e3-be24-cd112eb64810/relationships/items", "related": "/orders/361f0399-9a6b-43e3-be24-cd112eb64810/items" } }

Listing 25: Fetching a relationship

By sending a GET request to the related link given in the representation of a relationship, a response containing a collection of the resources linked in the relationship will be returned. In the example in Listing 25, the response will contain a collection of the two items.

A.5 Querying Supported

The interface supports querying under the form of URL query string parameters. A query can be applied to any type of resource (collection, individual resource or relationship) in any request for which a response containing a representation of a resource is returned. However, some query parameters cannot be applied to any type of resource, as detailed below. If a request contains an incorrect query or if a query parameter which is not recognised by the interface is included, a 400 Bad Request response is returned containing details about the cause of the error, in the format mentioned in A.3. The interface offers the following querying functionality:

65 A.5.1 Inclusion of related resources The include parameter specifies which related resources to primary data will be included in the response. It can be applied to any type of resource. Example usage: GET /orders/1?include=items HTTP/1.1 An example response document for this request can be found in Section A.11, which was too large to include in this section. The parameter respects the semantics defined by the JSON:API specification[99].

A.5.2 Sparse Fieldsets The fields parameter specifies which fields of a resource will be included in aresource representation. It can be applied to any type of resource. Example usage:

GET /orders/1?include=items&fields[orders]=customer&fields[items]=name,stock ,→ HTTP/1.1 In the response for the previous example request, the order resource will only have the customer attribute included in the response while the items resources will only have name and stock fields returned. The parameter respects the semantics defined bythe JSON:API specification[100].

A.5.3 Sorting The sort parameter specifies the fields and order the resources in a collection’s primary data are sorted by. This parameter can only be applied to collection endpoints. Example: GET /items/1?sort=stock,-name HTTP/1.1 The - character in front of a field specifies descending order for the corresponding field. If the parameter is not present in a query, the interface will return the resource in a non-specified order. The parameter respects the semantics defined by the JSON:API specification[101].

A.5.4 Paging The page parameter specifies a limit for the number of resources returned in a response using offset and limit parameters. It can be applied only to collection endpoints. Example: GET /items/1?sort=stock&page[limit]=2&page[offset]=0 HTTP/1.1 JSON:API does not specify a precise pagination strategy (page and limit are also supported)[55]. For this interface, the offset and limit seemed to be the appropriate one as it is the most flexible and also imitates the one in SQL. If the parameter is present in a query, then the response will contain pagination links (prev, next, first, last) which offer the client the possibility to navigate forward or backwards through the result set. The complete example given in Section A.11 contains such pagination links.

66 A.5.5 Filtering The filter parameter filters the resources in a collection based on the values of their attributes Example: GET /items?filter[items][colour]=blue&filter[items][available]=1 HTTP/1.1 JSON:API is agnostic about filtering strategies[58]. This functionality was added to the interface in order to allow a basic form of filtering for resources.

A.6 Creating Resources

Resources can be created by sending a POST request to a collection endpoint, with the resource to be added to the collection. The type field of the resource must match the name of the collection in which the resource is added. The resource sent in the request must not include an id field, as the ID is generated by the server. If the request is successful, a 201 Created response is returned with the new resource, otherwise the client receives 400 Bad Request error response. Listing 26 shows how a new resource can be added to an existing collection.

67 HTTP request:

POST /items HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": { "type": "items", "attributes": { "name": "Product2", "price": 10.33 } } }

HTTP response:

HTTP/1.1 201 Created Location: /items/d9626845-a4dd-4d8d-92ed-86e07958ada3 Content-Type: application/vnd.api+json ETag: 0f8a9525baafe3c0371e125fe7c132067e9aeda3

{ "data": { "type": "items", "id": "d9626845-a4dd-4d8d-92ed-86e07958ada3", "attributes": { "name": "Product2", "price": 10.33 }, "links": { "self": "/items/d9626845-a4dd-4d8d-92ed-86e07958ada3" } } }

Listing 26: Creating a new resource

The client can also create new collections. This can be done by sending a POST request to the root endpoint with a new resource, as before. The server will respond with the newly created collection and its location on the server. An example is given in Listing 27. This behaviour is not part of JSON:API and is explained in Section 3.2.

68 HTTP request:

POST / HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": { "type": "items", "attributes": { "name": "Product2", "price": 10.33 } } }

HTTP response:

HTTP/1.1 201 Created Location: /items Content-Type: application/vnd.api+json

{ "data": [{ "type": "items", "id": "d9626845-a4dd-4d8d-92ed-86e07958ada3", "attributes": { "name": "Product2", "price": 10.33 }, "links": { "self": "/items/d9626845-a4dd-4d8d-92ed-86e07958ada3" } }], "links": { "self": "/items" } }

Listing 27: Creating a new collection

A.7 Updating Resources

Individual resources can be updated by sending a PATCH request to the server. The request must contain the updated version of the resource in the body and must also specify the ETag of the resource in the If-Match header. The reasons for this approach are explained in Section 3.2.5. If the ETag provided does not match the ETag on the server (of the resource in its

69 newest state), then the server will return a 412 Precondition Failed response. If the resource given in the request is the same as the resource stored on the server, a 304 Not Modified response will be returned. If the resource given in the request is not correct, then the server will return a 400 Bad Request response. If the update is successful, the server will return a 200 OK response. Examples of successful and unsuccessful update requests are given in Listings 28 and 29.

HTTP request:

PATCH /items HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json If-Match: 0f8a9525baafe3c0371e125fe7c132067e9aeda3

{ "data": { "type": "items", "attributes": { "name": "Product2", "price": 3.2 } } }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json ETag: 3ed595c9f58242555650d25cb9bbe1a6cb9d9a5e

{ "data": { "type": "items", "id": "d9626845-a4dd-4d8d-92ed-86e07958ada3", "attributes": { "name": "Product2", "price": 3.2 }, "links": { "self": "/items/d9626845-a4dd-4d8d-92ed-86e07958ada3" } } }

Listing 28: Updating an existing resource

70 HTTP request:

PATCH /items HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json If-Match: 0f8a9525baafe3c0371e125fe7c132067e9aeda3

{ "data": { "type": "items", "attributes": { "name": "Product9", "price": 3.2 } } }

HTTP response:

HTTP/1.1 412 Precondition Failed Content-Type: application/vnd.api+json ETag: 3ed595c9f58242555650d25cb9bbe1a6cb9d9a5e

{ "errors": [ { "code": 412, "status": 412, "title": "Conflict", "detail": "ETag given does not match the actual ETag of the ,→ resource: 3ed595c9f58242555650d25cb9bbe1a6cb9d9a5e", } ] }

Listing 29: Conflicted update request

A.8 Updating Relationships

Although relationships can be updated along with resources, they can also be updated independently by sending a PATCH, POST or DELETE request to a relationship endpoint. A PATCH request will replace the relationship members entirely and the client must supply the ETag of a relationship in the request in order to be able to perform the update. This operation is similar, therefore, to the update operation for an individual resource, presented in Section A.7. An example is given in Listing 30.

71 HTTP request:

PATCH /orders/361f0399-9a6b-43e3-be24-cd112eb64810/relationships/items ,→ HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json If-Match: 0f8a9525baafe3c0371e125fe7c132067e9aeda3

{ "data": [ { "type": "items", "id": "64335af0-57ad-4754-b007-8c69397a1968" }, { "type": "items", "id": "76eb9503-5160-4d91-a2fa-02737adafae7" } ], "links": { "self": "/orders/361f0399-.../relationships/items", "related": "/orders/361f0399-.../items" } }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json ETag: 3ed595c9f58242555650d25cb9bbe1a6cb9d9a5e

Listing 30: Complete update request for a relationship

72 A POST request can be used to add members to a relationship (Listing 31). The request must include the members to be added to the relationship, in the body. Because the request cannot overwrite changes made by other clients, the client is not required to provide an ETag.

HTTP request:

POST /orders/361f0399-9a6b-43e3-be24-cd112eb64810/relationships/items ,→ HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": [ { "type": "items", "id": "6999a62e-2420-4bf7-b606-cebfcd7908a5" } ] }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "data": [ { "type": "items", "id": "64335af0-57ad-4754-b007-8c69397a1968" }, { "type": "items", "id": "76eb9503-5160-4d91-a2fa-02737adafae7" }, { "type": "items", "id": "6999a62e-2420-4bf7-b606-cebfcd7908a5" } ], "links": { "self": "/orders/361f0399-.../relationships/items", "related": "/orders/361f0399-.../items" } }

Listing 31: Adding members to a relationship

73 A DELETE request can be used to remove members from a relationship (Listing 32). The request must include the members to be removed from the relationship, in the body. Similar to the POST request, the client is not required to provide an ETag.

HTTP request:

DELETE /orders/361f0399-9a6b-43e3-be24-cd112eb64810/relationships/items ,→ HTTP/1.1 Content-Type: application/vnd.api+json Accept: application/vnd.api+json

{ "data": [ { "type": "items", "id": "76eb9503-5160-4d91-a2fa-02737adafae7" } ] }

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json

{ "data": [ { "type": "items", "id": "64335af0-57ad-4754-b007-8c69397a1968" }, { "type": "items", "id": "6999a62e-2420-4bf7-b606-cebfcd7908a5" } ], "links": { "self": "/orders/361f0399-.../relationships/items", "related": "/orders/361f0399-.../items" } }

Listing 32: Removing members from a relationship

A.9 Deleting Resources

Collections and resources can be deleted using a DELETE request. If the update is successful, the server will respond with a 204 No Content code. Listing 33 shows an

74 example of a DELETE request for an individual resource.

HTTP request:

DELETE /orders/361f0399-9a6b-43e3-be24-cd112eb64810 HTTP/1.1

HTTP response:

HTTP/1.1 204 No Content Content-Type: application/vnd.api+json

Listing 33: Deleting an individual resource

A.10 HEAD and OPTIONS

The interface supports two additional HTTP methods, HEAD and OPTIONS, on all endpoints. This behaviour is not part of JSON:API but lies within the good practises of HTTP. A HEAD request sent by the client acts like a GET request. The only difference is that no response body will be provided by the server, which makes this type of request useful for checking if a resource exists, to retrieve the ETag, etc as it is quicker than a GET request. Listing 34 shows an example of a HEAD request.

HTTP request:

HEAD /orders/361f0399-9a6b-43e3-be24-cd112eb64810 HTTP/1.1

HTTP response:

HTTP/1.1 200 OK Content-Type: application/vnd.api+json ETag: 3ed595c9f58242555650d25cb9bbe1a6cb9d9a5e

Listing 34: Example of a HEAD request

OPTIONS can be used by a client to identify which methods are allowed on a specific endpoint. The server will respond a 200 OK response with no body and an Allow header which lists all the allowed methods for the endpoint. An example is given in Listing 35.

75 HTTP request:

OPTIONS /orders/361f0399-9a6b-43e3-be24-cd112eb64810 HTTP/1.1

HTTP response:

HTTP/1.1 200 OK Allow: GET, PATCH, DELETE, OPTIONS, HEAD

Listing 35: Example of an OPTIONS request

A.11 A Full JSON:API Document

{ "links": { "self": "/orders?sort=-total&page[offset]=0&page[limit]=2 &fields[orders]=total,payment-date&filter[orders][paid]=1&include=items", "next": "/orders?sort=-total&page[offset]=2&page[limit]=2 &fields[orders]=total,payment-date&filter[orders][paid]=1&include=items", "last": "/orders?sort=-total&page[offset]=8&page[limit]=2 &fields[orders]=total,payment-date&filter[orders][paid]=1&include=items" }, "data": [{ "type": "orders", "id": "0258aa19-e9ab-410c-96f6-c6c2fd2ae129", "attributes": { "payment-date": "2015-10-01 02:01:00", "total": 124.5 }, "relationships": { "items": { "links": { "self": "/orders/0258aa19-.../relationships/items", "related": "/orders/0258aa19-.../items" }, "data": [ { "type": "items", "id": "550e8400-e29b-41d4-a716-446655440000" } ] } }, "links": { "self": "/orders/0258aa19-e9ab-410c-96f6-c6c2fd2ae129" } },{ "type": "orders", "id": "e49a065d-3f0a-4dd1-a6d2-ec732c9ee085",

76 "attributes": { "payment-date": "2015-10-05 05:01:00", "total": 65.0 }, "relationships": { "items": { "links": { "self": "/orders/e49a065d-.../relationships/items", "related": "/orders/e49a065d-.../items" }, "data": [ { "type": "items", "id": "d9626845-a4dd-4d8d-92ed-86e07958ada3" } ] } }, "links": { "self": "/orders/e49a065d-3f0a-4dd1-a6d2-ec732c9ee085" } }], "included": [ { "links": { "self": "/items/550e8400-e29b-41d4-a716-446655440000" }, "data": { "type": "items", "id": "550e8400-e29b-41d4-a716-446655440000", "attributes": { "name": "Product1", "price": 22.12 } } }, { "data": { "type": "items", "id": "d9626845-a4dd-4d8d-92ed-86e07958ada3", "attributes": { "name": "Product2", "price": 10.33 }, "links": { "self": "/items/d9626845-a4dd-4d8d-92ed-86e07958ada3" } } } ]

77 }

The document above is an example collection represented in JSON:API, which could be the result of a GET request to the orders collection endpoint. It also shows examples of the query parameters supported by the interface, which are present in the links. The collection includes the items resources which are referenced by the relationships of the orders resources, in the included array.