The API revolution continues! API Whitepaper Selection 25+ pages of knowledge for API enthusiasts. Learn about exciting topics such as API Development, Management, Design and Platforms & Software as a Service API Conference @api_conference @apiconf #APICON apiconference.net content API Design The demand for realtime APIs 3 Realtime API Family helps us to reduce complexity by Kieran Kilbride-Singh API Development What APIs Can and Cannot Do 6 How to avoid common API misconceptions and the resulting disillusionment by Erik Wilde Go for gRPC? 11 Develop gRPC services with Go by Rainer Stropek API Platforms & Software-as-a-Service X-Road – The Free and Open Source Data Exchange Layer 17 A centrally managed distributed Data Exchange Layer between information systems by Petteri Kivimäki Regulation as a Service 21 Rethinking APIs by Khallai Taylor API Management Shopware 6 and the new API-First Approach 23 Putting the API on the Acid Test by Thomas Eiling APIs have Properties – find and test them! 26 Taking executable specifications seriously by Dr. Michael Sperber WHITEPAPER API Design Realtime API Family helps us to reduce complexity The demand for realtime APIs Looking to 2020 and beyond, the proportion of data produced and consumed in realtime is growing exponentially by Kieran Kilbride-Singh defines it as anything that happens in under 100ms as this is typically the threshold for perceiving something IDC predict that by 2025 1/3 of all data produced glob- as happening immediately. ally will be realtime. The engineers and organizations Any API designed such that data flowing from pro- that make up the realtime ecosystem, Ably included, ducers to consumers happens in the shortest amount of have yet to agree on how we describe the APIs we’re time possible can therefore be described as realtime. creating and consuming that are powering this growth. Ably has settled on realtime API as an umbrella for The problem is that there are various ways to describe event-driven, streaming, pub/sub, push, and other APIs. APIs that provide realtime functionality. We’ve seen many more than four ways of describing API evangelist Kin Lane, and many others, have been this, but these are the most popular. It’s why we call it writing and talking about these new types of APIs for a the Realtime API Family (Image 1). while. Event-driven seems to be the most common descriptor. Gartner has adopted the term, stating that by 2020 50 % of all managed APIs will be event-driven. But still today there is no consistency or consensus between what terms like realtime API, event-driven API, or streaming API refer to. Often they’re used interchangeably. As an engineering team and API provid- er working on a global pub/sub messaging platform, we work with these ideas every- day. Talking to our users we see the lan- guage they use to describe both what we do and what they achieve using our APIs. And we’ve watched discussions around event-driven architecture, webhooks, and streaming data proliferate. Over the years we’ve thought extensively about the best terminology to use and arrived at what we call the Realtime API Family Defining realtime When it comes to realtime in the context of user interaction, it can be defined as a function or interaction perceived as imme- diate. As a concept in this context then, re- altime is easy enough to understand. Ably Image 1: Realtime API Family www.apiconference.net 3 WHITEPAPER API Design Image 2: Ably’s Pub/Sub Channels API acts as a serverless message bus Streaming APIs Push APIs A stream is a means of transporting data. Streaming is a Push is another umbrella term for every API that is real consumer pattern that describes how consumers receive time. events through a stream. But it’s just a producer messaging pattern. It means A streaming API will commonly address issues of data that data is pushed upstream over a connection, versus integrity with: a pull mechanism used by the request/ response pat- tern. • Message ordering - ensure messages are delivered in An example is helpful. A push notifications API (Im- the order they were published age 3) is an example of a push API. Or a sporting event • Stream continuity / resume - upon disconnection, might send a single score update that traverses multiple resume from the disconnected point within a set pe- systems via push APIs, resulting in millions of messages riod of time to global fans within a few hundred milliseconds. • Contiguous serial numbers - a simple series of `ACK` Push can also refer to triggering a new request as with a or `NACK` responses, each addressing a contiguous Webhook. Or it can mean ‘push subscription’ – i.e. a pro- sequence of messages ducer needs to reach out to a consumer. WebSub is an ex- ample of this. But that’s beyond the scope of this article. Kafka offers a streaming API following this model for internal systems. Event-driven APIs Event-driven is an architectural design pattern that de- Pub/Sub APIs fines how a system processes data. Pub/Sub (Publisher/Subscriber) is an infamous messag- ing design pattern. When it debuted in 1987 it solved loose decoupling of servers. The earliest distributed systems used to send events between servers using Pub/Sub, but all within reliable networks. This pattern operates by publishing API 365 Workshop: messages on a topic (or channel) within a message bus, Building APIs and managing and subscribers can listen for events based on those top- them with serverless, containers, ics. Pub/Sub is over 30 years old now. The design makes API gateways and Azure no attempt to define semantics around ordering or Dr. Roberto Freato (Freelance) continuity (loss of connection). While it’s still a pat- tern used widely today, these are issues that must be The main focus of this workshop is to considered and addressed in our world of unreliable have a look into the API ecosystem connections. to approach the problem with good insights and to organize things to be Ably provides a pub/sub (Image 2) API following this reliable, available and to reduce the model but with guarantees around ordering, continuity, overall governance effort. idempotency, and more. www.apiconference.net 4 WHITEPAPER API Design event in an event-driven world triggers a chain of events which must be processed down- stream, extending through the entire data supply chain. All components in this sup- ply chain are reactive, rapidly responding to events and per- forming onwards processing. The time-bound, reactive nature of this data supply chain causes increased engi- neering and infrastructure complexity. And when com- pared to REST APIs following the request/response model, the complexity is inverted and put onto API producers rather than API consumers. However, that’s a conver- sation beyond the scope of this article. But it’s something we’ve spoken about before at API Image 3: Ably’s unified push notifications API And so we arrive at Realtime APIs It simply states a system should be responding/re- Streaming, Pub/Sub, and Push are all patterns that can acting to events as they happen. Streaming, Pub/Sub, be delivered through event-driven architecture. But the and Push are all messaging patterns that can delivered outcomes of streaming, push, pub/sub, and event-driven through an event-driven architecture. As such they can are all realtime functionality. They’re all means of get- all fall under the umbrella of event-driven APIs. ting data from producer to subscriber in the shortest Most of us know what an event-driven API is but possible time. So we can call them realtime APIs. an example never hurts. Unlike traditional request/re- But you can’t have a realtime API unless it’s delivered sponse APIs where data is requested, event-driven APIs as an event-driven, pub/ sub, streaming, or push API. push data from a producer to a consumer. They can be Hence how we’ve settled on the umbrella term of real- quite simple or very complex. time API to encompass them all. In the above example a retailer needs an event-driven architecture to respond to things as they happen. An Navigating complexity and fragmentation As you can see, it’s easy to use these terms interchangea- bly despite them all being different. The realtime ecosys- tem is still maturing and changes all the time. Eventually we believe developers and organizations will naturally come to a shared and standardized terminology for APIs Do we have a Packaging designed to deliver real time functionality. Problem? Until then, the Realtime API Family helps us to reduce complexity when we explain our cloud infrastructure Maxim Zaks (Self Employed) and APIs to the developer community, our users, and our potential customers. Designing an API is a complex ende- avour. It contains of many details and considerations. One of such conside- rations is: How do we package data we are sending around? In this talk, we Kieran Kilbride-Singh (Product Marketing Manager) will discuss different packaging techniques like CSV, is writer and marketer with enough technical know- XML, JSON, Protocol Buffers and FlatBuffers. We will how to be dangerous in a GitHub repo. He first flexed evaluate the benefits and pitfalls those techniques his fingers writing about topics like interoperability in embedded IoT devices. Nowadays he thinks and bring to the table. writes about realtime infrastructure, event-driven APIs, and next-generation connected experiences for Ably.io. www.apiconference.net 5 WHITEPAPER API Development How to avoid common API misconceptions and the resulting disillusionment What APIs Can and Cannot Do APIs sometimes are perceived as having almost magical properties: Since they are so essential to digital transformation, and all of our digital experiences to- day are powered by APIs, the conclusion is that once you have and use APIs, all the wonders of digital transformation are simply going to happen.