Cross-Platform Data Models and API Using gRPC

Sebastian Hagedorn, Felix Lamouroux

SumUp. A better way to get paid Outline

1. Motivation & Goals 2. Choosing the Right Cross-Platform Technology 3. Introduction to Protocol Buffers and gRPC 4. Collaborative API Design 5. Here Be Dragons

SumUp. A better way to get paid Create a Playful & Performant App that Feels Native

SumUp. A better way to get paid Starting Point

● Green field: New project, clean slate, no legacy components, engineering-driven decisions ● Team of experienced developers per platform ● Preferred languages: iOS (Swift), Android (Kotlin), Web (Elm), Server-Backend (Go) ● No complex business logic on mobile/desktop clients

SumUp. A better way to get paid Goals

● Ensure consistent API usage ● Reduce boilerplate per platform ○ Parsing & serialization ○ Transport layer ● No critical 3rd party dependencies above transport layer ● Integrate seamlessly with native languages & tools

SumUp. A better way to get paid Options Considered: JNI/Djinni

● Interfaces defined in platform-agnostic Interface Definition Language (IDL) ● Platform-specific interfaces (Java or C++/Objective-C) are generated ● Service/shared code must be implemented in C++ ● Decided against it: ○ Not usable for web client ○ Our team lacks C++ experience for large/critical code bases ○ Little client-side business logic in our UI-heavy project

https://github.com/dropbox/djinni

SumUp. A better way to get paid Options Considered: React Native

● Allows cross-platform code (way) beyond data model and networking ● Enables JavaScript developers to write native apps ● Decided against it: ○ Does not help to define a common model unless all components use React Native (backend must use node.js) ○ Our team consists of experienced platform developers (with little JS experience) ○ Does not align with our goal of playful & performant apps

SumUp. A better way to get paid Technology Stack: Protocol Buffers

● Binary serialization format developed by Google ● Model types are defined in simple Protobuf language ● Per-platform/per-language code is generated from Proto files ○ Generates interface and implementation for (de)serialization ○ Official support for many languages: Java, Python, Objective-C, C++, Go, Ruby, and C# ○ More can be added via plugins, as Apple did for Swift: https://github.com/apple/swift-protobuf

SumUp. A better way to get paid Auto-generated types

Golang

Java Proto

Objective-C

Swift code generator available from Apple: https://github.com/apple/swift-protobuf

Swift SumUp. A better way to get paid Technology Stack: gRPC

● “A high performance, open-source universal RPC framework" (grpc.io) ● API endpoints are defined in IDL (Protobuf by default) ● Server and client libraries are generated ○ Protobuf handles (de)serialization ○ gRPC handles networking ● Apps call generated endpoint APIs in native language ● gRPC-ProtoRPC generators & runtimes available in Swift and ObjC ○ We chose ObjC for better stability (Swift support is declared experimental) ○ Interdependency between Protobuf (Swift support implemented by Apple) and gRPC (Swift support implemented by official project maintainers): Cannot (necessarily) update components individually ○ Apple uses gRPC-ProtoRPC in private CloudKit server-to-server API

SumUp. A better way to get paid Code Speaks Louder than Words

SumUp. A better way to get paid Login Example: API Specs (Service)

▪ Support JSON fall-back via proxy ▪ Shared documentation also provided in auto-generated clients. ▪

SumUp. A better way to get paid Login Example: API Specs (Model)

SumUp. A better way to get paid Login Example: iOS Callsite (Swift)

SumUp. A better way to get paid Login Example: Backend Implementation (Go)

SumUp. A better way to get paid Collaborative API Design: Tooling & Setup

● Bridge gap between frontend & backend teams: Design API collaboratively ● Cross-platform API submodule ○ Central source of truth for model and API definitions (Proto files) ○ Tooling to generate code for all platforms (using Makefiles) ○ Integration of runtimes and generated files solved per platform repository ● All developers invited to contribute via Pull Requests to shared API specs ○ On-the-fly code generation allows local iterations ○ Review & verification on all platforms ● Shared vocabulary about types (user, account, profile, …)

SumUp. A better way to get paid Don't Let Clients Send Messages With Ignored Fields

▪ Messages that are used in requests should only contain fields that are mutable by the client. ▪ Information about an entity that is only mutable by the server should only be available as part of responses. ▪ Combine messages to decorate them (composition over inheritance)

SumUp. A better way to get paid Avoid Duplicate Fields

Avoid repeating fields by wrapping decorating messages around existing ones.

SumUp. A better way to get paid Here Be Dragons

SumUp. A better way to get paid Beware: All Fields are Optional

Protocol buffers 3 no longer allows marking fields as required

▪ Makes backwards/forward-compatibility easier: ▪ required fields invalidate the entire message regardless of context ▪ parts of an application could pass around partially-complete messages internally ▪ whole applications might simply be interested in passing message through (see routers) ▪ Ensure that each callsite using a message performs sensible validation (no blanket rejections)

▪ Provide invalid default values to fields if you need to ensure that clients set them explicitly (see enums) ▪ Only validate messages when you care about the content

See also: https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers SumUp. A better way to get paid Optionality & Versioning

“We’re mitigating [backward and forward compatibility issues] by using Protocol Buffers and controlling most new features with flags, which remain turned off during a partition upgrade, and are enabled only after all hosts are updated.”

From Apple’s CloudKit paper (http://www.vldb.org/pvldb/vol11/p540-shraer.pdf)

SumUp. A better way to get paid Check Your Defaults

▪ Explicit UNINITIALIZED_* values in enums ensure that unset values are caught

▪ Check if your business logic makes sense for other type defaults, such as empty string or zero integers

SumUp. A better way to get paid Enforce Client-Side Non-Optional Values

▪ Linter enforces usage of “convenience” initializers ▪ Unit tests make sure that model changes do not go unnoticed

SumUp. A better way to get paid Tooling Quirks

● Custom Makefile to process Proto files ○ Not trivial to write ○ Integration is different per platform/IDE ● Make modifies output file timestamps regardless of content diff ○ Swift compiler re-builds the entire module ○ Solution: Extra script using rsync that only makes actual changes available to Xcode ● Integration of gRPC runtime is not trivial ○ …except when using CocoaPods ● Protobuf optionality is not reflected by ObjC header annotations ○ All properties are force-unwrapped

SumUp. A better way to get paid We Are Hiring

We are building a product team here in Cologne.

▪ Looking for iOS, Android, Web, Backend, UI/UX design…. ▪ Strong mobile focus with many senior developers.

http://sumup.com/careers/

SumUp. A better way to get paid Sebastian Hagedorn Felix Lamouroux iOS Lead Developer Project Lead - New Ventures (and iOS dev)

▪ @hagidd ▪ @felixlamouroux ▪ [email protected] ▪ [email protected]

SumUp. A better way to get paid