Cross-Platform Data Models and API Using gRPC Sebastian Hagedorn, Felix Lamouroux SumUp. A better way to get paid Outline 1. Motivation & Goals 2. Choosing the Right Cross-Platform Technology 3. Introduction to Protocol Buffers and gRPC 4. Collaborative API Design 5. Here Be Dragons SumUp. A better way to get paid Create a Playful & Performant App that Feels Native SumUp. A better way to get paid Starting Point ● Green field: New project, clean slate, no legacy components, engineering-driven decisions ● Team of experienced developers per platform ● Preferred languages: iOS (Swift), Android (Kotlin), Web (Elm), Server-Backend (Go) ● No complex business logic on mobile/desktop clients SumUp. A better way to get paid Goals ● Ensure consistent API usage ● Reduce boilerplate per platform ○ Parsing & serialization ○ Transport layer ● No critical 3rd party dependencies above transport layer ● Integrate seamlessly with native languages & tools SumUp. A better way to get paid Options Considered: JNI/Djinni ● Interfaces defined in platform-agnostic Interface Definition Language (IDL) ● Platform-specific interfaces (Java or C++/Objective-C) are generated ● Service/shared code must be implemented in C++ ● Decided against it: ○ Not usable for web client ○ Our team lacks C++ experience for large/critical code bases ○ Little client-side business logic in our UI-heavy project https://github.com/dropbox/djinni SumUp. A better way to get paid Options Considered: React Native ● Allows cross-platform code (way) beyond data model and networking ● Enables JavaScript developers to write native apps ● Decided against it: ○ Does not help to define a common model unless all components use React Native (backend must use node.js) ○ Our team consists of experienced platform developers (with little JS experience) ○ Does not align with our goal of playful & performant apps SumUp. A better way to get paid Technology Stack: Protocol Buffers ● Binary serialization format developed by Google ● Model types are defined in simple Protobuf language ● Per-platform/per-language code is generated from Proto files ○ Generates interface and implementation for (de)serialization ○ Official support for many languages: Java, Python, Objective-C, C++, Go, Ruby, and C# ○ More can be added via plugins, as Apple did for Swift: https://github.com/apple/swift-protobuf SumUp. A better way to get paid Auto-generated types Golang Java Proto Objective-C Swift code generator available from Apple: https://github.com/apple/swift-protobuf Swift SumUp. A better way to get paid Technology Stack: gRPC ● “A high performance, open-source universal RPC framework" (grpc.io) ● API endpoints are defined in IDL (Protobuf by default) ● Server and client libraries are generated ○ Protobuf handles (de)serialization ○ gRPC handles networking ● Apps call generated endpoint APIs in native language ● gRPC-ProtoRPC generators & runtimes available in Swift and ObjC ○ We chose ObjC for better stability (Swift support is declared experimental) ○ Interdependency between Protobuf (Swift support implemented by Apple) and gRPC (Swift support implemented by official project maintainers): Cannot (necessarily) update components individually ○ Apple uses gRPC-ProtoRPC in private CloudKit server-to-server API SumUp. A better way to get paid Code Speaks Louder than Words SumUp. A better way to get paid Login Example: API Specs (Service) ▪ Support JSON fall-back via proxy ▪ Shared documentation also provided in auto-generated clients. ▪ SumUp. A better way to get paid Login Example: API Specs (Model) SumUp. A better way to get paid Login Example: iOS Callsite (Swift) SumUp. A better way to get paid Login Example: Backend Implementation (Go) SumUp. A better way to get paid Collaborative API Design: Tooling & Setup ● Bridge gap between frontend & backend teams: Design API collaboratively ● Cross-platform API submodule ○ Central source of truth for model and API definitions (Proto files) ○ Tooling to generate code for all platforms (using Makefiles) ○ Integration of runtimes and generated files solved per platform repository ● All developers invited to contribute via Pull Requests to shared API specs ○ On-the-fly code generation allows local iterations ○ Review & verification on all platforms ● Shared vocabulary about types (user, account, profile, …) SumUp. A better way to get paid Don't Let Clients Send Messages With Ignored Fields ▪ Messages that are used in requests should only contain fields that are mutable by the client. ▪ Information about an entity that is only mutable by the server should only be available as part of responses. ▪ Combine messages to decorate them (composition over inheritance) SumUp. A better way to get paid Avoid Duplicate Fields Avoid repeating fields by wrapping decorating messages around existing ones. SumUp. A better way to get paid Here Be Dragons SumUp. A better way to get paid Beware: All Fields are Optional Protocol buffers 3 no longer allows marking fields as required ▪ Makes backwards/forward-compatibility easier: ▪ required fields invalidate the entire message regardless of context ▪ parts of an application could pass around partially-complete messages internally ▪ whole applications might simply be interested in passing message through (see routers) ▪ Ensure that each callsite using a message performs sensible validation (no blanket rejections) ▪ Provide invalid default values to fields if you need to ensure that clients set them explicitly (see enums) ▪ Only validate messages when you care about the content See also: https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers SumUp. A better way to get paid Optionality & Versioning “We’re mitigating [backward and forward compatibility issues] by using Protocol Buffers and controlling most new features with flags, which remain turned off during a partition upgrade, and are enabled only after all hosts are updated.” From Apple’s CloudKit paper (http://www.vldb.org/pvldb/vol11/p540-shraer.pdf) SumUp. A better way to get paid Check Your Defaults ▪ Explicit UNINITIALIZED_* values in enums ensure that unset values are caught ▪ Check if your business logic makes sense for other type defaults, such as empty string or zero integers SumUp. A better way to get paid Enforce Client-Side Non-Optional Values ▪ Linter enforces usage of “convenience” initializers ▪ Unit tests make sure that model changes do not go unnoticed SumUp. A better way to get paid Tooling Quirks ● Custom Makefile to process Proto files ○ Not trivial to write ○ Integration is different per platform/IDE ● Make modifies output file timestamps regardless of content diff ○ Swift compiler re-builds the entire module ○ Solution: Extra script using rsync that only makes actual changes available to Xcode ● Integration of gRPC runtime is not trivial ○ …except when using CocoaPods ● Protobuf optionality is not reflected by ObjC header annotations ○ All properties are force-unwrapped SumUp. A better way to get paid We Are Hiring We are building a product team here in Cologne. ▪ Looking for iOS, Android, Web, Backend, UI/UX design…. ▪ Strong mobile focus with many senior developers. http://sumup.com/careers/ SumUp. A better way to get paid Sebastian Hagedorn Felix Lamouroux iOS Lead Developer Project Lead - New Ventures (and iOS dev) ▪ @hagidd ▪ @felixlamouroux ▪ [email protected] ▪ [email protected] SumUp. A better way to get paid.