tech talk

Going gRPC: Open sourcing the definitions for our management APIs

As part of a talk on our journey and lessons in going gRPC, we're sharing the API definitions for our management APIs.

Pim Nauts, Co-Founder

Reading time: about 7 minutes.

About a year ago, we decided to test and setup some of our newer apps in gRPC as an experiment. Well, that escalated quickly: we switched all our management APIs and communication from REST to gRPC.

At the last OpenValue meetup, our engineers Robin and Jan-Kees gave an overview of this journey and our lessons. As part of the talk we’re open sourcing our proto API definitions. Read along for a summary of our journey, find the repo at the end of this post and watch the talk.

Going gRPC with STRM

We’re going to assume you know what gRPC is. If not, the gRPC docs have you covered.

At STRM, we’re building a privacy-focused data processing platform. We develop a concept of what we call privacy streams, which is a data pipeline that receives, transforms and delivers data according to a strict and pre-approved definition of the data (data contracts).


Back to tech. Our back-end has quite a broad range of responsiblities: user management, authentication and roles, registering and validating schemas and contracts, a gateway, streams management (an abstraction of a Kafka topic for streaming, or more traditional batch pipeline for file-based data). And it’s not a sequential model: most apps interact with most other apps. With such a high degree of internal communication we concluded our best option was to build API First from the beginning.

At the start, we wrote our APIs as we were used to: REST through OpenAPI specs.

But as the cascade through services is quite high, the shortcoming of a traditional REST approach quickly became clear. With all the communication between apps (like a network of nodes), every app needs to be aware of almost every other app’s API spec. With OpenAPI, the repositories themselves had to include the API spec of every other repository. Over time we started to discover discrepancies between the Swagger pages (the “public” spec) and the actual server implementations. This lead to quite a few “da fuq” errors, where you really can’t map the behavior to your expectation or the error message. All in: we were losing time over it.

From a strategic perspective, we take a platform and language agnostic approach to new functionality, where we look at the goal to be achieved first and the tool or language only second - e.g. while many of our services are Kotlin, our CLI is written in Go as we found more mature and convenient tooling for it, and a visual interface is more logical in TypeScript (React) than a server-side render with HTML templates. With this “broadening” of languages and platforms, we would have to write separate clients for each new choice we made!

(this is the main reason many CTOs choose to centralize around just a few platforms, which is a very sensible thing to do. We could’ve simply limited ourselves, but as we were eyeing gRPC already, this provided a nice experimentation ground).

Enter gRPC

So, we needed a high performant, API-first protocol with high emphasis on compatibility, and low effort to operate over different platforms. gRPC offered this, and some implementation specifics (HTTP/2, binary serialization) are also attractive for the kind of work we are doing.

And so we got to work on writing and setting up our first protos.

All our eggs in one basket

Writing RPCs and exposing them is well documented. One of the most opinionated moves we made was putting ALL API definitions together.

Although we’re not in big company territory (yet), we run quite a few services (it’s micro services after all). By centralizing the API defintions, we don’t have to check every repository and build/keep a mental model of each API involved for a task that spans multiple.

Next to that, there’s quite some reuse of common entities across our apps, such as our (Privacy) Stream object, which is used by a lot of our apps. Moreover, gRPC/proto forces us to be API-first, because you can’t even expose an app without starting from the API! This fosters a lot of interoperability and means we first need to think about how a consumer will interact with the service. With a well-defined API and gRPC’s client code generation, we can truly focus on the information exchange model instead of specific implementation details (and just generate the client stubs and server implementation classes!).

Finally, as everything is centralized, we have to achieve a high level of “working product” before we merge into the specification, helping us to find unintended breaks early on (as it detects how even small changes in specs cascade over services). Doesn’t that eat into velocity, we hear you ask? Well, no. While the first merge is (slightly) later compared to a merge-quickly model, overall the time-to-live for a production feature is actually shorter (and the quality higher!) as you get more feedback from the team earlier on, minimizing rework (software is iterative!).

A centralized API definition, distributed to any platform

So with all this in place - an API first way of working and a cross-language compatible setup - the next question was how to distribute these proto definitions to every app. We saw two possible routes:

  • distribute the files itself through cloning, and generate client code inside each application
  • compile for different applications centrally, and distribute the result of the proto compiler (artifacts) using the tools used in other languages (like a Maven Repository, or NPM for JS/TS, etc.)

And you guessed it: we went for that second option.

But, aren’t APIs about decentralization in the first place? They are, and at first this might seem a bit of an antipattern in that regard.

But from the perspective of the entire landscape, as every app and service by itself has become independent of the way we choose to specify the API, it’s not: the definitions are clear, and there’s working code to interact with any other component of the landscape. So in our experience you can (more easily) achieve the level of decloupling you strive for as part of being “API-first”.

What we learned

And with this setup running for over a year (including adding services, refactoring, and even changing our deployment model), we can confidently say we learned a thing or two about going gRPC for your (management) APIs:

  • It works! With the learning curve for junior or incoming developers limited to a few days of understanding concepts and adjusting to the way of working (if they come from REST).
  • gRPC (in our setup) removes a lot of possible friction as your landscape grows: you trade some extra work up front per service or release for velocity over time (which is evidently the higher exponent!)
  • Interoperability! You don’t have to write all those clients! Not today, not for Java, and not when Julia or Rust or Crystal or whatevah finally become mainstrain (as cross-language generation is included or becomes available for basically any dominant language).
  • Backwards compatibility is a blessing. Breaking changes are always a new version. And because code is generated, this allows for a lot of flexibility in having different versions in co-existence (and so your deprecation policies can be much more relaxed at lower maintenance cost to your own team!)

But there’s some caveats, too. Some mistakes to avoid for free:

  • Check yo defaults for generated code! A string variable of “” is an empty string, not a null!
  • The linter and rules that come packaged are very powerful BUT quite elaborate and focused on the Googleplex, you probably don’t need all (or even a lot) of them.
  • Don’t try to understand the generated code or expect it to be legible to humans. Just don’t.

Open-sourcing our gRPC API defitions

As we leverage a lot of open source ourselves, this was a nice opportunity to give back a litte. That’s why we’ve open sourced (under MIT) our gRPC API definitions as our small contribution to your next API.

It is currently a push-only mirror of our API Definitions repo we keep in GitLab, but we hope this serves as inspiration and/or a quick way for you to get started with gRPC yourself!

Head over the Github repository and fork away 🙂.

PS We’re hiring!

Want to help data teams build awesome data products without sacrificing privacy in the process? There’s plenty of cool work left. Did we mention we are hiring!?

Decrease risk and cost, increase speed encode privacy inside data with STRM.