Protocol buffers are a binary content type that reduces network congestion. It comes with an Interface Description Language (IDL) to define structured data and code generators will provide you the code to transform between the binary representation and objects of a given language. The Protocol compiler has default plugins for most languages and is extensible with custom plugins to support all other languages.

The biggest benefit is that the Interface Description Language also allows you to describe an RPC service. This allows code generators to provide you with a completely typed server or client. The same IDL can be used to generate services in different technologies such as Google's gRPC or Twitch's Twirp. gRPC is sophisticated - it uses HTTP/2 to provide bi-directional streaming and has extensive plugins for auth, tracing, load balancing, etc. Twirp is simple - it combines HTTP/1.1 with a lean runtime at the cost of bi-directional communication.

The default JavaScript code generator is not great for TypeScript. Its class-based approach is far from idiomatic TypeScript, does not have optional types and needs a custom plugin since type declarations are not included in the default generator. Plenty of plugins started to appear that either augment the default generator or replace it altogether. As a result, the Protobuf ecosystem for TypeScript is a bit messy and difficult to navigate.

Define your service

Let's look at a .proto file that uses the Interface Description Language.

// blog.proto
syntax = "proto3";
package zagrit.v1;

service BlogService {
  rpc Blogs(Empty) returns (BlogsResponse);
  rpc BlogCreate(BlogCreateRequest) returns (Empty);
}

message Blog { string title = 1; string content = 2; }

message Empty {}
message BlogsResponse { repeated Blog blogs = 1; }
message BlogCreateRequest { string title = 1; string content = 2; }

This code snippet shows you an example of a service. Your RPC methods use messages. Messages are mandatory for in- and output, so it's common to use an Empty message. The field number behind each property identifies fields within the binary format. You can read more about the proto3 language here.

Performance. In JSON you would send 7 bytes to transfer the key "title" over the wire while in Protocol buffers this is replaced by a single byte to transfer the field number - a modest reduction in network congestion.

Linting. You should lint your Proto3 files to avoid common mistakes. Use buf to lint with the command below. It first generates a configuration file in which you can customize lint rules.

brew tap bufbuild/buf
brew install buf

buf config init
buf build

Implement your service

Time to get hands-on by building a Twirp and gRPC server. Our compiler of choice is buf and we will avoid dynamic code generation. You can read more about this below or skip directly to the implementation.

Protocol compilers. There are plenty of tools available to parse the Protobuf grammar and generate code. The most relevant ones for TypeScript are protoc, buf and protobuf.js. Protoc is the original Protocol Compiler developed by Google and it's implemented in C++. Buf reimplements the Protoc compiler in Go to enhance it with linting - it is fully compatible with all protoc plugins. Finally, protobuf.js is a pure JavaScript implementation with its own solution to parse Protobuf and generate TypeScript.

Avoid dynamic code generation. JavaScript being a dynamically typed language in combination with Protobuf.js being implemented in JavaScript opens the door to remove the need for a separate code generation step. You can boot Node.Js which dynamically reads a .proto file and generates the code just-in-time before your server is started. This technique becomes difficult once you add TypeScript into the mix, instead I recommend avoiding this and adding an explicit build step.

Implement a Twirp server

Let's explore code generation by generating a Twirp server in TypeScript.

You start by installing the dependencies:

yarn add twirp-ts @protobuf-ts/plugin

Afterwards configure buf with the desired code generation plugins:

# buf.gen.yaml
version: v1
plugins:
  - name: ts
    path: ./node_modules/.bin/protoc-gen-ts
    out: src/twirp/__generated__
    opt:
      - client_none
      - generate_dependencies
  - name: twirp
    path: ./node_modules/.bin/protoc-gen-twirp_ts
    out: src/twirp/__generated__
    opt:
      - index_file

All that is left is to generate the code with buf generate and to implement the server as follows:

import * as http from "http";
import { Blog } from "./__generated__/proto/blog";
import { createBlogServiceServer } from "./__generated__/proto/blog.twirp";

let blogs: Blog[] = [{ title: "hello", content: "world" }];

const server = createBlogServiceServer({
  Blogs: async () => {
    return { blogs };
  },
  BlogCreate: async (_ctx, request) => {
    blogs.push({ ...request });
    return {};
  },
});

http.createServer(server.httpHandler()).listen(8080);

Finally, you can start your program with ts-node ./src/main.ts and try it with an HTTP library like curl or HTTPie. Twirp also supports JSON - see what happens when you change the content type to application/json! This is very useful for debugging and exploring APIs.

http POST http://localhost:8080/twirp/zagrit.v1.BlogService/Blogs Content-Type:application/protobuf

Alternatives. Currently twirp-ts is the only viable implementation in TypeScript. It can be used with either ts-proto or @protobuf-ts - which simply changes the look and feel of generated message interfaces. TwirpScript is an alternative which recently appeared but it is still rather experimental. All plugin combinations summarized:

Implement a gRPC server

Let's also take a look at a gRPC server to show you the strength of the Interface Description Language. We'll use the default JavaScript plugin to explore its previously mentioned deficits.

Within a new workspace, you once again start by installing the dependencies. Note that grpc-tools is currently not available on Darwin arm64 and you will receive a node-pre-gyp error (see GitHub issue 1405).

yarn add @grpc/grpc-js grpc-tools grpc_tools_node_protoc_ts mali

Similarly configure buf with the desired code generation plugins. See how we use both the default JavaScript plugin augmented with types through the protoc-gen-ts plugin.

version: v1
plugins:
  - name: js
    out: src/__generated__
    opt:
      - import_style=commonjs
      - binary
  - name: ts
    path: ./node_modules/.bin/protoc-gen-ts
    out: src/__generated__
    opt:
      - grpc_js
  - name: grpc
    path: ./node_modules/.bin/grpc_tools_node_protoc_plugin
    out: src/__generated__
    opt: grpc_js

Once again you can generate the code with buf generate. Below serves as an example of what the implementation could look like. Observe the mandatory mapping between your types and Protobuf classes.

import { createGrpcServer } from "./createGrpcServer";
import { BlogServiceService } from "./__generated__/proto/blog_grpc_pb";
import * as pb from "./__generated__/proto/blog_pb";

type Blog = { title: string; content: string };
let blogs: Blog[] = [{ title: "hello", content: "world" }];

const server = createGrpcServer({
  service: BlogServiceService,
  name: "BlogService",
  methods: {
    blogs: async (ctx) => {
      const blogsList = blogs.map((blog) =>
        new pb.Blog().setTitle(blog.title).setContent(blog.content)
      );

      ctx.res = new pb.BlogsResponse().setBlogsList(blogsList);
    },
    blogCreate: async (ctx) => {
      const newBlog = {
        title: ctx.req.getTitle(),
        content: ctx.req.getContent(),
      };

      blogs.push(newBlog);

      ctx.res = new pb.Empty();
    },
  },
});

server.start("0.0.0.0:8080");

Finally, you can also start this program with ts-node ./src/main.ts. It's a bit more tedious to validate gRPC servers through curl, instead I recommend downloading BloomRPC:

Closing thoughts

Throughout this guide, you learned what Protocol Buffers are and how to use its Interface Description Language to define both messages and a service definition. Afterwards you saw how you can use buf to lint and generate code. Finally, we also explored hands-on how to implement a gRPC and Twirp server and how to validate it respectively with BloomRPC and HTTPie.

I hope that the guide highlights that Protocol Buffers and server technology are closely entangled because of the IDL - yet completely decoupled and different topics. Once you understand that, it becomes easier to use an array of plugins to generate a solution that works for you.

You can try out the gRPC and Twirp servers by checking out this repository.

TypeScript meets Protobuf, gRPC and Twirp

Define your service

Implement your service

Implement a Twirp server

Implement a gRPC server

Closing thoughts