Back

Build and Maintain a Custom Protoc Docker Image

This post explains the process of building a custom Protoc image to generate files for Golang. This image is comprised of three parts

Motivation

Maintaining a custom protoc docker image is really useful as

  • The image contains only the necessary plugins and only for the required language.
  • It allows the user to specify a particular version of the compiler and the plugins. This means that the image is unaffected by bugs introduced in newer versions.
  • Multiple images, with different combinations of plugins and different languages can be created.
  • Nothing (apart from docker) is installed in the local machine, thereby decluttering the local environment.
  • Sharing a customized generator for the project with other collaborators is extremely easy.

Pre-requisites

This post assumes that the following are present/setup/available.

  • Docker
  • Basic Knowledge of Dockerfile syntax.
  • A repository to push your images (Docker Hub/Github Container Registry/Gitlab Registry/ECR etc).
  • An IDE/Text editor of choice.

Prebuilt image

If you want to skip the (slow) build process and directly use my image, it’s available on Github Container Registry.

$ docker pull ghcr.io/krishnaiyer/protoc-go:latest
$ docker run --rm -v $(pwd)/proto:/proto -v $GOPATH/src:/go-out  krishnaiyer/protoc-go <file>.proto

Build

In order to easily explain the process, it’s decomposed into multiple steps in this section. However, in reality, the image is built using a single docker file in a single (user visible) step.

Advanced users may skip this explanation and check the full Dockerfile directly.

Step 1: Preparation

ARG ALPINE_VERSION=3.12
FROM alpine:${ALPINE_VERSION}

RUN apk --update --no-cache add ca-certificates go build-base curl automake autoconf git libtool zlib-dev

RUN addgroup -g 1000 protoc && adduser -u 1000 -S -G protoc protoc

RUN mkdir -p /tmp/protobuf

WORKDIR /tmp

Here, the Alpine base image is chosen and some necessary packages and folders are setup. For better security, a non-root user is created which will be used when the image is run.

Step 2: Compiling Protoc

ARG PROTOBUF_VERSION=3.14.0
RUN curl -L https://github.com/google/protobuf/archive/v${PROTOBUF_VERSION}.tar.gz | tar xvz --strip-components=1 -C /tmp/protobuf

RUN cd protobuf && autoreconf -f -i -Wall,no-obsolete && \
    ./configure --prefix=/usr --enable-static=no && \
    make -j2 && make install

RUN rm -rf /tmp/protobuf

Here, the Protoc compiler is fetched and complied from source. This step takes a while, so I recommend taking a coffee break at this point.

Step 3: Compiling protoc-gen-go

ENV GOPATH=/go

RUN mkdir -p ${GOPATH} ${GOPATH}/src/github.com/golang/protobuf

ARG PROTOC_GEN_GO_VERSION=1.4.3
RUN curl -sSL https://api.github.com/repos/golang/protobuf/tarball/v${PROTOC_GEN_GO_VERSION} | tar xz --strip 1 -C ${GOPATH}/src/github.com/golang/protobuf
   
WORKDIR  ${GOPATH}/src/github.com/golang/protobuf

RUN go build -ldflags '-w -s' -o /golang-protobuf-out/protoc-gen-go ./protoc-gen-go

RUN install -Ds /golang-protobuf-out/protoc-gen-go /usr/bin/protoc-gen-go

Here, the protoc-gen-go plugin is fetched, compiled and installed.

Step 4: Additional Plugins

Any optional additional plugins can be installed at this stage. The example Dockerfile omits this step for simplicity.

Step 5: Setting the Entrypoint

RUN chmod a+x /usr/bin/protoc

RUN mkdir -p /proto /go-out

RUN apk del ca-certificates go curl automake autoconf git libtool zlib-dev

ENTRYPOINT [ "/usr/bin/protoc", "-I=/proto", "--go_out=/go-out"]

USER protoc:protoc

The entrypoint to this image is /usr/bin/protoc. Additionally, the image sets default bindings for the input proto folder and the generate output folder. Also, unnecessary packages are removed and the default user is set.

Additional Plugins

The Dockerfile in this post only uses the base protoc-gen-go plugin. In reality, however, the generate protobuf files use other plugins. These can be included as part of the image in Step 4. If the plugin is written in golang, the same commands to build and install protoc-gen-go will usually work;

  • Either use a tarball (for a specific version) or use go get for the latest master.
  • Build
$ go build -ldflags '-w -s' ...
  • Install
$ install -Ds <build-folder> /usr/bin/<name>

Here’s a (non-exhaustive) list of note-worthly plugins

Usage

This image is run using docker. The input proto files and the location to generate the go output files (*.pb.go) are mounted as volumes. The docker entrypoint is preconfigured to make this easier.

  • Mount the local source folder ( with the proto files) to the /proto folder on the image.
  • Mount the local target folder to the /go-out folder on the image.

It’s recommended to use the go_package declaration in the proto files. This not only defines the go package name for the generated files, but also provides the relative path for protoc to generate the output files.

Consider the following definition.

option go_package = "github.com/user/project/gen-folder";

By setting $GOPATH/src as the target, the generated files are perfectly generated in the right folder inside a go project.

The generated folder can also be manually specified by using the volume mount to /go-out.

So, this is effectively

$ docker run --rm -v $(pwd)/proto:/proto -v $GOPATH/src:/go-out  <username>/protoc-go  <file>.proto

The output files will be generated at the location $GOPATH/src/github.com/user/project/gen-folder.

Source

The snippets used in this post are available in the custom-protoc folder of my go-snippets Github Repository.

References

  1. Official Golang Protobuf Documentation