This post explains the process of building a custom Protoc
image to generate files for Golang
. This image is comprised of three parts
- The Protoc Compiler
- A plugin to generate golang files;
protoc-gen-go
- (Optional) Other Plugins
Motivation
Maintaining a custom protoc
docker image is really useful as
- The image contains only the necessary plugins and only for the required language.
- It allows the user to specify a particular version of the compiler and the plugins. This means that the image is unaffected by bugs introduced in newer versions.
- Multiple images, with different combinations of plugins and different languages can be created.
- Nothing (apart from docker) is installed in the local machine, thereby decluttering the local environment.
- Sharing a customized generator for the project with other collaborators is extremely easy.
Pre-requisites
This post assumes that the following are present/setup/available.
- Docker
- Basic Knowledge of
Dockerfile
syntax. - A repository to push your images (Docker Hub/Github Container Registry/Gitlab Registry/ECR etc).
- An IDE/Text editor of choice.
Prebuilt image
If you want to skip the (slow) build process and directly use my image, it’s available on Github Container Registry.
$ docker pull ghcr.io/krishnaiyer/protoc-go:latest
$ docker run --rm -v $(pwd)/proto:/proto -v $GOPATH/src:/go-out krishnaiyer/protoc-go <file>.proto
Build
In order to easily explain the process, it’s decomposed into multiple steps in this section. However, in reality, the image is built using a single docker file in a single (user visible) step.
Advanced users may skip this explanation and check the full Dockerfile directly.
Step 1: Preparation
ARG ALPINE_VERSION=3.12
FROM alpine:${ALPINE_VERSION}
RUN apk --update --no-cache add ca-certificates go build-base curl automake autoconf git libtool zlib-dev
RUN addgroup -g 1000 protoc && adduser -u 1000 -S -G protoc protoc
RUN mkdir -p /tmp/protobuf
WORKDIR /tmp
Here, the Alpine base image is chosen and some necessary packages and folders are setup. For better security, a non-root user is created which will be used when the image is run.
Step 2: Compiling Protoc
ARG PROTOBUF_VERSION=3.14.0
RUN curl -L https://github.com/google/protobuf/archive/v${PROTOBUF_VERSION}.tar.gz | tar xvz --strip-components=1 -C /tmp/protobuf
RUN cd protobuf && autoreconf -f -i -Wall,no-obsolete && \
./configure --prefix=/usr --enable-static=no && \
make -j2 && make install
RUN rm -rf /tmp/protobuf
Here, the Protoc compiler is fetched and complied from source. This step takes a while, so I recommend taking a coffee break at this point.
Step 3: Compiling protoc-gen-go
ENV GOPATH=/go
RUN mkdir -p ${GOPATH} ${GOPATH}/src/github.com/golang/protobuf
ARG PROTOC_GEN_GO_VERSION=1.4.3
RUN curl -sSL https://api.github.com/repos/golang/protobuf/tarball/v${PROTOC_GEN_GO_VERSION} | tar xz --strip 1 -C ${GOPATH}/src/github.com/golang/protobuf
WORKDIR ${GOPATH}/src/github.com/golang/protobuf
RUN go build -ldflags '-w -s' -o /golang-protobuf-out/protoc-gen-go ./protoc-gen-go
RUN install -Ds /golang-protobuf-out/protoc-gen-go /usr/bin/protoc-gen-go
Here, the protoc-gen-go
plugin is fetched, compiled and installed.
Step 4: Additional Plugins
Any optional additional plugins can be installed at this stage. The example Dockerfile
omits this step for simplicity.
Step 5: Setting the Entrypoint
RUN chmod a+x /usr/bin/protoc
RUN mkdir -p /proto /go-out
RUN apk del ca-certificates go curl automake autoconf git libtool zlib-dev
ENTRYPOINT [ "/usr/bin/protoc", "-I=/proto", "--go_out=/go-out"]
USER protoc:protoc
The entrypoint to this image is /usr/bin/protoc
. Additionally, the image sets default bindings for the input proto folder and the generate output folder. Also, unnecessary packages are removed and the default user is set.
Additional Plugins
The Dockerfile
in this post only uses the base protoc-gen-go
plugin. In reality, however, the generate protobuf files use other plugins. These can be included as part of the image in Step 4
. If the plugin is written in golang, the same commands to build and install protoc-gen-go
will usually work;
- Either use a tarball (for a specific version) or use
go get
for the latest master. - Build
$ go build -ldflags '-w -s' ...
- Install
$ install -Ds <build-folder> /usr/bin/<name>
Here’s a (non-exhaustive) list of note-worthly plugins
Usage
This image is run using docker. The input proto
files and the location to generate the go output files (*.pb.go
) are mounted as volumes. The docker entrypoint is preconfigured to make this easier.
- Mount the local source folder ( with the proto files) to the
/proto
folder on the image. - Mount the local target folder to the
/go-out
folder on the image.
It’s recommended to use the go_package
declaration in the proto files. This not only defines the go package name for the generated files, but also provides the relative path for protoc to generate the output files.
Consider the following definition.
option go_package = "github.com/user/project/gen-folder";
By setting $GOPATH/src
as the target, the generated files are perfectly generated in the right folder inside a go project.
The generated folder can also be manually specified by using the volume mount to
/go-out
.
So, this is effectively
$ docker run --rm -v $(pwd)/proto:/proto -v $GOPATH/src:/go-out <username>/protoc-go <file>.proto
The output files will be generated at the location $GOPATH/src/github.com/user/project/gen-folder
.
Source
The snippets used in this post are available in the custom-protoc folder of my go-snippets Github Repository.