Keeping Build Secrets out of your Golang Images

I recently started a new position and one of the fun things about changing companies, is responding to new and interesting challenges. I started working at a predominantly Golang shop that leverages many private repositories for their go modules. One of my first tasks was to secure the build process for our Golang applications. The docker build process at the time was leveraging a single stage build process that required arguments from the build command. This build command exposed our service accounts username and password to the final Docker image. I can already feel the security minded fokes cringing at the thought of this. Thankfully our image registry was private and only accessible from within our VPC, but this was still a major security concern. I set out to find a better way to secure our build process and keep our service account credentials out of our final images. Below is an idea of how the Dockerfile looked before we started working on it.

  

  FROM golang:1.19

  ARG SERVICE_ACCOUNT_USERNAME
  ARG SERVICE_ACCOUNT_PASSWORD

  COPY . /app

  RUN git config --global \
    url."https://${SERVICE_ACCOUNT_USERNAME}:${SERVICE_ACCOUNT_PASSWORD}@privategitlab.com".insteadOf \
    "https://privategitlab.com"
  RUN go mod download

  RUN cd /app/cmd && go build -tags musl -a -race -o /app/dkr-test

  CMD ["/app/dkr-test"]

  

Incase you were not aware, Docker layers used to build the final image save a copy of the command and secrets used to in the build process. While Docker does have a way to pass secrets to the build process without saving them to a layers, I wanted to find a solution that would also help us speed up our build times. After a bit of research, I began to look deeper into vendoring our go modules and teaming them with a multi-stage build process.

Just to give you an idea of why build args arent the best way to pass secrets to your build process, lets take a look this simple example below. First we define a Dockerfile that taks a SUPER_SECRET build arg and uses it in a trite example.

  

  FROM golang:1.19

  ARG SUPER_SECRET

  RUN echo "SUPER_SECRET=${SUPER_SECRET}"

  CMD ["/bin/bash]

  

  

  $ podman build -t unsecure --build-arg SUPER_SECRET=P@$$W0Rdls .

  

  

  podman history ff24f69e0463 -H
  ID            CREATED         CREATED BY                                     SIZE                     COMMENT
  64c5ff67bd96  57 seconds ago  /bin/sh -c #(nop) CMD ["/bin/bash]             0B                       FROM 64c5ff67bd96
       58 seconds ago  |1 SUPER_SECRET=P@16878W0Rdls /bin/sh -c e...  4.61kB                   FROM 0b74fff196dd
  80b76a6c918c  58 seconds ago  /bin/sh -c #(nop) ARG SUPER_SECRET             0B                       FROM docker.io/library/golang:1.19
       5 months ago    /bin/sh -c #(nop) WORKDIR /go                  0B

  

From the command above you can see that we send the SUPER_SECRET build arg to the build process. Once the build has finished and our image is pushed to our registry we can use the history of the container to reveal our mistake. Podman/Docker history along with the container image id will show us an output of the commands used to build the image. On the second line of the output, we can see clear as day, the SUPER_SECRET build arg we used to build the image. This is a major security concern and should be avoided at all costs. Private registries offer some protection, but it is not a good idea to rely on them to keep your secrets safe.

By vendoring our go modules, we were able to solve a few problems in one go. First, we were able to cut down on the time the build process took. We no longer had to download our go modules every time we built our images. With the modules saved in our github repositories, we were now able to cache the modules and only download them when they changed. This cut our build times significantly. Second, we were able to remove our dependency on a Service Account to access our private go modules. By having developers vendor the modules directly, the service account with ever creeping permissions was no longer needed, resulting in a security win for us.

  

  $ go mod vendor

  

Running the command above in the root of our repository will create a vendor directory that contains all of the go modules we used to create the final binary. Once these files are commited to our repository we can make some major tweaks to our Dockerfile. We can remove a good amount of the build process and only copy the final binary into a smaller base image. Our final Dockerfile will look something like the following. Note that we need to update our go build command to use the newly added vendor directory.

  

  FROM golang:1.19 AS builder

  COPY . /app
  RUN cd /app/cmd && go build -mod vendor -tags musl -a -race -o /app/dkr-test

  # Final image stage
  FROM golang:1.19
  COPY --from=builder /app/dkr-test /app/
  CMD ["/app/dkr-test"]

  

With the above changes in place, we moved on to the multi-stage build process. Now that builds were faster and more secure, we could focus on improving pull times by only copying the final binary into a final smaller base image. From here we can experiment with even smaller base images on the final stage of the build process.

There are many ways to secure your build process, but I found that vendoring go modules and using a multi-stage build process worked wonders for us. As always the approach and best path forward will depend on your specific use case, but I hope this article has given you some ideas on how to secure your Golang build process.