Intro

Dependency management in Golang got a lot better since go1.5, however, we still need tools to manage it.

For a long time, I used Godep which was working great, but handles everything based on your local GOPATH which result in massive change sets in git each time a different team member updates them, which makes the code review difficult.

Here comes Glide . A “newcomer” which uses a yaml config in order to explicitly set the dependency version needed. It is based on semver and allow for automatic update of path/minor version, in a similar way as npm.

Once the initial config is set (and glide allows to automatically generate it), glide up will generate a .lock file with the expected commit, based on the remote version specified in the yaml config.

Docker

For many reasons, Docker is a great tool and is a time saver. However, when it comes to develop in Go in a Docker environment, things quickly become slow, especially when using a lot of dependencies.

Let's take a naive Dockerfile:

1
2
3
4
5
6

FROM       golang:1.7
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
ADD        . $APP_DIR
RUN        go install

Each time something changes in the local directory, the ADD instruction will have its cache invalidated, resulting in the following go install to recompile the whole code, including all dependencies.

This is a major inconvenience when actively developing when we need to often recompile and/or run the tests, especially when dealing with statically linked, CGO disabled program.

Godep

With Godep, in go1.4, a simple solution is to add the Godeps directory first, compile it and then add the rest of the app. In order to do that, we iterate over the dependency list and install them. As Godep uses json, we'll need jq , an awesome tool in order to play with json in the shell.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

FROM       golang:1.4
# Install jq and Godep.
RUN        apt-get update && apt-get install -y jq && go get github.com/tools/godep
ENV        APP_DIR $GOPATH/src/github.com/org/myapp
WORKDIR    $APP_DIR
ENTRYPOINT ["myapp"]
# Add Godeps and precompile.
ADD        Godeps/ $APP_DIR/Godeps
RUN        for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do \
             godep go install $pkg; \
           done
# Add App and install.
ADD        . $APP_DIR
RUN        godep go install

This is nice and saves up quite a lot of time, however, since go1.5, the vendor model changed and the imported packages are now scoped within the package itself instead of using the GOPATH one, which make this method obsolete.

If you are curious about the magic line for pkg in $(cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath'); do godep go install -ldflags -d $pkg; done, here is what it does: Godep stores the known dependencies in the json file Godeps/Godeps.json which contains a json object with a Deps key which contains an array of dependencies. Each of which are a json object with the key ImportPath which is the value that interest us. cat Godeps/Godeps.json | jq -r '.Deps[].ImportPath' returns a list of values from the json file, which we iterate on via the for loop and then install the dependency.

Glide

With glide, in go1.5 and up, we need to rethink a bit the process. It will be similar, however, the first issue is that glide uses a yaml config. How to extract the values from a shell command?

yaml2json

I looked for tools similar to jq for yaml but didn't find much so I built yaml2json which is a small go util which simply translate yaml to json using github.com/ghodss/yaml.

It can be installed via the go toolchain:

1
2

go get github.com/creack/yaml2json
yaml2json < glide.yaml > glide.json

or via Docker:

1
2

alias yaml2json='docker run -i --rm creack/yaml2json'
yaml2json < glide.yaml > glide.json

FYI, this Docker image contains only the statically linked, stripped down binary and weight only 3Mb!

1
2
3

$> docker images creack/yaml2json
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
creack/yaml2json    latest              a8e3e1fff7bb        3 weeks ago         3.007 MB

Caching

Now that we can have a json version of the yaml config, we can simply use jq in order to play with it.

With the new vendor model, the dependencies are now install as: $APP_DIR/vendor/$DEP_PATH rather than in the GOPATH directly.

Example: yaml2json is in github.com/creack/yaml2json and depends on github.com/ghodss/yaml so it will be installed as github.com/creack/yaml2json/vendor/github.com/ghodss/yaml

Another difficulty resides with the sub-packages, they are glide lists them as directory names under the parent's imports section. We need to use a bit more advanced jq query to construct the full list to be installed.

Let's see:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

FROM       golang:1.7
# Install yaml2json and jq.
RUN        apt-get update && apt-get install -y jq && go get github.com/creack/yaml2json
ENV        APP_DIR  github.com/org/myapp
ENV        APP_PATH $GOPATH/src/$APP_DIR
WORKDIR    $APP_PATH
ENTRYPOINT ["myapp"]
# Add glide lock file and precompile.
ADD        glide.lock $APP_PATH/glide.lock
ADD        vendor     $APP_PATH/vendor
RUN        yaml2json < glide.lock | \
           jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
           jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
           while read pkg; do \
             echo "$pkg...";  \
             go install $APP_DIR/vendor/$pkg 2> /dev/null; \
           done

# Add App and install.
ADD        . $APP_PATH
RUN        go install

First, we convert the lock file to json using yaml2json, then we extract the main import list as well as the test import list from which we need the name and the sub-packages if any. As some dependencies will not have sub-package, we manually add + [""] to facilitate the next step. Now that we have this list, we forge the full package names from the package list: we keep the “main” name and join it with the list of sub-packages (and "" for the main package itself). Finally, we trim down the trailing / if any and install each dependency.

Alternative

Alternatively, instead of trying to pre-compile the dependencies, one could use the mount-bind feature of Docker in order to mount the local directory in a long running container and run the build/test there, which would allow to have the native caching of the go toolchain, but looses the reproducibility warranty of Docker.

Conclusion

This method might not be the most “straight forward” one, but gives us the ability to quickly iterate over our code without worrying about the toolchain.

Bonus: the actual Dockerfile that I use at Agrarian Labs for all our micro services:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

FROM            golang:1.7
MAINTAINER      Guillaume J. Charmes <guillaume@leaf.ag>

# Install linters, coverage tools and test formatters.
RUN             go get github.com/alecthomas/gometalinter && gometalinter -i && \
                go get github.com/axw/gocov/... \
                       github.com/AlekSi/gocov-xml \
                       github.com/jstemmer/go-junit-report \
                       github.com/matm/gocov-html

# Disable CGO and recompile the stdlib.
ENV             CGO_ENABLED 0
RUN             go install -a -ldflags -d std

# Install jq and yaml2json for parsing glide.lock to precompile.
RUN             apt-get update && apt-get install -y jq
RUN             go get github.com/creack/yaml2json

ARG             APP_DIR

ENV             APP_PATH $GOPATH/src/$APP_DIR

WORKDIR         $APP_PATH

# Precompile deps.
ADD             glide.lock $APP_PATH/glide.lock
ADD             vendor     $APP_PATH/vendor
RUN             yaml2json < glide.lock | \
                jq -r -c '.imports[], .testImports[] | {name: .name, subpackages: (.subpackages + [""])}' | \
                jq -r -c '.name as $name | .subpackages[] | [$name, .] | join("/")' | sed 's|/$||' | \
                while read pkg; do \
                  echo "$pkg...";  \
                  go install -ldflags -d $APP_DIR/vendor/$pkg 2> /dev/null; \
                done

ADD             .          $APP_PATH

RUN             make install