Dependency management in Go has never been fun. When I was starting this post, I was making updates to the Terraform ACME provider, and in dependency hell with govendor due to this issue, which still has yet to be fixed, or so it would seem.

You would think that fixing this would be as easy as just ripping out the vendor directory and installing every dependency again, version-locking the things you are concerned about (Terraform core in this case), but nope. Ultimately, I think one of the non-TF dependencies overwrote some UUID generation stuff that we depend on in Terraform, or perhaps one of the TF providers that we use for testing overwrote go-uuid, but in any case, TF core is now blocking the build and I can’t proceed.

So, what’s the only answer – burn it all down of course 🔥🔥🔥🔥🔥🔥

Since this is my personal project, I can pretty much use whatever dependency management tool I want, so I’m trying (and documenting for your pleasure) a run of using the new dep tool that will ultimately become the one dependency management system to rule them all in Go, finally.

With that said, let’s give it a spin. Since I like to live life on the edge, I’m using the bleeding edge copy fetched by go get, so if things are a bit off from the last binary release (I don’t know if they are or not, by the way), then you might want to check out master. There hasn’t been a new release since October.

(The specific install step is go get -u github.com/golang/dep/cmd/dep).

Initializing

With all of my import paths in place and raring to go, I deleted my vendor directory. You don’t have to do this, by the way, dep will back up your old vendor directory and also import the dependencies based off your existing tool’s metadata file – you can see the list of supported tools here. I just didn’t really trust my vendor.json given the amount of problems I was having trying to get it to actually converge a correct dependency set.

Here is the output of dep init on the ACME provider:

Locking in master (06020f8) for transitive dep github.com/mitchellh/mapstructure
Using master as constraint for direct dep github.com/xenolf/lego
Locking in master (b929aa5) for direct dep github.com/xenolf/lego
Locking in master (23c074d) for transitive dep github.com/hashicorp/hcl
Locking in v1.32.0 (32e4c1e) for transitive dep gopkg.in/ini.v1
Locking in v1.1.0 (879c588) for transitive dep github.com/satori/go.uuid
Locking in v0.1.0 (4aabc24) for transitive dep github.com/bgentry/speakeasy
Locking in v1.32.0 (32e4c1e) for transitive dep github.com/go-ini/ini
Locking in v9.7.0 (8c58b47) for transitive dep github.com/Azure/go-autorest
Locking in master (7554cd9) for transitive dep github.com/hashicorp/errwrap
Locking in master (d23ffcb) for transitive dep github.com/mitchellh/copystructure
Locking in master (44bad6d) for transitive dep github.com/hashicorp/hcl2
Locking in v1.2.1 (5f10fee) for transitive dep github.com/agext/levenshtein
Locking in master (30785a2) for transitive dep golang.org/x/oauth2
Locking in v1.9.0 (f3955b8) for transitive dep google.golang.org/grpc
Locking in master (3d48364) for transitive dep github.com/jen20/awspolicyequivalence
Locking in master (709e403) for transitive dep github.com/zclconf/go-cty
Locking in master (42fe2e1) for transitive dep golang.org/x/net
Locking in master (dd9ec17) for transitive dep golang.org/x/sys
Locking in v1.1.0 (346938d) for transitive dep github.com/davecgh/go-spew
Locking in master (1e59b77) for transitive dep github.com/golang/protobuf
Locking in v0.9.1 (87b6919) for transitive dep github.com/hashicorp/vault
Locking in master (683f491) for transitive dep github.com/hashicorp/yamux
Locking in v12.1.1-beta (57da600) for transitive dep github.com/Azure/azure-sdk-for-go
Using ^1.6.0 as constraint for direct dep github.com/terraform-providers/terraform-provider-aws
Locking in v1.6.0 (ddd2205) for direct dep github.com/terraform-providers/terraform-provider-aws
Locking in master (64130c7) for transitive dep github.com/hashicorp/go-uuid
Locking in master (1fca145) for transitive dep github.com/armon/go-radix
Locking in v1.1 (dc2bc5a) for transitive dep github.com/posener/complete
Locking in master (fa9f258) for transitive dep github.com/hashicorp/hil
Locking in master (d5fe4b5) for transitive dep github.com/hashicorp/go-cleanhttp
Locking in master (f33a2c6) for transitive dep github.com/decker502/dnspod-go
Locking in v1.0.1 (787fb05) for transitive dep github.com/miekg/dns
Locking in master (59fac50) for transitive dep github.com/juju/ratelimit
Locking in v0.17.0 (050b16d) for transitive dep cloud.google.com/go
Locking in master (53e6ce1) for transitive dep github.com/google/go-querystring
Locking in v0.0.3 (0360b2a) for transitive dep github.com/mattn/go-isatty
Locking in master (2d22b6e) for transitive dep github.com/keybase/go-crypto
Locking in master (2bd8b58) for transitive dep github.com/apparentlymart/go-cidr
Locking in master (2bca23e) for transitive dep github.com/mitchellh/hashstructure
Locking in master (b1c0f9b) for transitive dep google.golang.org/api
Locking in master (37e8452) for transitive dep github.com/timewasted/linode
Locking in 1.15.0 (fa1c036) for transitive dep github.com/JamesClonk/vultr
Locking in (0b12d6b5) for transitive dep github.com/jmespath/go-jmespath
Locking in master (4fe82ae) for transitive dep github.com/hashicorp/go-version
Locking in v1.0.4 (d682213) for transitive dep github.com/sirupsen/logrus
Locking in master (994f50a) for transitive dep github.com/hashicorp/go-getter
Using ^0.11.1 as constraint for direct dep github.com/hashicorp/terraform
Locking in v0.11.1 (a42fdb0) for direct dep github.com/hashicorp/terraform
Locking in master (0dc08b1) for transitive dep github.com/hashicorp/logutils
Locking in master (02f7e94) for transitive dep github.com/ovh/go-ovh
Locking in v0.1.0 (5e23d5d) for transitive dep github.com/exoscale/egoscale
Locking in v0.15.0 (9641549) for transitive dep github.com/dnsimple/dnsimple-go
Locking in v3.1.0 (dbeaa93) for transitive dep github.com/dgrijalva/jwt-go
Locking in master (b836f5c) for transitive dep github.com/apparentlymart/go-textseg
Locking in master (ca137eb) for transitive dep github.com/hashicorp/go-hclog
Locking in master (e2fbc68) for transitive dep github.com/hashicorp/go-plugin
Locking in master (a61a995) for transitive dep github.com/mitchellh/go-testing-interface
Locking in v1.0.0 (150dc57) for transitive dep google.golang.org/appengine
Locking in master (63d60e9) for transitive dep github.com/mitchellh/reflectwalk
Locking in v3.5.1 (2ee8785) for transitive dep github.com/blang/semver
Locking in v2 (0e4404d) for transitive dep gopkg.in/yaml.v2
Locking in v1.1.0 (aa2e30f) for transitive dep gopkg.in/square/go-jose.v1
Locking in v1.0.3 (1563e62) for transitive dep github.com/edeckers/auroradnsclient
Locking in v1.2.0 (b91bfb9) for transitive dep github.com/stretchr/testify
Locking in master (a8101f2) for transitive dep google.golang.org/genproto
Using ^1.0.1 as constraint for direct dep github.com/terraform-providers/terraform-provider-tls
Locking in v1.0.1 (1be2f9e) for direct dep github.com/terraform-providers/terraform-provider-tls
Locking in master (b7773ae) for transitive dep github.com/hashicorp/go-multierror
Locking in v1.0.0 (15a30b4) for transitive dep github.com/beevik/etree
Locking in master (b8bc1bf) for transitive dep github.com/mitchellh/go-homedir
Locking in v1.12.56 (ce8d7a1) for transitive dep github.com/aws/aws-sdk-go
Locking in master (33edc47) for transitive dep github.com/mitchellh/cli
Locking in v2 (3c1c487) for transitive dep gopkg.in/ns1/ns1-go.v2
Locking in v0.5.4 (0c6b41e) for transitive dep github.com/ulikunitz/xz
Locking in master (ad45545) for transitive dep github.com/mitchellh/go-wordwrap
Locking in master (e19ae14) for transitive dep golang.org/x/text
Locking in master (553a641) for transitive dep github.com/golang/snappy
Locking in master (9fd32a8) for transitive dep github.com/bgentry/go-netrc
Using master as constraint for direct dep golang.org/x/crypto
Locking in master (0fcca48) for direct dep golang.org/x/crypto
Locking in v1.0.0 (792786c) for transitive dep github.com/pmezard/go-difflib

My initial impression here is the UX is much better than govendor. It actually tells you what versions it’s pulling in? What sorcery is this? 😂

Looking at the output, it seems to have done a pretty good (initial) job at giving me what I want. The most important thing is that it’s vendoring lego at master. The tool hasn’t had a release for a while, and I need some post-release fixes I specifically need.

Going even farther, it vendored Terraform at it’s most recent release, 0.11.1. This is amazing, because it’s really the only dependency that I was concerned with vendoring at a specific version lock!

Now, editing one of the files within the provider does not give me the UUID-related build issues I was having, or any of the other missing or outdated dependency issues, for that matter. Pack it up, we’re done here.

But wait, are we done? Let’s check out a few more things.

File Count

dep will – currently anyway – bloat your vendor directory pretty hard. The ACME project’s directory probably inflated about 5 times (from about 2800 files to over 16000).

This is due to some currently ongoing work in dep to ensure that the steps undertaken by dep prune are done by the rest of the process automatically. This removes unused packages and running it got the file count down to a much more manageable ~4700 files, which, I would imagine, is much easier to get by someone during code review (thankfully I don’t have to do that here 😉).

This still did not remove everything though. It appears the team is still working on making this work on a scale that will ultimately ensure that only what is truly needed remains – this includes removing _test.go files in necessary packages. This is being tracked in #944, it appears, or at the very least questions about the pruning functionality are being answered there. Note that as mentioned, and as the subject of the issue mentions, prune will soon be consolidated into ensure, with that command assuming prune’s functionality along with its own.

Metadata and Configuration Files

Two files control dep’s behavior.

The first one, Gopkg.toml, is a configuration file that only displays your direct dependencies – the packages your project directly references. This is where you will want to make changes if you need to increment a version of a particular dependency before running dep ensure. Personally, I like this approach over something like govendor where you need to manage this on the command line.

The documentation seems to indicate that you can make changes here, run dep ensure, and dep will automatically transition the entire vendor tree to properly satisfy the new dependency chain, if that’s possible.

The other file is Gopkg.lock. This is similar to any other lock file that you would see in a dependency management system (such as Bundler), and replaces other files like vendor.json (if you were using govendor, for example). This file you would normally not touch, and serves to catalog all of the dependencies the tool needs, including your transitive dependencies – the packages your direct dependencies require.

Viewing Dependency Status

dep status gives you information about the vendored dependencies.

Running it with no options gives you stats on every project that has been vendored, including versions and used package count within each project.

Also, running dep status -dot allows you to output a GraphViz diagram, much like terraform graph does, if you are familiar with that. If you are familiar with terraform graph though, you might know how ridiculous the graphs get at scale, and obviously vendored dependencies are going to be worse, so don’t expect the graphs to be entirely simple.

Local Work

Aniother thing to note is that dep does not work out of local paths at all currently, it seems. This means that any local work you do will be superseded when you run dep ensure, which I personally think is a great addition, because it removes the ambiguity that other tools currently have surrounding this behavior.

My hope is they they keep this default behavior if/when they do actually introduce support for working with local packages.

Version Constraints for Transitive Dependencies

The last thing I want to note is how to deal with transitive dependencies. These are dependencies that your direct dependencies (the packages your project directly references) depend on.

This probably cost me about half a day of debugging and troubleshooting, but one thing that did happen was the correctly vendored dependencies for copystructure and reflectwalk were not imported, and there are currently some “correct” behaviors in Terraform that depend on bugs that were present in these two packages. This was fixed by adding some extra override blocks in Gopkg.toml that locked these two packages at the correct commits. After adding these in, it was as easy as running dep ensure to get everything corrected.

Conclusion

dep is a great tool. It’s got a bit more work to go, but the UX is very pleasant, and everyone involved should give themselves a pat on the back for finally getting the Go community to a place where we might be able to put dependency management behind us. It’s always going to be somewhat of a struggle, but at the very least we can all struggle under one banner versus the dozen or so different ones that currently comprise the vendor tool ecosystem.

If you are curious as to how it looks in action, you can check out the code for the Terraform ACME provider, which is now being managed with dep!

(If you read all the way through this and notice that the TF being used is now constrained to master, even though I mentioned that I wanted it locked at 0.11.1, it’s because not only did I do that during my troubleshooting process, but there has also been some changes to the “provider SDK” in helper/schema that I figured would be good to have on hand.)