Dependency management in go has never been fun. When I was starting this post, I was making updates to the Terraform ACME provider, and in dependency hell with govendor due to this issue, which still has yet to be fixed, or so it would seem.
You would think that fixing this would be as easy as just ripping out the
vendor directory and installing every dependency again, version-locking the things you are concerned about (Terraform core in this case), but nope. Ultimately, I think one of the non-TF dependencies overwrote some UUID generation stuff that we depend on in Terraform, or perhaps one of the TF providers that we use for testing overwrote
go-uuid, but in any case, TF core is now blocking the build and I can’t proceed.
So, what’s the only answer – burn it all down of course 🔥🔥🔥🔥🔥🔥
Since this is my personal project, I can pretty much use whatever dependency management tool I want, so I’m trying (and documenting for your pleasure) a run of using the new dep tool that will ultimately become the one dependency management system to rule them all in Go, finally.
With that said, let’s give it a spin. Since I like to live life on the edge, I’m using the bleeding edge copy fetched by
go get, so if things are a bit off from the last binary release (I don’t know if they are or not, by the way), then you might want to check out master. There hasn’t been a new release since October.
(The specific install step is
go get -u github.com/golang/dep/cmd/dep).
With all of my import paths in place and raring to go, I deleted my
vendor directory. You don’t have to do this, by the way,
dep will back up your old
vendor directory and also import the dependencies based off your existing tool’s metadata file – you can see the list of supported tools here. I just didn’t really trust my
vendor.json given the amount of problems I was having trying to get it to actually converge a correct dependency set.
Here is the output of
dep init on the ACME provider:
Locking in master (06020f8) for transitive dep github.com/mitchellh/mapstructure Using master as constraint for direct dep github.com/xenolf/lego Locking in master (b929aa5) for direct dep github.com/xenolf/lego Locking in master (23c074d) for transitive dep github.com/hashicorp/hcl Locking in v1.32.0 (32e4c1e) for transitive dep gopkg.in/ini.v1 Locking in v1.1.0 (879c588) for transitive dep github.com/satori/go.uuid Locking in v0.1.0 (4aabc24) for transitive dep github.com/bgentry/speakeasy Locking in v1.32.0 (32e4c1e) for transitive dep github.com/go-ini/ini Locking in v9.7.0 (8c58b47) for transitive dep github.com/Azure/go-autorest Locking in master (7554cd9) for transitive dep github.com/hashicorp/errwrap Locking in master (d23ffcb) for transitive dep github.com/mitchellh/copystructure Locking in master (44bad6d) for transitive dep github.com/hashicorp/hcl2 Locking in v1.2.1 (5f10fee) for transitive dep github.com/agext/levenshtein Locking in master (30785a2) for transitive dep golang.org/x/oauth2 Locking in v1.9.0 (f3955b8) for transitive dep google.golang.org/grpc Locking in master (3d48364) for transitive dep github.com/jen20/awspolicyequivalence Locking in master (709e403) for transitive dep github.com/zclconf/go-cty Locking in master (42fe2e1) for transitive dep golang.org/x/net Locking in master (dd9ec17) for transitive dep golang.org/x/sys Locking in v1.1.0 (346938d) for transitive dep github.com/davecgh/go-spew Locking in master (1e59b77) for transitive dep github.com/golang/protobuf Locking in v0.9.1 (87b6919) for transitive dep github.com/hashicorp/vault Locking in master (683f491) for transitive dep github.com/hashicorp/yamux Locking in v12.1.1-beta (57da600) for transitive dep github.com/Azure/azure-sdk-for-go Using ^1.6.0 as constraint for direct dep github.com/terraform-providers/terraform-provider-aws Locking in v1.6.0 (ddd2205) for direct dep github.com/terraform-providers/terraform-provider-aws Locking in master (64130c7) for transitive dep github.com/hashicorp/go-uuid Locking in master (1fca145) for transitive dep github.com/armon/go-radix Locking in v1.1 (dc2bc5a) for transitive dep github.com/posener/complete Locking in master (fa9f258) for transitive dep github.com/hashicorp/hil Locking in master (d5fe4b5) for transitive dep github.com/hashicorp/go-cleanhttp Locking in master (f33a2c6) for transitive dep github.com/decker502/dnspod-go Locking in v1.0.1 (787fb05) for transitive dep github.com/miekg/dns Locking in master (59fac50) for transitive dep github.com/juju/ratelimit Locking in v0.17.0 (050b16d) for transitive dep cloud.google.com/go Locking in master (53e6ce1) for transitive dep github.com/google/go-querystring Locking in v0.0.3 (0360b2a) for transitive dep github.com/mattn/go-isatty Locking in master (2d22b6e) for transitive dep github.com/keybase/go-crypto Locking in master (2bd8b58) for transitive dep github.com/apparentlymart/go-cidr Locking in master (2bca23e) for transitive dep github.com/mitchellh/hashstructure Locking in master (b1c0f9b) for transitive dep google.golang.org/api Locking in master (37e8452) for transitive dep github.com/timewasted/linode Locking in 1.15.0 (fa1c036) for transitive dep github.com/JamesClonk/vultr Locking in (0b12d6b5) for transitive dep github.com/jmespath/go-jmespath Locking in master (4fe82ae) for transitive dep github.com/hashicorp/go-version Locking in v1.0.4 (d682213) for transitive dep github.com/sirupsen/logrus Locking in master (994f50a) for transitive dep github.com/hashicorp/go-getter Using ^0.11.1 as constraint for direct dep github.com/hashicorp/terraform Locking in v0.11.1 (a42fdb0) for direct dep github.com/hashicorp/terraform Locking in master (0dc08b1) for transitive dep github.com/hashicorp/logutils Locking in master (02f7e94) for transitive dep github.com/ovh/go-ovh Locking in v0.1.0 (5e23d5d) for transitive dep github.com/exoscale/egoscale Locking in v0.15.0 (9641549) for transitive dep github.com/dnsimple/dnsimple-go Locking in v3.1.0 (dbeaa93) for transitive dep github.com/dgrijalva/jwt-go Locking in master (b836f5c) for transitive dep github.com/apparentlymart/go-textseg Locking in master (ca137eb) for transitive dep github.com/hashicorp/go-hclog Locking in master (e2fbc68) for transitive dep github.com/hashicorp/go-plugin Locking in master (a61a995) for transitive dep github.com/mitchellh/go-testing-interface Locking in v1.0.0 (150dc57) for transitive dep google.golang.org/appengine Locking in master (63d60e9) for transitive dep github.com/mitchellh/reflectwalk Locking in v3.5.1 (2ee8785) for transitive dep github.com/blang/semver Locking in v2 (0e4404d) for transitive dep gopkg.in/yaml.v2 Locking in v1.1.0 (aa2e30f) for transitive dep gopkg.in/square/go-jose.v1 Locking in v1.0.3 (1563e62) for transitive dep github.com/edeckers/auroradnsclient Locking in v1.2.0 (b91bfb9) for transitive dep github.com/stretchr/testify Locking in master (a8101f2) for transitive dep google.golang.org/genproto Using ^1.0.1 as constraint for direct dep github.com/terraform-providers/terraform-provider-tls Locking in v1.0.1 (1be2f9e) for direct dep github.com/terraform-providers/terraform-provider-tls Locking in master (b7773ae) for transitive dep github.com/hashicorp/go-multierror Locking in v1.0.0 (15a30b4) for transitive dep github.com/beevik/etree Locking in master (b8bc1bf) for transitive dep github.com/mitchellh/go-homedir Locking in v1.12.56 (ce8d7a1) for transitive dep github.com/aws/aws-sdk-go Locking in master (33edc47) for transitive dep github.com/mitchellh/cli Locking in v2 (3c1c487) for transitive dep gopkg.in/ns1/ns1-go.v2 Locking in v0.5.4 (0c6b41e) for transitive dep github.com/ulikunitz/xz Locking in master (ad45545) for transitive dep github.com/mitchellh/go-wordwrap Locking in master (e19ae14) for transitive dep golang.org/x/text Locking in master (553a641) for transitive dep github.com/golang/snappy Locking in master (9fd32a8) for transitive dep github.com/bgentry/go-netrc Using master as constraint for direct dep golang.org/x/crypto Locking in master (0fcca48) for direct dep golang.org/x/crypto Locking in v1.0.0 (792786c) for transitive dep github.com/pmezard/go-difflib
My initial impression here is the UX is much better than
govendor. It actually tells you what versions it’s pulling in? What sorcery is this? 😂
Looking at the output, it seems to have done a pretty good (initial) job at giving me what I want. The most important thing is that it’s vendoring lego at master. The tool hasn’t had a release for a while, and I need some post-release fixes I specifically need.
Going even farther, it vendored Terraform at it’s most recent release, 0.11.1. This is amazing, because it’s really the only dependency that I was concerned with vendoring at a specific version lock!
Now, editing one of the files within the provider does not give me the UUID-related build issues I was having, or any of the other missing or outdated dependency issues, for that matter. Pack it up, we’re done here.
But wait, are we done? Let’s check out a few more things.
dep will – currently anyway – bloat your
vendor directory pretty hard. The ACME project’s directory probably inflated about 5 times (from about 2800 files to over 16000).
This is due to some currently ongoing work in
dep to ensure that the steps undertaken by
dep prune are done by the rest of the process automatically. This removes unused packages and running it got the file count down to a much more manageable ~4700 files, which, I would imagine, is much easier to get by someone during code review (thankfully I don’t have to do that here 😉).
This still did not remove everything though. It appears the team is still working on making this work on a scale that will ultimately ensure that only what is truly needed remains – this includes removing
_test.go files in necessary packages. This is being tracked in #944, it appears, or at the very least questions about the pruning functionality are being answered there. Note that as mentioned, and as the subject of the issue mentions,
prune will soon be consolidated into
ensure, with that command assuming
prune’s functionality along with its own.
Metadata and Configuration Files
Two files control
The first one,
Gopkg.toml, is a configuration file that only displays your direct dependencies – the packages your project directly references. This is where you will want to make changes if you need to increment a version of a particular dependency before running
dep ensure. Personally, I like this approach over something like
govendor where you need to manage this on the command line.
The documentation seems to indicate that you can make changes here, run
dep ensure, and
dep will automatically transition the entire
vendor tree to properly satisfy the new dependency chain, if that’s possible.
The other file is
Gopkg.lock. This is similar to any other lock file that you would see in a dependency management system (such as Bundler), and replaces other files like
vendor.json (if you were using
govendor, for example). This file you would normally not touch, and serves to catalog all of the dependencies the tool needs, including your transitive dependencies – the packages your direct dependencies require.
Viewing Dependency Status
dep status gives you information about the vendored dependencies.
Running it with no options gives you stats on every project that has been vendored, including versions and used package count within each project.
dep status -dot allows you to output a GraphViz diagram, much like
terraform graph does, if you are familiar with that. If you are familiar with
terraform graph though, you might know how ridiculous the graphs get at scale, and obviously vendored dependencies are going to be worse, so don’t expect the graphs to be entirely simple.
Aniother thing to note is that
dep does not work out of local paths at all currently, it seems. This means that any local work you do will be superseded when you run
dep ensure, which I personally think is a great addition, because it removes the ambiguity that other tools currently have surrounding this behavior.
My hope is they they keep this default behavior if/when they do actually introduce support for working with local packages.
Version Constraints for Transitive Dependencies
The last thing I want to note is how to deal with transitive dependencies. These are dependencies that your direct dependencies (the packages your project directly references) depend on.
This probably cost me about half a day of debugging and troubleshooting, but one thing that did happen was the correctly vendored dependencies for
reflectwalk were not imported, and there are currently some “correct” behaviors in Terraform that depend on bugs that were present in these two packages. This was fixed by adding some extra override blocks in
Gopkg.toml that locked these two packages at the correct commits. After adding these in, it was as easy as running
dep ensure to get everything corrected.
dep is a great tool. It’s got a bit more work to go, but the UX is very pleasant, and everyone involved should give themselves a pat on the back for finally getting the Go community to a place where we might be able to put dependency management behind us. It’s always going to be somewhat of a struggle, but at the very least we can all struggle under one banner versus the dozen or so different ones that currently comprise the vendor tool ecosystem.
If you are curious as to how it looks in action, you can check out the code for the Terraform ACME provider, which is now being managed with
(If you read all the way through this and notice that the TF being used is now constrained to master, even though I mentioned that I wanted it locked at 0.11.1, it’s because not only did I do that during my troubleshooting process, but there has also been some changes to the “provider SDK” in
helper/schema that I figured would be good to have on hand.)