Skip to content

Commit 2ffc3d6

Browse files
mattfarinamattfarina
committed
Merge pull request #161 from kardianos/master
advent-2015: add vendor-folder
2 parents e4408d0 + abde162 commit 2ffc3d6

File tree

1 file changed

+257
-0
lines changed

1 file changed

+257
-0
lines changed

‎content/advent-2015/vendor-folder.md

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
+++
2+
author = ["Daniel Theophanes"]
3+
date = "2015-12-28T00:00:00-08:00"
4+
linktitle = "vendor folder"
5+
series = ["Advent 2015"]
6+
title = "Understanding and using the vendor folder"
7+
+++
8+
9+
With the release of Go 1.5, there is a new way the go tool can discover go
10+
packages. This method is off by default and the surrounding tools, such as
11+
`goimports`, do not understand that folder layout. However in Go 1.6 this method
12+
will be on for everyone and other tools will have support for it as well. This
13+
new package discovery method is the `vendor` folder.
14+
15+
Before we look into the solution and semantics of the vendor folder, let's explore
16+
the problem that prompted it.
17+
18+
## The prompting problem
19+
20+
Go programs are often comprised of packages from many different sources.
21+
Each one of these sources are pulled in from the `GOPATH` or from the standard
22+
library. However, only their project was subject their own source control.
23+
Projects that cared about not breaking when their dependent packages changed
24+
or went away did one of the following:
25+
26+
1. Copied the dependent packages into the project source tree and rewrote imports
27+
that referenced it.
28+
2. Copied the dependent packages into the project source tree and modified
29+
the `GOPATH` variable to include a project specific sub-tree.
30+
3. Wrote the repository revision down in a file, then updated the existing
31+
`GOPATH` packages to be that revision.
32+
33+
Although different projects did slight variations of this, these were
34+
the major trends present in all.
35+
36+
There was one large problem and several smaller ones. The largest problem
37+
was that each of these were different. The individual problems included;
38+
39+
* Many people did not like modifying the import paths or being required to
40+
include dependent packages in their repository.
41+
* Modifying `GOPATH` implied using a bare `go build` would not be sufficient.
42+
Wrappers for the `go` command emerged, each slightly different.
43+
* Modifying packages in the normal `GOPATH` required each project
44+
to have a unique `GOPATH`.
45+
46+
## A solution to these problems
47+
48+
With Go 1.5 a new method to discover go packages was released.
49+
Nothing has been changed or added
50+
to the go language or the go compilers. Packages must still reside in `GOPATH`.
51+
However, if a package or a parent folder of a package contains folder named
52+
`vendor` it will be searched for dependencies using the `vendor` folder as an
53+
import path root. While `vendor` folders can be nested, in most cases it is
54+
not advised or needed.
55+
Any package in the `vendor` folder will be found *before* the standard library.
56+
57+
To enable this in Go 1.5 set the environment variable `GO15VENDOREXPERIMENT=1`.
58+
In Go 1.6 this will be on by default without an environment variable.
59+
60+
## An example
61+
62+
This simple package lives in `$GOPATH/src/github.com/kardianos/spider`.
63+
```
64+
$ cat main.go
65+
package main
66+
67+
import (
68+
"bytes"
69+
"flag"
70+
"io"
71+
"io/ioutil"
72+
"log"
73+
"net/http"
74+
"net/url"
75+
"os"
76+
"path"
77+
"path/filepath"
78+
"strings"
79+
"sync"
80+
"time"
81+
82+
"github.com/andybalholm/cascadia"
83+
"github.com/tdewolff/parse/css"
84+
"golang.org/x/net/html"
85+
)
86+
...
87+
```
88+
89+
```
90+
$ tree
91+
.
92+
├── css_test.go
93+
├── main.go
94+
└── vendor
95+
├── github.com
96+
│ ├── andybalholm
97+
│ │ └── cascadia
98+
│ │ ├── LICENSE
99+
│ │ ├── parser.go
100+
│ │ ├── README.md
101+
│ │ └── selector.go
102+
│ └── tdewolff
103+
│ ├── buffer
104+
│ │ ├── buffer.go
105+
│ │ ├── lexer.go
106+
│ │ ├── LICENSE.md
107+
│ │ ├── reader.go
108+
│ │ ├── README.md
109+
│ │ ├── shifter.go
110+
│ │ └── writer.go
111+
│ └── parse
112+
│ ├── common.go
113+
│ ├── css
114+
│ │ ├── hash.go
115+
│ │ ├── lex.go
116+
│ │ ├── parse.go
117+
│ │ ├── README.md
118+
│ │ └── util.go
119+
│ ├── LICENSE.md
120+
│ ├── README.md
121+
│ └── util.go
122+
├── golang.org
123+
│ └── x
124+
│ └── net
125+
│ └── html
126+
│ ├── atom
127+
│ │ ├── atom.go
128+
│ │ ├── gen.go
129+
│ │ └── table.go
130+
│ ├── const.go
131+
│ ├── doc.go
132+
│ ├── doctype.go
133+
│ ├── entity.go
134+
│ ├── escape.go
135+
│ ├── foreign.go
136+
│ ├── node.go
137+
│ ├── parse.go
138+
│ ├── render.go
139+
│ └── token.go
140+
└── vendor.json
141+
```
142+
143+
## How to use the vendor folder
144+
145+
Advice on how to use the vendor folder is varied. Some will shutter at
146+
the thought of including dependencies in the project repository. Others
147+
hold it is unthinkable to *not* include the dependencies in the project
148+
repository. Some will just want to include a manifest or lock file and
149+
fetch the dependencies before building.
150+
151+
Regardless of what *you* choose to do with it, the vendor folder enables you to
152+
do more.
153+
154+
Two general guidelines:
155+
156+
* If you vendor a single package, you probably want to vendor all your dependencies.
157+
* You probably want a flat vendor structure (you probably don't want nested vendor
158+
folders).
159+
160+
## Tools that use this.
161+
162+
There are [many tools](https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment) that use the vendor folder today. There are even more
163+
tools that have support for the vendor folder in a feature branch, so the future
164+
is bright for the `vendor` folder.
165+
166+
I am the author of [govendor](https://github.com/kardianos/govendor) and
167+
[vendor-spec](https://github.com/kardianos/vendor-spec). The primary goal
168+
for `govendor` was to prove and use a common vendor specification file.
169+
The secondary goals were to make a tool that worked at the package level
170+
and could provide quick insight into the status of vendor packages.
171+
It also has always ensured the dependencies are "flattened" and only contains
172+
a single vendor folder. Today
173+
`govendor list` and `govendor list -v` are some of my favorite commands.
174+
If you are coming from `godep`, you just need to run `govendor migrate`
175+
to start using the vendor folder today.
176+
177+
### Example govendor commands
178+
Here are some examples of govendor commands on the `github.com/kardianos/spider` repo.
179+
```
180+
$ govendor list
181+
v github.com/andybalholm/cascadia
182+
v github.com/tdewolff/buffer
183+
v github.com/tdewolff/parse
184+
v github.com/tdewolff/parse/css
185+
v golang.org/x/net/html
186+
v golang.org/x/net/html/atom
187+
p github.com/kardianos/spider
188+
189+
$ govendor list -no-status +v
190+
github.com/andybalholm/cascadia
191+
github.com/tdewolff/buffer
192+
github.com/tdewolff/parse
193+
github.com/tdewolff/parse/css
194+
golang.org/x/net/html
195+
golang.org/x/net/html/atom
196+
197+
$ govendor list -v
198+
v github.com/andybalholm/cascadia ::/vendor/github.com/andybalholm/cascadia
199+
└── github.com/kardianos/spider
200+
v github.com/tdewolff/buffer ::/vendor/github.com/tdewolff/buffer
201+
└── github.com/kardianos/spider/vendor/github.com/tdewolff/parse/css
202+
v github.com/tdewolff/parse ::/vendor/github.com/tdewolff/parse
203+
└── github.com/kardianos/spider/vendor/github.com/tdewolff/parse/css
204+
v github.com/tdewolff/parse/css ::/vendor/github.com/tdewolff/parse/css
205+
└── github.com/kardianos/spider
206+
v golang.org/x/net/html ::/vendor/golang.org/x/net/html
207+
├── github.com/kardianos/spider
208+
└── github.com/kardianos/spider/vendor/github.com/andybalholm/cascadia
209+
v golang.org/x/net/html/atom ::/vendor/golang.org/x/net/html/atom
210+
└── github.com/kardianos/spider/vendor/golang.org/x/net/html
211+
p github.com/kardianos/spider
212+
213+
$ govendor remove +v
214+
$ govendor list
215+
p github.com/kardianos/spider
216+
e github.com/andybalholm/cascadia
217+
e github.com/tdewolff/buffer
218+
e github.com/tdewolff/parse
219+
e github.com/tdewolff/parse/css
220+
e golang.org/x/net/html
221+
e golang.org/x/net/html/atom
222+
223+
$ tree
224+
.
225+
├── css_test.go
226+
├── main.go
227+
└── vendor
228+
└── vendor.json
229+
230+
$ govendor add +e
231+
$ govendor list
232+
v github.com/andybalholm/cascadia
233+
v github.com/tdewolff/buffer
234+
v github.com/tdewolff/parse
235+
v github.com/tdewolff/parse/css
236+
v golang.org/x/net/html
237+
v golang.org/x/net/html/atom
238+
p github.com/kardianos/spider
239+
240+
$ govendor update -n golang.org/x/net/html/...
241+
Copy "/home/daniel/src/golang.org/x/net/html" -> "/home/daniel/src/github.com/kardianos/spider/vendor/golang.org/x/net/html"
242+
Ignore "escape_test.go"
243+
Ignore "node_test.go"
244+
Ignore "parse_test.go"
245+
Ignore "entity_test.go"
246+
Ignore "render_test.go"
247+
Ignore "token_test.go"
248+
Ignore "example_test.go"
249+
Copy "/home/daniel/src/golang.org/x/net/html/atom" -> "/home/daniel/src/github.com/kardianos/spider/vendor/golang.org/x/net/html/atom"
250+
Ignore "atom_test.go"
251+
Ignore "table_test.go"
252+
253+
```
254+
255+
As a side note, the test files are ignored in this repo because the "vendor/vendor.json" file
256+
has a field called `"ignore": "test"`. Multiple tags can be ignored. I also often
257+
ignore files with appengine and appenginevm build tags.

0 commit comments

Comments
 (0)