proposal: cmd/go: support embedding static assets (files) in binaries

[bradfitz]

There are many tools to embed static asset files into binaries:

• https://godoc.org/perkeep.org/pkg/fileembed / perkeep.org/pkg/fileembed/genfileembed
• https://godoc.org/github.com/gobuffalo/packr
• https://godoc.org/github.com/knadh/stuffbin
• https://github.com/rakyll/statik
• Bazel go_embed_data

Actually, https://tech.townsourced.com/post/embedding-static-files-in-go/ lists more:

• vfsgen - https://github.com/shurcooL/vfsgen
• go.rice - https://github.com/GeertJohan/go.rice
• statik - https://github.com/rakyll/statik
• esc - https://github.com/mjibson/esc
• go-embed - https://github.com/pyros2097/go-embed
• go-resources - https://github.com/omeid/go-resources
• statics - https://github.com/go-playground/statics
• templify - https://github.com/wlbr/templify
• gnoso/go-bindata - https://github.com/gnoso/go-bindata
• shuLhan/go-bindata - https://github.com/shuLhan/go-bindata
• fileb0x - https://github.com/UnnoTed/fileb0x
• gobundle - https://github.com/alecthomas/gobundle
• parcello - https://github.com/phogolabs/parcello

Proposal

I think it's time to do this well once & reduce duplication, adding official support for embedding file resources into the cmd/go tool.

Problems with the current situation:

• There are too many tools
• Using a go:generate-based solution bloats the git history with a second (and slightly larger) copy of each file.
• Not using go:generate means not being go install-able or making people write their own Makefiles, etc.

Goals:

• don't check in generated files
• don't generate *.go files at all (at least not in user's workspace)
• make go install / go build do the embedding automatically
• let user choose per file/glob which type of access is needed (e.g. []byte, func() io.Reader, io.ReaderAt, etc)
• Maybe store assets compressed in the binary where appropriate (e.g. if user only needs an io.Reader)? (edit: but probably not; see comments below)
• No code execution at compilation time; that is a long-standing Go policy. go build or go install can not run arbitrary code, just like go:generate doesn't run automatically at install time.

The two main implementation approaches are //go:embed Logo logo.jpg or a well-known package (var Logo = embed.File("logo.jpg")).

go:embed approach

For a go:embed approach, one might say that any go/build-selected *.go file can contain something like:

//go:embed Logo logo.jpg

Which, say, compiles to:

func Logo() *io.SectionReader

(adding a dependency to the io package)

Or:

//go:embedglob Assets assets/*.css assets/*.js

compiling to, say:

var Assets interface{
     Files() []string
     Open func(name string) *io.SectionReader
} = runtime.EmbedAsset(123)

Obviously this isn't fully fleshed out. There'd need to be something for compressed files too that yield only an io.Reader.

embed package approach

The other high-level approach is to not have a magic //go:embed syntax and instead just let users write Go code in some new "embed" or "golang.org/x/foo/embed" package:

var Static = embed.Dir("static")
var Logo = embed.File("images/logo.jpg")
var Words = embed.CompressedReader("dict/words")

Then have cmd/go recognize the calls to embed.Foo("foo/*.js") etc and glob do the work in cmd/go, rather than at runtime. Or maybe certain build tags or flags could make it fall back to doing things at runtime instead. Perkeep (linked above) has such a mode, which is nice to speed up incremental development where you don't care about linking one big binary.

Concerns

• Pick a style (//go:embed* vs a magic package).
• Block certain files?
  • Probably block embedding ../../../../../../../../../../etc/shadow
  • Maybe block reaching into .git too

[ianlancetaylor]

It's worth considering whether embedglob should support a complete file tree, perhaps using the ** syntax supported by some Unix shells.

Some people would need the ability to serve the embedded assets with HTTP using the http.FileServer.

I personally use either mjibson/esc (which does that) or in some cases my own file embedding implementation which renames files to create unique paths and adds a map from the original paths to the new ones, e.g. "/js/bootstrap.min.js": "/js/bootstrap.min.827ccb0eea8a706c4c34a16891f84e7b.js". Then you can use this map in the templates like this: href="{{ static_path "/css/bootstrap.min.css" }}".

[cespare]

I think a consequence of this would be that it would be nontrivial to figure out what files are necessary to build a program.

The //go:embed approach introduces another level of complexity too. You'd have to parse the magic comments in order to even typecheck the code. The "embed package" approach seems friendlier to static analysis.

(Just musing out loud here.)

[bradfitz]

@opennota,

would need the ability to serve the embedded assets with HTTP using the http.FileServer.

Yes, the first link above is a package I wrote (in 2011, before Go 1) and still use, and it supports using http.FileServer: https://godoc.org/perkeep.org/pkg/fileembed#Files.Open

[bradfitz]

@cespare,

The //go:embed approach introduces another level of complexity too. You'd have to parse the magic comments in order to even typecheck the code. The "embed package" approach seems friendlier to static analysis.

Yes, good point. That's a very strong argument for using a package. It also makes it more readable & documentable, since we can document it all with regular godoc, rather than deep in cmd/go's docs.

[agnivade]

@bradfitz - Do you want to close this #3035 ?

[bradfitz]

@agnivade, thanks for finding that! I thought I remembered that but couldn't find it. Let's leave it open for now and see what others think.

[balasanjay]

If we go with the magic package, we could use the unexported type trick to ensure that callers pass compile-time constants as arguments: https://play.golang.org/p/RtHlKjhXcda.

(This is the strategy referenced here: https://groups.google.com/forum/#!topic/golang-nuts/RDA9Hag8RZw/discussion)

[AlexRouSg]

One concern I have is how would it hanle invividual or all assets being too big to fit into memory and whether there would be maybe a build tag or per file access option to choose between pritorizing access time vs memory footprint or some middle ground implementation.

[urandom]

the way i've solved that problem (because of course i also have my own implementation :) ) is to provide an http.FileSystem implementation that serves all embedded assets. That way, you don't to rely on magic comments in order to appease the typechecker, the assets can easily be served by http, a fallback implementation can be provided for development purposes (http.Dir) without changing the code, and the final implementation is quite versatile, as http.FileSystem covers quite a bit, not only in reading files, but listing directories as well.

One can still use magic comments or whatever to specify what needs to be embedded, though its probably easier to specify all the globs via a plain text file.

[ianlancetaylor]

@AlexRouSg This proposal would only be for files which are appropriate to include directly in the final executable. It would not be appropriate to use this for files that are too big to fit in memory. There's no reason to complicate this tool to handle that case; for that case, just don't use this tool.

[bradfitz]

@ianlancetaylor, I think the distinction @AlexRouSg was making was between having the files provided as global []bytes (unpageable, potentially writable memory) vs providing a read-only, on-demand view of an ELF section that can normally live on disk (in the executable), like via an Open call that returns an *io.SectionReader. (I don't want to bake in http.File or http.FileSystem into cmd/go or runtime... net/http can provide an adapter.)

[urandom]

@bradfitz both http.File itself is an interface with no technical dependencies to the http package. It might be a good idea for any Open method to provide an implementation that conforms to that interface, because both the Stat and Readdir methods are quite useful for such assets