How //go:embed Works: From Source Files to Binary

How //go:embed Works: From Source Files to Binary

Since its release in Go 1.16 (2021), //go:embed has quickly proven to be one of the most loved features of the Go language. Russ Cox and Brad Fitzpatrick originally designed the feature in 2020 to embed files and directories into Go binaries using a go directive (more on what that is later). It allows engineers to quickly embed anything they want instead of using awkwardly formatted constant string literals or reaching for flaky third-party tooling to bake data directly next to their program.

Feedback on the proposal was overwhelmingly positive. It was quickly coined as one of the stand-out features of Go, not only because it’s an easy way to embed strings and byte slices, but because it also shipped with a read-only file system. That was the part that set it apart from other languages; D and Zig shipped expressions, but Go shipped a file system you could interact with easily, straight from the standard library.

I’m unreasonably excited about this feature. I feel like I always have some big ol’ JSON or template or something chilling in a constant making the file looking more unruly than it actually is, but don’t want to ship the file with the binary or bring in some sort of 3rd party tool like packr into the mix.

u/RyanTheKing

So let’s have a look into //go:embed, to see how to use it, but also see how it works under the hood; so you’re able to make more informed decisions when shipping static assets alongside your programs.

Contents

• The directive: What it is
• The variable: How to use it
• The binary: How it’s built
• The runtime: How it’s read
• The tradeoffs: What you take on

The directive: What it is

So how do we use it? Here’s a simple example to whet your appetite:

//go:embed version.txt
var version string

• We have what looks like a go comment, but it’s actually a Go directive, sometimes called pragmas (if you’re working with the compiler). You may have seen these before as go:generate or go:build.
• After writing //go:embed we specify the relative file path to where the file is located.
• In the next line, we specify the variable name we want to use. Which is where the go compiler will read the file contents and bake all of that lovely data into the binary before your program runs.
• The variable is just a pointer to the file contents; the magic happens in the build.

Before we get into any of that, it’s worth remembering the problem this was built to solve.

We’ve all seen this?

I’m sure we’ve all seen something like this before:

$ docker run myapp
2026/09/09 open ./static/app.css: no such file or directory

How annoying is this to debug? It’s usually not difficult or cumbersome, but fiddly and time-consuming. You usually have to debug some weird absolute/relative path fiasco or go into some glob patterns. This is exactly why an embed feature was needed, it makes static assets incredibly easy to ship.

What it is not

We’ve talked about what //go:embed is, but let’s just touch on what it isn’t for a moment.

• Don’t think that it’s a way to compress your files; a megabyte is a megabyte. Go doesn’t care or manipulate your data in any way.
• It’s also not a build pipeline, as mentioned above. Go doesn’t minify, transpile or fingerprint anything; it just copies bytes.
• It’s also not dynamic, it’s run at compile time, not run time. The bytes are locked in when you run go build. So if you tweak a template and expect it to update, it won’t. There’s no hot reload anywhere.
• And it’s not a media server, please don’t be going embedding your favourite Netflix films. Coincidentally, there’s actually a 2GB limit (I tried, believe me). But seriously, use object storage or something else for storing large files.

The variable: How to use it

So now we’ve seen what it is, let’s remind ourselves how to use it on a high level. If you’re familiar with the concepts of how to use it, feel free to skip this section .

There are three data types we can use with //go:embed; a string, a byte slice and a read-only file system. Here are all three of them in one program:

import "embed" // Or import _ "embed"

//go:embed version.txt
var version string

//go:embed schema.sql
var schema []byte

//go:embed assets/*
var assets embed.FS

func main() {
	fmt.Println(version) // v1.4.2
	db.Exec(string(schema))

	css, _ := assets.ReadFile("assets/index.css")
	fmt.Println(string(css)) // body { margin: 0 }
}

We first need to import the embed package. The import isn’t blank here because we’re naming embed.FS below, so we need the package itself. A blank import (import _ "embed") is for when you’re only embedding into strings and byte slices. Either way, it sends a signal to the compiler that we need to embed data, and you’ll see exactly how that works later on.

Then it’s one directive per variable, one type each:

• A string, our version.txt from the last section. Good for anything you’re going to read as text; a version number, a template, a licence.
• A byte slice, here an SQL schema. But we could really use it with anything, a config file, TLS cert, font, image, and much more. Use bytes when it makes sense for how the data is used in your program.
• An embed.FS, a whole folder. This is the part that sets Go apart from most languages, and it’s great for web assets, templates, migrations and whatever else you fancy.

And then we just use them. They’re variables. No os.Open, no path fiddling, no “did I ship the migrations folder?”. The contents are already there, just like any other variable in your program. Note that a file inside an FS keeps the path it was embedded with, which is why we’re asking for assets/index.css rather than just index.css.

embed.FS

The FS type has a bit more going on with it, so it’s worth a proper look. You can see the methods below: Open, ReadDir and ReadFile.

// An FS is a read-only collection of files.
type FS struct {}

func (f FS) Open(name string) (fs.File, error) // Satisfies fs.FS
func (f FS) ReadDir(name string) ([]fs.DirEntry, error)
func (f FS) ReadFile(name string) ([]byte, error)

The most important thing to note is that it satisfies the fs.FS interface, which means we can use it in many different ways. It makes it easy to plug and play across packages, such as http.FileServer to serve it as a static site or template.ParseFS to parse HTML templates.

// An FS provides access to a hierarchical file system.
type FS interface {
	Open(name string) (File, error)
}

Working example

Before we dive any deeper, let’s give ourselves a folder to work with. This is the example we’ll be coming back to all the way down to the linker, so it’s worth a proper look.

We’re serving a normal website. An index.html at the root, and some css and js in a folder called assets. No more, no less.

.
├── main.go
├── index.html (<h1>Hello</h1>)
└── assets
    ├── index.css
    └── index.js

To embed that into the program, we create a string variable for the contents of index.html, and an embed.FS for the contents of the assets folder, and then just print them. In a production environment you’d serve these through a handler, but we’ve kept it simple for now.

package main

import (
	"embed"
	"fmt"
)

//go:embed index.html
var indexHTML string

//go:embed assets/*
var static embed.FS

func main() {
	fmt.Println(indexHTML) // <h1>Hello</h1>
	css, _ := static.ReadFile("assets/index.css")
	fmt.Println(string(css)) // body { background: red; }
}

The assets/* pattern gets us every file in that folder; naming the directory on its own (//go:embed assets) takes the whole tree instead. The difference isn’t only depth: naming the directory skips every file beginning with . or _ anywhere beneath it, where the /* glob still pulls those in at the top level. Write all:assets when you want the whole tree, dotfiles and all. There are more gotchas in the pkg.go.dev/embed docs.

The binary: How it’s built

Now that we know what //go:embed is, we’ve seen how it works. Let’s dive a little bit deeper to see how it’s constructed under the hood. If I were to show you a bunch of code from the Go compiler out of context in the next section, I’ll probably have you lost; like it did me for a few days researching into how Go is built (TLDR; it’s challenging but worth it).

Everything below was read against Go 1.27.1, and every file and line number links to that tag on GitHub, so the lines quoted here stay put even after the toolchain moves on.

Compiler basics

So before we go anywhere near how go is built, let’s zoom all the way out. When you type go build, six steps run in order and out the other end comes a binary. This is the path that matters for //go:embed; it isn’t everything the toolchain does, just the part your files travel through.

I’m aware that there’s a whole heap of stuff that we’re not going into here such as SSA, inlining, escape analysis, etc… But none of it really touches the embed packages, because your data never becomes instructions; they are just bytes after all.

The first thing worth knowing is that there isn’t one program doing all of this. There are several, each responsible for its own thing, and go build is the one that runs the other two.

go build: six steps, three programs

So what are we looking at here?

1. Packages: The go command starts by reading your .go files and working out what package each one is in.
2. Lexer: The lexer chops that text up into tokens, and while it’s in there it reads the comments looking for pragmas, which is where your //go:embed lines get spotted.
3. Patterns: Turns each //go:embed pattern into a list of real files on the disk and hands that list to the compiler.
4. Parser: Will then arrange those tokens into a syntax tree (AST) that describes the structure of your code.
5. Compiler: Then checks the tree and turns it into machine code. What comes out is one object archive per package, which is code that can’t run on its own yet.
6. Linker: Is what ties everything together, along with the Go runtime, into the binary you build.

Notice the three different sections, that’s the bit that tripped me up for the longest, go runs once for the whole build, compile runs once per package, and link runs once at the end. It’s an important nuance to understand.

Check out go.dev/src/cmd/compile/README for more information.

Packages (go build)

So now, onto the juicy bits. Let’s start with how Go looks at your project. When you run go build it needs to know what files and packages exist, so it knows what to parse and build.

A package directory goes in, and a *build.Package comes out. So it walks your project, finds every .go file and then works out which package each one belongs to. For every file, it notes down where it lives, what it imports, and any directives (such as //go:embed) that it carries.

Packages: walking the package directory

You can see our assets/ folder and index.html being skipped, because they aren’t Go files and this pass only cares about Go. main.go makes it through the condition into matchFile, and out the other side comes a Package which details its name, its Go files, and the position of every //go:embed pattern in it.

func (ctxt *Context) Import(path string, srcDir string, mode ImportMode) (*Package, error) {
	for _, d := range ctxt.readDir(p.Dir) {          // walk the package dir
		name := d.Name()
		if filepath.Ext(name) != ".go" {             // main.go, index.html, assets/…
			continue
		}
		info, _ := ctxt.matchFile(p.Dir, name, ...)  // reads header, scans comments, gets embed
		p.GoFiles = append(p.GoFiles, name)			 // ["main.go"]
		p.Name = info.parsed.Name.Name               // main
	
		for _, emb := range info.embeds {
			// {"index.html": [main.go:10], "assets/*": [main.go:13]}
			embedPos[emb.pattern] = append(embedPos[emb.pattern], emb.pos)
		}
	}
}

Reference: src/go/build/build.go ((*Context).Import, simplified)

Our //go:embed index.html and //go:embed assets/* have been spotted and recorded along with their file names and token positions.

But how did it spot the comment originally? This is where the lexer comes into play.

Lexer (go build)

The lexer (or scanner, or tokeniser, whatever you fancy) is the first thing that actually reads your code as Go, rather than as a blob of bytes. It scans the file one character at a time and chops it up into little tokens, which are the pieces of code that mean something on their own. Such as func, var, =, a string literal, or a comment.

This is the stage that makes //go:embed actually possible. In Go, a comment is a real token (token.COMMENT).

Lexer: one character at a time, into tokens

Characters go in and tokens come out. The whole //go:embed index.html line becomes a single COMMENT token, and the two words of var indexHTML string become a VAR and two IDENTs.

The go command’s scan is deliberately inexpensive as it stops at the end of the import block, because that’s normally all it needs. Unless, however it saw import "embed" on the way past, in which case it reads on to the end of the file, picking out every comment starting //go:embed:

if hasEmbed {
    r.readRest() // keep going: read the rest of the file after all
    sc.Init(file, r.buf, nil, scanner.ScanComments) // get comments as tokens
	
    for {
        pos, tok, lit := sc.Scan()
        if tok == token.COMMENT && strings.HasPrefix(lit, "//go:embed") {
            // in:  `//go:embed index.html`
            // out: {pattern: "index.html", pos: ...}
            embs, _ := parseGoEmbed(fset, pos, lit)
            info.embeds = append(info.embeds, embs...)
        }
    }
}

Reference: src/go/build/read.go#L374-L392 (readGoInfo, simplified)

That hasEmbed flag is the real reason import _ "embed" is mandatory even when you never type the package name. If you don’t import the package, we’ll never go into this code block.

What comes out is a list of the patterns it found, and where each one was written:

[]fileEmbed{
    {pattern: "index.html", pos: main.go:10:12},
    {pattern: "assets/*",   pos: main.go:13:12},
}

We now have patterns to file names and line numbers, but there are no files? This is where resolving embed patterns comes in.

Patterns (go build)

Within the go command we have a fairly important function called resolveEmbed which resolves patterns into actual files and trees. File patterns go in, and real file paths come out.

Patterns: resolveEmbed turns patterns into files

The glob is where it gets interesting. Our index.html was only ever going to match itself, but assets/* fans out to assets/index.css and assets/index.js, neither of which we named anywhere. That’s the first time anything in this whole process has actually gone and looked at your disk.

So these patterns get expanded, globbing it against the disk and walking any directory it matches, into a list of actual file names. You can see this in action below:

func resolveEmbed(pkgdir string, patterns []string) (files []string, pmap map[string][]string, err error) {
    pmap = make(map[string][]string)
    have := make(map[string]bool) // every file, once

    // patterns = []string{"index.html", "assets/*"}
    for _, pattern := range patterns {
        matches, _ := fsys.Glob(pkgdir + "/" + pattern)

        var list []string
        for _, match := range matches {
            if fsys.IsDir(match) {
                fsys.WalkDir(match, func(path string, ...) { list = append(list, path) })
            } else {
                list = append(list, match)
            }
        }

        pmap[pattern] = list // what this pattern matched
        for _, f := range list {
            have[f] = true   // ...and the running set of all of them
        }
    }

    for f := range have {
        files = append(files, f)
    }

    return files, pmap, nil
}

Reference: src/cmd/go/internal/load/pkg.go#L2165 (resolveEmbed, simplified)

Output will now be:

// pmap:
map[string][]string{
     "assets/*":   {"assets/index.css", "assets/index.js"},
     "index.html": {"index.html"},
}

// files:
[]string{"assets/index.css", "assets/index.js", "index.html"}

Those two return values become the two data points that the compiler needs:

• Patterns answers “what did this pattern match?” and keeps the duplicates, because that’s the tree your embed.FS gets, and two different patterns can easily match the same file.
• Files answers “where do I read those bytes?” and cannot have duplicates, because the compiler should open each file exactly once.

The go command marshals them into JSON, writes it to a file called embedcfg in the build’s temporary work directory, and passes that path to the compiler as -embedcfg, once per package.

Why a file, why not just hand it over directly? Because go build and the compiler are two separate processes, as we saw in that first diagram, so it gets written down instead.

For our example project, the generated embedcfg looks like:

{
	"Patterns": {
		"assets/*": [
			"assets/index.css",
			"assets/index.js"
		],
		"index.html": [
			"index.html"
		]
	},
	"Files": {
		"assets/index.css": "/home/user/project/assets/index.css",
		"assets/index.js": "/home/user/project/assets/index.js",
		"index.html": "/home/user/project/index.html"
	}
}

Reference: work/exec.go#L893-L906 builds and marshals it, work/gc.go#L148-L152 writes it and passes -embedcfg (simplified)

(Tip: go list -json=EmbedPatterns,EmbedFiles . surfaces these exact matched patterns and files straight from the package metadata, any time you want to look.)

Parser (cmd/compile)

We’re in a different program now. Tokens go in, along with the path to that embedcfg, and what comes out is a tree with your directive attached to the right node.

The parser is where the real magic happens in the Go compiler. It takes those tokens and little bits of Go code and works out their structure. A bit like breaking a sentence down into a subject, verb and object.

The result is a syntax tree; you may have heard of the term AST before, which is short for abstract syntax tree. It’s the building blocks of our program. Every piece of your code (a function, a variable, an if statement) becomes a node within that tree.

Directives such as //go:embed get attached to the node, which is how Go knows which variable to put your embedded file in.

The below example is captured from the two lines from our example above:

//go:embed index.html
var indexHTML string

Parser: the directive and the variable are one node

• GenDecl: Is the declaration as a whole, it’s short for a generic declaration node which represents an import, constant, type or variable declaration. In our example, it’s
  both the comment, and the variable.
• Doc: Is the comment, held as a *ast.CommentGroup. Our //go:embed index.html is in there, kept word for word in Text.
• Specs: Is what that declaration actually declares which is an *ast.ValueSpec holding the name indexHTML and the type string, our variable.

Both the Doc and Specs come under GenDecl. The comment and the variable aren’t two things that happen to be near each other in a file; they’re two branches of one node. That’s how the compiler knows the contents of index.html go into indexHTML, and not into whatever is declared next.

Which looks like this. The dump is go/ast, which is what ast.Print gives you and what most tooling reads. The compiler’s own parser is cmd/compile/internal/syntax and holds the same shape under different names, which is why checkEmbed in the next section takes a *syntax.VarDecl:

1: *ast.GenDecl {
.  Doc: *ast.CommentGroup {
.  .  List: []*ast.Comment (len = 1) {
.  .  .  0: *ast.Comment {
.  .  .  .  Slash: main.go:8:1
.  .  .  .  Text: "//go:embed index.html"
.  .  .  }
.  .  }
.  }
.  TokPos: main.go:9:1
.  Tok: var
.  Specs: []ast.Spec (len = 1) {
.  .  0: *ast.ValueSpec {
.  .  .  Names: []*ast.Ident (len = 1) {
.  .  .  .  0: *ast.Ident {
.  .  .  .  .  NamePos: main.go:9:5
.  .  .  .  .  Name: "indexHTML"
.  .  .  .  }
.  .  .  }
.  .  .  Type: *ast.Ident {
.  .  .  .  NamePos: main.go:9:15
.  .  .  .  Name: "string"
.  .  .  }
.  .  }
.  }
}

Clever stuff, right? This was the aha moment for me. The comment and variable are one declaration in Go’s eyes.

Compiler (cmd/compile)

Now onto the nuts and bolts, the compiler. A tree goes in, and what comes out is an object archive, one per package, which is your code compiled but not yet runnable on its own. This is also the stage where your data is actually allocated.

Everything so far has been bookkeeping; finding files, expanding patterns and writing lists, but not one byte of your index.html has actually been allocated into your program. However, in this step, it does.

The compiler lays your variable out in memory and fills it with the file’s contents, as static data, that’s there before your program runs a single line of your app’s code.

Although its first job isn’t your files at all, it’s the rules. Remember, the go command spotted your directive and wrote the pattern down, but it never checked anything about it, which is mentioned in the comments:

// The compiler will enforce the mapping of comments to
// declared variables. We just need to know the patterns.

Reference: src/go/build/read.go#L369-L373 (readGoInfo)

Now it checks what you’ve written is actually valid. Nearly every //go:embed error message you’ve probably seen comes from this switch statement:

func checkEmbed(decl *syntax.VarDecl, haveEmbed, withinFunc bool) error {
	switch {
	case !haveEmbed: // The file never imported "embed".
		return errors.New(`go:embed only allowed in Go files that import "embed"`)
	case len(decl.NameList) > 1: // var a, b string
		return errors.New("go:embed cannot apply to multiple vars")
	case decl.Values != nil: // var s string = "hello"
		return errors.New("go:embed cannot apply to var with initializer")
	case withinFunc: // Declared inside a func body.
		return errors.New("go:embed cannot apply to var inside func")
	}
	return nil
}

Reference: src/cmd/compile/internal/noder/noder.go#L462-L481 (checkEmbed, simplified)

But how is the data read?

Compiler: WriteEmbed lays <h1>Hello</h1> out as 14 bytes of read-only data

// WriteEmbed, for our `var indexHTML string`.
case embedString, embedBytes:
	file := files[0]                        // "index.html", from the embedcfg JSON
	path := base.Flag.Cfg.Embed.Files[file] // "/home/user/project/index.html"
	fsym, size, _ := fileStringSym(v.Pos(), path, ...) // read it, name the bytes

	sym, off := v.Linksym(), 0                 // indexHTML itself
	off = objw.SymPtr(sym, off, fsym, 0)       // field 1: pointer to those bytes
	off = objw.Uintptr(sym, off, uint64(size)) // field 2: length
	if kind == embedBytes {
		objw.Uintptr(sym, off, uint64(size))   // field 3: cap, because a slice has one
	}
case embed.FS:
	// See source files below...

Reference: src/cmd/compile/internal/staticdata/embed.go#L101-L134 (WriteEmbed, simplified)

Here we can see WriteEmbed switching on the variable’s data type. It then goes on to call a function called fileStringSym, which opens your file and returns a symbol for its contents, as well as the size.

Okay, but what’s a symbol? Forget []byte for a second. A symbol is a box with a name on it, what it’s called, how big it is, and where it lives in the binary. That’s what the linker actually needs to do its job, and it’s why fileStringSym hands one back rather than the bytes themselves. This is where your asset has stopped being a path and has actually become data.

The rest is ordinary compilation. The tree gets turned into machine code (cmd/compile/internal/ssagen), and out comes one object archive per package, which is code that can’t run yet, which is where the linker comes into play.

Linker (cmd/link)

Although the linker has little to do with //go:embed, it plays a crucial part in the build process. It’s a separate program, located under cmd/link. Many .a files go in, and one executable comes out. All of the object archives that we mentioned above, now are joined together. It fills in all the gaps so every piece can find the others.

Which is where the Go runtime is injected. The runtime is built-in code that manages memory, goroutines and all of the stuff that makes Go, Go.

Linker: object archives and the runtime become one binary

There’s main.a still carrying our 14 bytes, along with fmt.a, embed.a, every other package you pulled in, and runtime.a. On the right is what they become, one binary, and our <h1>Hello</h1> sitting in the rodata section, which is where read-only data lives.

Binary

Now run it (./myapp) and voilà! We have your finished program; maybe we should take a breather…

Build Summary

I know that was a lot to take in, so let’s go back to where we started and walk it once more, now that all the names mean something. I always find this easier to read as pseudocode, so here we go. It isn’t real or runnable; it just connects all the dots:

import (
	"go/build"                        	// 1. Packages: what's in this package?
										// 2. Lexer:    text -> tokens
	"cmd/go/internal/load"            	// 3. Patterns: "assets/*" -> real paths on disk
	"cmd/compile/internal/syntax"     	// 4. Parser:   tokens -> syntax tree
	"cmd/compile/internal/staticdata" 	// 5. Compiler: file bytes -> read-only data
	"cmd/link/internal/ld"            	// 6. Linker:   object files -> the binary
)

func goBuild(dir string) Binary {
	pkg := build.ImportDir(dir) // files, imports, embed patterns
	
	var objs []Object
	for _, p := range pkg.Deps {
		embeds := load.ResolveEmbed(p.Dir, p.EmbedPatterns) // "assets/*" -> [assets/index.css, ...]
		tree := syntax.Parse(p.GoFiles)                     // code + directives -> tree
	
		for _, v := range tree.EmbedVars {
			data := os.ReadFile(embeds[v.Pattern]) // the compiler opens your file
			staticdata.WriteEmbed(v, data)         // and bakes the bytes in
		}
		objs = append(objs, compile(tree)) // tree -> object file
	}
	
	return ld.Link(objs, runtime) // -> the binary you ship
}

Three separate processes, and they don’t run the same number of times, go runs once for the whole build, compile runs once per package and link runs once at the end. That’s why embedcfg is written per package, immediately before each compile invocation.

And you don’t have to take my word for any of it. go build -x prints the real command lines and the temporary directory, so you can watch embedcfg being written and passed along, and ls $(go env GOROOT)/pkg/tool/$(go env GOOS)_$(go env GOARCH)/ shows you that compile and link really are separate binaries sitting on your disk. Every Reference link above points at the exact lines in Go 1.27.1 if you’d like to read the real thing.

The runtime: How it’s read

So the bytes are in the binary. The last question is what happens when your program actually opens them, because that depends on which of the three types you picked.

For a string or a []byte, there’s almost nothing to tell, and that’s rather the point. The compiler wrote a pointer and a length, and the pointer aims at bytes sitting in the rodata section we just watched the linker build. Reading the variable is an ordinary variable read.

An embed.FS is more involved, and it’s the more interesting half. Remember it’s a single pointer to a hidden list, and here’s what’s on the end of it:

// An FS is a read-only collection of files.
type FS struct {
	files *[]file
}

// A file is a single file in the FS.
type file struct {
	name string
	data string
	hash [16]byte // truncated SHA256 hash
}

Reference: src/embed/embed.go#L151-L212 (FS and file, simplified)

Each file’s contents are a string, pointing into the same read-only data our indexHTML did, so a 40MB asset isn’t sitting in your heap waiting for you.

Asking for a file is a search, not a syscall. The list is sorted, so lookup does a binary search over it, and readDir does two of them to find where a directory starts and ends, then hands back that stretch of the slice. There’s no disk involved at any point, which is why ModTime on an embedded file is the zero time. There’s nothing there to have been modified.

There’s one detail worth the whole section, though, and it’s the difference between the two ways of getting at your bytes. Open hands you back an openFile that reads straight out of that string, copying only as much as fits the buffer you gave it. ReadFile ends looking a bit like this:

return []byte(ofile.f.data), nil

Reference: src/embed/embed.go#L322-L332 (ReadFile)

That conversion is a copy, and a fresh one on every call. So the 40MB asset that wasn’t sitting in your heap a moment ago lands there the moment you call ReadFile on it. Open doesn’t; it fills your buffer a chunk at a time, which is the path http.FileServer and io.Copy take. Neither touches the disk, but only one of them allocates the file.

The tradeoffs: What you take on

You’ve seen how it works and how it’s used, so let’s talk about what it costs you, because embedding isn’t free.

Rebuild to change a byte

Remember what WriteEmbed did with those bytes. They’re read-only data in the object file now, which means your file isn’t really a file any more. So change one line of CSS, and you have to rebuild.

This can be a bit painful because nobody wants a full build on every save. The fix is the last in this section, so hold onto that.

No secrets

And here’s the one that catches people out. Embedded bytes are just data in the binary, which means they’re readable:

$ strings myapp | grep -i api_key
API_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc

Anyone with your binary has your file. So no .env, no private keys, no credentials. It might seem obvious, but it’s an easy one to forget when you’re embedding a whole config directory.

What the compiler catches

Now for the good news. Remember checkEmbed, that big switch statement? Bad patterns, no matching files, the wrong type, a variable inside a function; all of them are build failures. But the compiler knows the file exists. It doesn’t know it’s the file you meant. That’s where the standard library has a package that can help you out.

Testing what you embedded

testing/fstest is the handy little package of the standard library that doesn’t get enough attention, which is a shame, as it works perfectly alongside embed. fstest.TestFS takes your FS and a list of paths you expect to find within it, and then does two jobs at once:

• Asserts every path you named is really there.
• Cross-checks the file system against itself, walking the whole thing to confirm that Open, ReadDir and ReadFile all agree with each other.

The trick is what you list. It shouldn’t be a whole inventory of the FS, just a handful of the files your code actually opens:

func TestAssetsArePresent(t *testing.T) {
	// Extras are ignored, so adding a file needs no test change.
	err := fstest.TestFS(static,
		"assets/index.css",
		"assets/index.js",
	)
	if err != nil {
		t.Fatalf("files are not found: %v", err)
	}
}

Only name what you’d notice missing; the template your handler executes, the migration directory, the default config. Everything else is just noise that makes the test brittle.

Depend on fs.FS

And this is the one thing I’d really like you to take away, and the answer to the rebuild problem we started this section with.

// Take the interface, not the implementation.
func NewServer(assets fs.FS) *Server {
	return &Server{assets: assets}
}

func TestIndexHandlesAnEmptyTemplate(t *testing.T) {
	srv := NewServer(fstest.MapFS{
		"templates/index.html": &fstest.MapFile{Data: []byte("")},
	})

	// No files on disk, no embed, no build.
	if err := srv.Render(io.Discard, "index.html"); err == nil {
		t.Fatal("expected an error for an empty template")
	}
}

Take fs.FS in your constructors, not embed.FS. Your server doesn’t care where the files came from; it cares that it can open them. Which means a test can hand you an fstest.MapFS instead, with the files declared inline as a map.

And now you can ask the questions a good unit test should ask. What if this file is empty? What if it’s corrupt, or 40 megabytes? You can answer all of that without touching the disk or waiting on a build. It’s also how you sidestep the rebuild-to-change-a-byte problem in development: the same constructor takes os.DirFS("./assets") while you’re working and your embed.FS in production, and nothing else in your program needs to know which one it got.

Wrapping up

So that’s the whole journey, from a comment in your source to bytes in the rodata section of a binary you can copy anywhere.

//go:embed is one of those quiet features about Go that makes it so easy to reason about. It’s a kind of feature that you didn’t know you needed until it came along. It makes for a great experience when you’re building a package that needs to ship some data with it.

I would really encourage you to have a look at the Go source code, although it can look scary at first (I spent some time dazed and confused looking at it). But it really makes you a better engineer for knowing how these constructs work.

Please reach out to me at [email protected] if you have any questions! I hope you’ve enjoyed reading this post as much as I have writing it.

Source code

A reminder for each section, its program, package and line numbers are below:

Further reading

Embed

• Draft design: Go command support for embedded static assets , Russ Cox & Brad Fitzpatrick, July 2020
• The original proposal (#41191)
• The io/fs proposal (#41190) , the companion that makes embed.FS compose with everything
• embed package documentation
• CL 243942 , embed: implement FS

Go command

• CL 243941 , go/build: recognise and report //go:embed lines
• CL 243945 , cmd/go: add //go:embed support
• go/build/read.go#L314 , the imports-only parse the go command starts with

Compiler

• Go compiler README
• cmd/compile/internal/syntax/parser.go , the compiler’s own parser, with the directive attached by noder
• CL 243944 , cmd/compile: add //go:embed support
• staticdata/embed.go, where the bytes are written

Linker

• Go linker documentation
• Go linker source code

Runtime

• src/embed/embed.go , the whole embed.FS implementation, and a genuinely short read
• testing/fstest , TestFS and MapFS
• Go runtime package documentation
• Go runtime internals

Other languages

• D, import("file.txt"), an expression, documented since D 1.0 (spec )
• Zig, @embedFile, in the docs since 0.1.1 (ziglang/zig#14637 )

Community

• Embed files in Go using the new embed package , the thread the quote at the top comes from
• Go 1.16 release notes on Hacker News
• JetBrains, How to use go:embed in Go

This email was sent as part of the Internals for Interns newsletter.
Visit the blog • Read this post online

You're receiving this because you subscribed to our newsletter.