Getting `go: inconsistent vendoring` error when a module is located within function

[Dentrax]

Expected Behaviour

It should work? 🤷‍♂️

Current Behaviour

Step 17/32 : RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Running in 56939001fb88
go: inconsistent vendoring in /go/src/handler:
	github.com/openfaas/templates-sdk@v0.0.0-20200723092016-0ebf61253625: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1
[0] < Building notifier-fn done in 2.64s.
[0] Worker done.

Total build time: 2.64s
Errors received during build:
- [notifier-fn] received non-zero exit code from build, error: The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1

Steps to Reproduce (for bugs)

We were following the template documentation guide as described here.

1. $ faas-cli template store pull golang-http
2. $ faas-cli new notifier-fn --lang golang-http --prefix Dentrax
3. $ cd notifier-fn
4. $ go mod init github.com/Dentrax/foo
5. Use a dependency (i.e. github.com/slack-go/slack) inside handler.go

Context

go.mod

module github.com/Dentrax/foo

go 1.16

require (
    github.com/openfaas/templates-sdk v0.0.0-20200723092016-0ebf61253625
    github.com/slack-go/slack v0.8.1
)

notifier-fn.yml:

version: 1.0
provider:
  name: openfaas
  gateway: http://foo
functions:
  notifier-fn:
    lang: golang-http
    handler: ./notifier-fn
    image: Dentrax/notifier-fn:latest
    build_args:
      GO111MODULE: on
    environment:
      SLACK_WEBHOOK_URL: https://bar

What we have tried so far that did not work

Each step we applied these in sequential:

• faas-cli up -f notifier-fn.yml

• faas-cli up -f notifier-fn.yml --build-arg GO111MODULE=on

• Single handler.go file:

• $ go mod init ...

• $ go mod vendor + $ go mod tidy? (w/ vendor dir)

• Using GO_REPLACE.txt in parent dir

Your Environment

FaaS-CLI version

CLI: commit: 6b5e7a14a598527063c7c05fb4187739b6eb6d38 version: 0.13.6

Docker version

Client: Docker Engine - Community Cloud integration: 1.0.7 Version: 20.10.2 API version: 1.41 Go version: go1.13.15 Git commit: 2291f61 Built: Mon Dec 28 16:12:42 2020 OS/Arch: darwin/amd64 Context: default Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0

• Are you using Docker Swarm (FaaS-swarm ) or Kubernetes (FaaS-netes)? NO

• Operating System and version (e.g. Linux, Windows, MacOS): macOS 11.0.1

We are not sure why the given template documentation does not work as we expected. I think we are missing some important detail on the documentation. Actually, the command that failing above is passing on our local workspace: $ CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build --ldflags "-s -w" -a -installsuffix cgo -o handler . Any ideas?

cc: @developer-guy

[alexellis]

@LucasRoesler ptal

[alexellis]

My initial thoughts are:

Turn go modules off (the build arg, but don't delete anything), and it should continue to work.

Or remove vendoring and it will continue to work.

Vendoring and modules together (in the build) have started to fail in recent Go versions.

Please try both and report back.

[Dentrax]

Hmm, now I just turn off GO111MODULE flag (by setting off) and removed it from the build flags. Vendored the function afterward: $ go mod vendor

Although $ go build --ldflags "-s -w" -a -installsuffix cgo -o handler . works well manually in the function directory, I get another error during the Docker build phase:

$ faas-cli up -f notifier-fn.yml

# handler
./main.go:102:39: cannot use req (type "handler/vendor/github.com/openfaas/templates-sdk/go-http".Request) as type "handler/function/vendor/github.com/openfaas/templates-sdk/go-http".Request in argument to function.Handle

Setting the Go version to 1.15 did not resolve the issue. 🤔

[LucasRoesler]

@Dentrax i have noticed this issue as well. I ran into it last week, but I have not found a backwards compatible solution. Last weekend when I ran in to the issue, i opted to use the http middleware template instead as a quick fix. But I am still investigating how to fix this template. I need to determine if the middleware template is actually working or just lucky because it doesn't import any external packages.

[Dentrax]

@LucasRoesler Thanks for the detailed explanation. We have moved the function from golang-http to golang-middleware, as you said. Now it rocks. ^^

$ faas-cli new notifier2-fn --lang golang-middleware --prefix Dentrax

I think it was some kind of specific bug at the golang-http template. Version: github.com/openfaas/templates-sdk@v0.0.0-20200723092016-0ebf61253625

[LucasRoesler]

@alexellis I started a looking at a fix that would simply aligned the http and the middleware templates. But there is an issue.

The http template includes a vendor folder, this is a problem in vendor module. It produces this error due to Go validating that the mod file and the vendor folder are in sync, this produces the following error

#25 0.437 go: inconsistent vendoring in /go/src/handler:
#25 0.437 	github.com/openfaas/templates-sdk@v0.0.0-20200723092016-0ebf61253625: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
#25 0.437 	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt
#25 0.437
#25 0.437 run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory

To work around this error, you can delete the vendor folder that is included in the template. When you remove the vendor folder, I have a PR that works in these two scenarios

1. pure modules downloaded at build time

faas-cli build -f publisher.yml --build-arg GO111MODULE=on

2. modules disabled and using vendor

cd publisher && go mod vendor && cd ..
faas-cli build -f publisher.yml --build-arg GO111MODULE=off

But you can't use modules and vendor, for example this will fail

cd publisher && go mod vendor && cd ..
faas-cli build -f publisher.yml --build-arg GO111MODULE=on

This behavior is effectively implemented by using

RUN if [ -d ./function/vendor ] && [ "${GO111MODULE}" == "off" ]; then \
    echo "moving handler vendor" && \
    mv -f ./function/vendor .; \
    else \
    echo "using modules or vendor not found "; \
    fi

the biggest issue is that the template currently defaults to modules "off", which means the default vendor folder that i just removed is required to make the default "echo" handler work. To make it work we would need to

1. turn modules on by default
2. leave modules off, but push the vendor folder from the template base and into the handler folder

[LucasRoesler]

One thing to note, i tried using go mod tidy to synchronize the go.mod and vendor during the Docker build, but it still results in and error

#20 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#20 sha256:48ad39e8ada37838ac158b13cce56b5bb4cd5dc818c058f9c1f1bc40c024a10f
#20 0.308 cat: can't open 'function/GO_REPLACE.txt': No such file or directory
#20 DONE 0.3s

#21 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor . && go mod tidy;     else     echo "using modules or vendor not found ";     fi
#21 sha256:6300638d9169e81111fb8f382fd8a274d66ee6ec190eb958125fe17c0a0361ed
#21 0.357 moving handler vendor
#21 0.539 go: downloading github.com/openfaas/templates-sdk v0.0.0-20200723092016-0ebf61253625
#21 0.631 go: found handler/function in handler/function v0.0.0-00010101000000-000000000000
#21 2.024 go: downloading github.com/openfaas/templates-sdk v0.0.0-20200723110415-a699ec277c12
#21 2.037 go: downloading github.com/nats-io/nats.go v1.10.0
#21 2.140 go: downloading github.com/nats-io/jwt v1.1.0
#21 2.140 go: downloading github.com/nats-io/nkeys v0.1.4
#21 2.140 go: downloading github.com/nats-io/nuid v1.0.1
#21 2.141 go: downloading github.com/nats-io/nats-server/v2 v2.1.9
#21 2.141 go: downloading github.com/golang/protobuf v1.4.3
#21 2.141 go: downloading google.golang.org/protobuf v1.25.0
#21 2.182 go: downloading golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59
#21 2.388 go: downloading github.com/google/go-cmp v0.5.0
#21 2.487 go: downloading golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e
#21 2.517 go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
#21 DONE 2.9s

#22 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#22 sha256:310a7c685f0ed22d53cc736699d8299ff5543963680ef2eefbb5349f9c7dc1ac
#22 DONE 0.4s

#23 [build 11/14] WORKDIR /go/src/handler/function
#23 sha256:7c444425d527f8de94638a1c47c4fe729f0f40b6ca22c8c74f391558668fdded
#23 DONE 0.0s

#24 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#24 sha256:0c7c7857b64200523e078c51cca979b4108c14c4694b7e51299949370e2f4c68
#24 4.240 ?   	function	[no test files]
#24 DONE 4.3s

#25 [build 13/14] WORKDIR /go/src/handler
#25 sha256:f548c022b481872d9036a04d81a4cd4f559d77290190882ef0fd5bf90e13b517
#25 DONE 0.0s

#26 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#26 sha256:63f06c1f4ebe4dd80dbd0f24d262bb9d0cb9f39f60acfc1fda7eb06bbbb9ad72
#26 0.492 go: inconsistent vendoring in /go/src/handler:
#26 0.492 	handler/function@v0.0.0-00010101000000-000000000000: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
#26 0.492 	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt
#26 0.492 	github.com/nats-io/nats.go@v1.10.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
#26 0.492
#26 0.492 run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
#26 ERROR: executor failed running [/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .]: exit code: 1

The cause is the replace we use in the template go.mod

[alexellis]

Do you know if these steps affect caching? Can you try some things and find out.

[LucasRoesler]

Do you know if these steps affect caching? Can you try some things and find out.

@alexellis what are some things that you consider as proof? just running the build twice and seeing that step reported as cached?

[LucasRoesler]

Here is a second build of my test function, all of the steps report as cached

$ faas-cli build -f publisher.yml --build-arg GO111MODULE=on
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:21a3073c4b533f5a50a7af91c6fc092fd2e018454c212c837e9a49b96a9ee5d5
#1 transferring dockerfile: 2.01kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:10bc899bd437f5c8b9341111d3de92a13762288bed34430aa2ff9bafe5407c5a
#2 transferring context: 102B done
#2 DONE 0.0s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 ...

#4 [internal] load metadata for ghcr.io/openfaas/of-watchdog:0.8.4
#4 sha256:a36b81af230f4aa9af61173e476a504ddf679531cc6fb84c21a9f883d305ce12
#4 DONE 0.2s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 DONE 0.6s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 DONE 0.6s

#6 [stage-2 1/7] FROM docker.io/library/alpine:3.13@sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
#6 sha256:ac117f80319f3dfc5f3a89bed8fde570a7095ef011b4aa4f1832954fe5fae646
#6 DONE 0.0s

#10 [build  1/14] FROM docker.io/library/golang:1.15-alpine3.13@sha256:a025015951720f3227acd51b0a99a71578b574a63172ea7d2415c60ae5e2bc0a
#10 sha256:945f28b33676aa8e1e61782ed1c14580cb6577c9dae69198435ffaa8642d6acc
#10 DONE 0.0s

#12 [watchdog 1/1] FROM ghcr.io/openfaas/of-watchdog:0.8.4@sha256:9a0b595f77724005ef7336d68a5a6347426d1931c82f238916496f27d7ea2e74
#12 sha256:0b83d079b8d19e155cadf89ef4c3782ffd5a248d98dbd78ee94c18245505455c
#12 DONE 0.0s

#17 [internal] load build context
#17 sha256:f377bbe6ad24eb025bf76e33d6219983965cc07c9dd78e02186bb94c2e8061ba
#17 transferring context: 16.92kB done
#17 DONE 0.0s

#15 [build  5/14] RUN mkdir -p /go/src/handler
#15 sha256:49827b6b29acd15ceb89996f6858464d41ef41d45bf4439caa09ad778a541c9b
#15 CACHED

#27 [stage-2 6/7] COPY --from=build --chown=app /usr/bin/fwatchdog         .
#27 sha256:a15f3a00af062c11966cc8bf9b62eec7189a69a7a5d6bd292268f5c04e62e80f
#27 CACHED

#13 [build  3/14] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
#13 sha256:3d4b56eeab2f493ee25bcffa0a77d52bf329e8d5bcf1a7f4f9e60d76207b024c
#13 CACHED

#16 [build  6/14] WORKDIR /go/src/handler
#16 sha256:2d57cb8be74105b3bf574dde9b3acfb2afb9ef7c3db0babaf2976e90bd189f0a
#16 CACHED

#9 [stage-2 4/7] WORKDIR /home/app
#9 sha256:8d07b104cc02b839e2f8b214eb64d1906c976d87c5178fc380b99ff4ee79f52d
#9 CACHED

#14 [build  4/14] RUN chmod +x /usr/bin/fwatchdog
#14 sha256:0ce54538eaf598232623e0d0131601ae53ed928d280203d11398984ad325296d
#14 CACHED

#26 [stage-2 5/7] COPY --from=build --chown=app /go/src/handler/handler    .
#26 sha256:96883fc3eb65c24f97b0347d0c5bfb829b3bd29c2f2df68a7681a0257c0a09d3
#26 CACHED

#11 [build  2/14] RUN apk --no-cache add git
#11 sha256:e895da5279e30155cbfeb55e93d54399f92e5b71ef96937c7955731c811f752e
#11 CACHED

#25 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#25 sha256:f103a02f229f25654f54cbde12afa99aa0c721c85804b51c23e539cb95e044e9
#25 CACHED

#21 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#21 sha256:b1d524d09ab70ce20b7f57efa46db80d3cf0e49ea734b78f152828451eacc2cd
#21 CACHED

#23 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#23 sha256:2d9eb2ff7aff1812f647f4c7f9e1b420821e818717ce1145553f6b44635b0073
#23 CACHED

#19 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#19 sha256:4d72ddb9b23a8417c2db4826354610584a497c771deef7debf52befd032d9638
#19 CACHED

#20 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor . && go mod tidy;     else     echo "using modules or vendor not found ";     fi
#20 sha256:d1efda4d0f61d94cc51ed7d079ca194f3ff8a5dae60a6d9af48ffe4bc320efb8
#20 CACHED

#22 [build 11/14] WORKDIR /go/src/handler/function
#22 sha256:10fbc5207db3831f3e06a80dc0c6f960ca19f7dbd8192ff2435b41aaa663a832
#22 CACHED

#24 [build 13/14] WORKDIR /go/src/handler
#24 sha256:ee155e1a79e01bab85a5c637cbf925c23746c9aedd5b1288b142e5628df20daf
#24 CACHED

#8 [stage-2 3/7] RUN mkdir -p /home/app     && chown app /home/app
#8 sha256:392db714e3258ded26fd35e24dbf9da4b34d5f1b72bafa6651edb2c67c6e337a
#8 CACHED

#7 [stage-2 2/7] RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
#7 sha256:f3920e39e6cb6756252ef8376a8a6a562e10f4156f6387b0e5aa5f3204bbbeb5
#7 CACHED

#18 [build  7/14] COPY . .
#18 sha256:b2081521828c5a6f4f2df32505d0dfa0fd8e3b7f26e1e4e23ec1cfbdc2e60fb5
#18 CACHED

#28 [stage-2 7/7] COPY --from=build --chown=app /go/src/handler/function/  .
#28 sha256:0f57dc249c06f1ea6f7b251e043aaa88a4977b712ae899c41a93513ed9e7578a
#28 CACHED

#29 exporting to image
#29 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#29 exporting layers done
#29 writing image sha256:ae862458b5fc8f845899815ba391511beb488b07239a5d79c2f284f60a351e9b done
#29 naming to docker.io/library/publisher:latest done
#29 DONE 0.0s
Image: publisher:latest built.
[0] < Building publisher done in 2.03s.
[0] Worker done.

Total build time: 2.03s

[LucasRoesler]

Same with modules off and a vendor folder exists

$ faas-cli build -f publisher.yml --build-arg GO111MODULE=off
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:e574877995c10303eb346abe88f875984739ad086d155a9a1760ba721dc461b9
#1 transferring dockerfile: 1.99kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:c2879ad2d201614c4ca5f0b0a93e9bd95ddcd2d3244b2afe07de6e6ac0eda3b2
#2 transferring context: 102B done
#2 DONE 0.0s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 ...

#4 [internal] load metadata for ghcr.io/openfaas/of-watchdog:0.8.4
#4 sha256:a36b81af230f4aa9af61173e476a504ddf679531cc6fb84c21a9f883d305ce12
#4 DONE 0.2s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 ...

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 DONE 0.6s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 DONE 1.4s

#6 [stage-2 1/7] FROM docker.io/library/alpine:3.13@sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
#6 sha256:ac117f80319f3dfc5f3a89bed8fde570a7095ef011b4aa4f1832954fe5fae646
#6 DONE 0.0s

#10 [build  1/14] FROM docker.io/library/golang:1.15-alpine3.13@sha256:a025015951720f3227acd51b0a99a71578b574a63172ea7d2415c60ae5e2bc0a
#10 sha256:945f28b33676aa8e1e61782ed1c14580cb6577c9dae69198435ffaa8642d6acc
#10 DONE 0.0s

#12 [watchdog 1/1] FROM ghcr.io/openfaas/of-watchdog:0.8.4@sha256:9a0b595f77724005ef7336d68a5a6347426d1931c82f238916496f27d7ea2e74
#12 sha256:0b83d079b8d19e155cadf89ef4c3782ffd5a248d98dbd78ee94c18245505455c
#12 DONE 0.0s

#17 [internal] load build context
#17 sha256:22e4a759ec2fb76c5ae310d48e674f49bb5419d3eeca06c2b27c8987047be299
#17 transferring context: 496.45kB done
#17 DONE 0.0s

#20 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor .;     else     echo "using modules or vendor not found ";     fi
#20 sha256:59dd69b01bd2f54de6b035b838eaf41765d90591e42e1add8f2597a30122102d
#20 CACHED

#22 [build 11/14] WORKDIR /go/src/handler/function
#22 sha256:0f0ac30cb75d30ea26d73d4c9c2f1fd1e5b0deec9cdb447c7136b29d4beba8f2
#22 CACHED

#18 [build  7/14] COPY . .
#18 sha256:5f280c8982f74eb09a72e9966ab26deb656de679d2275697d355679904bac9c7
#18 CACHED

#16 [build  6/14] WORKDIR /go/src/handler
#16 sha256:2d57cb8be74105b3bf574dde9b3acfb2afb9ef7c3db0babaf2976e90bd189f0a
#16 CACHED

#23 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#23 sha256:db1d87d696d6f1822011c62aa89d515dadbf2eb0b3eb304223308a73644ce7ef
#23 CACHED

#8 [stage-2 3/7] RUN mkdir -p /home/app     && chown app /home/app
#8 sha256:392db714e3258ded26fd35e24dbf9da4b34d5f1b72bafa6651edb2c67c6e337a
#8 CACHED

#26 [stage-2 5/7] COPY --from=build --chown=app /go/src/handler/handler    .
#26 sha256:3d4993c2f2577fd2fb316fc0705cdf656ddb2108d79f88520308dcc9bc09d238
#26 CACHED

#14 [build  4/14] RUN chmod +x /usr/bin/fwatchdog
#14 sha256:0ce54538eaf598232623e0d0131601ae53ed928d280203d11398984ad325296d
#14 CACHED

#21 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#21 sha256:700ed50eb6b399e541605fae19f2ba57943b7925d06b01eb79436008cc735ea6
#21 CACHED

#27 [stage-2 6/7] COPY --from=build --chown=app /usr/bin/fwatchdog         .
#27 sha256:361a3b5061edb01c935a8e442bfb9bf1cbc4f162ce840bba5b7f2071c3f50c1c
#27 CACHED

#13 [build  3/14] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
#13 sha256:3d4b56eeab2f493ee25bcffa0a77d52bf329e8d5bcf1a7f4f9e60d76207b024c
#13 CACHED

#15 [build  5/14] RUN mkdir -p /go/src/handler
#15 sha256:49827b6b29acd15ceb89996f6858464d41ef41d45bf4439caa09ad778a541c9b
#15 CACHED

#11 [build  2/14] RUN apk --no-cache add git
#11 sha256:e895da5279e30155cbfeb55e93d54399f92e5b71ef96937c7955731c811f752e
#11 CACHED

#19 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#19 sha256:e8470771778a6d5753f72db5f421edc35846630310fc09596f8dae241fda872d
#19 CACHED

#7 [stage-2 2/7] RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
#7 sha256:f3920e39e6cb6756252ef8376a8a6a562e10f4156f6387b0e5aa5f3204bbbeb5
#7 CACHED

#9 [stage-2 4/7] WORKDIR /home/app
#9 sha256:8d07b104cc02b839e2f8b214eb64d1906c976d87c5178fc380b99ff4ee79f52d
#9 CACHED

#24 [build 13/14] WORKDIR /go/src/handler
#24 sha256:ff1630159ddb306a357f8bcae32cdabc0d1b3852179bfba444e47dba0b644aee
#24 CACHED

#25 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#25 sha256:ef4ce81a267fab2bf49828dc2184db924b15090ed4b8241a8172996473624d2e
#25 CACHED

#28 [stage-2 7/7] COPY --from=build --chown=app /go/src/handler/function/  .
#28 sha256:18d8c2e6684e3b4a3ead499390a5c901e4dd1438d59332d61b0889d5a412add7
#28 CACHED

#29 exporting to image
#29 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#29 exporting layers done
#29 writing image sha256:7374b12db800120b5c6de39d05bb1abfc3d72edb7b96b09cc22b0b25f72a606f done
#29 naming to docker.io/library/publisher:latest done
#29 DONE 0.0s
Image: publisher:latest built.
[0] < Building publisher done in 2.38s.
[0] Worker done.

Total build time: 2.38s

[alexellis]

What about with buildkit disabled?

[LucasRoesler]

@alexellis i will post the logs in a moment, but the docker docs are prtty clear about cache invalidation for RUN statements

https://docs.docker.com/engine/reference/builder/#:~:text=The%20cache%20for%20RUN%20instructions%20isn%E2%80%99t%20invalidated%20automatically%20during%20the%20next%20build

The cache for RUN instructions isn’t invalidated automatically during the next build. The cache for an instruction like RUN apt-get dist-upgrade -y will be reused during the next build. The cache for RUN instructions can be invalidated by using the --no-cache flag, for example docker build --no-cache.

The cache for RUN instructions can be invalidated by ADD and COPY instructions.

There is no reason to expect the if/else to cause a cache invalidation any more than the line immediately before it to copy the go mod replace statements.

[LucasRoesler]

Here are the logs when buildkit is disabled

DOCKER_BUILDKIT=0 faas-cli build -f publisher.yml --build-arg GO111MODULE=off
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
Sending build context to Docker daemon  569.9kB
Step 1/36 : FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.8.4 as watchdog
 ---> 4715559e6ad8
Step 2/36 : FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.15-alpine3.13 as build
 ---> b8d8ad7b4ab7
Step 3/36 : ARG TARGETPLATFORM
 ---> Using cache
 ---> e54de4e3e329
Step 4/36 : ARG BUILDPLATFORM
 ---> Using cache
 ---> 002f6d91ad3b
Step 5/36 : ARG TARGETOS
 ---> Using cache
 ---> 003ab4a52c89
Step 6/36 : ARG TARGETARCH
 ---> Using cache
 ---> 0978e1807ae2
Step 7/36 : RUN apk --no-cache add git
 ---> Using cache
 ---> fd831584b24f
Step 8/36 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
 ---> Using cache
 ---> ce43ffb34a27
Step 9/36 : RUN chmod +x /usr/bin/fwatchdog
 ---> Using cache
 ---> 26f87ae9e6d1
Step 10/36 : ENV CGO_ENABLED=0
 ---> Using cache
 ---> 03e5437e99fc
Step 11/36 : RUN mkdir -p /go/src/handler
 ---> Using cache
 ---> 89964a100c26
Step 12/36 : WORKDIR /go/src/handler
 ---> Using cache
 ---> 633c9382007b
Step 13/36 : COPY . .
 ---> Using cache
 ---> 8e51bfc34593
Step 14/36 : ARG GO111MODULE="off"
 ---> Using cache
 ---> 02efddd77e3d
Step 15/36 : ARG GOPROXY=""
 ---> Using cache
 ---> 638e050b3780
Step 16/36 : ARG GOFLAGS=""
 ---> Using cache
 ---> 2844cdb0943e
Step 17/36 : RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
 ---> Using cache
 ---> bdce1d761aa5
Step 18/36 : RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor .;     else     echo "using modules or vendor not found ";     fi
 ---> Using cache
 ---> ba70a5a38292
Step 19/36 : RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }
 ---> Using cache
 ---> ccf7a5efd4ab
Step 20/36 : WORKDIR /go/src/handler/function
 ---> Using cache
 ---> 89d0e8bc709e
Step 21/36 : RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
 ---> Using cache
 ---> d94f2ed7084d
Step 22/36 : WORKDIR /go/src/handler
 ---> Using cache
 ---> 59aac535b520
Step 23/36 : RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Using cache
 ---> 33eff2c16a04
Step 24/36 : FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.13
 ---> 28f6e2705743
Step 25/36 : RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
 ---> Using cache
 ---> 16d4eccf075b
Step 26/36 : RUN mkdir -p /home/app     && chown app /home/app
 ---> Using cache
 ---> 7d98c7f0dc19
Step 27/36 : WORKDIR /home/app
 ---> Using cache
 ---> c9a5f5a1b0c0
Step 28/36 : COPY --from=build --chown=app /go/src/handler/handler    .
 ---> Using cache
 ---> 9146e303eade
Step 29/36 : COPY --from=build --chown=app /usr/bin/fwatchdog         .
 ---> Using cache
 ---> 3f68b293e5be
Step 30/36 : COPY --from=build --chown=app /go/src/handler/function/  .
 ---> Using cache
 ---> 5f63bdb1bbc1
Step 31/36 : USER app
 ---> Using cache
 ---> 75ae534882fd
Step 32/36 : ENV fprocess="./handler"
 ---> Using cache
 ---> bb0f94d490d7
Step 33/36 : ENV mode="http"
 ---> Using cache
 ---> c51108fc209d
Step 34/36 : ENV upstream_url="http://127.0.0.1:8082"
 ---> Using cache
 ---> 114a47a9d7e2
Step 35/36 : ENV prefix_logs="false"
 ---> Using cache
 ---> 40b392698500
Step 36/36 : CMD ["./fwatchdog"]
 ---> Using cache
 ---> 2fa0b5044715
Successfully built 2fa0b5044715
Successfully tagged publisher:latest
Image: publisher:latest built.
[0] < Building publisher done in 2.99s.
[0] Worker done.

Total build time: 2.99s