Docker build log should be printed when docker build fails

[jie-lyu]

Currently, when CI=true, when docker build fails in copilot svc package, only

upload resources required for deployment for frontend: build and push the image "frontend": build Dockerfile at /path/to/Dockerfile: building image: exit status 1

is printed. AFAIK, there is no method to investigate what really is happening.

When CI != true, a few more lines is printed, but still not enough information is provided.

[iamhopaul123]

Hello @jie17. This is a very good call. We are prioritizing this enhancement now.

[geoff-mwi]

Is there any workaround either by addons, overrides, forks or build emulation that can expose the Docker build messages/errors?

Our pipeline intermittently fails on the Docker build step (even though local Docker build works perfectly) and similar to @jie17 we have no way of understanding what went wrong.

For our case, rerunning our pipeline shortly after with no changes results in a failure. Waiting until the next day and rerunning the pipeline again results in a success. My assumption for our particular error is there is some rate limiting happening maybe with docker pull.

The docker build messages are also outputted as if being run interactively which is not ideal. If there is an enhancement to output the docker build messages, would it be possible to change the docker build progress output to plain instead of tty?

[KollaAdithya]

Hello @geoff-mwi !

one way I can think of is to see if the docker build is successful before deploying the service. i.e;
you can run docker build . -f Dockerfile. Even though it adds an extra burden on your side.

We are currently working on to solve this issue on docker build output. I had a PR(5148) that will solve the issue if you just have single main container image in the manifest. This will show the output of docker build if it fails as well.

In case of multiple container images i.e (sidecars + main container) images issue still persists.

Regarding the docker pull rate issue, you can refer to this workaround that will help you solve this issue.

would it be possible to change the docker build progress output to plain instead of tty

If you want to have docker build output to be plain you need to set the CI=true.

I also had a question which version of copilot are you currently using?

[geoff-mwi]

Hi @KollaAdithya, thank you for the response.

docker build . -f Dockerfile and copilot svc deploy succeed locally and Dockerfile hasn't changed since the copilot pipeline was created. The pipeline build step often succeeds but intermittently fails. This is why it's unfortunate not to see the Docker build errors since the problem lies within the codebuild environment, copilot binary or the buildspec.yml.

What we see is an exit 1 from Docker build within the copilot-linux svc package ... command and stdout from Docker that stops on STEP 1/17 which is why I'm assuming that docker pull might be failing - but this is just assumption without seeing the messages. We use containers available on the Amazon ECR Public Gallery so I'm not sure it really is the rate limiting.

We use an unchanged buildspec.yml template as created by copilot pipeline init in v1.28.0. I have noticed that v1.29.0 has been released and I will update the buildspecs accordingly.

With regards to CI=true, I see now that in v1.29.0 CI=true has been added to the buildspec.yml template which makes the output a lot cleaner, thank you.

Luckily we have a local/manual workaround via copilot svc deploy and we don't use sidecars so we're looking forward to your PR(#5148) being released so that we can debug this :)

[dannyrandall]

Hey all! We just released a fix for this issue in v1.30.0!🚀🎉 Services with a single image no longer have the scrolling output, so you should be able to see errors from docker easier. We'll track the issue for multiple image builds on #5067.