Skip to content

Add Docker BuildKit to CI and test pip caching #934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

dshemetov
Copy link
Contributor

@dshemetov dshemetov commented Jun 18, 2022

Closes #933.

Prerequisites:

  • Unless it is a documentation hotfix it should be merged against the dev branch
  • Branch is up-to-date with the branch to be merged with, i.e. dev
  • Build is successful
  • Code is cleaned up and formatted

Summary

This is a test to see if we can use Docker BuildKit in CI.

@dshemetov dshemetov force-pushed the ds/ci-docker-buildkit branch 2 times, most recently from 5c3d7e6 to a27de83 Compare June 18, 2022 01:24
* Add mount=type=cache to Dockerfiles
* Add actions/cache for pip to CI
@dshemetov dshemetov force-pushed the ds/ci-docker-buildkit branch from a27de83 to e39eae5 Compare June 18, 2022 01:45
@dshemetov dshemetov marked this pull request as draft June 18, 2022 06:03
@dshemetov
Copy link
Contributor Author

dshemetov commented Jun 29, 2022

This is currently complicated by the interaction between actions/cache in CI and --mount=type=cache,target=/root/.cache/pip in Docker. It is unclear where Docker stores its cache, but it is certainly not placing those files in root/.cache/pip.

Todo:

  • try using actions/cache to cache /var/lib/docker/buildkit/cache.db
  • try following the instructions here

Refs:

  • Someone found an approach to this, but it's not clean and uses advanced Docker features I'd rather not touch.
  • This blog post explains more about how Docker mount cache works in the context of pip, but doesn't use it in a Github CI context.
  • In case you need to figure out where your Python packages are stored (this is not really relevant, since we are really concerned about where Docker's cache files are, we already know that /root/.cache is the right pip artifacts directory from the blog post above).
  • The RUN --mount command reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cache Python packages in CI Docker runs
1 participant