Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching docker directory to speed up image loading #22

Open
fenech opened this issue Aug 28, 2019 · 3 comments
Open

Caching docker directory to speed up image loading #22

fenech opened this issue Aug 28, 2019 · 3 comments

Comments

@fenech
Copy link
Contributor

fenech commented Aug 28, 2019

Hi, I have a Concourse build task that looks like

#!/bin/bash

source /docker-lib.sh
start_docker

docker load -i image1.tar
docker load -i image2.tar
docker load -i image3.tar

docker-compose up

The problem is that some of the images are quite large and this makes the task slow to run.

I thought that it might be possible to cache /scratch/docker in my build configuration:

caches: [{path: /scratch/docker}]

But it seems that this doesn't have any effect - docker images returns an empty list before I run docker load.

How can I cache the images loaded in previous builds to speed up my build task?

@dsawa
Copy link

dsawa commented Jun 5, 2020

@fenech Have you managed to find some kind of workaround ?
I have similar problem now.

@fenech
Copy link
Contributor Author

fenech commented Jun 5, 2020

@dsawa No, I couldn't find a way to persist the state between tasks runs using this image.

In the end I decided to change my approach: instead of using this task image, I SSH into a VM with Docker running there and pull the images rather than load them, so that layer caching can speed things up.

@georggrab
Copy link

Note that the concourse documention on caches states:

The path to a directory to be cached.

Paths are relative to the working directory of the task. Absolute paths are not respected.

I usually set up caches in concourse like this:

script in the task amounting to (.sbt is just an example directory):

rm -rf ~/.sbt
ln -fs sbt-cache ~/.sbt

cache configuration in the task declaration

caches:
- path: sbt-cache

Perhaps you could confirm if that approach works for you (applied to the directory /scratch/docker)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants