Add model/dataset cache to GH actions #26
Conversation
|
Amazing, thanks! Can you cache dependencies as well? Currently they're the bottleneck I believe. |
|
bizarre that README got formatted by something. Did you use something else than text editor? |
|
Ah, I think it was Prettier. I've disabled it for this workspace. |
Added via the integrated caching functionality in |
|
So good news/bad news: Good news: This does work, pip packages are saved and don't need to be downloaded again. First run writes to cache, second run reads from cache. Bad news: This only saves about 30 seconds, most of the time is spent on the actual install Good news: It's theoretically possible to save the post-install state, see https://archive.ph/WAex2 Bad news: This potentially breaks a lot of integrations by Doing Things In An Unexpected Way. And discussion of the relevant open issue on the action itself suggests that people haven't really gotten too much benefit out of it: actions/setup-python#330 Tentative conclusion: Take the small win and revisit later if and only if this is enough of a blocker that we're willing to dedicate a lot of person-hours to improving it instead of working on something else. |
This PR enables caching for huggingface models/datasets (#1). It does require users to manually increment a cache identifier when we want to change what's going into the cache, since I couldn't think of a good way to automatically check what will be needed in the pipeline ahead of time. Instructions have been added to README.md.