-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
126 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
![CI run](https://github.com/dimikot/ci-storage/actions/workflows/ci.yml/badge.svg?branch=main) | ||
|
||
# CI Storage | ||
|
||
This action quickly stores the content of the work directory in the storage with | ||
the provided slot id on a remote host, or loads the content from the storage to | ||
that directory. The tool makes some smart differential optimizations along the | ||
way to operate as fast as possible (typically, 4 seconds for a 1.5G directory | ||
with 200K files in it on Linux EXT4). | ||
|
||
Under the hood, the tool uses rsync. When storing to the remote storage, it uses | ||
rsync's "--link-dest" mode pointing to the most recently created slot, to reuse | ||
as many existing files in the storage as possible (hoping that almost all files | ||
to be stored in the current slot are the same as the files in the recent slot, | ||
which is often times true for e.g. node_modules directories). If a slot with the | ||
same id already exists, it is overwritten in a transaction-safe fashion. | ||
|
||
When loading the files from a remote storage slot to a local directory, implies | ||
that the local directory already contains almost all files equal to the remote | ||
ones, so rsync can run efficiently. | ||
|
||
> [!NOTE] | ||
> | ||
> This tool makes sense only when using it with [Self-Hosted | ||
> Runners](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners). | ||
> The main idea is to reuse the previous build artifacts in work directories, | ||
> which is not possible with default GitHub-Hosted Runners (GitHub always boots | ||
> the virtual machines with empty work directories). | ||
# Usage | ||
|
||
<!-- start usage --> | ||
```yaml | ||
- uses: actions/ci-storage@v1 | ||
with: | ||
# What to do (store or load). | ||
# Required. | ||
action: '' | ||
|
||
# Storage host in the format [user@]host; it must have password-free | ||
# SSH key access. | ||
# Required. | ||
storage-host: '' | ||
|
||
# Storage directory on the storage host. | ||
# Default: "ci-storage/" in the storage host's home directory. | ||
storage-dir: '' | ||
|
||
# Remove slots created earlier than this many seconds ago. | ||
# Default: 14400 (4 hours) | ||
storage-max-age-sec: '' | ||
|
||
# Id of the slot to store to or load from; use "*" to load a random | ||
# most recent slot. | ||
# Default: "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" | ||
slot-id: '' | ||
|
||
# Local directory path to store from or load to. | ||
# Default: "." (the current work directory) | ||
local-dir: '' | ||
|
||
# Newline separated exclude pattern(s) for rsync. | ||
# Default: empty | ||
exclude: '' | ||
|
||
# If set, prints the list of transferred files. | ||
# Default: false | ||
verbose: '' | ||
``` | ||
<!-- end usage --> | ||
## Example: Build, then Store Work Directory in the Storage | ||
```yaml | ||
jobs: | ||
build: | ||
name: Build | ||
runs-on: [self-hosted] | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
- name: Build | ||
run: npm i && npm run build | ||
- name: Store build artifacts, work directory and .git directory | ||
uses: actions/ci-storage@v1 | ||
with: | ||
action: store | ||
storage-host: my-hostname.com | ||
test: | ||
# ... | ||
lint: | ||
# ... | ||
``` | ||
|
||
## Example: Load Work Directory from the Storage and Run Tests | ||
|
||
The benefit in speed can only be achieved if you use self-hosted Action Runners. | ||
In this case, the content of work directory will be reused across the jobs runs, | ||
and ci-storage action will be very quick in loading the files and build | ||
artifacts (it will skip most of unchanged files). | ||
|
||
```yaml | ||
jobs: | ||
build: | ||
# ... | ||
test: | ||
name: Test | ||
needs: build | ||
runs-on: [self-hosted] | ||
steps: | ||
- name: Load build artifacts, work directory and .git directory | ||
uses: actions/ci-storage@v1 | ||
with: | ||
action: load | ||
storage-host: my-hostname.com | ||
- name: Run tests | ||
run: npm run test | ||
lint: | ||
# ... | ||
``` | ||
|
||
# License | ||
|
||
The scripts and documentation in this project are released under the [MIT License](LICENSE). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters