Skip to content

Commit

Permalink
Updated fetch_shas utility to use the channel manifest tomls (#2840)
Browse files Browse the repository at this point in the history
Updating shas was taking much longer than it used to. I've refactored
the shell script into python so to manage increased complexity. The
script now fetches the `channel-rust-*.toml` files which contains
additional sha256 values. This script still falls back to brute force
fetching of artifacts to account for any that are missing. The runtime
was reduced from ~20 min to ~80s.

```bash
python3.11 ./util/fetch_shas/fetch_shas.py
```
```
2024-09-06 08:57:22 - INFO - Fetching known sha256 data...
2024-09-06 08:57:22 - INFO - Downloading data...
2024-09-06 08:57:29 - INFO - Done.
2024-09-06 08:57:29 - INFO - Deserializing 119 tomls...
2024-09-06 08:57:36 - INFO - Done.
2024-09-06 08:57:36 - INFO - Parsing artifacts...
2024-09-06 08:57:36 - INFO - Done. Identified 52790 artifacts.
2024-09-06 08:57:36 - INFO - Checking for 81514 missing artifacts...
2024-09-06 08:58:26 - INFO - Done.
2024-09-06 08:58:29 - INFO - Downloading 404 missing artifacts...
2024-09-06 08:58:29 - INFO - Done.
2024-09-06 08:58:36 - INFO - Done. Wrote rust/known_shas.bzl
```

Note that `beta` sha256 values have been removed in this change.
  • Loading branch information
UebelAndre authored Sep 10, 2024
1 parent 42f327f commit dca03c9
Show file tree
Hide file tree
Showing 11 changed files with 4,128 additions and 1,610 deletions.
5,243 changes: 3,738 additions & 1,505 deletions rust/known_shas.bzl

Large diffs are not rendered by default.

6 changes: 0 additions & 6 deletions util/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
sh_binary(
name = "fetch_shas",
srcs = ["fetch_shas.sh"],
tags = ["manual"],
)

alias(
name = "collect_coverage",
actual = "//util/collect_coverage",
Expand Down
95 changes: 0 additions & 95 deletions util/fetch_shas.sh

This file was deleted.

14 changes: 14 additions & 0 deletions util/fetch_shas/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
name = "fetch_shas",
srcs = ["fetch_shas.py"],
data = [
"fetch_shas_HOST_TOOLS.txt",
"fetch_shas_NIGHTLY_ISO_DATES.txt",
"fetch_shas_TARGETS.txt",
"fetch_shas_TOOLS.txt",
"fetch_shas_VERSIONS.txt",
],
tags = ["manual"],
)
Loading

0 comments on commit dca03c9

Please sign in to comment.