Inefficient Builds (15 mins) #461
Comments
krisnova
added
the
Good First Issue
|
Was able to get a 30 minute build.. ouch |
|
naive attempt: unfortunately this build fails due to tonic generating code that fails the nightly clippy checks. adding an
works but only for the final crate. let's go deeper: oh look. a 2.4 GB JSON file for the chrome profiler to play with. which it rightly refused to load. increasing the minimum duration to summary: so the slowness is likely that we have:
each of which define their own set of dependencies that need checking, maybe downloading, and maybe building. if each takes 2mins, we're already over 30mins. naively i'd expect that if we merge these into a single crate we'd end up spending less time dealing with dependencies but having less flexibility to build things individually. |
The guidance for Rust is usually to break projects up into small crates since crates that don't depend on each other can be built in parallel. By looking at an action that missed the cache and picking a single dependency (I'm picking The following log lines appear for serde: cargo build --target x86_64-unknown-linux-musl -p auraed
Downloaded serde v1.0.152
Compiling serde v1.0.152
...
cargo build -p auraescript
Compiling serde v1.0.152
...
cargo build -p aer
Compiling serde v1.0.152
...
cargo clippy --all-features --workspace --exclude auraed --exclude auraescript --exclude aer -- -D clippy::all -D warnings
Checking serde v1.0.152
...
cargo clippy --target x86_64-unknown-linux-musl -p auraed --all-features -- -D clippy::all -D warnings
Checking serde v1.0.152
...
cargo clippy -p auraescript --all-features -- -D clippy::all -D warnings
Checking serde v1.0.152
...
cargo clippy -p aer --all-features -- -D clippy::all -D warnings
# serde does not get checked even though `aer` has a dependency on it
...
cargo install --target x86_64-unknown-linux-musl --path ./auraed --debug --force
Downloaded serde v1.0.158
Compiling serde v1.0.158
...
cargo install --path ./auraescript --debug --force
Compiling serde v1.0.158Notes:
It looks like we can eliminate two
It is also possible that we can add |
|
TL;DR: We should discuss why, what, and when we run certain CI commands. More to the point, here are some potential places we can improve the build times:
A script for replicating the analysis from this post can be found at the bottom. AnalysisFrom my initial investigation, it seems that the most timing-consuming operations are dependency management (particularly on fresh builds, as you might expect), the tests, and linking. Preliminary detailsDuring this investigation, it became apparent that I didn't understand cargo's caching mechanism and future-highway's comment confirmed it. I couldn't figure out how to handle the cargo build cache across the aurae workspace, so I'll be using sccache. This analysis was performed on a qemu Ubuntu VM with 4 vCPUs from a i9-9980HK and 16GBs of memory allocated to the VM. Results and notesCargo test timingsResults from Results from Results from Linker timingsLinking time with the default linker: Linking time with Linking time with An important note regarding linkers is that cargo will create a binary for each integration test file, which requires the whole linking rigamarole, for each test file. This is why I included the Another important note is that I couldn't get Better cachingAs I mentioned in the beginning, I don't quite understand how cargo caches the dependencies across a workspace, but at this point, I suspect it doesn't. We might be able to figure out a better keying method for our current caching mechanism in our CI, or swap to sccache after better understanding how all these caches work. Putting it all togetherThis isn't in the script below. Here's the timing after implementing the caching with sccache, linking via lld, and combining the integration tests (showing all 5s+ units): Total time: 201.1s (3m 21.1s) The "what," "why," and "when" in aurae's CIA great constructive exercise we can do as part of this ticket is to discuss how we're using CI. I was watching Nova's stream when this ticket was made, and it seemed like the root problem was that Nova couldn't get a
ScriptAssumes you have followed the aurae build instructions, have
progs=(
"aer"
"auraed"
"auraescript"
)
linkers=(
"lld"
"mold"
)
echo "INFO: Running analysis"
mkdir -p profiling
echo "INFO: Cleaning"
make clean-gen clean-crates
echo "INFO: Finished cleaning"
echo "INFO: First run of cargo test --workspace --timings (no cache)"
cargo test --workspace --timings
mv target/cargo-timings/cargo-timing.html profiling/cargo-timing-no-cache.html
echo "INFO: Done with first run of cargo test"
echo "INFO: Second run of cargo test --workspace --timings (using cache)"
cargo test --workspace --timings
mv target/cargo-timings/cargo-timing.html profiling/cargo-timing-with-cache.html
echo "INFO: Done with second run of cargo test"
echo "INFO: Third run of cargo test --workspace --timings (using lld and cache)"
RUSTFLAGS="-Clinker=clang -Clink-arg=-fuse-ld=lld" cargo test --workspace --timings
mv target/cargo-timings/cargo-timing.html profiling/cargo-timing-lld-with-cache.html
echo "INFO: Done with third run of cargo test"
for prog in "${progs[@]}"; do
echo "INFO: Running $prog + default"
cargo +nightly rustc -p "$prog" --bins -- -Z time-passes &> "profiling/$prog-default.txt"
echo "INFO: Done with $prog + default"
done
for linker in "${linkers[@]}"; do
for prog in "${progs[@]}"; do
echo "INFO: Running $prog + $linker"
RUSTFLAGS="-Clinker=clang -Clink-arg=-fuse-ld=$linker" cargo +nightly rustc -p "$prog" --bins -- -Z time-passes &> "profiling/$prog-$linker.txt"
echo "INFO: Done with $prog + $linker"
done
done
echo "INFO: Results for timings with no cache can be found at $(pwd)/profiling/cargo-timing-no-cache.html"
echo "INFO: Results for timings with cache can be found at $(pwd)/profiling/cargo-timing-with-cache.html"
for prog in "${progs[@]}"; do
echo "INFO: Results for $prog + default time-passes can be found at $(pwd)/profiling/$prog-default.txt"
done
for linker in "${linkers[@]}"; do
for prog in "${progs[@]}"; do
echo "INFO: Results for $prog + $linker time-passes can be found at $(pwd)/profiling/$prog-$linker.txt"
done
done
echo "INFO: View in browser via file://$(pwd)/profiling" |
|
my builds are still slow, but i'm not sure what we can do. maybe we should think about breaking out the monorepo... |
I think there is opportunity for newcomers to the project to help with our current build times.
Right now our current builds can take up to 15 minutes, and I believe there is a lot of repeated and duplicated work. The more compiling and downloading and linking commands we do, the longer the builds.
How do we get build times down to just a few seconds? Maybe less than 60 seconds for the entire project?
The text was updated successfully, but these errors were encountered: