Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[6.4.0] Add diff_against_dynamic_baseline option to experimental_output_direc… #19514

Merged
merged 3 commits into from
Sep 14, 2023
Merged

[6.4.0] Add diff_against_dynamic_baseline option to experimental_output_direc… #19514

merged 3 commits into from
Sep 14, 2023

Conversation

fmeum
Copy link
Collaborator

@fmeum fmeum commented Sep 13, 2023

…tory_naming_scheme.

To support, a new SkyFunction is introduced, BaselineOptionsFunction, with corresponding SkyValue and SkyKey. Also, a new function ExecutionTransitionFactory.createTransition was added to immediately create an ExecutionTransition.

When --experimental_output_directory_naming_scheme=diff_against_dynamic_baseline, the behavior is similar to diff_against_baseline, except when "is Exec" is true (that is, an ExecutionTransition has previously been applied), the baseline is chosen to be result of the ExecutionTransition being applied to the normal baseline.

This is meant to resolve an issue with performance when using remote execution engines that collate requests from multiple users. They were seeing varying output paths for actions in an execution configuration depending on how much 'resetting' the exec transition was doing because of varying user commandlines.

This relates to #14023 and #18002

PiperOrigin-RevId: 537097551
Change-Id: I72966406b7b8af0174a81b638bf0d1fb62466653

@fmeum fmeum requested a review from a team as a code owner September 13, 2023 22:36
@fmeum fmeum mentioned this pull request Sep 13, 2023
Closed
@iancha1992 iancha1992 changed the title Add diff_against_dynamic_baseline option to experimental_output_direc… [6.4.0] Add diff_against_dynamic_baseline option to experimental_output_direc… Sep 13, 2023
@iancha1992 iancha1992 added team-Configurability platforms, toolchains, cquery, select(), config transitions awaiting-review PR is awaiting review from an assigned reviewer labels Sep 13, 2023
@iancha1992 iancha1992 requested a review from sdtwigg September 13, 2023 23:07
@iancha1992 iancha1992 enabled auto-merge (squash) September 13, 2023 23:07
@sdtwigg
Copy link
Contributor

sdtwigg commented Sep 13, 2023 •
edited
Loading

Is this a backport from HEAD to an older release?

Edit: I just want some context b/c on first glance I'm getting deja vu.

@iancha1992 iancha1992 mentioned this pull request Sep 13, 2023
Closed
auto-merge was automatically disabled September 14, 2023 06:37

Head branch was pushed to by a user without write access

@fmeum fmeum force-pushed the release-6.4.0-diff branch from 76a4ad7 to 55d1d34 Compare September 14, 2023 06:37
@fmeum
Copy link
Collaborator Author

fmeum commented Sep 14, 2023

Is this a backport from HEAD to an older release?

Edit: I just want some context b/c on first glance I'm getting deja vu.

Yes, this is a cherry-pick of the diff_against_dynamic_baseline implementation to Bazel 6.4.0. This does not include the flip, just the implementation of the new option value so that users can flip themselves if needed.

fmeum
fmeum commented Sep 14, 2023
View reviewed changes
src/main/java/com/google/devtools/build/lib/skyframe/BaselineOptionsFunction.java
// Some test infrastructure only creates mock or partial top-level BuildOptions such that
// PlatformOptions or even CoreOptions might not be included.
// In that case, is not worth doing any special processing of the baseline.
if (!rawBaselineOptions.getFragmentClasses().contains(PlatformOptions.class)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On master this is rawBaselineOptions.hasNoConfig(), which was introduced by https://cs.opensource.google/bazel/bazel/+/ea049e46c84fdcef2e1acd83d07053b7fa0a5cb5. I "inlined" it based on the comment, but please let me know whether you would want me to cherry-pick the full function into BuildOptions.

@fmeum fmeum force-pushed the release-6.4.0-diff branch from 55d1d34 to 37a3232 Compare September 14, 2023 06:41
@fmeum
Add diff_against_dynamic_baseline option to experimental_output_direc…
09b3398 
…tory_naming_scheme.

To support, a new SkyFunction is introduced, BaselineOptionsFunction, with corresponding SkyValue and SkyKey. Also, a new function ExecutionTransitionFactory.createTransition was added to immediately create an ExecutionTransition.

When --experimental_output_directory_naming_scheme=diff_against_dynamic_baseline, the behavior is similar to diff_against_baseline, except when "is Exec" is true (that is, an ExecutionTransition has previously been applied), the baseline is chosen to be result of the ExecutionTransition being applied to the normal baseline.

This is meant to resolve an issue with performance when using remote execution engines that collate requests from multiple users. They were seeing varying output paths for actions in an execution configuration depending on how much 'resetting' the exec transition was doing because of varying user commandlines.

This relates to bazelbuild#14023 and bazelbuild#18002

PiperOrigin-RevId: 537097551
Change-Id: I72966406b7b8af0174a81b638bf0d1fb62466653
@fmeum fmeum force-pushed the release-6.4.0-diff branch from 37a3232 to 09b3398 Compare September 14, 2023 08:25
@iancha1992 iancha1992 enabled auto-merge (squash) September 14, 2023 17:31
@iancha1992 iancha1992 removed the awaiting-review PR is awaiting review from an assigned reviewer label Sep 14, 2023
@iancha1992 iancha1992 merged commit 448809e into bazelbuild:release-6.4.0 Sep 14, 2023
1 check passed
@fmeum fmeum deleted the release-6.4.0-diff branch September 14, 2023 20:46
copybara-service bot pushed a commit that referenced this pull request Oct 19, 2023
@iancha1992
Release 6.4.0 (2023-10-19)
844c1d6 
Baseline:  0f231ac

Release Notes:

+ Support multiple remote execution digest functions (#19042)
+ Release 6.4.0 remote (#18959)
+ Move BazelFileSystemModule into bazel package (#19043)
+ Fix a bug where frozen targets list was mutated while expanding env attribute (#19053)
+ Mark isolated extension usages as experimental (#19065)
+ Add the remote_require_cached flag (#19075)
+ Advertise CcInfo from cc_import (#19086) (#19088)
+ Update java_tools version to 12.6 (#19092)
+ Set the digest_function field as part of all relevant gRPC requests (#19049)
+ Merge `use_repo` buildifier fixups into a single command (#19134)
+ Ensure that extension unique names followed by `~` are prefix-free (#19164)
+ Lockfile updates & fixes (#19153)
+ Switch xcode_autoconf to use 'configure = True' (#19174)
+ Cherry pick Add a Starlark flag that allows disabling proguard. This will be useful for testing later. (#19179)
+ Update CODEOWNERS for 6.4.0 (#19194)
+ Friendlier error message for `bazel_dep`s without `version` (#19196)
+ Always check `$config_dependencies` visibility at use (#19197)
+ Add support for the BLAKE3 digest function (#19191)
+ Enable cc toolchain resolution when cross compiling to windows arm64. (#19198)
+ Ensure disk cache root exists (#19225)
+ Fix valid json when using jsonproto output in queries  with new `--ouput=streamed_jsonproto` implementation. (#19226)
+ Add toolchain type for Java bootstrap runtime (#19220)
+ Add Starlark implementation for several CcCommon methods. (#19076)
+ Rename `cc_test_wrapper` to `cc_test` (#19231)
+ Cherry-pick commits to fix a Windows issue (#19232)
+ Add support for more workspace boundary files to bash completion (#19281)
+ Use `debugPrint` instead of `str` for `fail` arguments (#19283)
+ Include name in `repr` of exported `rule`s (#19229)
+ Download `BazelRegistryJson` only once per registry (#19300)
+ Make module extension tag's `debugPrint` useful for error messages (#19285)
+ Intern repository mapping entries (#19293)
+ Add `additional_linker_inputs` option to `cc_library` rule (#19264)
+ Do not rerun module extensions when only imports or locations change (#19284)
+ Add profiling for Bzlmod operations (#19313)
+ Retry on javax.net.ssl.SSLException ... BAD_DECRYPT (#19346)
+ Fetch `RepoSpecs` in parallel (#19354)
+ Make `MODULE.bazel.lock` deterministic (#19370)
+ Ensure lockfile is updated after reset to pre-build state (#19371)
+ build-runfiles: remove temporary file prior to creating it (#19386)
+ Always fail on unknown attributes (#19404)
+ Ignore Starlark options on commands with `allowResidue = False` (#19417)
+ Separate PackageSpecificationProvider from its target (PackageGroupConfiguredTarget) (#19420)
+ Expose PackageSpecificationInfo provider as a top level symbol (#19422)
+ Revert "Report remote execution messages as events" (#19415)
+ [6.4] Add --incompatible_disable_objc_library_transition (#19393)
+ Create .bazelversion to address postsubmit timeout issues (#19435)
+ Add `contains` method inside `PackageSpecificationProvider`  (#19425)
+ Wrong include path to Clang 16 on Windows (#19430)
+ Simplify release notes by just printing the first line of the commit … (#19448)
+ Remove PackageGroupConfiguredTarget.isAvailableFor function (#19444)
+ Remove default -s flag from macOS libtool invocation (#19454)
+ Turn off lockfile feature by default (#19462)
+ Take the no-remote-exec tag into account when computing the action salt (#19457)
+ Add `--incompatible_merge_fixed_and_default_shell_env` (#19319)
+ Improve error when a label is provided in `config_setting`'s `values` (#19484)
+ Mark tool inputs in the execution log. (#19483)
+ Add visionOS support (#19436)
+ Intern empty `Depset`s (#19443)
+ Do not allow applicable_licenses on platform. (#19426)
+ Cherry pick Bzlmod fixes (#19494)
+ Optimize classpath pre-processing in java_stub_template.txt (#19491)
+ Add output name to CacheNotFoundException (#19452)
+ feat: add option to exit early if analysis cache is discarded (#19503)
+ Cherry pick platform dependent lockfile (#19498)
+ Print Passed and Failed methods in detailed test summary (#19505)
+ Add formatted timestamp entries to volatile workspace status file. (#19499)
+ Write an explicit line ending to the lockfile (#19519)
+ Only use `/showIncludes` if supported (#19521)
+ Also apply `NestedSet` optimizations to `Depset` (#19492)
+ Add diff_against_dynamic_baseline option to experimental_output_direc… (#19514)
+ Update java_tools to v12.7 (#19522)
+ Advertise CcInfo from cc_proto_library (#19534)
+ Update unknown Xcode version error message and provide an environment variable to force re-evaluation (#19540)
+ Print dep chain leading to a module that is in error (#19543)
+ Show fetch progress for the `mod` command (#19542)
+ Revert "Switch xcode_autoconf to use 'configure = True' (#19174)" (#19550)
+ Explain the use of `str(Label(...))` in the docs (#19554)
+ Add `--consistent_labels` flag to all query commands (#19567)
+ Inject builtin modules at the end of the MODULE.bazel file (#19573)
+ Disable bzlmod_query_test for RBE build (#19585)
+ Clear runfiles environment variables for `bazel run` (#19606)
+ cc_library: propagate data dependencies via implementation_deps. (#19590)
+ Error on potential unsupported `/showIncludes` lines (#19611)
+ Use case-insensitive comparison for Windows paths in `runfiles.bash`  (#19626)
+ Show test labels in summaries in display form (#19625)
+ Remove stale extension entries from lockfile (#19683)
+ Keep leading zero in formatted date (#19694)
+ Bzlmod lockfile: fix pretty printing for attributes  (#19691)
+ MODULE.bazel.lock file contains user specific paths (#19698)
+ Consider RCs equivalent to release for `bazel_compatibility` (#19689)
+ Use `Label` in `@bazel_tools//tools/jdk` macros (#19675)
+ Remove stale extension entries from lockfile if module order changes (#19730)
+ Update rules_java 5.5.1 (#19701)
+ Fix Java compilation for jdk21 (#19735)
+ Fix output materialized as symlink when building without the bytes. (#19739)
+ Make lockfile's `RepoSpec` attributes more readable (#19748)
+ Merge rule and aspect validation output groups (#19745)
+ Fix handling of non-ASCII characters in archive entry file names (#19765)
+ Raise an early error on invalid labels in transitions inputs/outputs (#19764)
+ Handle synthetic method parameters entries that don't have names (#19758)
+ Bazel release process: Fix push justification. (#19768)
+ Flip --experimental_cc_implementation_deps (#19751)
+ Add blake3 NEON instructions on linux arm64 (#19804)
+ Fix crash when `environ` contains duplicate entries (#19827)
+ Bump c++ standard to c++14 per default (#19794)
+ Collect debug info context from implementation deps (#19836)
+ Fix unconditional Skyframe invalidation with --lockfile_mode=… (#19848)

Acknowledgements:

This release contains contributions from many people at Google, as well as Andreas Herrmann, bazel.build machine account, Brentley Jones, buildbreaker2021, Chirag Ramani, David Ostrovsky, Ed Schouten, Fabian Meumertzheim, George Gensure, Greg, John Laxson, Julio Merino, Keith Smiley, Matt Mackay, Mauricio G, NelsonLi0701, nglevin, Nicholas Junge, oquenchil, Orion Hodson, Roman Salvador, Ted Kaplan, Thi Doan, Thi Don, Tyler Williams, Xùdōng Yáng.
tdibattista pushed a commit to SibrosTech/bazel that referenced this pull request Oct 19, 2023
@iancha1992
Release 6.4.0 (2023-10-19)
50b61e3 
Baseline:  0f231ac

Release Notes:

+ Support multiple remote execution digest functions (bazelbuild#19042)
+ Release 6.4.0 remote (bazelbuild#18959)
+ Move BazelFileSystemModule into bazel package (bazelbuild#19043)
+ Fix a bug where frozen targets list was mutated while expanding env attribute (bazelbuild#19053)
+ Mark isolated extension usages as experimental (bazelbuild#19065)
+ Add the remote_require_cached flag (bazelbuild#19075)
+ Advertise CcInfo from cc_import (bazelbuild#19086) (bazelbuild#19088)
+ Update java_tools version to 12.6 (bazelbuild#19092)
+ Set the digest_function field as part of all relevant gRPC requests (bazelbuild#19049)
+ Merge `use_repo` buildifier fixups into a single command (bazelbuild#19134)
+ Ensure that extension unique names followed by `~` are prefix-free (bazelbuild#19164)
+ Lockfile updates & fixes (bazelbuild#19153)
+ Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174)
+ Cherry pick Add a Starlark flag that allows disabling proguard. This will be useful for testing later. (bazelbuild#19179)
+ Update CODEOWNERS for 6.4.0 (bazelbuild#19194)
+ Friendlier error message for `bazel_dep`s without `version` (bazelbuild#19196)
+ Always check `$config_dependencies` visibility at use (bazelbuild#19197)
+ Add support for the BLAKE3 digest function (bazelbuild#19191)
+ Enable cc toolchain resolution when cross compiling to windows arm64. (bazelbuild#19198)
+ Ensure disk cache root exists (bazelbuild#19225)
+ Fix valid json when using jsonproto output in queries  with new `--ouput=streamed_jsonproto` implementation. (bazelbuild#19226)
+ Add toolchain type for Java bootstrap runtime (bazelbuild#19220)
+ Add Starlark implementation for several CcCommon methods. (bazelbuild#19076)
+ Rename `cc_test_wrapper` to `cc_test` (bazelbuild#19231)
+ Cherry-pick commits to fix a Windows issue (bazelbuild#19232)
+ Add support for more workspace boundary files to bash completion (bazelbuild#19281)
+ Use `debugPrint` instead of `str` for `fail` arguments (bazelbuild#19283)
+ Include name in `repr` of exported `rule`s (bazelbuild#19229)
+ Download `BazelRegistryJson` only once per registry (bazelbuild#19300)
+ Make module extension tag's `debugPrint` useful for error messages (bazelbuild#19285)
+ Intern repository mapping entries (bazelbuild#19293)
+ Add `additional_linker_inputs` option to `cc_library` rule (bazelbuild#19264)
+ Do not rerun module extensions when only imports or locations change (bazelbuild#19284)
+ Add profiling for Bzlmod operations (bazelbuild#19313)
+ Retry on javax.net.ssl.SSLException ... BAD_DECRYPT (bazelbuild#19346)
+ Fetch `RepoSpecs` in parallel (bazelbuild#19354)
+ Make `MODULE.bazel.lock` deterministic (bazelbuild#19370)
+ Ensure lockfile is updated after reset to pre-build state (bazelbuild#19371)
+ build-runfiles: remove temporary file prior to creating it (bazelbuild#19386)
+ Always fail on unknown attributes (bazelbuild#19404)
+ Ignore Starlark options on commands with `allowResidue = False` (bazelbuild#19417)
+ Separate PackageSpecificationProvider from its target (PackageGroupConfiguredTarget) (bazelbuild#19420)
+ Expose PackageSpecificationInfo provider as a top level symbol (bazelbuild#19422)
+ Revert "Report remote execution messages as events" (bazelbuild#19415)
+ [6.4] Add --incompatible_disable_objc_library_transition (bazelbuild#19393)
+ Create .bazelversion to address postsubmit timeout issues (bazelbuild#19435)
+ Add `contains` method inside `PackageSpecificationProvider`  (bazelbuild#19425)
+ Wrong include path to Clang 16 on Windows (bazelbuild#19430)
+ Simplify release notes by just printing the first line of the commit … (bazelbuild#19448)
+ Remove PackageGroupConfiguredTarget.isAvailableFor function (bazelbuild#19444)
+ Remove default -s flag from macOS libtool invocation (bazelbuild#19454)
+ Turn off lockfile feature by default (bazelbuild#19462)
+ Take the no-remote-exec tag into account when computing the action salt (bazelbuild#19457)
+ Add `--incompatible_merge_fixed_and_default_shell_env` (bazelbuild#19319)
+ Improve error when a label is provided in `config_setting`'s `values` (bazelbuild#19484)
+ Mark tool inputs in the execution log. (bazelbuild#19483)
+ Add visionOS support (bazelbuild#19436)
+ Intern empty `Depset`s (bazelbuild#19443)
+ Do not allow applicable_licenses on platform. (bazelbuild#19426)
+ Cherry pick Bzlmod fixes (bazelbuild#19494)
+ Optimize classpath pre-processing in java_stub_template.txt (bazelbuild#19491)
+ Add output name to CacheNotFoundException (bazelbuild#19452)
+ feat: add option to exit early if analysis cache is discarded (bazelbuild#19503)
+ Cherry pick platform dependent lockfile (bazelbuild#19498)
+ Print Passed and Failed methods in detailed test summary (bazelbuild#19505)
+ Add formatted timestamp entries to volatile workspace status file. (bazelbuild#19499)
+ Write an explicit line ending to the lockfile (bazelbuild#19519)
+ Only use `/showIncludes` if supported (bazelbuild#19521)
+ Also apply `NestedSet` optimizations to `Depset` (bazelbuild#19492)
+ Add diff_against_dynamic_baseline option to experimental_output_direc… (bazelbuild#19514)
+ Update java_tools to v12.7 (bazelbuild#19522)
+ Advertise CcInfo from cc_proto_library (bazelbuild#19534)
+ Update unknown Xcode version error message and provide an environment variable to force re-evaluation (bazelbuild#19540)
+ Print dep chain leading to a module that is in error (bazelbuild#19543)
+ Show fetch progress for the `mod` command (bazelbuild#19542)
+ Revert "Switch xcode_autoconf to use 'configure = True' (bazelbuild#19174)" (bazelbuild#19550)
+ Explain the use of `str(Label(...))` in the docs (bazelbuild#19554)
+ Add `--consistent_labels` flag to all query commands (bazelbuild#19567)
+ Inject builtin modules at the end of the MODULE.bazel file (bazelbuild#19573)
+ Disable bzlmod_query_test for RBE build (bazelbuild#19585)
+ Clear runfiles environment variables for `bazel run` (bazelbuild#19606)
+ cc_library: propagate data dependencies via implementation_deps. (bazelbuild#19590)
+ Error on potential unsupported `/showIncludes` lines (bazelbuild#19611)
+ Use case-insensitive comparison for Windows paths in `runfiles.bash`  (bazelbuild#19626)
+ Show test labels in summaries in display form (bazelbuild#19625)
+ Remove stale extension entries from lockfile (bazelbuild#19683)
+ Keep leading zero in formatted date (bazelbuild#19694)
+ Bzlmod lockfile: fix pretty printing for attributes  (bazelbuild#19691)
+ MODULE.bazel.lock file contains user specific paths (bazelbuild#19698)
+ Consider RCs equivalent to release for `bazel_compatibility` (bazelbuild#19689)
+ Use `Label` in `@bazel_tools//tools/jdk` macros (bazelbuild#19675)
+ Remove stale extension entries from lockfile if module order changes (bazelbuild#19730)
+ Update rules_java 5.5.1 (bazelbuild#19701)
+ Fix Java compilation for jdk21 (bazelbuild#19735)
+ Fix output materialized as symlink when building without the bytes. (bazelbuild#19739)
+ Make lockfile's `RepoSpec` attributes more readable (bazelbuild#19748)
+ Merge rule and aspect validation output groups (bazelbuild#19745)
+ Fix handling of non-ASCII characters in archive entry file names (bazelbuild#19765)
+ Raise an early error on invalid labels in transitions inputs/outputs (bazelbuild#19764)
+ Handle synthetic method parameters entries that don't have names (bazelbuild#19758)
+ Bazel release process: Fix push justification. (bazelbuild#19768)
+ Flip --experimental_cc_implementation_deps (bazelbuild#19751)
+ Add blake3 NEON instructions on linux arm64 (bazelbuild#19804)
+ Fix crash when `environ` contains duplicate entries (bazelbuild#19827)
+ Bump c++ standard to c++14 per default (bazelbuild#19794)
+ Collect debug info context from implementation deps (bazelbuild#19836)
+ Fix unconditional Skyframe invalidation with --lockfile_mode=… (bazelbuild#19848)

Acknowledgements:

This release contains contributions from many people at Google, as well as Andreas Herrmann, bazel.build machine account, Brentley Jones, buildbreaker2021, Chirag Ramani, David Ostrovsky, Ed Schouten, Fabian Meumertzheim, George Gensure, Greg, John Laxson, Julio Merino, Keith Smiley, Matt Mackay, Mauricio G, NelsonLi0701, nglevin, Nicholas Junge, oquenchil, Orion Hodson, Roman Salvador, Ted Kaplan, Thi Doan, Thi Don, Tyler Williams, Xùdōng Yáng.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sdtwigg sdtwigg sdtwigg approved these changes

Assignees
No one assigned
Labels
team-Configurability platforms, toolchains, cquery, select(), config transitions
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fmeum @sdtwigg @iancha1992