fix: make sys.executable work with script bootstrap

[rickeylev]

When --bootstrap_impl=script is used, PYTHONPATH is no longer used to set the import
paths, which means subprocesses no longer inherit the Bazel paths. This is generally a
good thing, but breaks when sys.executable is used to directly invoke the interpreter.
Such an invocation assumes the interpreter will have the same packages available and works
with the system_python bootstrap.

To fix, have the script bootstrap use a basic virtual env. This allows it to intercept
interpreter startup even when the Bazel executable isn't invoked. Under the hood, there's
two pieces to make this work. The first is a binary uses declare_symlink() to write a
relative-path based symlink that points to the underlying Python interpreter. The second
piece is a site init hook (triggered by a .pth file using an import line) performs
sys.path setup as part of site (import site) initialization.

Fixes #2169

[aignas]

FYI, I am curious if the next version of this and #2359 combined could make it easy to have a venv for any target in the monorepo. This would mimic what aspect has built for their users - a venv for any target so that IDE integration works smoothly.

[rickeylev]

FYI, I am curious if the next version of this and #2359 combined could make it easy to have a venv for any target in the monorepo. This would mimic what aspect has built for their users - a venv for any target so that IDE integration works smoothly.

I'm not sure I follow. With this PR, each binary will have its own venv, so an editor could be pointed to that directly. Do you mean something like, you do a one time setup to point your editor to some "well known path", and then run bazel run @rules_python//tools/whatever --binary=//my:bin to "populate" that well known path? And thus, you can "switch" between the venvs your editor sees by running the bazel command?

[aignas]

I think you understood it.

I was thinking that you could do either:

bazel run @rules_python//python:bin/activate --binary=//my:bin
vim

or tell VSCode or similar that the Python interpreter for //my:bin is in $(bazel run --run_under=echo @rules_python//python:bin/python --binary=//my:bin).

We could also symlink it to a well known location using https://github.com/buildbuddy-io/bazel_env.bzl

[rickeylev]

Ok, ready for review

[rickeylev]

Addressed most, still have 1 or 2 left. Thanks!

[aignas]

OK, LGTM in general.

[ewianda]

❤️

[ewianda]

hi @rickeylev this doesn't seem to be working, Not sure if it is my system or what.

~/projects/rules_python/examples/bzlmod (main =)
$ bazelisk test //tests:version_default_test  --@rules_python//python/config_settings:bootstrap_impl=script --cache_test_results=no
INFO: Invocation ID: ac4e384e-27c6-47c2-b5a6-6f6b35eb16e8
WARNING: Build option --enable_runfiles has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //tests:version_default_test (0 packages loaded, 3996 targets configured).
FAIL: //tests:version_default_test (see /home/ewianda/.cache/bazel/_bazel_ewianda/229007d1af36718f54c73697ae3aff83/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/version_default_test/test.log)
INFO: From Testing //tests:version_default_test:
==================== Test output for //tests:version_default_test:
ERROR: Python interpreter not found: /home/ewianda/.cache/bazel/_bazel_ewianda/229007d1af36718f54c73697ae3aff83/sandbox/linux-sandbox/47/execroot/_main/bazel-out/k8-fastbuild/bin/tests/version_default_test.runfiles/_main/tests/_version_default_test.venv/bin/python3
================================================================================
INFO: Found 1 test target...
Target //tests:version_default_test up-to-date:
  bazel-bin/tests/version_default_test
INFO: Elapsed time: 0.343s, Critical Path: 0.10s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//tests:version_default_test                                             FAILED in 0.0s

[ewianda]

Hmm actually it fails on WSL Ubuntu and is fine on a regular Linux Machine. Says a broken symbolic link

file  /home/ewianda/.cache/bazel/_bazel_ewianda/6091a940c516b5ac88940b423928af22/sandbox/linux-sandbox/7/execroot/_main/bazel-out/k8-fastbuild/bin/benchsci/test_pg13.runfiles/_main/benchsci/_test_pg13.venv/bin/python3
/home/ewianda/.cache/bazel/_bazel_ewianda/6091a940c516b5ac88940b423928af22/sandbox/linux-sandbox/7/execroot/_main/bazel-out/k8-fastbuild/bin/benchsci/test_pg13.runfiles/_main/benchsci/_test_pg13.venv/bin/python3: broken symbolic link to ../../../../../rules_python~~python~python_3_11_6_x86_64-unknown-linux-gnu/bin/python3