Add --test={skip|native|native-and-emulated}

[hadim]

See #1175 for context.

It adds a new flag to build that allow skipping the tests if cross compiling: --no-test-if-emulate

I have a few questions:

• should I prefer "emulate" or "cross compiling" for the variables as well as the CLI doc?
• how should noarch should be handled here? or when should we consider the build is a cross compilation or not?

[wolfv]

maybe --only-test-native or some variation like that? Or --no-emulation or --no-test-emulation?

Or we could also make this opt-in, by saying: --has-emulation. Or something like --run-test.

[wolfv]

If we make it opt-in, we can print a big fat warning and then say: to run tests in non-native environments, use --run-test or something liek that.

[hadim]

Do you prefer this to be opt-in? I originally wanted to keep the default behavior the same but up to you.

• if we go with opt-in then --run-tests or --run-tests-non-native makes sense to me.
• if we go opt-out then I like --only-test-native

I guess the question (to decide the best default) is whether "on average" running tests on non-native will work more often than it will fail.

Let me know what you prefer and I will adapt the code.

[baszalmstra]

I think opt-in makes sense here. Looking from the perspective of someone who just wants to invoke rattler-build. I would not expect a big fat error.

[hadim]

So, would --run-tests-non-native work for you?

[hadim]

Also should I add a test in test_simple.py?

[wolfv]

We had a build tools meeting and I brought up this question :)

One suggestion was to copy what conda-forge does: https://conda-forge.org/docs/maintainer/conda_forge_yml/#test

--test=native

--test=native-and-emulated

Although we don't really know when there is an "emulator". We could however potentially infer that if there are any host dependencies configured in a test ...

We could also retire the current --skip-test flag and do

--test=skip
# and maybe the inverse
--test=always

Anyways, just ideas.

[hadim]

I like the idea of something more expressive than a boolean flag.

I also think it would be nice to propose something related to emulation/cross-compilation so we don't have to compute that logic "outside" of rattler-build. Even if it's opinionated, we can always finetune and adapt it moving forward.

So, considering all the comments above, we could do:

• --test=skip : replace --skip-test
• --test=native: run when host_platform != build_platform + always run target_platform == 'noarch'
• --test=native-and-emulated: always run tests

We can have --test=native as default, so it's opt-in.

WDYT?

[wolfv]

Looks good to me! I still would print a warning if tests are not run when cross compiling though.

[hadim]

LGTM

[baszalmstra]

Nice!

[wolfv]

Excellent work @hadim! I think we need to keep the old flag to not break conda-forge (and we remove it in a few months). The old flag can just override the test strategy to skip, right?

[wolfv]

Thanks!