Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[staging] python313Packages.pyzmq: disable flaky tests #367255

Open
wants to merge 1 commit into
base: staging
Choose a base branch
from

Conversation

paparodeo
Copy link
Contributor

@paparodeo paparodeo commented Dec 22, 2024

3 tests are marked as flaky and fail periodically on hydra, disable them. Alternatively, adding pytest-rerunfailures package will enable the tests to be rerun (they are marked as rerun=3) but just disable them all together.

before:
collected 364 items / 81 deselected / 1 skipped / 283 selected
after:
collected 364 items / 84 deselected / 1 skipped / 280 selected

https://github.com/pytest-dev/pytest-rerunfailures
https://cache.nixos.org/log/ja0w2jni5qzlgrzp2sjl67rcjzrxxz0r-python3.13-pyzmq-26.2.0.drv

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

3 tests are marked as flaky and fail periodically on hydra, disable
them. Alternatively, adding pytest-rerunfailures package will enable the
tests to be rerun (they are marked as `rerun=3`) but just disable them
all together.

before:
collected 364 items / 81 deselected / 1 skipped / 283 selected
after:
collected 364 items / 84 deselected / 1 skipped / 280 selected

https://github.com/pytest-dev/pytest-rerunfailures
https://cache.nixos.org/log/ja0w2jni5qzlgrzp2sjl67rcjzrxxz0r-python3.13-pyzmq-26.2.0.drv
@paparodeo
Copy link
Contributor Author

paparodeo commented Dec 22, 2024

rebuild counts:

{
  "darwin": 1681,
  "linux": 2767
}
git switch pyzmq-disable-flaky-tests
nix-build ci -A eval.full --max-jobs 4 --cores $(($(nproc)  / 4)) --arg chunkSize 10000 -o after
git checkout HEAD~1
nix-build ci -A eval.full --max-jobs 4 --cores $(($(nproc)  / 4)) --arg chunkSize 10000 -o before
nix-build ci -A eval.compare --arg afterResultDir ./after --arg beforeResultDir ./before
git switch -
jq .rebuildCountByKernel < result/changed-paths.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant