Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[action] [PR:15314] Stabilize pfcwd warm-reboot test on Mellanox platform #15398

Merged
merged 1 commit into from
Nov 6, 2024

Conversation

mssonicbld
Copy link
Collaborator

Description of PR

Summary:
test_pfcwd_wb is flaky on Mellanox platform because PFCWD is not triggered as expected. The error is as below.

tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzerError: match: 0
expected_match: 0
expected_missing_match: 1

Expected Messages that are missing:
.* detected PFC storm .*

The failure is because the pfc_gen.py script running on leaf-fanout can't generate continuous PFC pause to trigger PFCWD on DUT.
For each lossless queue, a separated pfc_gen.py will be running on leaf-fanout. If two_queues is true, then there will be two processes running for each port in test. Due to job schedule, it's not guaranteed that all processes can generate continuous PFC pause to trigger PFCWD.
To workaround the issue, the PR set two_queues to False for Mellanox platform if leaf-fanout is running EOS.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

Approach

What is the motivation for this PR?

This PR is to stabilize test_pfcwd_wb by limiting queue number to 1 on Mellanox platform if leaf-fanout is running EOS.

How did you do it?

Update fixture two_queues.

How did you verify/test it?

The change is verified on a Mellanox-SN2700 testbed. Test can pass after the change.

collected 3 items 

pfcwd/test_pfcwd_warm_reboot.py::TestPfcwdWb::test_pfcwd_wb[no_storm-str2-msn2700-spy-1] 
PASSED [ 33%]
pfcwd/test_pfcwd_warm_reboot.py::TestPfcwdWb::test_pfcwd_wb[storm-str2-msn2700-spy-1] 
PASSED [ 66%] 

Any platform specific information?

Mellanox platform specific.

Supported testbed topology if it's a new test case?

Not a new test.

Documentation

@mssonicbld
Copy link
Collaborator Author

/azp run

@mssonicbld
Copy link
Collaborator Author

Original PR: #15314

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld mssonicbld merged commit 56da31c into sonic-net:202405 Nov 6, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants