Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. #16199

rraghav-cisco · 2024-12-23T05:13:43Z

Description of PR

Summary:
Fixes the flakiness of DWRR testcase. The PR adds a new fixture that slows down the scheduler without changing the underlying algorithm. This allows the dWRR test to pass consitently.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2024-12-23T05:13:46Z

/azp run

azure-pipelines · 2024-12-23T05:13:56Z

Azure Pipelines successfully started running 1 pipeline(s).

yejianquan · 2024-12-24T00:08:09Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts?
Thanks!

rraghav-cisco · 2024-12-24T00:29:41Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

yejianquan · 2024-12-24T03:56:47Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

@rraghav-cisco is DwrrWeightChange because of the testbed issue or the script?

sdszhang

LGTM

sdszhang · 2025-01-03T00:56:12Z

Hi @rraghav-cisco , this new test failed on T0/T1 setup #16314. could you please fix it ASAP?

yejianquan · 2025-01-03T01:01:59Z

convert to draft because it fails on t0/t1 cisco devices #16314

sdszhang

pls fix the error on t0/t1 as well.

sdszhang · 2025-01-03T03:01:03Z

@rraghav-cisco can you include fix #16315 in this PR?

mssonicbld · 2025-01-03T03:13:42Z

/azp run

azure-pipelines · 2025-01-03T03:13:53Z

Azure Pipelines successfully started running 1 pipeline(s).

rraghav-cisco · 2025-01-03T03:13:59Z

@rraghav-cisco can you include fix #16315 in this PR?

@sdszhang , done.

rraghav-cisco · 2025-01-06T18:18:31Z

Hi @rraghav-cisco , could you please paste the test result with the code that resolved the conlficts? Thanks!

Hi @yejianquan , I am seeing pass in Dwrr, but fails in DwrrWeightChange:

___________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_asic] ___________________________________________________________________________________________
______________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_longlink_to_shortlink] _________________________________________________________________________________
_________________________________________________________________________________ TestQosSai.testQosSaiDwrr[multi_dut_shortlink_to_longlink] _________________________________________________________________________________
-------------------------------------------------------- generated xml file: /run_logs/19351-dwrr-only/2024-12-20-16-14-07/2/qos/test_qos_sai_2024-12-20-17-03-53.xml --------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
--------------------------------------------------------------------------------------------------- live log sessionfinish ---------------------------------------------------------------------------------------------------
17:53:46 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================== short test summary info ===================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_longlink_to_shortlink]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[multi_dut_shortlink_to_longlink]
SKIPPED [1] qos/test_qos_sai.py:1541: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
SKIPPED [1] qos/test_qos_sai.py:2091: Don't have 2 shortlink frontend nodes - so can't run multi_dut_shortlink_to_shortlinktests
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_longlink_to_shortlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
FAILED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[multi_dut_shortlink_to_longlink] - tests.common.errors.RunAnsibleModuleFail: run module shell failed, Ansible Results =>
======================================================================= 4 failed, 4 passed, 2 skipped, 235 deselected, 1 warning in 2991.24s (0:49:51) =======================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

@rraghav-cisco is DwrrWeightChange because of the testbed issue or the script?

@yejianquan : The issue was in my workspace. After fixing the workspace, all of these passed.

mssonicbld · 2025-01-06T20:16:55Z

/azp run

azure-pipelines · 2025-01-06T20:17:06Z

Azure Pipelines successfully started running 1 pipeline(s).

Porting 15718 to 202405.

13374fe

rraghav-cisco mentioned this pull request Dec 23, 2024

Adding a fixture to set scheduler to slower speeds and revert it back. #15718

Merged

8 tasks

sdszhang self-requested a review January 2, 2025 04:29

sdszhang approved these changes Jan 3, 2025

View reviewed changes

sdszhang mentioned this pull request Jan 3, 2025

[Bug]: dwrr failed on T0/T1 cisco platform. #16314

Open

yejianquan marked this pull request as draft January 3, 2025 01:01

sdszhang reviewed Jan 3, 2025

View reviewed changes

Include the single-asic fix from PR:16315 in the same PR.

ea5888b

Fix the ptf code to handle single and mAsic combos.

9a28bb0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. #16199

Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. #16199

rraghav-cisco commented Dec 23, 2024

mssonicbld commented Dec 23, 2024

azure-pipelines bot commented Dec 23, 2024

yejianquan commented Dec 24, 2024 •

edited

Loading

rraghav-cisco commented Dec 24, 2024

yejianquan commented Dec 24, 2024

sdszhang left a comment

sdszhang commented Jan 3, 2025

yejianquan commented Jan 3, 2025

sdszhang left a comment

sdszhang commented Jan 3, 2025

mssonicbld commented Jan 3, 2025

azure-pipelines bot commented Jan 3, 2025

rraghav-cisco commented Jan 3, 2025

rraghav-cisco commented Jan 6, 2025

mssonicbld commented Jan 6, 2025

azure-pipelines bot commented Jan 6, 2025

Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. #16199

Are you sure you want to change the base?

Porting 15718 to 202405: Adding a fixture to set scheduler to slower speeds and revert it back. #16199

Conversation

rraghav-cisco commented Dec 23, 2024

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented Dec 23, 2024

azure-pipelines bot commented Dec 23, 2024

yejianquan commented Dec 24, 2024 • edited Loading

rraghav-cisco commented Dec 24, 2024

yejianquan commented Dec 24, 2024

sdszhang left a comment

Choose a reason for hiding this comment

sdszhang commented Jan 3, 2025

yejianquan commented Jan 3, 2025

sdszhang left a comment

Choose a reason for hiding this comment

sdszhang commented Jan 3, 2025

mssonicbld commented Jan 3, 2025

azure-pipelines bot commented Jan 3, 2025

rraghav-cisco commented Jan 3, 2025

rraghav-cisco commented Jan 6, 2025

mssonicbld commented Jan 6, 2025

azure-pipelines bot commented Jan 6, 2025

yejianquan commented Dec 24, 2024 •

edited

Loading