Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[202405] Fix intermittent issue on reboot in test_lldp_syncd #15331

Conversation

veronica-arista
Copy link
Contributor

Description of PR

At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the lldp table output will not. From debugging, this is the case since the end of the previous test (I put a check on the keys vs lldp table output and observed the missing eth0 at the end of test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart). From our test results, this issue hits both fixed and modular, but appears to be more easily reproduced on fixed systems.

Added a wait_until at the start of test_lldp_entry_table_after_reboot to wait until the LLDP_ENTRY_TABLE keys match show lldp table output before the tests starts to reboot.

Summary:
Fixes # (issue)

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

Approach

What is the motivation for this PR?

Observed another intermittent issue in test_lldp_syncd in addition to the issues fixed by #15258.

How did you do it?

How did you verify/test it?

Verified the test on one of our fixed system by running test 50 times with this change and PR#15258 to ensure all the intermittent fails are gone.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

@mssonicbld
Copy link
Collaborator

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing tests/lldp/test_lldp_syncd.py

fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Passed
flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@StormLiangMS StormLiangMS requested a review from ZhaohuiS November 7, 2024 05:49
@ZhaohuiS
Copy link
Contributor

ZhaohuiS commented Nov 7, 2024

@veronica-arista there is conflict in your PR. could you please check?
I also had a PR to address this mismatch issue after restart lldp.
#15337
But I think your check is not harmful.

At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot
the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the
lldp table output will not.  From debugging, this is the case since the
end of the previous test (I put a check on the keys vs lldp table output
and observed the missing eth0 at the end of
test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart)

Added a wait_until at the start of test_lldp_entry_table_after_reboot to
wait until the LLDP_ENTRY_TABLE keys match show lldp table output before
the tests starts to reboot.
@veronica-arista veronica-arista force-pushed the master-fix-test_lldp_syncd-reboot-initi branch from ff7f19f to 24ed2ce Compare November 7, 2024 16:14
@veronica-arista
Copy link
Contributor Author

@ZhaohuiS thanks, I fixed the merge conflict.

@@ -322,6 +329,12 @@ def test_lldp_entry_table_after_reboot(
localhost, duthosts, enum_rand_one_per_hwsku_frontend_hostname, db_instance
):
duthost = duthosts[enum_rand_one_per_hwsku_frontend_hostname]

# Verify LLDP_ENTRY_TABLE keys match show lldp table output at the start of test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There will be a reboot afterwards. All lldp_entry_keys and show_lldp_table_int_list will be checked after reboot. Will this check here still useful or it is not necessary anymore with another fix got in already?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wumiaont Ye, you are right, that's why I move the collection and verification after reboot, please check this PR https://github.com/sonic-net/sonic-mgmt/pull/15337/files

@arlakshm
Copy link
Contributor

/Azp run Azure.sonic-mgmt

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Collaborator

@StormLiangMS StormLiangMS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@StormLiangMS StormLiangMS merged commit f3d2014 into sonic-net:master Nov 13, 2024
16 checks passed
mssonicbld pushed a commit to mssonicbld/sonic-mgmt that referenced this pull request Nov 13, 2024
At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot
the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the
lldp table output will not.  From debugging, this is the case since the
end of the previous test (I put a check on the keys vs lldp table output
and observed the missing eth0 at the end of
test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart)

Added a wait_until at the start of test_lldp_entry_table_after_reboot to
wait until the LLDP_ENTRY_TABLE keys match show lldp table output before
the tests starts to reboot.
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202405: #15535

mssonicbld pushed a commit that referenced this pull request Nov 13, 2024
At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot
the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the
lldp table output will not.  From debugging, this is the case since the
end of the previous test (I put a check on the keys vs lldp table output
and observed the missing eth0 at the end of
test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart)

Added a wait_until at the start of test_lldp_entry_table_after_reboot to
wait until the LLDP_ENTRY_TABLE keys match show lldp table output before
the tests starts to reboot.
sreejithsreekumaran pushed a commit to sreejithsreekumaran/sonic-mgmt that referenced this pull request Nov 15, 2024
At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot
the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the
lldp table output will not.  From debugging, this is the case since the
end of the previous test (I put a check on the keys vs lldp table output
and observed the missing eth0 at the end of
test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart)

Added a wait_until at the start of test_lldp_entry_table_after_reboot to
wait until the LLDP_ENTRY_TABLE keys match show lldp table output before
the tests starts to reboot.
yutongzhang-microsoft pushed a commit to yutongzhang-microsoft/sonic-mgmt that referenced this pull request Nov 21, 2024
At the start of test_lldp_syncd.py::test_lldp_entry_table_after_reboot
the test polls and checks that LLDP_ENTRY_TABLE keys match show lldp table output.

The eth0 port is added last so sometimes the entry keys will have it but the
lldp table output will not.  From debugging, this is the case since the
end of the previous test (I put a check on the keys vs lldp table output
and observed the missing eth0 at the end of
test_lldp_syncd.py::test_lldp_entry_table_after_lldp_restart)

Added a wait_until at the start of test_lldp_entry_table_after_reboot to
wait until the LLDP_ENTRY_TABLE keys match show lldp table output before
the tests starts to reboot.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants