Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[202405][test_wr_arp] Warm reboot is failing during execution of testWrArpAdvance due to docker exec failure #21236

Open
gpunathilell opened this issue Dec 20, 2024 · 0 comments

Comments

@gpunathilell
Copy link
Contributor

Description

During testWrArpAdvance in sonic-mgmt https://github.com/sonic-net/sonic-mgmt/blob/040d677e9355aa62c7a7ad1e05fc587a22641fa0/tests/arp/test_wr_arp.py#L90 the warm-reboot execution fails due to failed docker exec during the check_docker_exec function during the warm-reboot execution https://github.com/sonic-net/sonic-utilities/blob/2f8508f182566dd75a7c66cfd8dcccaa21470485/scripts/fast-reboot#L469. The docker exec call times out and due to which the warm-reboot execution fails, causing test failure.
Issue is currently observed only in 202405 and with a rare occurence ( 1 in 20 runs)
Log Errors:

2024 Dec  2 01:36:01.052167 sonic NOTICE root: Starting warm-reboot
2024 Dec  2 01:36:15.415931 sonic INFO dockerd[1063]: time="2024-12-02T01:36:15.414956087+02:00" level=error msg="Error running exec 5da6854722ff10dd3ce552ff9baba424ac1d6e7a245ca45ebdedc7bbe0521527 in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe" 
2024 Dec  2 01:36:15.481252 sonic ERR root: Error seen during warm-reboot shutdown process: Docker exec on lldp timedout
2024 Dec  2 01:36:22.245221 sonic NOTICE root: Cancel warm-reboot: code (0)

Only reproduced once in 20 runs, and the docker for which the docker exec fails is different in each case

Steps to reproduce the issue:

  1. Run testWrArpAdvance test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant