Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[action] [PR:15174] tests-common: Handle PortInUseException for SSHConsoleConn #15293

Merged
merged 1 commit into from
Nov 1, 2024

Conversation

matthew-soulsby
Copy link
Contributor

Description of PR

Addresses #15092

Summary:
This PR adds handling for the PortInUseException, which is thrown when an SSHConsoleConn is successfully established (most prominently in dut_console tests) but the console port is occupied, resulting in the connection being terminated by the host.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

Approach

What is the motivation for this PR?

This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases.

How did you do it?

By completing the following:

  • Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for
  • Add function to clear a DUT's console port
  • Add retry logic to add resilience to main DUT connection set up

How did you verify/test it?

This process was conducted on 5 testbeds, with at least one of each config menu type.

Steps conducted during testing
  1. SSH into testbed using the console IP and console port - simulating a blocked port
  2. In a separate terminal, run any dut_console test individually
  3. During test setup, the connection from step 1 was terminated successfully
  4. The test then runs successfully
Example test output
  • Successful port reset (Digi config):

Screenshot 2024-10-25 165601

  • Successful port reset (Sonic config):

Screenshot 2024-10-25 183646

  • Sample failure
    • Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing:

Screenshot 2024-10-25 153008

Any platform specific information?

N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.

Supported testbed topology if it's a new test case?

N/A

Documentation

N/A

…#15174)

What is the motivation for this PR?
This PR is intended to add more resilience to the dut_console tests, as
if a testbed with a blocked port was selected for a nightly test, it
would cause the dut_console tests to fail during setup. This provides a
method for auto-recovery for these cases.

How did you do it?
By completing the following:

Add connection types for each of the configuration menu types (Digi,
Cisco, and Sonic) to allow for different command sequences to be
accounted for
Add function to clear a DUT's console port
Add retry logic to add resilience to main DUT connection set up
How did you verify/test it?
This process was conducted on 5 testbeds, with at least one of each
config menu type.

Steps conducted during testing
SSH into testbed using the console IP and console port - simulating a
blocked port
In a separate terminal, run any dut_console test individually
During test setup, the connection from step 1 was terminated
successfully
The test then runs successfully
Example test output
Successful port reset (Digi config):
Screenshot 2024-10-25 165601

Successful port reset (Sonic config):
Screenshot 2024-10-25 183646

Sample failure
Simulating config menu type is not defined (causing the port clear
function to exit early), and a more descriptive error message is thrown
regarding why the connection to the DUT is failing:
Screenshot 2024-10-25 153008

Any platform specific information?
N/A - this is a generalised solution which accounts for as many types of
configuration menus as possible.
@bingwang-ms
Copy link
Collaborator

Cherry-pick for #15174

@bingwang-ms bingwang-ms merged commit 071eafa into sonic-net:202405 Nov 1, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants