Try to Fix Flaky Test

[smallinsky]

What:

Try to fix test flakiness issues: #13623 and #13944 caused by racate between t.TmpDir cleanup and session recording handler:

ERROR: access denied to invalidUser connecting to localhost:24915

    testing.go:1206: TempDir RemoveAll cleanup: unlinkat /tmp/TestTSHConfigConnectWithOpenSSHClientnode_recording_mode_with_TLS_routing_disabled2058613825/001/log/upload/streaming/default: directory not empty
--- FAIL: TestTSHConfigConnectWithOpenSSHClient/node_recording_mode_with_TLS_routing_disabled (3.93s)

• Force the node-sync and add missing node.Wait() call
• Move dynamicServiceAddr to common package and reuse. (TestTerminalRequireSessionMfa)
• Fix flaky way of acquiring open local port by using dynamicServiceAddr

[Tener]

I'll need to dig it out, but from memory, the common fix was to remove session recording from happening in tests? @zmb3

[zmb3]

I'll need to dig it out, but from memory, the common fix was to remove session recording from happening in tests? @zmb3

Yes, addressed that in #28518

[smallinsky]

Friendly ping @Tener

[Tener]

Is this PR still needed? I thought #28518 would provide a fix already. If not, do let me know, and I'll review it.

[smallinsky]

Is this PR still needed? I thought #28518 would provide a fix already. If not, do let me know, and I'll review it.

Yea this is still needed. This pr aims to fix the TestTSHConfigConnectWithOpenSSHClientnode_recording_mode_with_TLS_routing directory not empty flaky test issue where #28518 addressed only TestTerminalRequireSessionMFA

[zmb3]

@smallinsky do we want to merge this still?

[smallinsky]

@zmb3

do we want to merge this still?
Yes thanks for reminding me about this. Can I ask for UT "Flaky Tests Detector" skip due to test --count=100 timeout ?

[zmb3]

/excludeflake TestTokens TestConnect TestCreateSAMLIdPServiceProvider TestCreateClusterAuthPreference_WithSupportForSecondFactorWithoutQuotes TestCreateDatabaseInInsecureMode TestCreateLock TestIntegrationResource TestAppResource TestDatabaseServerResource TestDatabaseResource TestDatabaseServiceResource

[zmb3]

/excludeflake *

[zmb3]

@smallinsky flake detector bypass should be good now. Let's get this rebased and merged.