Remaining Tumbleweed broken tests

[SludgeGirl]

Explain what happens

Hey!

Sorry it's been a little bit, but I'm back on it now

So, just to give you some extra visibility, these are all the current tests that fail with Tumbleweed after all the work we've been doing:

- TestApps.testBasic - Packagekit segfaulting with zypp
- TestApps.testOsMap - Packagekit segfaulting with zypp
- TestApps.testWithUrlRoot - Packagekit segfaulting with zypp
- TestClient.testBeibootNoBridge - /usr/bin/pmpython seems to be missing?
- TestClient.testBeibootWithBridge - /usr/bin/pmpython seems to be missing?
- TestHistoryMetrics.testEvents - Seems to be a time zone issue? It's expecting 1:00 but actually gets 12:00
- TestHostSwitching.testBasic - Hosts not pairing right?
- TestKdumpNFS.testBasic - fails to mount /kdump/mnt
- TestLogin.testBasic - who not showing admin's session despite successful login
- TestLogin.testLogging - failed login attempts not appearing in text
- TestLogin.testSSH - didn't find the expected login error message
- TestMetricsPackages.testBasic - cockpit-pcp wasn't installed
- TestMultiMachine.testTroubleshooting - assertion error after having failed auth
- TestNetworkingMAC.testBondMac - ip configuration could not be reserved
- TestNetworkingMAC.testMac - ip configuration could not be reserved
- TestNetworkingMTU.testMtu - ip configuration could not be reserved
- TestNetworkingSettings.testNoConnectionSettings - ip configuration could not be reserved
- TestNetworkingSettings.testOtherSettings - ip configuration could not be reserved
- TestPages.testHistory - seems to be a lack of lastlog2 support
- TestReverseProxy.testNginxNoTLS - setsebool erroring due to no managed policy
- TestReverseProxy.testNginxTLS - setsebool erroring due to no managed policy
- TestRHEL8.test - struggling to find and download rhel image?
- TestServices.testServicesThemeConsistency - switching theme failed
- TestSshDialog.testPassword - this gives assertion error instead of expected error text
- TestStorageLvm2.testLvmOnLuks - disk never seems to appear in menu
- TestStorageNBDE.testBasic - tanged.socket not found
- TestStorageNBDE.testRootReboot - tanged.socket not found
- TestSuperuser.testMultipleBridgeConfig - waiting for "switch to admin access"
- TestTeam.testActive - No add team button
- TestTeam.testBasic - No add team button
- TestUpdates.testBasic - tracer error
- TestUpdates.testFailServiceRestart - Couldn't find success table
- TestUpdates.testInfoSecurity - No security info shown on suse platforms
- TestUpdates.testInfoTruncation - Incorrect package text
- TestUpdates.testPackageKitCrash - Packagekit doesn't crash like is intended
- TestUpdates.testSecurityOnly - No security info shown on suse platforms
- TestUpdates.testTracer - Couldn't find success table
- TestUpdates.testUpdateError - Update didn't error
- TestWsUpdate.testBasic - progress bar updating is different on tw

In general I think we're in a really good place considering the starting point at least. One or two of these I think are just to do with my environment and quite a large part are todo with packaging and packagekit which requires some work from us in future to help improve packagekits zypp support

In the next few weeks I think I can get this list massively reduced in number, however I think some of these would have to be marked as naughty, since I don't think I can get the time to properly resolve them currently. That list would look something like the following:

- TestApps.testBasic - Packagekit segfaulting with zypp
- TestApps.testOsMap - Packagekit segfaulting with zypp
- TestApps.testWithUrlRoot - Packagekit segfaulting with zypp
- TestHostSwitching.testBasic - Hosts not pairing right?
- TestKdumpNFS.testBasic - fails to mount /kdump/mnt
- TestNetworkingMAC.testBondMac - ip configuration could not be reserved
- TestNetworkingMAC.testMac - ip configuration could not be reserved
- TestNetworkingMTU.testMtu - ip configuration could not be reserved
- TestNetworkingSettings.testNoConnectionSettings - ip configuration could not be reserved
- TestNetworkingSettings.testOtherSettings - ip configuration could not be reserved
- TestStorageLvm2.testLvmOnLuks - disk never seems to appear in menu
- TestTeam.testActive - No add team button
- TestTeam.testBasic - No add team button
- TestUpdates.testBasic - tracer error
- TestUpdates.testFailServiceRestart - Couldn't find success table
- TestUpdates.testInfoSecurity - No security info shown on suse platforms
- TestUpdates.testInfoTruncation - Incorrect package text
- TestUpdates.testPackageKitCrash - Packagekit doesn't crash like is intended
- TestUpdates.testSecurityOnly - No security info shown on suse platforms
- TestUpdates.testUpdateError - Update didn't error
- TestWsUpdate.testBasic - progress bar updating is different on tw

Version of Cockpit

No response

Where is the problem in Cockpit?

None

Server operating system

None

Server operating system version

No response

What browsers are you using?

No response

System log

No response

[jelly]

A quick reply, I am not sure if we should naughty or not but when I added the Arch image we basically did a bunch of:

@skipImage("TODO: support opensuse-tumbleweed", "tumbleweed")

So for the tests below we can do the same as it sounds like a test dependency either does not exist on OpenSUSE or might not be available.

- TestStorageNBDE.testBasic - tanged.socket not found
- TestStorageNBDE.testRootReboot - tanged.socket not found
- TestTeam.testActive - No add team button
- TestTeam.testBasic - No add team button

tanged.socket, I can only find tang stuff in clevis, so I would skip and TODO this.
NM-team was dropped in RHEL 10, so maybe also in SUSE?

I would mark all packagekit related tests also as a TODO.

• TestStorageLvm2.testLvmOnLuks - disk never seems to appear in menu

LUKS is available in udiskctl dump, so can't tell what is going wrong here.

• TestMetricsPackages.testBasic - cockpit-pcp wasn't installed

These days the dependency is moved from cockpit-pcp to python3-pcp see ea0432f

[SludgeGirl]

tanged.socket, I can only find tang stuff in clevis, so I would skip and TODO this.

Yeah my plan is to do that, I just haven't reached down to those tests yet

I would mark all packagekit related tests also as a TODO.

Gotcha will make sure to!

These days the dependency is moved from cockpit-pcp to python3-pcp see ea0432f

Thanks for the heads up, looks like something went awry during one of the rebase's

[martinpitt]

Right, please start with generously sprinkling @skipImage over the tests -- it's much easier and more motivating to have a working baseline and skip 50 tests, and then investigate them one by one, than trying to fix all of them in one go.