Move test-frame to 0.8.0 + use sequential deletion

[kornys]

Could you please test it if it work properly?

[jiridanek]

Checking, upgrade test will run 30 mins, then we'll see if it helps. Thanks

[jiridanek]

And I still see

2024-10-22T08:58:33 INFO [Exec:213] Error from server (Forbidden): admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists

2024-10-22T08:58:33 INFO  [LoggerUtils:74] Deleting Namespace/redhat-ods-operator
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource Namespace/redhat-ods-operator
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource Namespace/test-odh-notebook-upgrade
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource PersistentVolumeClaim/test-odh-notebook-pre
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource PersistentVolumeClaim/test-odh-notebook-post
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource OperatorGroup/odh-group
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource Subscription/rhods-operator
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource Notebook/test-odh-notebook-post
2024-10-22T08:58:33 INFO  [Wait:66] Waiting for: Resource condition: deletion to be fulfilled for resource Notebook/test-odh-notebook-pre
2024-10-22T08:58:33 ERROR [KubeResourceManager:429] Deleting DSCInitialization/default-dsci
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://api.crc.testing:6443/apis/dscinitialization.opendatahub.io/v1/dscinitializations/default-dsci. Message: admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists
. Received status: Status(apiVersion=v1, code=403, details=null, kind=Status, message=admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists
, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleDelete(OperationSupport.java:320)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.deleteAll(BaseOperation.java:535)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.delete(BaseOperation.java:500)
	at io.odh.test.framework.manager.resources.DataScienceInitializationType.delete(DataScienceInitializationType.java:52)
	at io.odh.test.framework.manager.resources.DataScienceInitializationType.delete(DataScienceInitializationType.java:20)
	at io.skodjob.testframe.resources.KubeResourceManager.deleteResource(KubeResourceManager.java:424)
	at io.skodjob.testframe.resources.KubeResourceManager.deleteResource(KubeResourceManager.java:403)
	at io.skodjob.testframe.resources.KubeResourceManager.lambda$pushToStack$2(KubeResourceManager.java:216)
	at io.skodjob.testframe.resources.KubeResourceManager.lambda$deleteResources$12(KubeResourceManager.java:525)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1796)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: https://api.crc.testing:6443/apis/dscinitialization.opendatahub.io/v1/dscinitializations/default-dsci. Message: admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists
. Received status: Status(apiVersion=v1, code=403, details=null, kind=Status, message=admission webhook "operator.opendatahub.io" denied the request: Cannot delete DSCI object when DSC object still exists
, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

[jiridanek]

I think I see

odh-e2e/src/test/java/io/odh/test/e2e/upgrade/UpgradeAbstract.java

Line 44 in 42a1f33

@ResourceManager

This one needs to be configured too, maybe?

[kornys]

@jiridanek yes it has to be, I thought that it inherid values but it overrides it, so it has to be added to others, let me do a fix

[jiridanek]

Tried this out. Out of 6 CI runs, three got stuck and died on jenkins timeout, two passed, and one timeouted with

2024-10-22T13:44:07 ERROR [Wait:106] Exception waiting for: DataScienceCluster upgrade-dsc readiness, Failure executing: GET at: https://api.crc.testing:6443/apis/datasciencecluster.opendatahub.io/v1/datascienceclusters/upgrade-dsc. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).
2024-10-22T13:44:07 ERROR [Wait:116] Timeout after 600000 ms waiting for DataScienceCluster upgrade-dsc readiness
io.skodjob.testframe.wait.WaitException: Timeout after 600000 ms waiting for DataScienceCluster upgrade-dsc readiness

I think this is good to go, there weren't any passes before; I'll investigate more on how to improve the job success rate.