✨ Add upgrade E2E

[m1kola]

Description

Testing upgrade from the latest release to the current commit.

Fixes #856

Reviewer Checklist

• API Go Documentation
• Tests: Unit Tests (and E2E Tests, if appropriate)
• Comprehensive Commit Messages
• Links to related GitHub Issue(s)

[netlify]

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	2cf7d2e
🔍 Latest deploy log	https://app.netlify.com/sites/olmv1/deploys/668fa113c9ce1000084be3f6
😎 Deploy Preview	https://deploy-preview-1003--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.60%. Comparing base (db5a75c) to head (2cf7d2e).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1003   +/-   ##
=======================================
  Coverage   77.60%   77.60%           
=======================================
  Files          18       18           
  Lines        1268     1268           
=======================================
  Hits          984      984           
  Misses        202      202           
  Partials       82       82           

Flag	Coverage Δ
e2e	56.44% <ø> (ø)
unit	52.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tmshort]

My general question is... do we want to use bash scripts for this, or do we want to use golang?

[joelanford]

Can we also run the standard e2e after an upgrade as well?

[m1kola]

We were considering this, but decided not to because:

1. It will be an equivalent of running E2E on the current commit and we are doing it anyway in a separate E2E job.
2. It will increase feedback time and has potential to add noise to the signal (e.g. upgrade was successful, but post-upgrade E2E test flaked) - you have to re-test and wait again.

[joelanford]

Do we want to use bash scripts for this, or do we want to use golang?

This is a good question. I don't necessarily think its an either-or thing. I could see something like:

sh <installPreviousRelease>
sh <setupPreUpgradeState>
sh <upgradeIt>
go test ./test/upgrade-e2e/...

[joelanford]

Another interesting upgrade question: At what point do we need to actually define an upgrade process? An upgrade is generally:

1. Create objects that are net new in the new release
2. Update objects that are changed from the old release to the new release
3. Delete objects that were present in the old release, but not the new release

We don't do step 3 right now, right? Is doing step 3 a prereq for valid upgrade testing?

[m1kola]

Another interesting upgrade question: At what point do we need to actually define an upgrade process? An upgrade is generally:

1. Create objects that are net new in the new release
2. Update objects that are changed from the old release to the new release
3. Delete objects that were present in the old release, but not the new release

We don't do step 3 right now, right? Is doing step 3 a prereq for valid upgrade testing?

We were talking about this with @ankitathomas. It sounds a bit like we are testing something that doesn't exist at this moment (the upgrade process).

We decided not to open a can of worms for now and assume that the upgrade process is just:

• For operator-controller - run the install script on top of what already exists
• For catalogd - apply manifests on top of what already exists

This should be enough short-term: it should signal when someone adds something breaking to manifests/install script.

But long term we probably want to define upgrade process and probably create tool for that. E.g. we will likely need to some migration tool where it is possible to clean up old in-cluster objects from previous releases. We can use OpenShift's Cluster Version Operator and how it handles deletions.

But IMO - this deserves its own epic. What do you think? Should we put this on hold and define upgrade process & create necessary tools first? Or should we proceed with this naive approach of just applying things on top?

Do we want to use bash scripts for this, or do we want to use golang?

This is a good question. I don't necessarily think its an either-or thing. I could see something like:

sh <installPreviousRelease>
sh <setupPreUpgradeState>
sh <upgradeIt>
go test ./test/upgrade-e2e/...

I decided to keep bash in the draft for now. We can switch to Go when/if we decide to do something more sophisticated here.

[joelanford]

Already have an epic for proper upgrade support. #982

I agree that this epic should not be blocked on that one.

I think naive approach is good enough for now, but I suspect there could be some cases where leaving around old manifests causes the upgrade tests to pass when cleaning them up would have made them fail. As an example: a PR that deletes our issuer Certificate from our manifest with no other changes.

[m1kola]

I suspect there could be some cases where leaving around old manifests causes the upgrade tests to pass when cleaning them up would have made them fail. As an example: a PR that deletes our issuer Certificate from our manifest with no other changes.

Yes, there will be cases like that, but in this specific example, I think, we will catch the issue in the E2E job because a clean setup will fail.

[joelanford]

Anything else you're thinking of doing before taking this out of draft? At a quick glance, I think this is looking really good!

[m1kola]

Anything else you're thinking of doing before taking this out of draft? At a quick glance, I think this is looking really good!

@joelanford I just wanted to take another fresh look in the morning and clean some things up. Nothing major: adjusting test logs for consistency, a bit of comment changes and moving variables around. Here is the diff between the version you looked at and the latest state.

Should be good now. Marking as ready for review.

[tmshort]

/lgtm
Just one question on the location of a file... it doesn't necessarily stop this from being merged.

[tmshort]

Would it make sense to put this under the test/tools directory? There are a few other testing scripts there, we might want to consolidate?

[tmshort]

This could be done later (if we desire)