How to update major version of tf module without causing production outage?

[james64]

Scenario:

• k8s cluster is deployed in cloud via third party terraform module
• current version of the module is getting old. We want to use newer version to use new features etc
• assuming usage of semver new module version requires major version upgrade

In general new major version contains substantial changes in:

• parameters accepted by module
• names of TF resources
• structure of what actual resources are created

After updating version string one needs to edit what parameters are passed in (rename, re-organize...). This part is unavoidable. Once this is done, plan is run. Chances are the plan will be substantial. Including such big guns as re-creating actual k8s cluster (control plane + nodes).

Options how to proceed:

1. Try to manipulate a state (state mv/rm import export) to reduce plan size.
   • Time consuming
   • Usually only part of the plan is eliminated. The core problem of sizable plan causing big changes remains
   • Not an option with git-ops
2. Re-create all resources
   • Applying the plan "as is" is very disruptive and rarely acceptable (in prod at least)
   • Staged replace should be possible (add new, switch, drop old) if whole infra is designed well (databases etc)
3. Abandon the upgrade. Continue using old version indefinitely

Out of this list staged replace seems like only option which is somewhat acceptable.

Are there some other options I have not considered? Or does big plan on major upgrade automatically raises a red flag to stop using given tf module? This happened to me with aws-eks-module (upgrade 17.24 -> 18.0) for example...

This just has to be a very common scenario out there. I am sure there is some good approach to how to handle this. I am very interested in hearing opinions on this from more experienced folks 👍

---

Tracked in ticket #111270