Skip to content

Commit

Permalink
feat: autorecover from stuck situations
Browse files Browse the repository at this point in the history
Signed-off-by: Gernot Feichter <[email protected]>
  • Loading branch information
gerrnot committed Oct 15, 2024
1 parent 74c3e94 commit 7b94ce8
Showing 1 changed file with 19 additions and 6 deletions.
25 changes: 19 additions & 6 deletions hips/hip-9999.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,14 @@ After implementation, the --timout parameter will be stored in the helm release
have an indirect impact on possible parallel processes.

`helm ls -a` shows two new columns, regular `helm ls` does NOT show those:
- LOCKED TILL
<datetime> calculated by the helm client: k8s server time + timeout parameter value
- SESSION ID
Unique, random session id generated by the client
- `LOCKED TILL`
datetime calculated by the helm client: current time + timeout parameter value
Originally k8s server time was intended as "current time", but since helm exclusively uses the
client time everywhere else, we do not change that via this HIP, such a refactoring would need
to be performed via a separate HIP against the entire codebase.
- `SESSION ID`

Unique, random session id generated by the client.

Furthermore, if the helm client process gets killed (SIGTERM), it tries to clear the LOCKED TILL value,
SESSION ID and sets the release into a failed state before terminating in order to free the lock.
Expand Down Expand Up @@ -88,7 +92,16 @@ None
- [ ] HIP status `accepted'
- [x] Reference implementation
- [x] Test for concurrent upgrade (valid lock should still block concurrent upgrade attempts)
- [ ] Test for kill scenario (forever stuck in pending)
- [ ] Backwards compatibility check (looking good already)
- [x] Test for upgrading from pending state
- [x] Test for upgrading from failed state
- [ ] Decision: Helm ls -> which flag should show the new fields `LOCKED TILL` and `SESSION ID`?
- [ ] Decision: k8s Lease object vs helm relesae secret for storing the `LOCKED TILL` and `SESSION ID`
- [x] Backwards compatibility check (part of acceptance tests repo, looking good already, even when storing the state in the release object)

## References

https://github.com/helm/helm/issues/7476

https://github.com/rancher/rancher/issues/44530

https://github.com/helm/helm/issues/11863

0 comments on commit 7b94ce8

Please sign in to comment.