Pipeline in failed state "can't resume"

[juliev0]

Describe the bug

This is related to when there's a Failed Pipeline. We've decided before that a Failed Pipeline does not require pausing.

(Note that a Failed pipeline can occur when an isbsvc is upgrading, which is what we observed.)

I think there may be 2 issues which are in some ways canceling each other out:

1. The Failed Pipeline is supposed to be okay to resume, but there's a check to prevent resuming if isbsvc or numaflow controller is requesting pause, which there shouldn't be.
2. After we try and fail to resume, we leave the PPND "in progress strategy", which we probably shouldn't. Then we direct apply the change which removes the "desiredPhase" field.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

[juliev0]

cc @whynowy