Async calls get stuck if controller dies

[matt2e]

Steps to reproduce

• Comment out db cleanup code used in dev: https://github.com/TBD54566975/ftl/blob/main/backend/controller/sql/databasetesting/devel.go#L77
• Start FTL and trigger an async call that takes some time. Here's what I did:
  • Make this verb basically just do time.Sleep(time.Minute); return nil
  • ftl dev --log-level=DEBUG backend/controller/pubsub/testdata/go/subscriber backend/controller/pubsub/testdata/go/publisher --recreate
  • Go to the console and trigger the verb publisher.publishOneToTopic2
• Wait a few seconds so we know the async call has started
• Kill FTL (ctrl-C)
• Re-run FTL but without --recreate

Expected result:

• The lease associated with the previous attempt of the async call with expire, and the async call will be attempted again or fail, depending on the retry policy.

Actual result:

• The lease gets reaped
• The async call row loses its reference to the lease, but remains in an executing state forever
• This blocks the associated subscription / FSM forever

I think this happens for any case where async call leases expire, like:

• controller losing connection to the db momentarily
• controller dying

It think it might also happen in this case:

• controller executes executeAsyncCalls()
• it successfully acquires a call with `AcquireAsyncCall()
• it defers release of the call lease
• CompleteAsyncCall fails for any reason (eg db connection error)
  Similar to what's outlined above, we now have a released lease and a async call row that will sit in an executing state forever

[matt2e]

I think in other use cases for leases we have special logic to handle the necessary knock on effects, but it seems like we didn't do it for async calls, or it's not working