Module leases fail to expire

[matt2e]

Steps to reproduce:

• Alter examples/go/time/time.go to obtain a lease but forget to release it:

func Time(ctx context.Context, req TimeRequest) (TimeResponse, error) {
   _, err := ftl.Lease(ctx, 10*time.Second, "time")
   if err != nil {
   	return TimeResponse{}, fmt.Errorf("could not obtain time lease: %w", err)
   }
   return TimeResponse{Time: time.Now()}, nil
}

• Go to the console and trigger the time.time verb. Do it once a second...

Expected result:

• The first call to time.time should succeed
• Subsequent calls to time.time should fail with could not obtain time lease for the next 10 seconds or so (waiting for the lease to expire)

Actual result:

• First call succeeds
• Subsequent calls fail indefinitely...
• Trigger a new deploy of time by modifying any code in time.go
• Taking down the old deployment will cause the lease to finally expire
• Next call to time.time will succeed, and then the cycle continues...

From the digging I've done, here's why:

• When we obtain a lease, the module connects to the controller and sends a message to the controller every so often in an attempt to heartbeat the lease
  • When the verb call ends, the module stops sending these heartbeat messages to the controller
  • But the stream is not closed, as that only happens when Release() is called
• The controller receives this connection, then:
  • creates a lease which automatically renews itself
  • responds back to the module each time it receives a message in the stream
• Once the module stops sending heartbeats, the controller doesn't stop the automatic renewal of the lease

Leaving that connection alive sounds like a resource leak. Is making sure that gets closed enough to make this safe, or should the controller also only heartbeat the lease when it receives the heartbeat message from the module?

[alecthomas]

I might be misunderstanding, but this is user error, not a bug. Leases must be released, similarly to how file handles must be closed.