From a957549dae6a6acb3a6f0782fdca74b1c6a90305 Mon Sep 17 00:00:00 2001 From: Matt Toohey Date: Fri, 30 Aug 2024 17:23:33 +1000 Subject: [PATCH] fix: prevent tiny buffer preventing subscriber acking message quickly (#2563) fixes #2554 Current theory is this: - cluster has a low number of modules active (let's say 0) - gRPC call comes into controller to PullSchema - this causes us to subscribe to schema changes, with a chan with the length of the count of the current active modules (in our example, 0) - The lib then creates an extra buffer to queue up messages while the subscriber processes messages, with the buffer being the same size as the provided chan (again, 0) - As a change schema change message is received by the subscriber, we stream the message back over gRPC. This may take time. - While this is happening, another schema update may occur. the lib will not be able to ack the message because the buffer size is too small (0) so it will wait for the message to be received. - the lib will timeout if the networking is not done fast enough. Repro'd by doing the following: - Added a sleep step before sending the schema update over the network. - Start up FTL without any modules - Ran `ftl schema get --watch` - Ran `ftl deploy` with a bunch of module - Hit the `ack timout` panic Could not repro after making the chan length always be a decent size. --- backend/controller/controller.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/controller/controller.go b/backend/controller/controller.go index 84ac0e87e7..85cbb4417b 100644 --- a/backend/controller/controller.go +++ b/backend/controller/controller.go @@ -1856,7 +1856,7 @@ func (s *Service) watchModuleChanges(ctx context.Context, sendChange func(respon return err } initialCount := len(seedDeployments) - deploymentChanges := make(chan dal.DeploymentNotification, len(seedDeployments)) + deploymentChanges := make(chan dal.DeploymentNotification, 32) for _, deployment := range seedDeployments { deploymentChanges <- dal.DeploymentNotification{Message: optional.Some(deployment)} }