Skip to content

Commit

Permalink
fix: prevent tiny buffer preventing subscriber acking message quickly (
Browse files Browse the repository at this point in the history
…#2563)

fixes #2554

Current theory is this:
- cluster has a low number of modules active (let's say 0)
- gRPC call comes into controller to PullSchema
- this causes us to subscribe to schema changes, with a chan with the
length of the count of the current active modules (in our example, 0)
- The lib then creates an extra buffer to queue up messages while the
subscriber processes messages, with the buffer being the same size as
the provided chan (again, 0)
- As a change schema change message is received by the subscriber, we
stream the message back over gRPC. This may take time.
- While this is happening, another schema update may occur. the lib will
not be able to ack the message because the buffer size is too small (0)
so it will wait for the message to be received.
- the lib will timeout if the networking is not done fast enough.

Repro'd by doing the following:
- Added a sleep step before sending the schema update over the network.
- Start up FTL without any modules
- Ran `ftl schema get --watch`
- Ran `ftl deploy` with a bunch of module
- Hit the `ack timout` panic
Could not repro after making the chan length always be a decent size.
  • Loading branch information
matt2e authored Aug 30, 2024
1 parent 14d12e6 commit a957549
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion backend/controller/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -1856,7 +1856,7 @@ func (s *Service) watchModuleChanges(ctx context.Context, sendChange func(respon
return err
}
initialCount := len(seedDeployments)
deploymentChanges := make(chan dal.DeploymentNotification, len(seedDeployments))
deploymentChanges := make(chan dal.DeploymentNotification, 32)
for _, deployment := range seedDeployments {
deploymentChanges <- dal.DeploymentNotification{Message: optional.Some(deployment)}
}
Expand Down

0 comments on commit a957549

Please sign in to comment.