Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix: prevent tiny buffer preventing subscriber acking message quickly (…
…#2563) fixes #2554 Current theory is this: - cluster has a low number of modules active (let's say 0) - gRPC call comes into controller to PullSchema - this causes us to subscribe to schema changes, with a chan with the length of the count of the current active modules (in our example, 0) - The lib then creates an extra buffer to queue up messages while the subscriber processes messages, with the buffer being the same size as the provided chan (again, 0) - As a change schema change message is received by the subscriber, we stream the message back over gRPC. This may take time. - While this is happening, another schema update may occur. the lib will not be able to ack the message because the buffer size is too small (0) so it will wait for the message to be received. - the lib will timeout if the networking is not done fast enough. Repro'd by doing the following: - Added a sleep step before sending the schema update over the network. - Start up FTL without any modules - Ran `ftl schema get --watch` - Ran `ftl deploy` with a bunch of module - Hit the `ack timout` panic Could not repro after making the chan length always be a decent size.
- Loading branch information