-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transferring data through the RPC stream skips the second message #70
Comments
Temporarily, to allow deployment for continuing work on Polykey#838, I have commented out the tests. From what can be observed, nothing is failing outside the tests, so it seems safe to do. |
Can we fastcheck the sequential testing? |
Should be able to. Currently, the testing is taking in a randomly generated value for However, this is a result of a very brief testing and investigation. I will properly go through this when I will actually do this issue. |
This deserves plenty of fuzz testing, use chatgpt to help. |
Previously, it already had The sequential testing is mostly for convenience, to be able to easily locate where things went wrong. |
This is a regression? |
This is a bug, the fastcheck has discovered a bug here. |
Why is this not a bug issue? |
This should be added into linear todos. |
The 2 test names should be more specific:
|
The reason this happens is most likely due to how the header messages are handled. Currently, a new transform stream is created to fetch the header, then the stream is cancelled to allow changing the consumer from the header consumer to another consumer which parses the actual content. The weird decision of creating a stream and cancelling it just after getting the first message needs to be changed for another approach, which handles this more elegantly. Instead of cancelling the stream, a promise can be returned for the header, and something like an async iterable can be returned for content data. This would remove the janky solution and implement an elegant one, and should fix the CI issues.
This is probably not the case, as fastcheck won't discover the same bug when running multiple tests. Moreover, the second message is skipped, which is something fastcheck can't induce, so there is most likely some other underlying reason for this.
The current fastcheck tests are already pretty robust and extensive in checking, so I don't believe this to be the issue here. |
I have done the following in Polykey#847 to extract a header message from an async iterable. // Extracts the header message from the iterator
const headerMessage = await (async () => {
const iterator = input[Symbol.asyncIterator]();
const header = (await iterator.next()).value;
if (header.type === 'VaultNamesHeaderMessage') {
if (header == null) throw new clientErrors.ErrorClientInvalidHeader();
return header;
}
})();
// Do stuff on rest of the messages
for await (const message of input) {
// The header has been consumed, so all other messages will be
// returned from the loop.
} |
Why do |
There are easier ways to do this, but most of them include something like this The code I've provided creates and calls an async function inline. It's basically doing a bunch of calculations inside a block to return a singular value. This lets us avoid needing to type-check everywhere. The syntax looks weird because we await an async function that we have created. It is functionally similar to this, but more concise. const extractHeader = async (): HeaderType => {
const iterator = input[Symbol.asyncIterator]();
const header = (await iterator.next()).value;
if (header.type === 'VaultNamesHeaderMessage') {
if (header == null) throw new clientErrors.ErrorClientInvalidHeader();
return header;
}
}
const header = await extractHeader(); This makes it easier to see how this looks spread out, and how it would look like if |
You shouldn't have to check it's type everywhere even if you unwrapped the logic out of that arrow function. Also, does that code even work? Why check Isn't this simpler? const head = await input.next();
if (head.done) utils.never();
if (head.value.type === 'VaultNamesHeaderMessage') throw new clientErrors.ErrorClientInvalidHeader();
const headerMessage = head.value; |
Yes that code structure still smells to me. If you need to do some asynchronous work it should always be doable in a linear way without having to create anonymous async IIFEs. It is just not needed ever. It's too clever. Make it dumber. |
I haven't merged the PR yet, so I will make this change in it before merging it. |
Description
The second value is always skipped when the messages are being processed by
RPCServer
. This might be happening due to a potential race condition which happens when we try to disconnect the stream from one pipe and connect it to another. When we disconnect the stream, the stream has already partially loaded the next chunk of the message. This results in the data being lost.This is also reflected in the tests. Weirdly, all the tests in the feature branch pass for multiple commits, but anytime we merge, this failure is brought back. More interestingly, the MacOS machines are the only machines which have a failure due to this. However, once I pulled down the code from staging and ran the tests, I consistently got the same error.
Interestingly, this would have caused basically all RPC calls to be non-functional, but that is not the case. For Polykey and Polykey CLI, the RPC system has been working pretty reliably, so it's even more weird why this is happening only in the tests, but is not an issue anywhere else.
I went back to the first commit in this repo, ran
npm install
, then ran the tests again, but the tests still had the same result of failing in the same way, so this issue could be in the repo from the beginning, but no one ever ran into this for some reason.To Reproduce
staging
Additional context
Details of the commented out test
Platform
Methods of Resolution
Notify Maintainers
@aryanjassal @tegefaulkes
The text was updated successfully, but these errors were encountered: