[BUG] When performing distributed cache checks if there is an error with the call the check is marked as CHECK_RESULT_DENIED

[stheiven]

Describe the bug
We have observed that when distributed cache checks are made several parallel checks are made for the action we are checking against ,and if one of those checks errors out before the other checks finish then the remaining checks are cancelled. The entire check is deemed CHECK_RESULT_DENIED (which can lead to incorrect results).

Example Application
Here is a simplified schema

entity stock {
	relation seller @org
	
        action view = seller.view
}

entity org {
	relation accessgroup_orgs @accessgroup_org
	
        action view = accessgroup_orgs.viewFromOrg
}

If we check if a rep can view a stock several parallel checks for viewFromOrg (one for each access group the org is part of) are generated, and if one of them fails with rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination before the other checks have a change to complete then the view action is denied. The original call to check if the rep can view the stock comes back as denied with no errors.

Expected behavior
When an error occurs during a check operation the entire operation does not result in denied but an error is propagated back to the caller.

Additional context
I saw this and it seems like this might be expected behaviour not only for distributed cache checks but any checks. Is this correct? If so how would a caller distinguish between a denied caused by an error and a genuine one. In our case it looks like the denied might be getting cached so if we attempt the call again we get back denied and that 2 or 3 checks were performed.

Environment (please complete the following information, because it helps us investigate better):

• Version 1.1.1