feat: segregate retry decision for read and write APIs

[pratik151192]

• chore: add tests for retry strategies
• feat: segregate retry decision for read and write APIs

[anitarua]

lgtm!

[malandis]

Few questions:

1. We currently only retry idempotent RPCs. On a CANCELLED, shouldn't we retry any idempotent RPC (vs only read APIs)? If we can retry any idempotent RPC, then we won't have to maintain separate read vs write RPC lists.

2. I understand this came up because we want the client to be tolerant to deploys. Will the retry schedule we have now be long enough for outages due to deployments?

[pratik151192]

@malandis

1. Good call I thought of that too but wanted to be a bit safe but I think we can do that now you mention it as well. "Safe" and "idempotent" go hand in hand, set is also safe to retry as if it was accepted will just overwrite the same value same as unavailable and internal.

2. Nah these are very brief; similar to internal and unavailable.

I'll revert the source file and just uncomment cancelled

[pratik151192]

@malandis To add more context, not all requests are bound to this so the length of deployment is unrelated. It's more to do with how grpc-js is handling GOAWAY frames and in some cases (not totally known) it doesn't respect it and cancels the RPC instead. More details here grpc/grpc-node#2694

These are low enough in volume and historically has been to not be impacted by deployment length.