[KV] "failed to get rate limit token" when running under load

[arielshaqed]

Under load KV gets "DynamoDB: GetItem, failed to get rate limit token". But capacity appears to be available on this DynamoDB.

Message:

{
  "error": "get item: operation error DynamoDB: GetItem, failed to get rate limit token, retry quota exceeded, 4 available, 5 requested",
  "file": "build/pkg/api/controller.go:2118",
  "func": "pkg/api.(*Controller).handleAPIErrorCallback",
  "host": "***.us-east-1.lakefscloud.io",
  "level": "error",
  "method": "GET",
  "msg": "API call returned status internal server error",
  "operation_id": "StatObject",
  "path": "/api/v1/repositories/REPO/refs/REF/objects/stat?path=/PATH/TO/SOME/PARTITIONED/PARQUET&user_metadata=false&presign=false",
  "request_id": "12345-678",
  "service": "api_gateway",
  "service_name": "rest_api",
  "time": "2023-09-23THH:MM:SS",
  "user": "USER"
}

AFAIU this is an internal client-side error coming from the AWS SDK. I don't understand how this can work without knowing the desired rate, which of course depends on how many RCUs we've purchased or even how many RCUs DynamoDB autoscaling has given us.

[arielshaqed]

We see limited throttling but before this message. I believe that a client-side rate limit is actively harmful here. But if we do keep it, we obviously need a much more generous back-off.

[N-o-Z]

@arielshaqed How is this different from #6621 (aws/aws-sdk-go-v2#1665)?
Is there a different call to action here?

[github-actions]

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

[arielshaqed]

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

SRSLY?

[arielshaqed]

@arielshaqed How is this different from #6621 (aws/aws-sdk-go-v2#1665)? Is there a different call to action here?

That added some retries, which fixed a P0: it was happening all the time under load. The underlying issue remains: lakeFS should manage client-side rate limiting in a better way. It is incorrect to fail a user operation because of client-side rate limiting.