[store] Result list is silently limited to 10000 items

[barroco]

Currently, the DSS returns a maximum of 10000 items to clients when listing resources for RID and SCD. This value is set by the MaxResultLimit constant.
It seems that it was introduced to address OOM issues few years ago.

This issue has been created to discuss possible alternative approaches and make sure responses of the DSS are exhaustives.

[barroco]

This issue was discussed during the contributors sync call on 17/09/2024 and it was agreed the fact that the DSS is trimming results silently is an issue. The preferred approach discussed was to fail the response with a 400 or more appropriate error code in case the number of items is too big.

Pagination was mentioned as well but it would require changing the API. Requesting clients to better scope their request to reduce the number of items was preferred.