Gateway: add a custom endpoint to retrieve total object count

[mobyvb]

The endpoint should return values for:

• total object count (includes encrypted + decrypted objects) - this can be done with iteratively calling "libuplink list" with the "encrypted" flag
• encrypted object count

It should also be limited in some way (e.g. if there are 10s of millions of objects). Add a config value.

See "list buckets with attribution" for an example of a custom endpoint on gateway: https://github.com/storj/gateway-mt/blob/267c4fc8f7fdbb1953a8636ea305368a503d5506/pkg/minio/api-response.go#L25

Context for underlying issue: storj/storj#6080

AC:

• new endpoint added to gateway for listing total object counts on the bucket level
• add configuration for restricting this functionality in situations where the object count is very high

[storj-gerrit]

Change pkg/{server, minio}: custom endpoint to return total object count mentions this issue.

[halkyon]

Is there any reason we decided not to add something to the satellite API instead of going through gateway with ListObjects calls? I may be missing some context, but just looking at the gateway change it seems like calling ListObjects might not be an efficient way of doing it.

BTW: it would be helpful if background info is added to the issue and the commit message to aid in reviewing.

[storj-gerrit]

Change pkg/{server, minio}: custom endpoint to return total object count mentions this issue.

[ferristocrat]

@halkyon - The context is in storj/storj#6080 (added to description as well)

[halkyon]

@ferristocrat Unfortunately it doesn't have everything, there's at least a couple of Slack threads with more details I had to go hunt down. Part of my problem generally is our reliance on Slack too much (it's horrible as a documentation source, and it's exclusive to open source contributions), but that's for another discussion ;)

Anyway, judging by some discussions there's still some work to figure out what to do on this.

[ferristocrat]

GitHub Thread discussing options: https://storj.slack.com/archives/CAQV0AE2Z/p1698676617000929

[NiaStorj]

11/19/24

@ferristocrat is this ticket still needed?