fix(cli): org:search:dump issues when dump is fetched across multiple queries

[fbeaudoincoveo]

Proposed changes

Context:

The org:search:dump command executes queries under the hood. When we execute this command, it's possible that we hit the Search API maximum response size. The likelihood of hitting that limit is increased in the HIPAA environment, as the HIPAA maximum response size is 4x smaller than in PROD.

When the dump is executed across multiple queries, we ensure that results are fetched from the same index by passing the indexToken of the initial query to each subsequent queries. This is, by definition, incompatible with using the index cache, as stated in the indexToken parameter documentation:

The Base64 encoded identifier of the index mirror to forward the request to. See also the index parameter.
If you do not specify an indexToken (or index) value, any index mirror could be used.
Note: Passing an indexToken (or index) value has no effect when the results of a specific request can be returned from cache (see the maximumAge parameter).

Another issue is that in order to know what result to start from in each subsequent query after the first, we request results to be ordered by ascending rowid value, and we request only results whose rowid is greater than the rowid of the last result returned in the previous query. However, the rowid value is returned as a very large integer (greater than the maximum safe integer size JavaScript can handle without losing precision). Therefore, the expression rowid>ROW_ID_OF_LAST_RESULT typically behaves like rowid>=ROW_ID_OF_LAST_RESULT because JavaScript rounds the rowid during the JSON.parse conversion of the response body.

Fix:

• Setting maximumAge: 0 on every query performed for fetching source dumps trivially addresses the index cache issue.

• Using response handler to the platform client that uses the new context parameter of the JSON.parse reviver (see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse#parameters) allows us to avoid losing precision when converting rowid values. However, this new parameter is only available from Node 21, so we have to polyfill it for now. EDIT: After discussion with the DX team, this change was made directly in the platform-client project. See fix(success response handler): parse unsafe integers as strings platform-client#834.

Breaking changes

None

Testing

• Unit Tests: Yes
• Functionnal Tests: No, unit tests are sufficient to ensure that maximumAge is set to 0
• Manual Tests: Tested the org:search:dump command locally before / after platform-client update against problematic HIPAA test organization. The duplicate / inconsistent dump result issue is no longer present after the fix.

[github-actions]

Pull Request Report

PR Title

✅ Title follows the conventional commit spec.

[fbeaudoincoveo]

I moved the bulk of the changes to the platform-client project: coveo/platform-client#834

So basically all I'm doing in this PR now is setting maximumAge: 0 on dump queries.

[louis-bompart]

LGTM, after the release of the platform-client and its update in our project ofc

[github-actions]

Sorry @fbeaudoincoveo, this PR does not satisifies all conditions of mergeability:

• Required status check "Lint" is in progress.
• You're not authorized to push to this branch.
• Visit https://docs.github.com/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches for more informat

Modify your pull request to satisfies them and check the box below to try again:

• Merge!

You can also reach out to a maintainer if you think your contribution should be merged regardless of the reported issues.