could not clone from the remote: stream error: stream ID 407; CANCEL; received from peer

[kishaningithub]

Describe the bug
Getting the following error when running multi-gitter in an github org with more than 1000 repos.

could not clone from the remote: stream error: stream ID 407; CANCEL; received from peer

To Reproduce
Steps to reproduce the behavior:

1. Have an organization that has more than 1000 repos
2. Run multi-gitter run "./change.sh" --concurrent "$(($(nproc) * 2))" --org "<< org name>>" -m "Update" -B "update" --pr-title 'Update' --conflict-strategy replace
3. See error could not clone from the remote: stream error: stream ID 407; CANCEL; received from peer

Expected behavior
I would expect errors like these should be handled internally by multi-gitter using a resilient retry logic.

[lindell]

Please provide more context:

1. What platform are you using?
2. Confirm this is not happening when running on only one repository or in a very small org
3. When is this happening? Right at the start or after x number of repos?
4. What does nproc return on your system? You seem to run this with very high concurrency, can you run it with no concurrency or a low one?

[kishaningithub]

1. What platform are you using?

$ multi-gitter version

multi-gitter version: 0.54.1
Release-Date: 2024-11-21
Go version: go1.23.2
OS: darwin
Arch: arm64
Commit: 5841986bcc7e4db734349878435e4ef33e70aa2c

2. Confirm this is not happening when running on only one repository or in a very small org

This happens for like ~15 repos out of the ~1000+ repos. When i ran it with a filter expression (--repo-include) using selecting like 5 repos this issue did not occur.

3. When is this happening? Right at the start or after x number of repos?

This started happening after ~60th repo

4. What does nproc return on your system? You seem to run this with very high concurrency, can you run it with no concurrency or a low one?

The result of "$(($(nproc) * 2))" in my system is 20.

I ran it now without specifying the concurrency flag, i did not get the above issue.

[Allon-Guralnek]

This project uses go-git, and we're experiencing the same issues with go-git where we have intermittent errors as described. This may not be a multi-gitter issue.

[kishaningithub]

@Allon-Guralnek Can you link the respective go-git issue(s) in this thread ?

[lindell]

@Allon-Guralnek Thanks for pitching in. When are you experiencing this? Is it the same as for @kishaningithub, with something that the git server could block because of multiple requests, or in general?

@kishaningithub What is the result if you run with --git-type=cmd instead?