Any way to self-throttle downloading?

[toydotgame]

I'd imagine when doing an exportall and using --media would cause a tremendous amount of GET requests or something similar to the Discord servers. Speed isn't a big requirement for me because I just want a general archive of the last 4 years I've spent on Discord, but I have sent nearly 90k messages and likely have access to around 700k-ish messages throughout all my servers and DMs.

This would be an insane amount of data to request at these inhuman speeds from the servers, and I feel like I might get flagged for this. I don't want to lose my account so I'm wondering if there's a way to impose a rate limit on myself that's more at the speed of "some insane person who decided to go back and read the entire channel history," potentially over the course of a week or two, rather than requesting it all as fast as possible over a few hours (or however long it takes).

[Tyrrrz]

There's no need, the app respects rate limits

[toydotgame]

How fast are the rate limits? I'm pretty paranoid about being flagged for anything that would seem unusual. I might try downloading a few channels from big public servers on an alt to calm my nerves.

[Tyrrrz]

It's based on response headers:

DiscordChatExporter/DiscordChatExporter.Core/Discord/DiscordClient.cs

Lines 54 to 83 in c945c98

	// If this was the last request available before hitting the rate limit,
	// wait out the reset time so that future requests can succeed.
	// This may add an unnecessary delay in case the user doesn't intend to
	// make any more requests, but implementing a smarter solution would
	// require properly keeping track of Discord's global/per-route/per-resource
	// rate limits and that's just way too much effort.
	// https://discord.com/developers/docs/topics/rate-limits
	var remainingRequestCount = response
	.Headers
	.TryGetValue("X-RateLimit-Remaining")?
	.Pipe(s => int.Parse(s, CultureInfo.InvariantCulture));
	var resetAfterDelay = response
	.Headers
	.TryGetValue("X-RateLimit-Reset-After")?
	.Pipe(s => double.Parse(s, CultureInfo.InvariantCulture))
	.Pipe(TimeSpan.FromSeconds);
	if (remainingRequestCount <= 0 && resetAfterDelay is not null)
	{
	var delay =
	// Adding a small buffer to the reset time reduces the chance of getting
	// rate limited again, because it allows for more requests to be released.
	(resetAfterDelay.Value + TimeSpan.FromSeconds(1))
	// Sometimes Discord returns an absurdly high value for the reset time, which
	// is not actually enforced by the server. So we cap it at a reasonable value.
	.Clamp(TimeSpan.Zero, TimeSpan.FromSeconds(60));
	await Task.Delay(delay, innerCancellationToken);
	}

Also:

DiscordChatExporter/DiscordChatExporter.Core/Utils/Http.cs

Lines 42 to 50 in c945c98

	// If rate-limited, use retry-after header as the guide.
	// The response can be null here if an exception was thrown.
	if (result.Result?.Headers.RetryAfter?.Delta is { } retryAfter)
	{
	// Add some buffer just in case
	return retryAfter + TimeSpan.FromSeconds(1);
	}
	return TimeSpan.FromSeconds(Math.Pow(2, i) + 1);

https://discord.com/developers/docs/topics/rate-limits

[toydotgame]

Thanks! Hopefully rate limits are all that Discord will do. It doesn't look like Discord would investigate every single rate limit exceeded so I should be okay running this. 👍