Remote request retries and testing internet IO

[iboB]

What triggered this are the ahttp tests. From time to time some test fails because of lack of service from the test server. It's a free one: https://httpbin.org/ and we can't hold them accountable for occasional downtime.

This on its own is a problem as CI jobs would be prone to fail for reasons beyond our influence.

In this particular case we have two options:

• Allow the test to fail
• Set up our own test server (it could even be localhost as part of the CI test)

But this is HTTP and it's easy to set up. If we do exotic protocols like AWS S3 or others, setting up our own service might be hard to impossible.

Moreover this got me thinking that we would likely want to employ some retry logic in asset sources which require network IO. Networks (and especially Internet) are prone to sporadic temporary failures, where a retry is a good idea.