fetchurl: add flag to try other URLs on hash mismatch

[mschwaig]

Problem Statement

Normally when give a bunch of URLs to fetchurl and fetching the first one leads to a hash mismatch that makes the build fail.

This is almost always sensible behavior, because we expect those URLs to all serve the same content.

I know of one specific example where this is not the case: Maven repositories. It is common for different Maven repositories to offer the same version of the same library, but actually serve slightly different artifacts with a different hash.

This has been a problem for people writing Nix code to deal with Gradle builds (tmcl/minimal-android-nix-example#1), because because Gralde records hashes of artifact but not where it got them from. In that case this means roughly they would have to

• reconstruct Gradle's dependency resolution to try the correct URL from Nix, (not posible according to Leverage Gradle Lockfile tadfisher/gradle2nix#38 (comment))
• teach Gradle to dump that information voluntarily or
• have Nix try to download from a number of URLs (for the differnt Repos) and not fail if the first one does not deliver an artifact with the correct hash, but try the others as well.

Proposed Solution

I have prototyped a tryOtherUrlsOnHashMismatch flag for fetchurl that implements the last option, but I am unsure

• if something like this would be a desired feature and
• what the best way would be to implement it.

Maybe there are some totally different ways to get around this problem in a more elegant way.

You can find my implementation at https://github.com/mschwaig/nixpkgs//try-mirrors-prototype, but beware that it only supports sha-256 in a really wonky way.

The difficult thing about implementing what I did in a less hacky way is that it looks like a real pain to have fetchurl depend on nix for the hash computation because that introduces a bunch of circular dependencies. Maybe someone can suggest a strategy for getting around that by putting this feature somewhere else. I was also a bit unsure if this repo is the right place to discuss the issue, because the circular dependency issues might be something that's best fixed in Nix.

The reason why I spent some time on this even though it might be too niche of an issue to spend a lot of effort on fixing it is that without this things 'kind of work'. If you permutate the order in which you specify your URLs the fetchurl will succeed at some point and that might be worse for user experience than if it was not working at all.

---

I would appreciate any comments or suggestions (Maybe @tadfisher has an opinion on how sensible it is to add something like this).

[NickCao]

Can the difference be normalized with some extra post processing?

[mschwaig]

Can the difference be normalized with some extra post processing?

That's a good point. I don't know that right now. I will have to take a closer look sometime. Thanks!