Allow all external download URLs to be aliased

[rarkins]

Some users need to build their own containerbase-derived images in their product environments which do not have direct internet access. Instead, they need to set up generic proxies on Artifactory for each distinct external host which is approved, such as github.com, nodejs.org, etc.

Here's some examples used by containerbase:

Host	Tool(s)
https://nodejs.org	node
https://api.adoptium.net	java
https://downloads.lightbend.com	scala
https://github.com	sbt, python, php,
https://dot.net	dotnet
https://cache.ruby-lang.org/	ruby

We can assume for now that such users can already redirect common registries like apt, npm, pip, etc - the primary concern is on arbitrary URLs like the above.

The best way to test this would be to have a build environment with external URLs blocked by default and then try to build a "full" image with all tools and find the missing URLs one by one.

The next thing we'd want is a simple way of defining alternative URLs using env.

One possibility would be to have a syntax where the full from/to is specified in separate, related variables. e.g.

URL_REPLACE_0_FROM=https://node.org
URL_REPLACE_0_TO=https://artifactory.company.com/something/nested

Such a syntax is verbose but then at least should cover every case.

A higher level approach would be like JAVA_BASE_URL=https://artifactory.company.com/something/java but has the downsides:

• One tool might try multiple hosts, and
• Multiple tools might use the same host (e.g. github.com)

---

Missing Tools

• bundler (gem) feat(ruby): convert gem tools and add install-gem command #1355
• cocoapods (gem) feat(ruby): convert gem tools and add install-gem command #1355
• dotnet feat(cli): convert dotnet #1347
• hashin (pip) 6b8670c
• pdm (pip) 6b8670c
• pip-tool (pip) 6b8670c
• pipenv (pip) 6b8670c
• poetry (pip) 6b8670c

[viceice]

will do this after conversation to higher level, as it's much easier to implement in typescript.

[rarkins]

I need this within maximum 4 weeks :-/

I think longer term we should take all these URLs and proxy them ourselves through our own CDN/host so that the user only needs to alias one or a few URLs and not 10+

[viceice]

working

• npm can be configured via npm_config_registry env or ~/.npmrc
• pip can be configured via PIP_INDEX_URL env or ~/.config/pip/pip.conf
• gem can be configured via RUBYGEMS_HOST env or ~/.gemrc ^{1 2}

Footnotes

1. https://docs.inedo.com/docs/proget-feeds-rubygem ↩

2. https://guides.rubygems.org/command-reference/#gem-environment ↩

[viceice]

related:

• Log hosts during build #7
• Convert tools to ts #1074

[randygeyer-ws]

Is it sufficient to replace just base urls or will the entire url path need to be set, perhaps minus the artifact file name/ext?

An alternate/simple way to test this is to add /etc/hosts entries for each default hostname to point back to localhost, vs actually blocking the outbound traffic.

[rarkins]

Is it sufficient to replace just base urls or will the entire url path need to be set, perhaps minus the artifact file name/ext?

By base URL I'm not meaning only the host, I mean it could be like https://host.com/some/path too. Is that what you're worried about?

[randygeyer-ws]

Yes. Good deal.

…

__________________________ Randy Geyer | Principal Solutions Architect, Mend | +1 214 926-4907 | ***@***.***

On Wed, Jun 21, 2023 at 12:44 AM Rhys Arkins ***@***.***> wrote: Is it sufficient to replace just base urls or will the entire url path need to be set, perhaps minus the artifact file name/ext? By base URL I'm not meaning only the host, I mean it could be like https://host.com/some/path too. Is that what you're worried about? — Reply to this email directly, view it on GitHub <#1067 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASV56G6L2IEPAAUNZAOVNBTXMKC4FANCNFSM6AAAAAAZI77INE> . You are receiving this because you commented.Message ID: ***@***.***>

[viceice]

The linked PR above will fix this issue for all tools beside the tools installed via gem, npm or pip. Those would need more effort to override the registry urls.

Would it be enough to document how to use the specific package manager environment variables?

Otherwise i would convert those tools and update the default registry urls on the fly, but that needs some more time to implement.

[rarkins]

I think those tools hopefully have their own way of using own registries. I was after examples of redirecting URLs like GitHub and downloads.apache.org

[viceice]

will add the docs and also missing gem, npm and pip support

[rarkins]

@viceice does this require any more code, or just docs?

[viceice]

needs more code too

[EXHades]

please support redirect/replace https://dl.google.com/go

in some company intranet，unable to access dl.google.com directly

[viceice]

please support redirect/replace https://dl.google.com/go

in some company intranet，unable to access dl.google.com directly

already supported

[viceice]

when this build is done, we should have all tools compatible with url replace

https://github.com/containerbase/base/actions/runs/5964382723

[viceice]

We should mention the abilllity to change the default corepack registry (can only be done at runtime)

• https://github.com/nodejs/corepack/tree/main#environment-variables