Support Local Development with Kubernetes with Docker Desktop

[rgaiacs]

No description provided.

[rgaiacs]

When I run

python deploy.py docker-desktop

I get the error

Error: UPGRADE FAILED: failed to create resource: Ingress.networking.k8s.io "static" is invalid: spec: Invalid value: []networking.IngressRule(nil): either `defaultBackend` or `rules` must be specified
Traceback (most recent call last):
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 543, in <module>
    main()
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 539, in main
    deploy(args.release, args.name, args.dry_run, args.diff)
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 241, in deploy
    check_call(helm, dry_run)
  File "/home/raniere/mybinder.org-deploy/deploy.py", line 49, in check_call
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['helm', 'upgrade', '--install', '--cleanup-on-fail', '--create-namespace', '--namespace=docker-desktop', 'docker-desktop', 'mybinder', '-f', 'secrets/config/common/bans.yaml', '-f', 'secrets/config/common/common.yaml', '-f', 'secrets/config/common/cryptnono.yaml', '-f', 'config/docker-desktop.yaml']' returned non-zero exit status 1.

@manics could you point me to the configuration that I'm missing? Thanks!

[manics]

Could it be

mybinder.org-deploy/config/staging.yaml

Lines 136 to 140 in 9a88e71

	static:
	ingress:
	hosts:
	- static.staging.mybinder.org
	- static.gke2.staging.mybinder.org

[manics]

Can we rename this to local or something similar? I don't think this should be specific to docker-desktop, e.g. it should work with a local k3s/microk8s etc. It also means we could run this in GitHub CI using https://github.com/jupyterhub/action-k3s-helm which will allow us to test some changes in PRs instead of having to deploy to staging first.

[rgaiacs]

Thanks @manics for the feedback. I fixed the previously mentioned failed to create resource: Ingress.networking.k8s.io. I also re-wrote the documentation page to be more generic.

Following the provided documentation, the user should be able to

• see the mybinder.org UI
• start the creation of a new container image (for example, based on https://github.com/binder-examples/requirements)
• start a container based on the newly created image
• access the running container

At the moment, the user will get the following error

[I 250108 14:56:29 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250108 14:56:29 launcher:198] Creating user binder-examples-requirements-xwjx20bu for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[E 250108 14:56:49 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-xwjx20bu): HTTP 599: Connection timed out after 20001 milliseconds
[E 250108 14:57:13 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-xwjx20bu): HTTP 599: Connection timed out after 20000 milliseconds

This should be solved by changing /etc/hosts.

[rgaiacs]

I try using

binderhub:
  config:
    BinderHub:
      hub_url: http://jupyterhub.localhost

to fix the Error accessing Hub API but it did not work.

@manics do you have a suggestion?

[manics]

I don't think we should recommend modifying the hosts file, it should be possible to either rely on

• kube-dns to resolve internal service names
• a service like https://nip.io/ so that IP addresses 1.2.3.4 can be treated as a hostname 1.2.3.4.nip.io which is required for Kubernetes ingress

[rgaiacs]

In 2a6514a, the launch requests produces the following log in the binder pod

[I 250109 09:26:27 build:563] Started build build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:27 build:565] Watching build pod build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:29 build:647] Watching logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:36 build:679] Finished streaming logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 09:26:38 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250109 09:26:38 launcher:198] Creating user binder-examples-requirements-c4vstg48 for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[E 250109 09:26:58 launcher:123] Error accessing Hub API (using hub/api/users/binder-examples-requirements-c4vstg48): HTTP 599: Connection timed out after 20001 milliseconds

My understanding is that the binder pods tries to contact the hub pod but the request fails because hub_url is not defined.

Define hub_url to localhost or 127.0.0.1 fails because the binder pod will make a request to itself. The solution here is to use the "public" IP of the machine. In 3f76268, the user is required to provide their "public" IP and the launch requests produces the following log in the binder pod

[I 250109 10:06:13 build:563] Started build build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:13 build:565] Watching build pod build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:18 build:647] Watching logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:19 build:679] Finished streaming logs of build-binder-2dexamples-2drequirements-55ab5c-505-2c
[I 250109 10:06:21 builder:669] Launching server for https://github.com/binder-examples/requirements: 0 other servers running this repo (0 total)
[I 250109 10:06:21 launcher:198] Creating user binder-examples-requirements-guutvpki for image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f
[I 250109 10:06:21 launcher:258] Starting server for user binder-examples-requirements-guutvpki with image binder-2dexamples-2drequirements-55ab5c:50533eb470ee6c24e872043d30b2fee463d6943f

The binder pod successfully contact the hub pod. Unfortunately, the user pod (jupyter-binder-examples-requirements-guutvpki) fails in the tc-init container with

+ tc qdisc add dev eth0 root handle 1: htb default 30
RTNETLINK answers: No such file or directory

Based on https://stackoverflow.com/a/16235349/1802726, the error is because of missing Linux kernel modules that I'm assuming are not available on Docker Desktop.

@manics does this looks good for you? Can we merge? I will try to test on my GNU/Linux machine next week.