Streamline deployment of GESIS stage server #3090

rgaiacs · 2024-09-06T15:33:27Z

This is related to #2797

The configuration in the ansible folder is working and GitLab CI at .gitlab-ci.yml is also working.

I'm trying to complete the Kubernetes cluster configuration in the Helm chart.

rgaiacs · 2024-09-06T15:39:04Z

@manics @sgibson91 @minrk could you help me to understand what Helm chart configuration is being loaded by mistake? The binder pod crashes with the following log

Loading /etc/binderhub/config/values.yaml
Loading extra config: 01-eventlog
Loading extra config: 01-template-variables
Loading extra config: 02-badge-base-url
Loading extra config: 02-event-loop-metric
[BinderHub] starting!
[BinderHub] WARNING | BinderHub.build_node_selector is deprecated, use KubernetesBuildExecutor.node_selector
[BinderHub] WARNING | BinderHub.build_docker_host is deprecated, use KubernetesBuildExecutor.docker_host
[W 240906 15:36:29 _metadata:139] Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
[W 240906 15:36:32 _metadata:139] Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: timed out
[W 240906 15:36:35 _metadata:139] Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
[W 240906 15:36:35 _default:338] Authentication failed using Compute Engine authentication due to unavailable metadata server.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/binderhub/__main__.py", line 4, in <module>
    main()
  File "/usr/local/lib/python3.11/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
  File "/usr/local/lib/python3.11/site-packages/binderhub/app.py", line 913, in initialize
    self.event_log = EventLog(parent=self)
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/binderhub/events.py", line 51, in __init__
    self.handlers = self.handlers_maker(self)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 18, in _make_eventsink_handler
  File "/usr/local/lib/python3.11/site-packages/google/cloud/logging_v2/client.py", line 122, in __init__
    super(Client, self).__init__(
  File "/usr/local/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 320, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "/usr/local/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 268, in __init__
    project = self._determine_default(project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 287, in _determine_default
    return _determine_default_project(project)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/cloud/_helpers/__init__.py", line 152, in _determine_default_project
    _, project = google.auth.default()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/google/auth/_default.py", line 691, in default
    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

GESIS runs the BinderHub server on bare metal.

manics

mybinder.org members send events to Google Stackdriver/Google Cloud Operations:

mybinder.org-deploy/mybinder/values.yaml

Lines 198 to 227 in 1940651

    
             extraConfig: 
        
               # Send Events to StackDriver on Google Cloud 
        
               # This doesn't need any extra permissions, since the GKE nodes have 
        
               # permission to write to StackDriver by default. We don't block access 
        
               # to cloud metadata in binderhub pod, so this should 'just work'. 
        
               01-eventlog: | 
        
                 import os 
        
                 import google.cloud.logging 
        
                 import google.cloud.logging.handlers 
        
                 from traitlets.log import get_logger 
        
                 # importing google cloud configures a root log handler, 
        
                 # which prevents tornado's pretty-logging 
        
                 import logging 
        
                 logging.getLogger().handlers = [] 
        
                 class JSONCloudLoggingHandler(google.cloud.logging.handlers.CloudLoggingHandler): 
        
                     def emit(self, record): 
        
                         record.name = None 
        
                         super().emit(record) 
        
                 def _make_eventsink_handler(el): 
        
                     client = google.cloud.logging.Client() 
        
                     # These events are not parsed as JSON in stackdriver, so give it a different name 
        
                     # for now. Should be fixed in https://github.com/googleapis/google-cloud-python/pull/6293 
        
                     name = os.environ.get("EVENT_LOG_NAME") or "binderhub-events-text" 
        
                     get_logger().info("Sending event logs to %s/logs/%s", client.project, name) 
        
                     return [JSONCloudLoggingHandler(client, name=name)] 
        
                 c.EventLog.handlers_maker = _make_eventsink_handler

If you're haven't disabled this in your existing deployment you should have a secret eventsArchiver.serviceAccountKey.

I noticed some of your Ansible roles include configuration values that will be specific to Gesis. I think we should move those into a Gesis specific subfolder in case we want to use Ansible for other members in future.

manics · 2024-09-07T14:18:36Z

ansible/roles/k8s-control-panel/files/usr/bin/orc2-fix-dind-bot.py

@@ -0,0 +1,146 @@
+"""Script to identify when Docker-in-Docker stop working."""


The workaround of removing an incorrect DinD directory was added into BinderHub in
jupyterhub/binderhub#1828

manics · 2024-09-07T14:23:39Z

mybinder/templates/ingress-nginx/ingress.yaml

+metadata:
+  name: {{ .Release.Name }}
+  annotations:
+    nginx.ingress.kubernetes.io/rewrite-target: /


Is this needed? Isn't / the default?

manics · 2024-09-07T14:27:29Z

mybinder/templates/ingress-nginx/ingress.yaml

@@ -0,0 +1,24 @@
+{{- $ingressType := index .Values "ingress-nginx" "controller" "service" "type" }}
+{{- if eq $ingressType "ClusterIP" }}


Do you have two layers of ingress? Since this is specific to your setup I think we should make it an explicit flag rather than auto-configuring based on an internal config value. This will make maintenance easier across the multiple deployments.

rgaiacs · 2024-09-10T07:41:29Z

Thanks @manics for the reply and comments. I was able to disable the the attempt to contact Google Cloud with

mybinder.org-deploy/config/curvenote.yaml

Lines 252 to 253 in 1940651

    
           analyticsPublisher: 
        
             enabled: false

The problem that I have is that all persistent volume claims are pending.

kubectl get -n gesis pvc

NAME                                        STATUS    VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
binderhub-grafana                           Pending                                                           24h
binderhub-harbor-jobservice                 Pending                                                           4d16h
binderhub-harbor-registry                   Pending                                                           4d16h
binderhub-prometheus-server                 Pending                                            standard       24h
data-binderhub-harbor-redis-0               Bound     alertmanager   5Gi        RWO                           4d16h
data-binderhub-harbor-trivy-0               Pending                                                           4d16h
database-data-binderhub-harbor-database-0   Pending                                                           4d16h
hub-db-dir                                  Pending                                                           24h

I know that I need to declare a correct persistent volume but I can't find where the persistent volume is declared for OVH or CurveNote. @manics can you point me to the persistent volume declaration? Thanks!

rgaiacs · 2024-09-10T14:49:18Z

I have the main pods running.

kubectl get -n gesis pods

NAME                                                     READY   STATUS             RESTARTS   AGE
binder-7c84c576c-2689p                                   1/1     Running            0          80m
binderhub-cryptnono-c9hrj                                2/2     Running            0          128m
binderhub-cryptnono-dgr4g                                2/2     Running            0          128m
binderhub-cryptnono-hqpzf                                2/2     Running            0          128m
binderhub-cryptnono-pbqlx                                2/2     Running            0          128m
binderhub-dind-ntxvs                                     1/1     Running            0          80m
binderhub-grafana-9d48bc74-qtn4x                         1/1     Running            0          62m
binderhub-image-cleaner-6zc9v                            1/1     Running            0          80m
binderhub-ingress-nginx-controller-6fdbf98688-j29w2      1/1     Running            0          47m
binderhub-ingress-nginx-defaultbackend-5d698c868-qh5zx   1/1     Running            0          128m
binderhub-kube-state-metrics-8547b9d4dd-rr4tw            1/1     Running            0          128m
binderhub-prometheus-node-exporter-4dv2s                 1/1     Running            0          128m
binderhub-prometheus-node-exporter-c8bv7                 1/1     Running            0          128m
binderhub-prometheus-node-exporter-gkxcf                 1/1     Running            0          128m
binderhub-prometheus-node-exporter-wfk7h                 1/1     Running            0          128m
binderhub-prometheus-server-7c59dd5d85-fwbqm             2/2     Running            0          128m
hub-6564cd475f-nxltz                                     1/1     Running            0          13m
minesweeper-bf58z                                        0/1     ImagePullBackOff   0          128m
minesweeper-fkjd6                                        0/1     ImagePullBackOff   0          128m
minesweeper-t2fs8                                        0/1     ImagePullBackOff   0          128m
proxy-f5b566ddc-j7l9l                                    1/1     Running            0          80m
proxy-patches-85b5998bdb-9mjw9                           1/1     Running            0          128m
static-6f64c6bc8-ndn2t                                   1/1     Running            0          128m
user-scheduler-55df956bcf-6b4m6                          1/1     Running            0          80m
user-scheduler-55df956bcf-db79g                          1/1     Running            0          80m

Ingress

The ingress is not working. The goal here is to have http://notebooks-test.gesis.org being answer by the NGINX Ingress pod. @manics can you help me?

ping -c 1 notebooks-test.gesis.org

PING notebooks-test.gesis.org (194.95.75.20) 56(84) bytes of data.
64 bytes from svko-css-backup-node.gesis.intra (194.95.75.20): icmp_seq=1 ttl=61 time=2.26 ms

--- notebooks-test.gesis.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.261/2.261/2.261/0.000 ms

kubectl -n gesis describe ingress binderhub

Name:             binderhub
Labels:           app.kubernetes.io/managed-by=Helm
Namespace:        gesis
Address:          10.100.230.222
Ingress Class:    <none>
Default backend:  <default>
TLS:
  kubelego-tls-binder-binderhub terminates notebooks-test.gesis.org
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  notebooks-test.gesis.org  
                            /   binder:80 (10.244.255.21:8585)
Annotations:                kubernetes.io/ingress.class: nginx
                            kubernetes.io/tls-acme: true
                            meta.helm.sh/release-name: binderhub
                            meta.helm.sh/release-namespace: gesis
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    54m   nginx-ingress-controller  Scheduled for sync
  Normal  Sync    54m   nginx-ingress-controller  Scheduled for sync
  Normal  Sync    53m   nginx-ingress-controller  Scheduled for sync

kubectl -n gesis describe service binderhub-ingress-nginx-controller

Name:              binderhub-ingress-nginx-controller
Namespace:         gesis
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=binderhub
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/part-of=ingress-nginx
                   app.kubernetes.io/version=1.11.2
                   helm.sh/chart=ingress-nginx-4.11.2
Annotations:       meta.helm.sh/release-name: binderhub
                   meta.helm.sh/release-namespace: gesis
Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=binderhub,app.kubernetes.io/name=ingress-nginx
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.100.230.222
IPs:               10.100.230.222
Port:              http  80/TCP
TargetPort:        http/TCP
Endpoints:         10.244.65.205:80
Port:              https  443/TCP
TargetPort:        https/TCP
Endpoints:         10.244.65.205:443
Session Affinity:  None
Events:            <none>

`minesweeper`

The image name is wrong. It is trying to pull jupyterhub/mybinder.org-minesweeper:set-by-chartpress.

kubectl -n gesis describe pod minesweeper-bf58z

Name:             minesweeper-bf58z
Namespace:        gesis
Priority:         0
Service Account:  minesweeper
Node:             svko-css-backup-node/194.95.75.20
Start Time:       Tue, 10 Sep 2024 14:27:52 +0200
Labels:           app=binder
                  component=minesweeper
                  controller-revision-hash=767d8795cc
                  heritage=Helm
                  name=minesweeper
                  pod-template-generation=1
                  release=binderhub
Annotations:      checksum/configmap: 7a857debb16fa8bcb22a5de6418a5ff319c9e06f4cfc010705caec539b9614cc
                  cni.projectcalico.org/containerID: a3415f68c66691989387a7ea9bc5c6dd5cfa8039affee823adfd0a9b8f0b7263
                  cni.projectcalico.org/podIP: 10.244.65.206/32
                  cni.projectcalico.org/podIPs: 10.244.65.206/32
Status:           Pending
IP:               10.244.65.206
IPs:
  IP:           10.244.65.206
Controlled By:  DaemonSet/minesweeper
Containers:
  minesweeper:
    Container ID:  
    Image:         jupyterhub/mybinder.org-minesweeper:set-by-chartpress
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      python
      /srv/minesweeper/minesweeper.py
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  250Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      NAMESPACE:  gesis
    Mounts:
      /etc/minesweeper from config (ro)
      /srv/minesweeper from src (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5wbfq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  src:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      minesweeper-src
    Optional:  false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      minesweeper-config
    Optional:  false
  kube-api-access-5wbfq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 hub.jupyter.org/dedicated=user:NoSchedule
                             hub.jupyter.org_dedicated=user:NoSchedule
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason   Age                     From     Message
  ----    ------   ----                    ----     -------
  Normal  BackOff  4m51s (x545 over 129m)  kubelet  Back-off pulling image "jupyterhub/mybinder.org-minesweeper:set-by-chartpress"

manics · 2024-09-10T16:19:26Z

Can you try running an ephemeral pod in the same namespace, and exec something like curl -v http://binderhub-ingress-nginx-controller/ from the pod? That should return a 404 from the Nginx controller default backend. You might need to add the internal service port. Note the existing pods may be restricted by NetworkPolicies, so best to create a new pod. I often use https://gist.github.com/manics/67efaed42d25cc1f830e0d5566652b03 as netshoot includes several useful tools for troubleshooting networks.

Then try curl -v --header 'Host: notebooks-test.gesis.org' http://binderhub-ingress-nginx-controller/ from the pod which should fool the ingress controller into thinking you've requested notebooks-test.gesis.org.

If that works it means the controller and your internal BinderHub/JupyterHub ingress is (probably!) working, and the problem is likely in the path between the external internet and the internal ingress.

manics · 2024-09-10T16:27:27Z

For the chartpress tag problem you'll need to first run chartpress --skip-build to update the set-by-chartpress placeholders:

mybinder.org-deploy/.github/workflows/cd.yml

Lines 322 to 324 in 1940651

    
                 - name: "Stage 3: Run chartpress to update values.yaml" 
        
                   run: | 
        
                     chartpress ${{ matrix.chartpress_args || '--skip-build' }}

The actual building and pushing of the container images is done in the staging workflow, and since chartpress deterministically generates the tag based on git commit hash it's fine to rerun it to update the tags.

rgaiacs · 2024-09-11T15:43:27Z

Thanks @manics for the reply. I will look into chartpress. And I believe the problem with traffic is because of the load balancer. I looking at MetalLB.

rgaiacs · 2024-09-25T09:14:38Z

@manics can I have a bit of help with the pre-commit CI? Anything that I could do for it to reformat the code automatically?

sgibson91 · 2024-09-25T09:47:59Z

@rgaiacs You can run pre-commit run -a locally and commit/push the result. I think prettier specifically doesn't write in CI for reasons that are documented somewhere but I will need to find the link.

sgibson91 · 2024-09-25T09:52:37Z

Add missing extends.

WIIP: Configure Access to Harbor See merge request methods-hub/interactive-environment!21

Update value of Ansible K8S_INGRESS See merge request methods-hub/interactive-environment!22

Read https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/

Read https://github.com/kubernetes/ingress-nginx

Steps to configure NGINX Ingress See merge request methods-hub/interactive-environment!23

Fix no chart version See merge request methods-hub/interactive-environment!24

Configure Ingress for Harbor See merge request methods-hub/interactive-environment!25

rgaiacs · 2025-01-07T17:26:20Z

I'm closing this as after some discussion with @arnim, will be better for us at GESIS to handle the Kubernetes deployment to our bare-metal server on a separate Git repository.

Thanks for all the help!

rgaiacs self-assigned this Sep 6, 2024

manics reviewed Sep 7, 2024

View reviewed changes

rgaiacs force-pushed the gesis branch from 556188d to 41b46ed Compare September 25, 2024 09:03

rgaiacs requested a review from manics September 25, 2024 09:10

rgaiacs added 17 commits January 2, 2025 19:13

Add Ansible inventory for GESIS stage cluster

ddbfe6d

Add k8s-common role

2eee2cb

Add Ansible playbook for GESIS

0b9eee2

Add GitLab CI Kubernetes agent for stage

751696b

Add GitLab CI

96e940a

Add Ansible vault

43bec3a

Remove manual option for GitLab CI

ce9454c

Remove ssh-key-type from GitLab CI

9944c78

Upgrade Ansible Component

239265c

Upgrade Ansible Component

12ae72e

Add smoke test

f58e1c8

Add EditorConfig

6048419

Fix warming from Ansible

abe6e30

Remove rules for debug

0db8dfe

Fix Ansible Vault

7633776

Fix SSH key type

36b439d

Enable root for Ansible

b279cdf

rgaiacs added 10 commits January 3, 2025 09:10

Fix lint errors

834df81

Remove white space from file name

cce9d24

Add missing information to Helm deploy

6e10155

Add missing variable

6685c92

Update storage class name

f37f52c

Fix name of storage unit

34eda4a

Use variable for harbor version

ac60a4f

Fix metadata for PersistentVolume

38ca8e4

Fix persistent volume

f30d28a

Update values for Harbor Helm chart

7bbbc30

rgaiacs force-pushed the gesis branch from 5b256aa to 7bbbc30 Compare January 3, 2025 08:11

rgaiacs and others added 17 commits January 3, 2025 09:13

Fix typo in GitLab CI

cf7dda7

Update GitLab CI rules

9850e9a

Improve GitLab CI

60b4269

Fix GitLab CI

306c7b3

Add missing extends.

Change type of expose for Harbor

1f15674

Merge branch '18-configure-access-harbor' into 'gesis'

7da0eee

WIIP: Configure Access to Harbor See merge request methods-hub/interactive-environment!21

Update value of Ansible K8S_INGRESS

34094e0

Merge branch '18-configure-access-harbor' into 'gesis'

7393b63

Update value of Ansible K8S_INGRESS See merge request methods-hub/interactive-environment!22

Change Harbor expose type to ClusterIP

4f516f7

Read https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/

Configure Ingress NGINX

67c29df

Read https://github.com/kubernetes/ingress-nginx

Merge branch '18-ingress' into 'gesis'

15e45d8

Steps to configure NGINX Ingress See merge request methods-hub/interactive-environment!23

Fix Kubernetes configuration file is group-readable

00446b2

Force update Helm repository

2d90927

Merge branch '18-ingress-fix' into 'gesis'

14e1087

Fix no chart version See merge request methods-hub/interactive-environment!24

Configure Ingress for Harbor

a201407

Merge branch '18-add-ingress-resource' into 'gesis'

26aec89

Configure Ingress for Harbor See merge request methods-hub/interactive-environment!25

Improve use of Ansible roles variables

60d4e61

rgaiacs closed this Jan 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streamline deployment of GESIS stage server #3090

Streamline deployment of GESIS stage server #3090

rgaiacs commented Sep 6, 2024

rgaiacs commented Sep 6, 2024

manics left a comment

manics Sep 7, 2024

manics Sep 7, 2024

manics Sep 7, 2024

rgaiacs commented Sep 10, 2024 •

edited

Loading

rgaiacs commented Sep 10, 2024

manics commented Sep 10, 2024 •

edited

Loading

manics commented Sep 10, 2024

rgaiacs commented Sep 11, 2024

rgaiacs commented Sep 25, 2024

sgibson91 commented Sep 25, 2024

sgibson91 commented Sep 25, 2024

rgaiacs commented Jan 7, 2025

	extraConfig:
	# Send Events to StackDriver on Google Cloud
	# This doesn't need any extra permissions, since the GKE nodes have
	# permission to write to StackDriver by default. We don't block access
	# to cloud metadata in binderhub pod, so this should 'just work'.
	01-eventlog: \|
	import os

	import google.cloud.logging
	import google.cloud.logging.handlers
	from traitlets.log import get_logger

	# importing google cloud configures a root log handler,
	# which prevents tornado's pretty-logging
	import logging
	logging.getLogger().handlers = []

	class JSONCloudLoggingHandler(google.cloud.logging.handlers.CloudLoggingHandler):
	def emit(self, record):
	record.name = None
	super().emit(record)

	def _make_eventsink_handler(el):
	client = google.cloud.logging.Client()
	# These events are not parsed as JSON in stackdriver, so give it a different name
	# for now. Should be fixed in https://github.com/googleapis/google-cloud-python/pull/6293
	name = os.environ.get("EVENT_LOG_NAME") or "binderhub-events-text"
	get_logger().info("Sending event logs to %s/logs/%s", client.project, name)
	return [JSONCloudLoggingHandler(client, name=name)]
	c.EventLog.handlers_maker = _make_eventsink_handler

		@@ -0,0 +1,146 @@
		"""Script to identify when Docker-in-Docker stop working."""

		@@ -0,0 +1,24 @@
		{{- $ingressType := index .Values "ingress-nginx" "controller" "service" "type" }}
		{{- if eq $ingressType "ClusterIP" }}

Streamline deployment of GESIS stage server #3090

Streamline deployment of GESIS stage server #3090

Conversation

rgaiacs commented Sep 6, 2024

rgaiacs commented Sep 6, 2024

manics left a comment

Choose a reason for hiding this comment

manics Sep 7, 2024

Choose a reason for hiding this comment

manics Sep 7, 2024

Choose a reason for hiding this comment

manics Sep 7, 2024

Choose a reason for hiding this comment

rgaiacs commented Sep 10, 2024 • edited Loading

rgaiacs commented Sep 10, 2024

Ingress

minesweeper

manics commented Sep 10, 2024 • edited Loading

manics commented Sep 10, 2024

rgaiacs commented Sep 11, 2024

rgaiacs commented Sep 25, 2024

sgibson91 commented Sep 25, 2024

sgibson91 commented Sep 25, 2024

rgaiacs commented Jan 7, 2025

rgaiacs commented Sep 10, 2024 •

edited

Loading

`minesweeper`

manics commented Sep 10, 2024 •

edited

Loading