bento-platform · mjdupont12 · Jan 6, 2025 · Dec 5, 2024 · Dec 9, 2024 · Dec 9, 2024
diff --git a/README.md b/README.md
@@ -39,6 +39,7 @@ that make up the Bento platform.
 * [Monitoring](./docs/monitoring.md)
 * [Public discovery configuration](./docs/public_discovery.md)
 * [Using a reverse proxy in front of Bento](./docs/reverse-proxy.md)
+* [MinIO object storage](./docs/minio.md)
 
 ### Data ingestion and usage
 

diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -14,6 +14,7 @@ include:
       - lib/gohan/docker-compose.gohan.yaml  # Optional feature; controlled by a compose profile
       - lib/katsu/docker-compose.katsu.yaml
       - lib/logs/docker-compose.logs.yaml
+      - lib/minio/docker-compose.minio.yaml
       - lib/notification/docker-compose.notification.yaml
       - lib/public/docker-compose.public.yaml  # Optional feature; controlled by a compose profile
       - lib/redis/docker-compose.redis.yaml

diff --git a/docs/deployment.md b/docs/deployment.md
@@ -9,6 +9,7 @@ BENTOV2_DOMAIN=bento.example.com
 BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
 BENTOV2_AUTH_DOMAIN=auth.${BENTOV2_DOMAIN}
 BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}
 ```
 
 For a real deployment, make sure that your `local.env` file uses valid domain names for which SSL certificates 

diff --git a/docs/img/minio_object_store.png b/docs/img/minio_object_store.png
diff --git a/docs/migrating_to_18.md b/docs/migrating_to_18.md
@@ -19,7 +19,9 @@ TODO
   above location, or copy `branding.png` to `branding.lightbg.png`
 
 
-TODO
+## 3. Enabling MinIO
+
+To enable the deployment of a MinIO server for S3 storage, refer to the documentation on [configuring MinIO for Bento](/docs/minio.md).
 
 
 ## TODO. Restart services

diff --git a/docs/minio.md b/docs/minio.md
@@ -0,0 +1,128 @@
+# Bento MinIO
+
+Bento can be deployed with a [MinIO](https://github.com/minio/minio) service.
+The MinIO service provides an [S3 compatible API](https://min.io/docs/minio/linux/reference/s3-api-compatibility.html) 
+for object storage.
+
+It is mostly intended for development and testing purposes with the S3 API, 
+as we are working on enabling S3 storage for Bento.
+
+For deployments, Bento instances requiring large storage capacity should rely on an external managed 
+S3 storage provider (AWS, Ceph, production MinIO cluster, etc). 
+This will allow Bento deployments to be decoupled from disk storage concerns.
+
+## Routing
+![MinIO local deployment](./img/minio_object_store.png)
+
+**Note:** As shown above, the MinIO service relies on disk storage, which is why
+production environments should consider external S3 storage instead.
+
+## Configuration
+
+To enable the MinIO service in a Bento deployment, please follow the instructions bellow.
+
+### Environment variables
+
+Enable MinIO by setting the feature flag and other required variables in `local.env`.
+
+```bash
+BENTO_MINIO_ENABLED='true'
+BENTO_MINIO_ROOT_USER=root                  # default value, can be changed
+BENTO_MINIO_ROOT_PASSWORD=secure-password   # change to a secure pw
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}  # MUST be a subdomain of BENTOV2_DOMAIN
+```
+
+### Domain resolution
+
+In a VM using a trusted certificate authority, there should be a DNS record for `BENTO_MINIO_DOMAIN`.
+
+In a local development environment, you must specify how `BENTO_MINIO_DOMAIN` should be resolved, 
+simulating a DNS record for self-signed certificates.
+
+Assuming `BENTO_MINIO_DOMAIN=minio.bentov2.local`, add the following line to your `/etc/hosts` file:
+
+```bash
+# /etc/hosts
+127.0.0.1       minio.bentov2.local
+```
+
+### Initialize MinIO certificates, networking and directories
+
+After enabling the MinIO feature flag for the first time and setting domain resolution, 
+you must initialize the Docker networks, mounted directories and certs.
+```bash
+./bentoctl.bash init-certs -f   # creates the self-signed certificate for MinIO
+./bentoctl.bash init-docker     # creates the Docker network for MinIO
+./bentoctl.bash init-dirs       # creates MinIO's data directory to be mounted
+```
+
+### Start MinIO
+
+If all previous steps were performed correctly, you are ready to restart the 
+gateway and start the MinIO service!
+
+```bash
+# Will recreate the gateway container automatically and start MinIO.
+./bentoctl.bash run
+```
+
+## Using the console
+
+The console can be accessed using a web browser, simply navigate to [minio.bentov2.local/minio/ui/](https://minio.bentov2.local/minio/ui/).
+
+Authenticate using `BENTO_MINIO_ROOT_USER` and `BENTO_MINIO_ROOT_PASSWORD`.
+
+Once logged in, you can issue access keys. These keys can be used to make 
+S3 API calls.
+
+## Using MinIO's S3 API
+
+Assuming you created an access key in the console and saved the values, 
+you are ready to make object storage operations through the S3 API.
+
+Interactions between clients and the S3 API all take place over HTTP.
+Many CLI tools and libraries are available to simplify these operations.
+
+### S3cmd
+
+[S3cmd](https://s3tools.org/s3cmd) is a popular CLI tool to interact with object stores that
+support the S3 protocol, including MinIO.
+
+Once S3cmd is installed on a machine, you can create a s3cmd configuration file for the S3 API
+endpoint of your choice.
+
+```bash
+# ~/.s3cfg-minio-local
+host_base = minio.bentov2.local     # S3 API endpoint (local here)
+host_bucket = minio.bentov2.local   
+use_https = True                    # Use HTTPS
+
+# For dev self-signed certs only
+check_ssl_certificate = False       # Enable if using trusted CA
+
+# Setup access keys
+access_key = <OBTAIN FROM MINIO CONSOLE>
+secret_key = <OBTAIN FROM MINIO CONSOLE>
+```
+
+With the S3cmd config file in place, you can start creating buckets, uploading files, and much more.
+
+```bash
+# list buckets (empty at first)
+s3cmd -c ~/.s3cfg-minio-local ls
+
+# Create a bucket named 'test'
+s3cmd -c ~/.s3cfg-minio-local mb s3://test
+
+# Upload a file to your new bucket!
+s3cmd -c ~/.s3cfg-minio-local put some-file.txt s3://test/some-file.txt
+```
+
+### Boto3
+
+[Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#installation) 
+is an official AWS Python package to interact with an S3 compatible object store.
+
+Like S3cmd, it must be configured to use access keys.
+
+Future work on DRS and Drop-Box will involve Boto3 to enable S3 storage in Bento.
diff --git a/etc/bento.env b/etc/bento.env
@@ -34,7 +34,7 @@ BENTOV2_GATEWAY_INTERNAL_CERTS_DIR=/usr/local/openresty/nginx/certs
 
 # Gateway
 BENTOV2_GATEWAY_IMAGE=ghcr.io/bento-platform/bento_gateway
-BENTOV2_GATEWAY_VERSION=0.13.2
+BENTOV2_GATEWAY_VERSION=edge
 BENTOV2_GATEWAY_VERSION_DEV=${BENTOV2_GATEWAY_VERSION}-dev
 BENTOV2_GATEWAY_CONTAINER_NAME=${BENTOV2_PREFIX}-gateway
 
@@ -472,3 +472,13 @@ BENTO_GRAFANA_SIGNOUT_REDIRECT_URL=https://${BENTOV2_AUTH_DOMAIN}/realms/${BENTO
 BENTO_PROMTAIL_IMAGE=grafana/promtail
 BENTO_PROMTAIL_IMAGE_VERSION=3.1.2
 BENTO_PROMTAIL_CONTAINER_NAME=${BENTOV2_PREFIX}-promtail
+
+# MinIO
+BENTO_MINIO_IMAGE=quay.io/minio/minio
+BENTO_MINIO_IMAGE_VERSION=RELEASE.2024-12-13T22-19-12Z
+BENTO_MINIO_CONTAINER_NAME=${BENTOV2_PREFIX}-minio
+BENTO_MINIO_DATA_DIR=${BENTO_SLOW_DATA_DIR}/minio/data
+BENTO_MINIO_NETWORK=${BENTOV2_PREFIX}-minio-net
+BENTO_MINIO_ROOT_USER=root
+BENTO_MINIO_INTERNAL_PORT=9000
+BENTO_MINIO_CONSOLE_PORT=9001
diff --git a/etc/bento_deploy.env b/etc/bento_deploy.env
@@ -14,6 +14,7 @@ BENTO_BEACON_NETWORK_ENABLED='false'
 BENTO_CBIOPORTAL_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
 BENTO_MONITORING_ENABLED='false'
+BENTO_MINIO_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -36,6 +37,8 @@ BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
 BENTOV2_AUTH_DOMAIN=bentov2auth.local
 # Unused if cBioPortal is disabled:
 BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+# Unused if MinIO is disabled
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}
 # ---------------------------------------------------------------------
 
 # Authn/z -------------------------------------------------------------
@@ -78,6 +81,9 @@ BENTOV2_KATSU_APP_SECRET=  # TODO: SET ME WHEN DEPLOYING!
 # Reference
 BENTO_REFERENCE_DB_PASSWORD=  # TODO: SET ME WHEN DEPLOYING!
 
+# MinIO
+BENTO_MINIO_ROOT_PASSWORD= # TODO: SET ME WHEN DEPLOYING!
+
 # HTTPS Certificates --------------------------------------------------
 BENTOV2_CERTS_DIR=${PWD}/certs
 BENTOV2_AUTH_CERTS_DIR=${BENTOV2_CERTS_DIR}/auth

diff --git a/etc/bento_dev.env b/etc/bento_dev.env
@@ -14,6 +14,7 @@ BENTO_BEACON_NETWORK_ENABLED='false'
 BENTO_CBIOPORTAL_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
 BENTO_MONITORING_ENABLED='false'
+BENTO_MINIO_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -32,6 +33,8 @@ BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
 BENTOV2_AUTH_DOMAIN=bentov2auth.local
 # Unused if cBioPortal is disabled:
 BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+# Unused if MinIO is disabled
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}
 # ---------------------------------------------------------------------
 
 # Authn/z -------------------------------------------------------------
@@ -79,6 +82,9 @@ BENTO_REFERENCE_DB_PASSWORD=devpassword456
 # BENTO_CBIOPORTAL_DATABASE_PASSWORD=somepassword  Required for CBIOPORTAL
 # BENTO_CBIOPORTAL_DATABASE_ROOT_PASSWORD=somepassword  Required for CBIOPORTAL
 
+# MinIO
+BENTO_MINIO_ROOT_PASSWORD=devpassword789
+
 # Development settings ------------------------------------------------
 
 # - Git configuration

diff --git a/etc/default_config.env b/etc/default_config.env
@@ -19,6 +19,7 @@ BENTO_BEACON_NETWORK_ENABLED='false'
 BENTO_CBIOPORTAL_ENABLED='false'
 BENTO_GOHAN_ENABLED='true'
 BENTO_MONITORING_ENABLED='false'
+BENTO_MINIO_ENABLED='false'
 
 #  - Switch to enable French translation in Bento Public
 BENTO_PUBLIC_TRANSLATED='true'
@@ -45,6 +46,8 @@ BENTOV2_PORTAL_DOMAIN=portal.${BENTOV2_DOMAIN}
 BENTOV2_AUTH_DOMAIN=bentov2auth.local
 # Unused if cBioPortal is disabled:
 BENTOV2_CBIOPORTAL_DOMAIN=cbioportal.${BENTOV2_DOMAIN}
+# Unused if MinIO is disabled
+BENTO_MINIO_DOMAIN=minio.${BENTOV2_DOMAIN}
 # ---------------------------------------------------------------------
 
 # Certificates --------------------------------------------------------
@@ -62,6 +65,8 @@ BENTOV2_GATEWAY_INTERNAL_CBIOPORTAL_FULLCHAIN_RELATIVE_PATH=/cbioportal_fullchai
 BENTOV2_GATEWAY_INTERNAL_CBIOPORTAL_PRIVKEY_RELATIVE_PATH=/cbioportal_privkey1.key
 BENTO_GATEWAY_INTERNAL_REDIRECT_FULLCHAIN_RELATIVE_PATH=/redirect_fullchain1.crt
 BENTO_GATEWAY_INTERNAL_REDIRECT_PRIVKEY_RELATIVE_PATH=/redirect_privkey1.key
+BENTO_GATEWAY_INTERNAL_MINIO_FULLCHAIN_RELATIVE_PATH=/minio_fullchain1.crt
+BENTO_GATEWAY_INTERNAL_MINIO_PRIVKEY_RELATIVE_PATH=/minio_privkey1.key
 # ---------------------------------------------------------------------
 
 # Authn/z -------------------------------------------------------------
@@ -105,6 +110,9 @@ BENTOV2_KATSU_DB_PASSWORD=
 BENTO_CBIOPORTAL_DATABASE_PASSWORD=
 BENTO_CBIOPORTAL_DATABASE_ROOT_PASSWORD=
 
+# MinIO
+BENTO_MINIO_ROOT_PASSWORD=
+
 # Development settings ------------------------------------------------
 
 # - Git configuration

diff --git a/lib/gateway/docker-compose.gateway.yaml b/lib/gateway/docker-compose.gateway.yaml
@@ -22,6 +22,7 @@ services:
       - BENTO_BEACON_ENABLED
       - BENTO_CBIOPORTAL_ENABLED
       - BENTO_GOHAN_ENABLED
+      - BENTO_MINIO_ENABLED
       - BENTO_MONITORING_ENABLED
 
       - BENTOV2_GATEWAY_CONTAINER_NAME
@@ -35,6 +36,8 @@ services:
       - BENTOV2_GATEWAY_INTERNAL_CBIOPORTAL_PRIVKEY_RELATIVE_PATH
       - BENTO_GATEWAY_INTERNAL_REDIRECT_FULLCHAIN_RELATIVE_PATH
       - BENTO_GATEWAY_INTERNAL_REDIRECT_PRIVKEY_RELATIVE_PATH
+      - BENTO_GATEWAY_INTERNAL_MINIO_FULLCHAIN_RELATIVE_PATH
+      - BENTO_GATEWAY_INTERNAL_MINIO_PRIVKEY_RELATIVE_PATH
 
       - BENTOV2_GATEWAY_PUBLIC_ALLOW_FRAME_DOMAINS
 
@@ -78,6 +81,10 @@ services:
       - BENTO_CBIOPORTAL_CONTAINER_NAME
       - BENTO_CBIOPORTAL_INTERNAL_PORT
       - BENTO_GRAFANA_CONTAINER_NAME
+      - BENTO_MINIO_CONTAINER_NAME
+      - BENTO_MINIO_INTERNAL_PORT
+      - BENTO_MINIO_CONSOLE_PORT 
+      - BENTO_MINIO_DOMAIN
     networks:
       - aggregation-net
       - auth-net
@@ -89,6 +96,7 @@ services:
       - event-relay-net
       - gohan-api-net
       - katsu-net
+      - minio-net
       - monitoring-net
       - notification-net
       - public-net
@@ -171,6 +179,9 @@ networks:
   katsu-net:
     external: true
     name: ${BENTO_KATSU_NETWORK}
+  minio-net:
+    external: true
+    name: ${BENTO_MINIO_NETWORK}    
   monitoring-net:
     external: true
     name: ${BENTO_MONITORING_NETWORK}

diff --git a/lib/minio/docker-compose.minio.yaml b/lib/minio/docker-compose.minio.yaml
@@ -0,0 +1,30 @@
+services:
+  minio:
+    image: ${BENTO_MINIO_IMAGE}:${BENTO_MINIO_IMAGE_VERSION}
+    container_name: ${BENTO_MINIO_CONTAINER_NAME}
+    expose:
+      - ${BENTO_MINIO_INTERNAL_PORT} 
+      - ${BENTO_MINIO_CONSOLE_PORT} 
+    networks:
+      - minio-net
+    volumes:
+      - ${BENTO_MINIO_DATA_DIR}:/data
+    environment:
+      - MINIO_ROOT_USER=${BENTO_MINIO_ROOT_USER}
+      - MINIO_ROOT_PASSWORD=${BENTO_MINIO_ROOT_PASSWORD}
+      - MINIO_DOMAIN=https://${BENTO_MINIO_DOMAIN}
+      - MINIO_BROWSER_REDIRECT_URL=https://${BENTO_MINIO_DOMAIN}/minio/ui
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      timeout: ${BENTO_HEALTHCHECK_TIMEOUT}
+      interval: ${BENTO_HEALTHCHECK_START_INTERVAL}
+      start_period: ${BENTO_HEALTHCHECK_START_PERIOD}
+      start_interval: ${BENTO_HEALTHCHECK_START_INTERVAL}      
+    profiles:
+      - minio
+    command: server /data --console-address ":${BENTO_MINIO_CONSOLE_PORT}"
+
+networks:
+  minio-net:
+    external: true
+    name: ${BENTO_MINIO_NETWORK}
diff --git a/py_bentoctl/config.py b/py_bentoctl/config.py
@@ -103,6 +103,8 @@ def __init__(self, enabled: bool, profile: str):
     enabled=_env_get_bool("BENTO_CBIOPORTAL_ENABLED", default=False), profile="cbioportal")
 BENTO_FEATURE_GOHAN = BentoOptionalFeature(
     enabled=_env_get_bool("BENTO_GOHAN_ENABLED", default=False), profile="gohan")
+BENTO_FEATURE_MINIO = BentoOptionalFeature(
+    enabled=_env_get_bool("BENTO_MINIO_ENABLED", default=False), profile="minio")
 BENTO_FEATURE_MONITORING = BentoOptionalFeature(
     enabled=_env_get_bool("BENTO_MONITORING_ENABLED", default=False), profile="monitoring")