docker.compose.up() nonfunctional

[loopyd]

Expected Behavior

docker.compose.up() runs successfully.

Current Behavior

The module crashes because ComposeCLI malforms the command for docker compose instead of the correct docker-compose due to docker api changes (missing the hyphen). This happens on modern versions of docker compose installed from official docker sources.

It is worth of note that many Api for docker compose V3 >= 3.4 are missing in the pydantic models and the developer has not accepted PR for them since October of 2023 (assuming project dead?): See #486 which partially fixes issues with some new Api.

Aside from minor github workflow changes that affect the master branch, there's not been much here lately.

This one is a critical bug that affects core functionality of docker compose for python-on-whales package for recent versions of docker.

Highly recommended to fix bugs affecting core functionality soon, or mark project archived if work should not continue.

Operating Schema

Docker version:

night@NIGHT-PC:~/python-on-whales-example$ docker --version
Docker version 24.0.7, build afdd53b

Compose Version:

night@NIGHT-PC:~/python-on-whales-example$ docker-compose --version
docker-compose version 1.29.2, build unknown

Running on WSL version (docker works otherwise):

night@NIGHT-PC:~/python-on-whales-example$ uname -a
Linux NIGHT-PC 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Python pyenv tool version

night@NIGHT-PC:~/python-on-whales-example$ pyenv --version
pyenv 2.3.32

Conda environment.yml with .python-version on 3.11.6:

name: python-on-whales-example
channels:
  - defaults
  - conda-forge
  - Microsoft
dependencies:
  - python=3.11.6
  - pip
  - wheel
  - setuptools
  - python-on-whales
  - colorama
  - pydantic
prefix: ./.conda

Tested run.py

import os
import sys
from typing import Annotated, Any, Callable, Optional, Type, TypeVar
from python_on_whales import docker
from logging import Formatter, Logger, FileHandler, StreamHandler
from logging import INFO as log_level_info
from logging import DEBUG as log_level_debug
from logging import WARNING as log_level_warning
from logging import ERROR as log_level_error
from logging import CRITICAL as log_level_critical
from pydantic import BaseModel, Field
from colorama import Fore, Back, Style, init as colorama_init

T = TypeVar('T', bound=BaseModel)
def auto_singleton(cls: Type[T]) -> Callable[..., T]:
    instances = {}
    def get_instance(*args, **kwargs) -> T:
        if cls not in instances:
            instances[cls] = cls(*args, **kwargs)
        return instances[cls]

    return get_instance

@auto_singleton
class AppConfig(BaseModel):
    class Config:
        validate_assignment = True
        extra = "forbid"
        arbitrary_types_allowed = True

    color: bool = Field(default=True, alias="color")
    log_level: int = Field(default=log_level_info, alias="log_level")
    log_file: str = Field(default=os.path.abspath(__file__).rsplit("/", 1)[0] + f"/{os.path.basename(__file__)}.log", alias="log_file")
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

class ColoredFormatter(Formatter):
    def __init__(self, log_level_colors: dict[str, dict[int, str]], *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.log_level_colors = log_level_colors

    def format(self, record):
        level_color = self.log_level_colors['level'].get(record.levelno, "")
        asctime_color = self.log_level_colors['asctime'].get(record.levelno, "")
        message_color = self.log_level_colors['message'].get(record.levelno, "")
        record.levelname = level_color + record.levelname + Style.RESET_ALL
        record.asctime = asctime_color + self.formatTime(record, self.datefmt) + Style.RESET_ALL
        record.message = message_color + record.getMessage() + Style.RESET_ALL
        return super().format(record)
    
class PlainFormatter(Formatter):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)

    def format(self, record):
        record.levelname = record.levelname
        record.asctime = self.formatTime(record, self.datefmt)
        record.message = record.getMessage()
        return super().format(record)
    
@auto_singleton
class AppData(BaseModel):
    class Config:
        validate_assignment = True
        extra = "forbid"
        arbitrary_types_allowed = True

    colorama_initialized: bool = Field(False, alias="colorama_initialized")
    logger_initialized: bool = Field(default=False, alias="logger_initialized")
    log_level_colors: dict = Field(default={
            "message": {
                log_level_info: Fore.GREEN + Style.NORMAL,
                log_level_debug: Fore.BLUE + Style.NORMAL,
                log_level_warning: Fore.YELLOW + Style.BRIGHT,
                log_level_error: Fore.RED + Style.BRIGHT,
                log_level_critical: Fore.RED + Back.WHITE + Style.BRIGHT,
            },
            "level": {
                log_level_info: Fore.GREEN + Style.NORMAL,
                log_level_debug: Fore.BLUE + Style.NORMAL,
                log_level_warning: Fore.YELLOW + Style.BRIGHT,
                log_level_error: Fore.RED + Style.BRIGHT,
                log_level_critical: Fore.RED + Back.WHITE + Style.BRIGHT,
            },
            "asctime": {
                log_level_info: Fore.GREEN + Style.NORMAL,
                log_level_debug: Fore.BLUE + Style.NORMAL,
                log_level_warning: Fore.YELLOW + Style.BRIGHT,
                log_level_error: Fore.RED + Style.BRIGHT,
                log_level_critical: Fore.RED + Back.WHITE + Style.BRIGHT,
            },
        }, alias="log_level_colors")
    root_logger: Logger = Field(default=Logger.root, alias="root_logger")
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.colorama_initialized is False:
            colorama_init(autoreset=True, convert=False, strip=False, wrap=True)
            self.colorama_initialized = True
        if self.logger_initialized is False:
            if os.path.exists(AppConfig.log_file) is True:
                os.remove(AppConfig.log_file)
            self.root_logger.setLevel(AppConfig.log_level)
            stream_handler = StreamHandler(stream=sys.stdout)
            stream_handler.setLevel(AppConfig.log_level)
            stream_handler.setFormatter(ColoredFormatter(self.log_level_colors, '[%(levelname)s] [%(asctime)s] %(message)s', datefmt='%Y-%m-%d %H:%M:%S'))
            file_handler = FileHandler(filename=AppConfig.log_file, mode='w')
            file_handler.setLevel(AppConfig.log_level)
            file_handler.setFormatter(PlainFormatter('[%(levelname)s] [%(asctime)s] %(message)s', datefmt='%Y-%m-%d %H:%M:%S'))
            self.root_logger.addHandler(stream_handler)
            self.root_logger.addHandler(file_handler)
            self.logger_initialized = True

AppConfig = AppConfig()
AppData = AppData()

def log(level=log_level_info, message=""):
    if AppConfig.color is True:
        AppData.root_logger.log(level, f"{AppData.log_level_colors['message'][level]}{message}")
    else:
        AppData.root_logger.log(level, message)
    
docker.compose.up(detach=True, color=False) 
log(log_level_info, f"{docker.compose.ps()}")
docker.compose.down()

A docker-compose.yml to use:

version: '3.8'
services:
  postgresdb:
    image: postgres:latest
    restart: always
    container_name: db-postgres
    environment:
      POSTGRES_USER: example
      POSTGRES_DB: example_db
      POSTGRES_PASSWORD: example
    ports:
      - "5432:5432"
    volumes:
      - ./data/postgres-data:/var/lib/postgresql/data
    networks:
      - app-network
  pgmyadmin:
    image: phpmyadmin/phpmyadmin:latest
    restart: always
    container_name: pgp-myadmin
    environment:
      PMA_HOST: postgresdb
      PMA_PORT: 5432
      PMA_USER: example
      PMA_PASSWORD: example
    ports:
      - "8080:80"
    networks:
      - app-network
    
networks:
  app-network:
    name: appnet
    driver: bridge

Crash Traceback

(/home/night/python-on-whales-example/.conda) night@NIGHT-PC:~/python-on-whales-example$ python run.py
unknown flag: --detach
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Common Commands:
  run         Create and run a new container from an image
  exec        Execute a command in a running container
  ps          List containers
  build       Build an image from a Dockerfile
  pull        Download an image from a registry
  push        Upload an image to a registry
  images      List images
  login       Log in to a registry
  logout      Log out from a registry
  search      Search Docker Hub for images
  version     Show the Docker version information
  info        Display system-wide information

Management Commands:
  builder     Manage builds
  buildx*     Docker Buildx (Docker Inc., v0.11.2)
  container   Manage containers
  context     Manage contexts
  image       Manage images
  manifest    Manage Docker image manifests and manifest lists
  network     Manage networks
  plugin      Manage plugins
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Swarm Commands:
  swarm       Manage Swarm

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  import      Import the contents from a tarball to create a filesystem image
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  wait        Block until one or more containers stop, then print their exit codes

Global Options:
      --config string      Location of client config files (default "/home/night/.docker")
  -c, --context string     Name of the context to use to connect to the daemon (overrides
                           DOCKER_HOST env var and default context set with "docker context use")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket to connect to
  -l, --log-level string   Set the logging level ("debug", "info", "warn", "error", "fatal")
                           (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default "/home/night/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default "/home/night/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default "/home/night/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Run 'docker COMMAND --help' for more information on a command.

For more help on how to use Docker, head to https://docs.docker.com/go/guides/

Traceback (most recent call last):
  File "/home/night/python-on-whales-example/run.py", line 124, in <module>
    docker.compose.up(detach=True, color=False) 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/night/python-on-whales-example/.conda/lib/python3.11/site-packages/python_on_whales/components/compose/cli_wrapper.py", line 1015, in up
    run(full_cmd, capture_stdout=quiet, capture_stderr=quiet)
  File "/home/night/python-on-whales-example/.conda/lib/python3.11/site-packages/python_on_whales/utils.py", line 194, in run
    raise DockerException(
python_on_whales.exceptions.DockerException: The docker command executed was `/usr/bin/docker compose up --detach --no-color`.
It returned with code 125
The content of stdout can be found above the stacktrace (it wasn't captured).
The content of stderr can be found above the stacktrace (it wasn't captured).

[LewisGaul]

It is worth of note that many Api for docker compose V3 >= 3.4 are missing in the pydantic models and the developer has not accepted PR for them since October of 2023 (assuming project dead?): See #486 which partially fixes issues with some new Api.

Such assumption ("project dead") comes across as a bit rude/accusatory, especially since you can see multiple merged PRs in the last few weeks, a couple of active discussions, and the linked PR is failing the CI.

Enhancements are welcome, it just needs someone to do the work.

[gabrieldemarmiesse]

From what I understand, it seems that you expect Python-on-whales to call docker compose v1 which is "dead" or "archived" if I'm using your own words. The source for what I claim is here: https://docs.docker.com/compose/migrate/#can-i-still-use-compose-v1-if-i-want-to You should not use it.

Python-on-whales only supports docker compose v2 and the documentation gives instructions to verify that docker compose v2 is installed on your system, which seems to not be the case from the error message you provided.

I do agree that the error message isn't great, and could be improved.

Furthermore, I don't know if english is your first language or not but as @LewisGaul said, the tone is a bit too strong for a public, open source project. I garantee that I review PRs with a CI passing in a timely manner, I don't garantee that I will fix the CI for the people who are submitting PRs.

[loopyd]

Thank you for documentation update with notes clarifying the dependency requirement of compose v2. That is useful. I'm happy to accept dependency on docker compose v2 rather than a compatibility layer to a deprecated V1.

Is a simple fix with addition of V2 package which clarification was needed. To help you I have created proper docker compose installation script: install-compose.sh for linux.

💁‍♀️ The cloud integration repository you have linked was archived November 27, 2023. Recommend edit to docs to link docker/compose repo releases as the place to obtain Compose V2.

For simple users

The instructions on linux, I taylor to make this install-compose.sh script for linux. When a task arises to do I automate it, as a result have quite a large toolchain. Have:

#!/bin/bash

# Docker Compose V2 installation script by DeityDurg
# Please run this script as root for installation of plugin system-wide.

VERSION='v2.24.0-birthday.10'
CLI_PLUGINS_PATH='/usr/local/lib/docker/cli-plugins'
ARCH='x86_64'

if [ ! -d "$CLI_PLUGINS_PATH" ]; then
  echo "Creating directory for docker cli plugins: $CLI_PLUGINS_PATH..."
  {
    mkdir -p $CLI_PLUGINS_PATH
    chown root:docker $CLI_PLUGINS_PATH
    chmod 0775 $CLI_PLUGINS_PATH
  } 2>/dev/null
  if [ $? -ne 0 ]; then
    echo "Failed to create directory: $CLI_PLUGINS_PATH"
  else
    echo "Directory $CLI_PLUGINS_PATH created successfully."
  fi
fi
if [ ! -f "$CLI_PLUGINS_PATH/docker-compose" ]; then
  echo "Installing docker compose binary version $VERSION to $CLI_PLUGINS_PATH"
  {
    curl -sSL "https://github.com/docker/compose/releases/download/$VERSION/docker-compose-linux-$ARCH" -o "$CLI_PLUGINS_PATH/docker-compose"
    chown root:docker "$CLI_PLUGINS_PATH/docker-compose"
    chmod 0770 "$CLI_PLUGINS_PATH/docker-compose"
  } 2>/dev/null
  if [ $? -ne 0 ]; then
    echo "Failed to download docker-compose binary version $CLI_PLUGINS_PATH"
  else
    echo "Installed docker compose binary $VERSION to $CLI_PLUGINS_PATH successfully."
  fi
fi

And now when running:

docker compose --help

It will be using composer-cli-plugin version V2 on Linux. Also recommend add this to docs, if you like.

In regard to harsh tone, it gets a maintainer talking. I make sure maintainers are on top of their stuff., and you are with very detailed information. This is a good project, and useful to me. Good job, you now have a star.

⭐