Docker Refactoring

[MichaelRipa]

Description

This PR introduces significant improvements to the Docker setup and the build process, enhancing both efficiency and reliability. The key changes include:

• Dockerfile Reorganization:

  • Before: Only base.dockerfile and service.dockerfile existed within the ./services directory. The Makefile navigated to services/[name]/ to execute builds using these Dockerfiles.
  • After: Moved all Dockerfiles to a centralized ./docker directory and introduced a third Dockerfile, dockerfile.conda. The three Dockerfiles are:
    • dockerfile.base
    • dockerfile.conda
    • dockerfile.service

• Optimized Conda Environment Setup:

  • Before: Each service individually installed its Conda environment from services/[name]/environment.yml in the base.dockerfile, leading to redundant installations and increased build times.
  • After:
    • dockerfile.base now creates a shared base environment with common dependencies.
    • dockerfile.conda updates this base environment with service-specific dependencies. This reuse of the base image significantly reduces build times by avoiding repeated installations of common packages.

• Makefile Enhancements:

  • Bug Fix: Resolved an issue where N_DEVICES directly utilized nvidia-smi, causing crashes on machines without Nvidia GPUs. The updated Makefile now safely checks for the presence of nvidia-smi before attempting to use it.
  • Build Process: Adapted the Makefile to work with the new Dockerfile structure, ensuring seamless integration and improved build efficiency.

Benchmarks

I conducted a performance comparison between the old and the refactored build setups on the dev deployment:

• Old Setup: 347 seconds
• Refactored Setup: 225 seconds

NOTE: This benchmark excludes the ray_worker service from the build process. Including it would likely result in even more substantial reductions in build duration.

[JohnGouwar]

Overall, this looks good, just two small changes in your Makefile dependencies.

[JohnGouwar]

build_conda should have a dependency on build_base

[JohnGouwar]

build_service should have a dependency on build_conda