Use distributed storage instead of container storage: S3

[spwoodcock]

Is your feature request related to a problem? Please describe.

• Currently we download, process, and serve files directly from the FastAPI container.
• This is an acceptable strategy if running a single container, but it does mean that container storage can fill up.
• We have no way to cache submissions Cache submissions #713
• We want to serve cloud-optimised formats if possible Replace geojson features with flatgeobuf #381, then fallback to offline (downloaded) files.
• We need to easily download generated files: Allow downloading of mbtiles per task #890 Allow downloading data extract from FMTM and support upload of the same data extract back to FMTM #889

Describe the solution you'd like

• Distributed storage via S3.
• I propose we use Garage as a lightweight alternative to S3 as part of our software stack.
  • For development and self-hosted installs of FMTM we would use Garage.
  • For production deploys on cloud infrastructure, we would use managed S3 buckets (AWS).
  • This needs to be configurable.
• Then whenever we process a file, it should be uploaded to S3 for later retrieval (by the frontend, another container, etc).

Additional context

Doing this would make it much easier to deploy with something like Kubernetes.

[spwoodcock]

#908 adds Minio to the compose stack, plus some wrapper functions for accessing the bucket.

Two buckets have been added by default:

• Basemaps: to store pmtiles, mbtiles, sqlite basemaps.
• Overlays: to store overlay vector data (outline flatgeobufs).

To use S3:

from app.config import settings
from app.s3 import add_file_to_bucket, add_obj_to_bucket, get_file_from_bucket, get_obj_from_bucket

# Available buckets
basemap_bucket = settings.S3_BUCKET_NAME_BASEMAPS
overlay_bucket = settings.S3_BUCKET_NAME_OVERLAYS

# Upload from filesystem
add_file_to_bucket(bucket_name, file_path, s3_path)

# Upload from python obj (BytesIO)
add_obj_to_bucket(bucket_name, file_obj, s3_path)

# Download to file
get_file_from_bucket(bucket_name, s3_path, file_path)

# Download to python obj (returns BytesIO object)
variable = get_obj_from_bucket(bucket_name, s3_path)

I will test this further and refactor a bit so we use the S3 buckets to store files.
@nrjadkry going forward we should:

• Use in-memory BytesIO objects whenever we would use /tmp files.
• For files that need to persist (e.g. basemaps), use S3 files as described above.

We can discuss this on our next FMTM call 😄