Upload a file to AWS S3 with FastAPI

[saeedesmaili]

I'm working on a FastAPI endpoint to upload the provided file from user to an AWS bucket. To make sure my aws credentials and region is valid, I first tried the following code outside FastAPI (in a simple python script) and it successfully uploads the file into S3 bucket:

import boto3

s3 = boto3.client(
    "s3",
    aws_access_key_id="my-access-key",
    aws_secret_access_key="my-secret-key",
    region_name="eu-west-1",
)
with open("sample-file.png", "rb") as f:
    s3.upload_fileobj(f, "my-bucket-name", "sample-file.png")

But I tried so many ways (after searching in StackOverflow and FastAPI github repo issues) and with all of them I get the following error:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://my-bucket-name.s3.eu-west-1.amazonaws.com/sample-file.png"

Here are two different endpoints I've tried and failed:

@app.post("/upload-new")
async def upload_file_new(file: UploadFile = File(...)):
    file_path = os.path.join("/tmp", file.filename)
    with open(file_path, "wb") as out_file:
        out_file.write(await file.read()) # I've confirmed that file is saved in /tmp

    s3 = boto3.client(
        "s3",
        aws_access_key_id="my-access-key",
        aws_secret_access_key="my-secret-key",
        region_name="eu-west-1",
    )
    with open(file_path, "rb") as f:
        s3.upload_fileobj(f, "my-bucket-name", file.filename)

    return {"filename": file.filename}

@app.post("/upload")
async def upload_file(file: UploadFile = File(...)):
    s3 = boto3.client(
        "s3",
        aws_access_key_id="my-access-key",
        aws_secret_access_key="my-secret-key",
        region_name="eu-west-1",
    )
    s3.upload_fileobj(file.file, "my-bucket-name", file.filename)
    return {"filename": file.filename}

I have also tried the following to make sure my AWS credentials are valid:

@app.post("/upload-new")
async def upload_file_new(file: UploadFile = File(...)):
    s3 = boto3.resource(
        "s3",
        aws_access_key_id="my-access-key",
        aws_secret_access_key="my-secret-key",
        region_name="eu-west-1",
    )
    for bucket in s3.buckets.all():
        print(bucket.name) # I can see my S3 bucket names printed in docker logs

I can't understand why I can upload the file outside FastAPI but not in my endpoint. My FastAPI app is running on docker (if it's important in this case).

Update: Running the fastapi (with python3 -m uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000 --proxy-headers) outside docker, it can upload files into s3, but the same exact code with the same hard-coded s3 credentials can't upload to s3 (but can read the existing s3 buckets for example).

Update 2: the code inside docker can read the list of buckets (with s3.buckets.all()) but it can't read the list of objects inside a bucket (with s3.Bucket("bucket-name").objects.all()) and fails with botocore.exceptions.EndpointConnectionError.

Update 3: I added boto3.set_stream_logger("") to see the debugging logs, and here is where it fails when running inside docker:

botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler functools.partial(<function _sentry_request_created at 0xffff787b9bc0>, service_id='s3')

botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://bucket-name.s3.eu-west-1.amazonaws.com/filename.png, headers={'User-Agent': b'Boto3/1.28.57 md/Botocore#1.31.57 ua/2.0 os/linux#5.10.93-0-virt md/arch#aarch64 lang/python#3.11.3 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.57', 'Content-MD5': b'DFjTgwskabM5kcjsrFT7UQ==', 'Expect': b'100-continue', 'X-Amz-Date': b'20231002T175345Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'----', 'amz-sdk-invocation-id': b'79525a28-8e18-4df6-b955-05b209119d66', 'amz-sdk-request': b'attempt=2; max=5', 'Content-Length': '11290'}>

botocore.httpsession [DEBUG] Certificate path: /usr/local/lib/python3.11/site-packages/certifi/cacert.pem

urllib3.connectionpool [DEBUG] Starting new HTTPS connection (2): bucket-name.s3.eu-west-1.amazonaws.com:443

botocore.endpoint [DEBUG] Exception received when sending HTTP request.

[sergey-gr]

Have you tried:

@app.put(
    "/{system_code}/{file_name}",
    tags=["API V1"],
    summary="Upload file to S3 storage.",
    description="Upload file to S3 storage."
)
async def upload_file(system_code: str, file: UploadFile = File(...)):
    if isSystemDefined(system_code) is True:
        system = next((system for system in access if system['system_code'] == system_code), None)
        s3 = S3Browser(system)

        temp_file = os.path.join("/tmp", file.filename)
        with open(temp_file, "wb") as out_file:
            out_file.write(await file.read())

        try:
            s3.fput_object(file.filename, temp_file)
...
...
...