Compatibility with aiofiles

[hwong557]

aiofiles is a Python project to perform IO asynchronously. I am able to asynchronously read files from s3, but I cannot write files to s3. The following code example fails:

import asyncio

import aiofiles
import aiohttp
from cloudpathlib import S3Path


async def main():
    chunk_size = 1000

    path = S3Path("s3://my-bucket/image.png")

    async with aiohttp.ClientSession() as session:
        async with session.get("http://httpbin.org/image/png") as resp:
            async with aiofiles.open(path, "wb") as f:
                async for chunk in resp.content.iter_chunked(chunk_size):
                    await f.write(chunk)


asyncio.run(main())

The error is:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpq4ls9jhw/....../image.png'

I am using cloudpathlib==0.16.0 and aiofiles==23.2.1.

If I add a path.touch() before initializing the ClientSession, the code runs without error, but the file in s3 is zero bytes.

Thank you for your work on such a clean and easy to use library! It truly is a joy!

[pjbull]

Thanks @hwong557, the issue here depends entirely on what aiofiles.open(path, "wb") does within their method with the S3Path object. This is another instance of the kinds of issues in #128.

If they see that path has an open method and they use that properly like a context manager, things could work. If not, all bets are off. I took a look, and they just pass it to the built-in open.

A few possible workaround:

• It's possible that the experimental work in WIP: Patch open, os and os.path builtins #322 makes this scenario work. That would be useful to know.
• Write to a temporary file and then use your cloud path's upload_from method to write to the cloud.
• Call path.open() before your async code and then path.close() afterwards to close and upload it (this likely breaks the async writing, but I'm not sure).
• Maybe aiofiles is willing to fix their library to be compatible with pathlike objects that provide open methods to call that instead of using the built-in open method assuming that everything is a filepath.