Skip to content
This repository has been archived by the owner on Dec 18, 2023. It is now read-only.

Add 'blob' record upload endpoint #34

Open
pinkwah opened this issue Mar 22, 2021 · 1 comment
Open

Add 'blob' record upload endpoint #34

pinkwah opened this issue Mar 22, 2021 · 1 comment
Labels

Comments

@pinkwah
Copy link
Collaborator

pinkwah commented Mar 22, 2021

In addition to POST /ensembles/../records/../file, we should have a POST /ensembles/../records/../blob.

The file endpoint uses standard HTTP file uploading scheme by using the multipart/form-data. The user uploads a file named file, as if by using an HTML form like:

<input type="file" name="file" />

This has the benefit of being usable from Swagger, where it creates such an entry, but it has drawbacks, the largest of which is that it seems that FastAPI consumes the entire stream before the endpoint function is invoked. This leads to a substantial waiting time while the file is being uploaded to FastAPI, before it gets forwarded to blob storage.

The /blob endpoint, however, uses Starlette's Request class, which is represents the raw request. One of its functions is async def stream(self) -> AsyncGenerator[bytes, None]), which reads the stream in chunks, and allows us to send it further, without having to load the entire document as an intermediate step.

This is currently blocked by Azure Blob Storage not supporting async data uploads: Azure/azure-sdk-for-python#17478

@mortalisk
Copy link
Contributor

We have an endpoint for this. It is a bit unclear what this issue is about. Particularly the title. Could you clarify a bit @pinkwah ?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants