Does FilePicker load files to memory first?

[sturlath]

@David-Moreira now that I´m able to try out the FilePicker (swapping out Syncfustion) to upload big files I´m finding it hard to figure out whether the file is first uploaded into memory or it can be streamed directly somewhere?

You commented this

So it is seeming to me that even tough they fixed the 2gb constraint you'll still have some kind of limitations related to memory management when handling bigger files on WASM I guess?

Here it is pointed out that the docs are totally not clear about what the limitations are

When used for uploading, BrowserHttpHandler.SendAsync (reference source) reads the file's bytes into a single JavaScript array buffer when marshalling the data from JavaScript to C#, so large file uploads (> 250 MB) may fail.

More here

But if I search around it looks like it should be possible "I just uploaded a 1GB file without the memory usage going through the roof." (no code but with pointers) then there is "Streaming requests with the fetch API" that seems to offer this functionality.

So my question is: Can I use FilePicker in some way to upload a 5GB file by streaming it directly (to Blob storage or an API) without first reading it into memory and is is "stable" (chunk/retry/etc)?

Update: The FilePicker isn´t using MS InputFile, correct? So were do the limitations lie (if large file upload streaming isn´t possible)? Sorry for being so thoroughly confused!!😝

[sturlath]

@mrpmorris you had this comment with link to twitter

XMLHttpRequest does stream it. I just uploaded a 1GB file without the memory usage going through the roof.

Any change you have some code examples or pointers since as I understand using fetch is not supported in all browsers so maybe using XMLHttpRequest could be used, correct?

I´m "atting" you here because the github issue is closed.

[David-Moreira]

Hello @sturlath ,
I remember having some technical difficulties with actually being able to stream upload the file. I believe I had the same exact issue as @mrpmorris in that issue, I would try StreamContent expecting it to stream the file as it goes, and it did not seem to apply streaming at all filling up internal memory, even if the underlying stream was supposedly capable of that.

Thing is I wasn't able to get it to work neither on our component or Blazor's native FileInput. But I think this to be more a matter of lack of knowledge of how to properly use dotnet to actually stream the file.

Our internal implementation to how the FileInput / FilePicker handle the files is now pretty similar to Microsoft's, so what works in native Blazor InputFile should work in our component. So if you guys figure out a way how to properly stream, please share your findings. :)

[sturlath]

I will share every finding! Its just super confusing what should be possible and what is possible. Does InputFile have an issue.. is it only with Blazor WASM or not. Is it possible to do it with javascript or XMLHttpRequest (and then how), is fetch working or not or are we waiting for some browser support.. 🙈

I at least finally found this comment

It applies to the InputFile component specifically.

so I don´t know how close your "handle the files is now pretty similar to Microsoft's" implementation is but that might be the root of your problem.

And then he continues saying

"If you have a large file upload request, I suggest you try and implement that using Http Ranges, not only will avoid this problem all-together, it will also be more reliable allowing you to avoid re-uploading entire files upon a failure (you would need to update only the missing chunk instead)"

And I got that feedback here also "Currently, Javier has recommended streaming Fetch API or HTTP Ranges ... basically he's saying you handle the entire upload via JS."

Are we talking about something like this?

<form>
  <input type="file" id="fileInput">
  <button type="button" onclick="uploadChunk()">Upload</button>
</form>

var fileInput = document.getElementById("fileInput");
var file = fileInput.files[0];
var chunkSize = 1024 * 1024; // 1 MB
var start = 0;
var end = chunkSize;

function uploadChunk() {
  var xhr = new XMLHttpRequest();
  xhr.open("POST", "/server/upload");
  xhr.setRequestHeader("Content-Range", "bytes " + start + "-" + (end - 1) + "/" + file.size);
  xhr.setRequestHeader("Content-Type", "application/octet-stream");

  xhr.upload.addEventListener("progress", function(evt) {
    if (evt.lengthComputable) {
      var percentComplete = (evt.loaded / evt.total) * 100;
      console.log(percentComplete + "% uploaded");
    }
  });

  xhr.addEventListener("load", function() {
    if (end < file.size) {
      start = end;
      end = start + chunkSize;
      if (end > file.size) {
        end = file.size;
      }
      uploadChunk();
    } else {
      console.log("Upload complete!");
    }
  });

  xhr.addEventListener("error", function() {
    console.log("Error uploading chunk.");
  });

  var chunk = file.slice(start, end);
  xhr.send(chunk);
}

uploadChunk();

Would this just work? There is for instance the POST to a server API I also need (I will be trying to upload directly to Azure Blob Storage).

But here I'm missing all your other goodness in FilePicker...😪

@inject IJSRuntime JSRuntime

<form>
  <input type="file" id="fileInput">
  <button type="button" @onclick="UploadFile">Upload</button>
</form>

@code {
    private async Task UploadFile()
    {
        await JSRuntime.InvokeVoidAsync("uploadChunk");
    }
}

[mrpmorris]

Any change you have some code examples or pointers since as I understand using fetch is not supported in all browsers so maybe using XMLHttpRequest could be used, correct?

If you'd asked last night the answer would have been yes, but I did a factory reset on my old laptop when I went to bed.

[sturlath]

Here is code for uploading files directly to blob storage directly from the browser (using javascript. The syntax is different from the other code we hade above.

Would you know if this code would also fill up the memory with big files?

And @David-Moreira would it make sense (I hope so 😅) to offer a inbuilt functionality to directly upload to (different) cloud storage? So have SAS toke etc.

[David-Moreira]

I am not sure about the benefit in this,
So we'd have provider specific configurations? Is that so much extra "plumbing" that you'd like a component library to support this itself?

Because I feel like this configuration can get stale with time, provider changes the way they work, or does not exist anymore, etc...

[mrpmorris]

Looks like you are in luck after all.

This isn't complete, but the JS is there.

https://github.com/mrpmorris/Morris.Blazor.Web.FileUpload/blob/master/Source/Lib/Morris.Blazor.Web.FileUpload/wwwroot/index.js#L28

[David-Moreira]

Well I'd stay with the current implementation. Because there are benefits in all the othe regular cases I believe, but perhaps we could provide either a way to override or an additional implementation purposed to make big files work, where it would be up to the dev which one he uses. @stsrki thoughts?

[stsrki]

Not sure TBH. With the current implementation, we're closer to what .NET team is doing, and if one day they fix the problem, then we could easily modify our code accordingly We could introduce a new upload mechanism but that means more features to maintain, @David-Moreira you can go and investigate what can be done, and then we will discuss it before going all in.

[sturlath]

And just to be 100% clear on this, the FilePicker (in current state) loads everything to memory before being able to send (chunked or not) and it looks like my only option is to use javascript and XMLHttpRequest?

[David-Moreira]

Yes I think we are currently getting to understand that the current dotnet apis that are supposed to stream + the browser apis current capabilities, that you will need a javascript implementation for this.

[sturlath]

Since I was going to replace this one Syncfusion control I have I asked about its limitation and it has the same limitations. I might start looking into creating a desktop app just to upload files. Maybe that makes sense.

Btw Dropbox has a size upload limitation of <50GB for the web and that is not loaded into the browser.