Allow a PATCH request to restart at an offset of 0

[cjhenck]

It would be useful to allow a PATCH request to start at an offset of 0. In particular, iOS's upload mechanism for closed/background apps aggressively retries uploads and will not call your application if there is a wire error, so I would like to use the concatenation extension in order to pre-chunk the data and send it to the background uploader.

[Acconut]

It would be useful to allow a PATCH request to start at an offset of 0.

Can you elaborate on why creating a new upload instance is not acceptable for you in this case?

In particular, iOS's upload mechanism for closed/background apps aggressively retries uploads and will not call your application if there is a wire error, so I would like to use the concatenation extension in order to pre-chunk the data and send it to the background uploader.

Are you referring to an actual issue in TUSKit's implementation here or a more general problem?

[cjhenck]

The way that TUSKit has background uploads implemented, they will not continue long after the app closes. The only way that you can continue background uploads if the app is closed for an extended period of time is to use the NSURLSession framework, which retries on wire failures. (See here.)

Here is an example of the problem I am encountering implementing TUS on that framework:

• User is offline and a file needs to be uploaded. A file creation request is placed on the NSURLSession and the user closes the application. However, the upload task is remembered by the OS, and when it eventually succeeds, the application is restarted and the call is returned to the application.
• The application puts a PATCH request for the full file on the wire.
• The PATCH requests fails halfway through because the user's connection drops. iOS does not restart the application, but tries the PATCH request again, even when it receives an error code from the server, because that error code causes the connection to be dropped (and is thus a wire failure).

The only way around this that I can see is to break up the file into chunks, which are either uploaded sequentially to a single resource (resetting the offset as appropriate) or to partial files that are reset to zero each time. Therefore, having a way to reset offsets would allow us to implement truly background uploads (e.g. your app is not running) on iOS, just like DropBox or other services do.

You can see the effect I describe in my example by running the background example app in our fork (there are some bugs in the example, so you can only select one file to upload each time you run it). If you set up tusd to limit the read to 1,000,000 bytes, then the connection is dropped which should result in a second PATCH request. However, iOS never makes the callback to the application because from iOS's perspective, the request has not succeeded and wire failures are ignored.

[Acconut]

Thank you for this detailed description and I do understand your struggles but are there no alternatives to using NSURLSession? Do the Long-Running Tasks (https://developer.apple.com/library/ios/documentation/iPhone/Conceptual/iPhoneOSProgrammingGuide/BackgroundExecution/BackgroundExecution.html#//apple_ref/doc/uid/TP40007072-CH4-SW4) not fit your needs? If not, I apologize since I am not an iOS developer for myself. Maybe @MMasterson could help with these questions?

[cjhenck]

The long-running tasks you've pointed to mostly just wake the application up when something has changed. You can't just run an arbitrary process in the background, sadly. You can request more time to complete an upload, but you can't expect it to be an extended period (this is what the latest TUSKit version does).

[Acconut]

Could alternative libraries, such as AFNetworking, help? For example, https://github.com/AFNetworking/AFNetworking#creating-an-upload-task

[MMasterson]

@cjhenck @Acconut AFNetworkign would require a little bit of a rewrite, but is do-able if if this will solve the problem @cjhenck's having using their background upload functionality - but it might be the just give us the same problem. I'll have to do a little more digging on coming up with a perfect solution.

The idea of a "locker" has been brewing up in my mind where the upload is cached with its progress if it loses connection and continues when connection is found again.

@cjhenck how big are the files you are uploading? Generally - I'd like to create a scenario that is close to yours

[cjhenck]

The files may be up to 70 or 80mb in size, but the problem is that we have a number of users in developing markets who may not have very good access to the internet, but I still want to support their uploads. E.g. I want them to be able to hit "upload" in the app when they have no internet, and to have that upload complete even if they go in and out of network connection. Because you cannot keep the app running indefinitely in the background, that means letting the OS (NSURLSessions) handle the uploads.

AFNetworking will still rely on the NSURLSessions, so will still have the same problems (or not work at all if your app is closed, since it is set up to use callbacks).

[kvz]

Thanks for pointing this out and making tus better @cjhenck! I think I understand what you're saying. I'm going to summarize in my own words so you can help spot mistakes in my version.
On iOS you want to outsource uploading to the OS, so it can happen in the background. There is no other way to let the uploading continue if the application goes to sleep (or whatever is the correct term here). Outsourcing uploads/requests to iOS, eliminates however the use of a real tus implementation there, as you cannot run your own code in these parts.

So the workaround is to split the 70mb file up in say 10x 7mb chunks with tus, and then basically have iOS do 10 non-tus uploads (they may look like tus uploads, but they are not resumable, only retryable, and this, iOS will fiercely attempt). For this reason, with every retry, the server needs to allow going back in time, discarding what it had already received, alloing the client (iOS) to retry its 7mb chunk from the beginning.

I wonder, since we're not doing actual tus uploads, if the protocol is the place to accomodate that. This may sound like I made a call in my mind already but I honestly wonder.

I would be very happy to have background uploading on iOS feature-wise. The use case you describe seems like that covers a large portion of mobile upload behavior, and it addresses the main problem why tus was created.

I could also see scenarios however where this could lead to confusion, bugs, corrupt files, and these uploads are not truly resumable. I guess I wanted to make super double extra sure there is no way to do actual resumable uploads in the background.

For this to make it in the protocol, I personally feel it would be helpful if we could also think of other scenarios where a Reset would be useful. I could still be tipped both ways. What do you think @Acconut @bhstahl?

[bhstahl]

I agree that we probably shouldn't modify the protocol just for one use case, but if we can think of other suitable worthwhile reasons, then I could be swayed.

I guess the real question is, if the client has complete control, unlike NSURLSessions, are there cases for resetting an upload instead of creating a new one?

[cjhenck]

@kvz your description is absolutely correct. At the moment I can't think of alternative uses, but I do have a revised "flow" for iOS that would make use of the full protocol while the app is running (though the code would be more complicated):

• While app is running in the foreground, create a partial upload and use the tus protocol normally on that partial upload. PATCHing, resuming, etc.
• When the app is transitioning into the background, chunk the upload into 10 (etc.) files and send POST requests to the system. The app will be re-launched when those POST requests complete.
• When the POST requests complete, send the new PATCH requests for those partial uploads which will be aggressively retried until they succeed
• When they succeed, send the POST to finalize the files.

I agree with everyone that this is a bit of a hack, but I cannot think of any other way to support rapid background uploads in iOS, though there are workarounds for occasional uploading that I have added to the TUSKit issue and which may be sufficient for most use cases.

[Acconut]

I guess there may even be a better approach to solving @cjhenck issues on iOS. In the past some people suggested to allow sending the first chunk for the upload already in the initial POST request (when using the Creation extension), mainly for reducing the overhead of a second attempt but since then it hasn't made its way into the protocol specification.

Anyway, the way this solution is meant to work is following:

1. Split your upload into multiple really small chunks
2. For each chunk you create a new NSURLSession which sends a POST request to create a new partial upload and include the entire chunks' data (Expect: 100-continue can be very handy in this case to allow early failures)
   • If the request fails, the OS will retry the upload creation request which causes an entirely new partial upload to be created
   • If the request succeeds, the OS will notify you and you can add it to a list of succeeds partial uploads
3. Once all partial uploads have succeeds, you send a new POST request for the final upload which simply concatenates all previous, partial uploads. The you are done.

What do you think?

[cjhenck]

I think this is an excellent solution. It solves the exact same problem and is more generally useful.

[kvz]

Love this solution @Acconut! ❤️ Should we explicitly update the protocol to accommodate this in a 1.1 version you think?

[Acconut]

Should we explicitly update the protocol to accommodate this in a 1.1 version you think?

Yes, this is a good opportunity to ship the 1.1 release alongside with some other discussion which are still open, e.g. #79 and #81

[Acconut]

@cjhenck @kvz @bhstahl @MMasterson I, finally (sorry), got the time to start adding this feature to the specification. I would be pleased to hear your feedback: #88

[Acconut]

I also plan to add this to tusd as soon as possible.

[Acconut]

Closing this in favor of #88.