Asset fetching and physical asset management

[iboB]

We call Assets files which are downloaded from a remote. Ideally to save disk space the asset directory on a local provider will be shared between applications/processes which use the SDK.

This means that multiple processes will concurrently want to touch it. Especially true on non-mobile platforms where processes are not suspended when switching tasks (and even mobile platforms have limited daemon support which pretty much negates the statement).

We have two types of concurrency issues here:

• File-level: two processes may want to write to the same file.
• Data-level: we have the possibility to have two processes download the same (potentially many-gigabytes-huge) asset.

To avoid them we need a state manager which can handle atomic transactions.

The easiest (and arguably most natural) way to do so would be to have an asset manager daemon but our specs forbid us from enforcing this.

We are forced to use the file system as a transactional database.

This can be done by having the index and state in a sqlite database.

We could try employing filesystem locks, but they are notoriously iffy and error prone. Not to mention that it will require a lot of platform-specific code. I don't even know how this works on mobile platforms.

With further specification we can use the filesystem itself as an index and state management, but not all operations can be supported. Notably merging directories. It is likely however that we would never need them. I can only envision us needing immutable file creation, which can be well handled by FS ops alone.

I'm leaning towards the last option. It seems the easiest to me and doesn't require us to use third party software (sqlite, or potential fs-lock libs). If we're willing to accept the limitations, I say we go with that.

[iboB]

Continuing on the topic.

It seems that plain atomic file io (creation and move) will not suffice. We've identified four states of an asset.

• Present
• Not present
• Download in progress
• and crucially and most annoyingly Downloaded partly but download is not in progress

The last state on desktop can happen if the downloading process was unexpectedly killed (crash, poweroff, etc) which typically are rare enough that we wouldn't care about them and would just live with the complete re-download.

However on mobile it is not the case. As mentioned above task-switching on mobile platforms leads to process suspension. So it will likely be a very common case to have

• App A request a 10GB model X
• Get to 80% downloaded
• Switch to app B
• A gets suspended and stops downloading
• B requests X

What now? Redownload the full 10GB? A bad idea. B needs to identify and differentiate between the asset being currently downloaded (and wait) or downloaded partly but with progress suspended (and continue from where we left off).

Not to mention that B can get suspended and we need to have A continue from where B left off.

This is a complex flow for which even if there is a atomic-fs-op-only solution it would be very error prone and hard to maintain.

It would seem that an sqlite database is the only solution

[iboB]

An alternative approach here would involve artificially splitting models into small chunks. Thus at worst we would re-download a single chunk (a price that we would be willing to pay)

[pminev]

That will work. We should verify the downloaded chunks before actually start downloading, at least that is what I've seen in some other projects. Then another question comes up in my head. Are we going to be adding the chunk directly in single file or all chunks are going to be different files and then on completion we'll merge them?

[iboB]

Platform-specific solutions also exist here. On mobile platforms there's an OS-level download manager, which could serve as the daemon we discarded above.

The problem with it is that it only supports a limited number of protocols (very likely only HTTP) which would prevent us from using custom ones. One that we're considering for example is AWS S3.

Another obvious benefit is that mobile downloaders have built-in handling of network resets as switching data providers (5g - 4g - wifi) is very common on mobile devices.

Should we try to incorporate it? Can we live with HTTP alone?

RnD is needed to see how the OS-level downloaders work and how we can make use of them.

[iboB]

Asset disposal also plays here.

Ideally we would have some garbage collection, but there needs to be a store for the ref counts.

[IvanFilipov]

Getting more and more interesting. I can imagine how to handle such tasks, when running server side.
We can clean up cron-job-like, but on edge in some embedded system scenarios - the storage space can be quite crucial.
We don't want to be in the case in, which a model not used anymore is preventing the task B of downloading another model, which B needs to run.

[IvanFilipov]

In general - the suggested approach with sqlite, sounds very reasonable.
The database engine is small, efficient & supported everywhere (perhaps all of the possible target devices).

Regarding the downloading part - up to this moment, I'm thinking of solving the problem by using our
custom-made service-like app, which makes the most of the host's primitives, but extends their functionality.
Can we try to look for some open-source solutions here? Are there any at all?
I definitely shall dig deeper into the topic, before making any conclusions.

[iboB]

General asset coherency is also part of this. If we have an shared sqlite database, we can use it for this as well, but if we don't coherency should be a part of whatever decision we do.

We should:

• Check asset sizes
• Check asset hash sums (probably lazily).
• Ideally have a way of storing the information that an asset is coherent (either by its presence in a path or by atomic entry in a database)

[iboB]

It is becoming increasingly obvious that sqlite is the best choice here. Moving on with rnd in #104