Support Storage Migration

[loreanvictor]

This PR adds support for migrating diagram storage from one storage backend to another in a slow and gradual manner, by using an intermediary storage driver that moves diagrams from the old storage to the new one upon load:

• When a diagram is loaded that is not present in the new (target) storage, it will be loaded from the old (source) storage, and also stored in the target storage.
• Updates to diagrams will only be stored in target storage.
• Newly created diagrams will be stored in the target storage only.

Which will look like the following:

storage = new MigratingStorageService({ sourceStorage, targetStorage })

This PR also adds support for environment variable APOLLON_REDIS_MIGRATE_FROM_FILE, which if set, instructs Apollon Standalone to use a migrating storage, and migrate from file storage to redis storage.

Why Not a Migration Script?

A migration script will be tightly coupled with storage logic (how a storage stores its keys, what metadata it uses for storing diagrams, etc.), while being kept separated and designed for independent execution. Subsequently, it will be deprecated over time as storage logic is updated but migration scripts are not kept in sync.

The migrating storage driver provides a storage-agnostic solution that is usable for any potential future migration as well (e.g. from Redis to Postgres). Considering the fact that stored diagrams have a limited time span (30 days), after this time any old diagram would have been deleted regardless, so using a migrating storage driver (through an environment variable, for example) for the diagram retention period will be sufficient for a clean migration.

[egenerse]

I tested this pr and it works as expected.
but during the migration these two env values must be there otherwise it breaks the server.
APOLLON_REDIS_URL="redis://localhost:6379" APOLLON_REDIS_MIGRATE_FROM_FILE="true" // this key can be anything then undefined.

there is also crash when apollon_standalone tries to patch with non existed diagram in anywhere something like https://apollon.ase.in.tum.de/xxxxxxxxxxxxxxxxxxx?view=COLLABORATOR
but it has nothing to do with this pr

[loreanvictor]

@egenerse thanks a lot for review and testing!

@FelixTJDietrich since this PR affects deployment and future migrations as well, I'd appreciate if you could also take a look. I should note that this is not urgent by any means. Whenever this is merged we can migrate the main apollon standalone instance to use redis as storage instead of files.

[FelixTJDietrich]

We will first need to update the deployment side. I how to do that in the next two weeks.

[FelixTJDietrich]

I could finally test it on the test server, it worked as expected and successfully migrated an old diagram. Thanks for this! I will soon deploy Redis in prod 🎉