RESPECT Editor: File Saving and Loading

[RyanCoulsonCA]

The Goal

We want to be able to upload product files to and pull files from a server (which may or may not be a server independant from the one the editor is hosted on).

Implementation & Proposals

Below I've written a section for what needs to be done, as well as two proposals: one for loading a product from the server, and one for normalizing how we save files within the editor. Feel free to leave feedback if you see any flaws in what I've proposed or if you have any other ideas.

At the moment I'm not sure whether the download proposal is going to be effected by CORS, if you have any insight on this please post it below!

TODO: Configuring the Editor

If we want to support the editor being hosted on one server and the files being hosted on another, we're going to need to add a configuration option in the editor that points toward the server with the files on it. This configuration setting can't be in a product file (since this would be on the remote server), so we'll need to find a place for it.

The fetch call in fetchConfig() in editor.vue will need to be updated so that it doesn't assume that we're fetching the config file from the same server (we can default to this if the other config option isn't set).

Proposal: Normalize How We're Saving Panels

As of right now, the way we're saving and accessing files in the editor is different depending on what panel type you're working on:

• Images are saved to the ZIP structure. At the same time we're adding a temporary field in the image panel config called temp that stores a blob object, which we use when accessing the picture.
• Chart files are saved to the ZIP structure. When accessing the chart configuration we're either fetching it from the server or we're temporarily storing the config in the object.
• Map configs are saved to the ZIP structure. When attempting to access the map for editing, the editor pulls the config file directly out of the ZIP file and then loads it into the map editor. If the file doesn't exist in the ZIP file (I don't believe this should happen in practice), then a fetch from the server is attempted.

I'm proposing that we use the "load from ZIP" method that I used for map panels for our other panel types. The ZIP library we are using (JSZip) has both read and write operations that will allow us to work with all of the files when we need them.

The JSZip read operations are asynchronous, so we will need to implement loading screens where necessary. From my experience with using this method for the map editor, it seems to load pretty quickly.

If we choose to go with this proposal, then we don't need to worry about removing properties from the configuration file before sending the ZIP to be uploaded.

Proposal: Loading A Product From The Server

I've written two different implementations below that we can use to retrieve a product from the server. The methods are different, but they both end up with the same end result from the editor's perspective: a ZIP file containing all of the product files which can be read as necessary.

Option A

This idea is similar to what Dan mentioned in the body of this issue:

One idea (maybe it's bad), pull the Storylines_ID and scrape its main JSON file for slide details, URLs, etc. Maybe this all needs to be pulled locally for editing and then pushed back to the server (overwriting what's already there).

When the user enters a Storylines ID into the editor, we perform the following steps:

• initialize our local ZIP folder for file storage (this.configFileStructure)
• we fetch the <PRODUCT_ID>_lang.json product configuration file from the remote server
• parse the product config file and determine which files need to be fetched (chart, image, and map sources)
• fetch each file from the server, and add them to this.configFileStructure

During this time, we can display some sort of loading screen on the editor so the user knows what's going on.

Advantages

• the parsing work is handled by the client side, not the server

Disadvantages

• the file server will be hit with a lot of requests for large products (100 images = 100 requests), which could make other requests slower (including people who just want to load a product for viewing)
• if we change the config structure of our panels, we need to change the parsing code too

Option B

Currently, the Express server has just one endpoint (a POST function), that accepts an incoming ZIP file and processes it. We can add a second endpoint to the server (a GET function) that performs the opposite sequence of steps: ZIP up the product folder on the server and send it to the client.

With this option, the server does the work to re-compress the project folder and sends it back. The editor will still need to display a loading screen while the file is being transferred, but once it is returned we can simply set this.configFileStructure to point to the returned ZIP.

Advantages

• the file server is hit exactly one time
• it doesn't matter how our configs are structured, they will be passed back and forth as-is

Disadvantages

• I'm not sure how much processing power is required on the server side to generate the ZIP file, and if this is more work than a ton of fetch requests coming in

---

That's all! Again, if you have any suggestions, concerns, feedback, or if anything here isn't completely clear, please feel free to post below!

[RyanCoulsonCA]

Related to #55 and parts of #54

[spencerwahl]

Option B seems to make sense to me, its likely that you'd need all the external configs and images for a project and would end up requesting it all with Option A anyways.

The only downside I see is that if there are some extra files that used to be in the project but now have been replaced by newer images, etc. they will be transferred back... I'm not sure how we should handle that with the editor. Possibly overwrite the image, charts, maps, etc. folders?

[RyanCoulsonCA]

I agree that B makes the most sense. I think it'd probably be ok to just overwrite the folders to fix that issue, and I don't think it'd be much work either.

[yileifeng]

TODO: Configuring the Editor

Maybe this could be a UI revamp around the load functionality - adding some type of secondary option to specify whether to fetch from a remote source (if left blank default to local).

Proposal: Normalize How We're Saving Panels

Makes sense to go through with this to clean up code, shouldn't be anything ground breaking I think.

Proposal: Loading A Product From The Server

Much prefer option B here since we already have setup the Express server so creating a GET route makes the most sense and there should be some type of library to help simplify the ZIP process (e.g. archiver).

[spencerwahl]

+1 to Normalize How We're Saving Panels