[Feature] Google Drive upload and download (includes a working demo)

[scatwang]

I just implemented some Google Drive integration. Check out this demo here:

https://scatwang.github.io/pybricks-code/
video

Basically, it use Google Drive API to upload and download files. The react app get oauth token and talk to Google API directly ( hope this will be easier for GDPR).

A very basic version control just works, it should be good enough for small projects.

• When the downloaded file has a name conflicts, the ReplaceImportDialog dialog will ask if user want to overwrite or rename.
• For uploaded files, it's easy to check historical version on Google Drive.

I will work on unit tests and documents if this looks good to you.

[dlech]

Thanks for working on this!

Probably the most important question to answer first before spending a bunch of time on this is if we are going to be willing an able to support such a feature (technically, legally, enough manpower, etc.). Even it we can technically make it work, there are a lot of other things to be considered.

And there are also a few technical things that we want to make sure we can overcome before continuing.

1. Storing the API key where anyone can see it (in the source code or in the client code that gets transferred to the browser) is not going to be acceptable. I think this is how it should be implemented perhaps?
2. Requesting access to all files in someones google drive is too much. Can we make it work with with just https://www.googleapis.com/auth/drive.appdata and/or https://www.googleapis.com/auth/drive.file instead?

[scatwang]

1. Storing the API key where anyone can see it (in the source code or in the client code that gets transferred to the browser) is not going to be acceptable. I think this is how it should be implemented perhaps?

API keys is designed to be available to public. It's restricted to certain domain name and referer URLs, so it won't be abused. Oauth token is commonly stored in user's browser stroage.

2. Requesting access to all files in someones google drive is too much. Can we make it work with with just https://www.googleapis.com/auth/drive.appdata and/or https://www.googleapis.com/auth/drive.file instead?

Agreed, I will update and make it work. Suppose user should only download files that were uploaded by pybricks.

[laurensvalk]

Thanks for working on this.

From a user experience point of view, would uploading and downloading individual files be the most practical?

If we had this feature, I would imagine the following could be quite nice:

• Sign in with google/dropbox. Select or create one destination folder*. No access beyond that required.
• Press sync button in Pybricks Code to upload/download to and from folder. Perhaps automatic sync.

In this case, we would ideally avoid reinventing the wheel on file conflicts, but follow Dropbox/Google standards. In Dropbox for example the conflicts just show as a timestamped copy of the file, so we would not need any new UIs. Mistakes can be recovered from the Dropbox/Google interface. Conflicts can be deleted by the user. Kids working together on school projects will be used to this.

* A real folder would be nice, but if we can achieve something similar with the APIs David mentioned that is just fine too. I'm just describing the use case here, not the technical implementation.

[scatwang]

• Sign in with google/dropbox. Select or create one destination folder*. No access beyond that required.

This simplify the folder lookup progress, users don't need to worry about finding the destination folder in drive, also limit the permission scope.
But this also blocks a team from change the upload destination to their shared folder.

So, maybe we can by default create and upload to \pybricks\, but also allow user to change the destination folder (and memorize the last choice)?

• Press sync button in Pybricks Code to upload/download to and from folder. Perhaps automatic sync.

I asked @dlech before about this question, he also lean on individual file rather than sync a whole folder.
Compare with bidirectional sync, individual file upload will be easier for users to manage when a FLL team work on their shared folder.

In this case, we would ideally avoid reinventing the wheel on file conflicts, but follow Dropbox/Google standards. In Dropbox for example the conflicts just show as a timestamped copy of the file, so we would not need any new UIs.

Yes, they both have good history tracking. Google Drive API by default treat files uploaded with same file name as different files, so if a file is uploaded multiple times, there will be multiple with same name there (we could also change the behavior to overwrite).

I'm think what will be easiest for kids. I actually considering if we can add a comment line for downloaded files, with their original remote location and a timestamp. Which will be easier for kids for handling conflictions when download a different version changed by their teammates. Like:

# 2024-10-14 09:52 Downloaded from https://drive.usercontent.google.com/download?id=1ERlNnSk5H0ASLXBeOnGYjmjx2wTLokFu

[laurensvalk]

I'm not sure if manual, individual upload and download is viable in a team with kids. It would be very easy to forget one file, creating a mismatch for the other team member, with compounds when the original team member continues. (This is why we added the download zip option earlier).

Google/Dropbox/Onedrive have years of experience dealing with shared folders. This seems like a pattern that people are familiar with at school and work. It's also really easy to explain. Since Pybricks Code also has a single folder with files, this seems like the perfect match.

• Sign in with google/dropbox. Select or create one destination folder*. No access beyond that required.

This simplify the folder lookup progress, users don't need to worry about finding the destination folder in drive, also limit the permission scope. But this also blocks a team from change the upload destination to their shared folder.

So, maybe we can by default create and upload to \pybricks\, but also allow user to change the destination folder (and memorize the last choice)?

Yes, this is what I meant. Then users can use Google/Dropbox UI to configure sharing that folder as they see fit.

I'm think what will be easiest for kids. I actually considering if we can add a comment line for downloaded files, with their original remote location and a timestamp. Which will be easier for kids for handling conflictions when download a different version changed by their teammates. Like:

# 2024-10-14 09:52 Downloaded from https://drive.usercontent.google.com/download?id=1ERlNnSk5H0ASLXBeOnGYjmjx2wTLokFu

I'm not sure this is easier since you have to open them up before you can see which it is, and manually compare dates. For conflicts, maybe we could follow the pattern that they see in their own local cloud drive folder. That is to say, let Google/Dropbox create the timestamp-named conflict files.

---

Thanks again for working on this. It would be really amazing to have something like this available at some point. I do want to re-iterate what @dlech shared earlier:

Probably the most important question to answer first before spending a bunch of time on this is if we are going to be willing and able to support such a feature (technically, legally, enough manpower, etc.).

That is to say, please don't be discouraged if we can't merge it for now or in the future. We really are quite short on people 🙂

[scatwang]

I think what you proposed sounds like fulltime sync model. Ideally all typing operation will be synced to remote immediately, local storage will be only used as a cache. The experience will be like the file system on chromebooks, not even need a sync button, everything is behand the sense.

I like that but there are also some question come to my mind:

• How to handle the scenario that both user make change to a same file?
  • If we want currently pybricks-code behavior, adding a lock to the file when it's opened in one editor. There is a API. If app lost connection, when to unlock is also a challange.
  • Collaborative editing could be an even better experience, especially for education scenarios. But it might be beyond Google Drive API's scope.
• If someone (like a teammate) changed the file I'm currently working on (it happens a lot in my kid's team when there is a nauty one), should I check the history on pybricks UI or Google Drive? Google Drive doesn't provide a diff-like UI for comparing source code files line by line.
• API quota challange, the limit to per API key is 12k per minute, assume auto sync requires 10 operations per user per minute, is it enough in peak hours?