Storing PDFs and images

[wipfli]

In recent pull requests there has been some discussion about how to best store PDFs and images.

What we are doing at the moment is to have some pdfs / images in this git repo and some we get from 3rd party sources such as GitHub or Google Drive.

I think having multiple places to store images and pdfs is bad.

What is the best solution to this? cc @maxammann and @louwers

[maxammann]

Hm, storing everything on GitHub does not work (google docs etc.).
Storing everything on Google Drive does also not work because images would not be embedable.

So there is probably no solution.

[louwers]

Binary blobs bigger than a few KB should not be stored in a git repo. It will make cloning the repo very slow and if they are ever deleted they are still part of the git history. Also, with GitHub Pages the whole site cannot be more than 1GB which you would hit very quickly, and there is also a 100 GB bandwidth limit.

What I like to do is drag-and-drop images in GitHub editors (or you can click the text "Attach files by dragging & dropping"). It will get uploaded to GitHub's CDN. Here is a visual guide:

Source

Another advantage is that GitHub will do compression for you so you don't have to do it manually (it's really sloppy to accompany a newsletter with a 3MB image). Oh, you can also upload PDFs this way.

If that feels a bit shaky (uploaded images and files have no expiry but they could in theory disappear) we can set up an S3 bucket to upload images and files there. I would be happy to set up an account and the bucket.

[louwers]

This tool can be used to delete existing binary files from the repo: https://rtyley.github.io/bfg-repo-cleaner/

[wipfli]

I think it is best to use Cloudflare R2 to store binary files.

[wipfli]

Thanks for the feedback everyone!

[louwers]

@wipfli Happy to use Cloudflare R2, but it is tied to [email protected], so I am unable to log in (2 factor auth).

[louwers]

My connection is not the fastest right now, but after changing a single text file, git took several minutes to push a branch (uploading 18.89 MiB with 88.00 KiB/s).

[josxha]

Binary blobs bigger than a few KB should not be stored in a git repo. It will make cloning the repo very slow and if they are ever deleted they are still part of the git history.

It's possible to work around this problem.
Although, to be honest, I'm not sure if this is a good or bad workaround.

1. create a branch
2. commit and push the image to that branch
3. copy the raw url of the image with the specific commit hash
4. revert the last commit that pushed the image.
5. embedd the image link in web website and create a pull request
6. squash the pull request into the default branch

The image won't be part of the history of the main branch but will be still available via a direct link.

For example:

https://raw.githubusercontent.com/josxha/flutter-maplibre/d8e2c7daf15221f716a2ebbd63e57a317cb089ab/docs/static/img/showcase-map.jpg

And it's possible to embedd the image:

This method could be used for all kind of binary data that doesn't exceed the maximum file size of 25 MiB.

[HarelM]

Wouldn't uploading images here in an issue thread much easier to achieve the same results?

[josxha]

Wouldn't uploading images here in an issue thread much easier to achieve the same results?

Last time I checked, images uploaded to issues can't be embedded outside of github.com, probably CORS related.

[HarelM]

I'm pretty sure I'm using images upload to github issues in my website...
Here's an example, made sure using the network tab:
https://israelhiking.osm.org.il/poi/OSM/way_1250716963?language=he

[louwers]

@josxha That would be your server's CORS blocking it then.

[josxha]

That would be your server's CORS blocking it then.

Good point. 👀 I checked and my assumption with CORS was not correct, too. Apparently images uploaded to a private repo stay as private after the repo became publicly accessable. Uploaded images, now in a public repo

Here's an example, made sure using the network tab

Thanks for the sample Harel. This means posting the binary data in the related issue or pull requests could be used as well. 👍