CDN providers not syncing plugin manifest as expected

[Or1g3n]

Per the ReadMe,
"Flow downloads the plugins manifest file from various CDN providers, so when your new plugin appears is dependant on when these providers next sync the updated file, and this can be anywhere from one to three hours.

Every three hours the CI in this repository will check for new updates from plugins and automatically update them to the latest version."

The CDN providers do not appear to be syncing as expected. My plugin was updated a week ago but the Flow-Launcher website is still showing the old version of the plugin which currently points to an archived repo:
https://www.flowlauncher.com/docs/#/plugins

After inspecting the page, it appears this is the cdn url that is pulling the information. Clicking the link below and inspecting the JSON output shows the outdated version:
https://cdn.jsdelivr.net/gh/Flow-Launcher/Flow.Launcher.PluginsManifest@plugin_api_v2/plugins.json

This issue appears to be intermittent as sometimes it randomly shows the correct plugin information (perhaps they have multiple servers and not all of them are in sync).

The point

Wanted to inform of this issue as plugin updates are not being synced within the expected 1-3 hr window. It has been 5 days and cdn providers appear to not have synced fully. In my case its a bit problematic, as the outdated version of the plugin points to an archived repo so I don't want users to think the plugin is no longer maintained.

[Or1g3n]

I checked this morning and the manifest was showing correctly. However, it is now showing the outdated manifest again (see image). So if I had to guess, it would seem the CDN provider has multiple servers on some type of load balancer and for some reason not all of the servers are in sync (sometimes the manifest shows latest information and other times it's showing the manifest over a week ago).

[jjw24]

Thank you for the thorough investigation and write up. I don't know if that's something within our control, likely not, but the way I see minimising this issue is probably implement a mechanism to check GitHub connection first before falling to the CDN alternatives, at least some users who have no issue connecting to GitHub will be able to get the latest updates sooner. At the moment iirc the code downloads the manifest from whichever endpoint returns the fastest, but does not guarantee the latest manifest at all.

[Or1g3n]

Very interesting. Prioritizing GitHub seems like a good option to minimize the issue. I will continue checking randomly each day and post back if it appears that the manifest is stable (or at least reliably showing a version that is no longer from a week ago), this way we can at least document how long it took for the CDN servers to fully sync.

[Or1g3n]

FYI, I have been periodically checking throughout the day, and using Todos plugin as a reference point, the manifest consistently flip flops between 3.06 (latest - released yesterday) and 3.05 (released 3 days ago). So at least the version from 8 days ago which was pointing to the archived repo is no longer showing up. So it would appear that in current state, it takes several days (up to a week) for CDN servers to fully sync.

[jjw24]

Thank you for the observation, perhaps if you could update readme to fix up the time period until we get around to addressing this issue.

[Or1g3n]

@jjw24
Submitted pull request for update to readme.md which clarifies expected sync time for plugin manifest. Please edit as needed; wasn't sure if you also wanted to mention that efforts are being made to improve sync time.
#401

[Or1g3n]

I investigated this issue a bit further and it appears that one out of the 4 providers is more less consistently the problem child.

Below are the four providers based on the FlowLauncher PluginManifest.cs file:

Using clipboard+ as a test case given its recent update, below are the responses from each provider:
https://raw.githubusercontent.com/Flow-Launcher/Flow.Launcher.PluginsManifest/plugin_api_v2/plugins.json

https://gcore.jsdelivr.net/gh/Flow-Launcher/Flow.Launcher.PluginsManifest@plugin_api_v2/plugins.json:

https://fastly.jsdelivr.net/gh/Flow-Launcher/Flow.Launcher.PluginsManifest@plugin_api_v2/plugins.json:

https://cdn.jsdelivr.net/gh/Flow-Launcher/Flow.Launcher.PluginsManifest@plugin_api_v2/plugins.json:

As you can see, cdn.jsdelivr.net is the only provider not in sync with GitHub. Based on past testing (see original post), this is the same url that was outdated. Not sure if its within your control to determine why that particular server is consistently lagging, but perhaps a viable solution would be to deprioritize it amongst the 4 options.

[jjw24]

Happy to deprioritise cdn.jsdelivr.net also.

[Yusyuriv]

@jjw24 Should I also replace it in https://www.flowlauncher.com/docs/#/plugins? If yes, should I replace it with one URL or should I make it try different URLs until it finds one that doesn't fail to load (in case users can't reach some of the servers for whatever reason)?

[taooceros]

I think cdn jsdelivr has cache (7 days?). We should manually invalidate that if plugins change (maybe through the action).

[jjw24]

@jjw24 Should I also replace it in https://www.flowlauncher.com/docs/#/plugins? If yes, should I replace it with one URL or should I make it try different URLs until it finds one that doesn't fail to load (in case users can't reach some of the servers for whatever reason)?

Are you also working on the flow side or is this just for the docs?

I think logic could be, try load GitHub first, set an appropriate timeout. If fails, use the cdns and whichever returns first.

[jjw24]

I think cdn jsdelivr has cache (7 days?). We should manually invalidate that if plugins change (maybe through the action).

Can we actually invalidate the CDN cache do you know? Do they provide API endpoint to do this?

[Yusyuriv]

Are you also working on the flow side or is this just for the docs?

No, just the website.

I think logic could be, try load GitHub first, set an appropriate timeout. If fails, use the cdns and whichever returns first.

GitHub as in try to load the file from raw.githubusercontent.com? I think we discussed something like this already and decided we don't want to create additional load on GitHub's servers, and this is why we use CDNs?

I don't think I've seen this behavior in web often, when the page tries to load data from all URLs at the same time and takes the data from the fastest response. People could be using mobile connection to open the page, and while the manifest is not that big, it would be rather wasteful to use 4 times more traffic than necessary.

Can we actually invalidate the CDN cache do you know? Do they provide API endpoint to do this?

cdn.jsdelivr.net provides this page for manual cache purge, but it required me to solve a captcha. They also provide free API access on request which might allow to do what we want, but I'm not completely sure, their wording is a little confusing.

[jjw24]

Are you also working on the flow side or is this just for the docs?

No, just the website.

Yeah no worries.

I think logic could be, try load GitHub first, set an appropriate timeout. If fails, use the cdns and whichever returns first.

I don't think I've seen this behavior in web often, when the page tries to load data from all URLs at the same time and takes the data from the fastest response. People could be using mobile connection to open the page, and while the manifest is not that big, it would be rather wasteful to use 4 times more traffic than necessary.

Yes true, thanks for reminding me.

GitHub as in try to load the file from raw.githubusercontent.com? I think we discussed something like this already and decided we don't want to create additional load on GitHub's servers, and this is why we use CDNs?

My understanding the root cause for using CDN is about reach, not all users have stable connection to GitHub. I don't think we would place any additional traffic that GitHub's infrastructure can't necessarily handle. But regardless, CDN approach is also good.

The logic is something to implement flow side.

@jjw24 Should I also replace it in https://www.flowlauncher.com/docs/#/plugins? If yes, should I replace it with one URL or should I make it try different URLs until it finds one that doesn't fail to load (in case users can't reach some of the servers for whatever reason)?

Yeah I think up to you, I'd say keep it simple, just replace it with fastly, probably no need to over-engineer this.

[jjw24]

Happy to deprioritise cdn.jsdelivr.net also.

I think unless there is a reason not to, we should just remove its usage from flow.

[taooceros]

I think cdn jsdelivr has cache (7 days?). We should manually invalidate that if plugins change (maybe through the action).

Can we actually invalidate the CDN cache do you know? Do they provide API endpoint to do this?

Yes there is https://www.jsdelivr.com/tools/purge

[taooceros]

I think the cdn link is the official cdn. Maybe we can measure the latency toward each node globally (the cdn link may select better server compared to others). Originally that was used for Chinese user, but now it won't work so I agree it's fine to remove it.