Cycle 2024.6: index explorers in data catalog search

[larsyencken]

Ensure that the data catalog covers most of our data.

Motivation

We recently shipped the friendly data catalog, which provides an index over all Grapher charts. However, it's not at a solid resting place until it has coverage over the bulk of our explorers.

Proposed scope

Must do

• Break explorers out of the main monorepo package into a subpackage
• Provide workers with access to indicator-based explorer configs
• Adapt worker to thumbnail indicator-based explorer views
• Index indicator-based explorers in data catalog search

Should do

• ❓ Break MD indicators out of the main monorepo package into a subpackage
• Provide workers with access to MD indicator config
• Adapt worker to thumbnail MD indicator views
• Index MD indicators in Algolia for data catalog search

Won't do

• Worry about the All Charts block for now
• Index CSV-based explorer views in the data catalog

Technical notes

• ⚠ A key challenge is that we don't have MySQL on the edge, so we may need to bake explorer configs to a known place in S3 (a static S3-based API) in order to have them for the worker
• An indicator based explorer view has a URL like: https://ourworldindata.org/explorers/population-and-demography?Metric=Population+growth+rate&Sex=Both+sexes&Age+group=Total&Projection+Scenario=Medium
  • Note that Metric, Sex, Age and Projection are GET params that are totally custom to this explorer
  • To thumbnail, the worker needs basically to be able to run the explorer code to decode these GET params into a particular configuration
• MD indicators use a YAML config that can override chart settings, so that code likewise needs to run at thumbnail time to properly thumbnail the view
• For the worker to be able to run the explorer code could possibly require some refactoring/repackaging of the code for the worker, if it is not already in a Lerna package that the worker is importing\

Technical notes from Ike and Daniel:

3 moving parts to make explorer thumbnails:

1. Explorer.tsx has to be able to run inside a worker, which should serve the html file that we bake, but rewrite the og:image value to point to another new worker that we've added. It needs to parse the query params and understand what kind of explorer it is (csv-based we won't be able to render), grapher-based and indicator-based will work
2. We'll need a dictionary of view (represented as queryparams) -> uuid-config (marcel did this for MDMs mdd: fetch variable-level config by uuid #3922)
3. Use the existing functions that, given a uuid, render a thumbnail

We need a new worker to handle /explorers/ (which requires extracting explorers into a lerna package)

grapherConfigEtl is when there's a partial config from the ETL (as YAML)

grapherConfigAdmin is when there's a partial config from the admin e.g. under "Grapher Config (edited in the admin)" on https://admin.owid.io/admin/variables/985021

https://admin.owid.io/admin/explorers/sample-indicator-explorer

when there's 1 yVariableId, that ID can be used to lookup the indicator's grapher config via the variables->grapher_configs UUID join

As a stretch goal:
when there's more than 1 yVariableId (e.g. a scatterplot) the grapher config that needs to be used is defined in the explorer config for that particular view, and so the grapher config can be constructed from that (there is presumably something in Explorer.tsx somewhere that does this)

Open questions

• ❓ Is a UUID enough to thumbnail a view of an indicator-based explorer? If yes, it would be very helpful; that would mean that any customisations in that explorer's config ended up in the DB.

[danyx23]

The open question here is a good one - I think we should treat this as it's own litte project to create these UUID charts for explorer views (and also mdim views). I created #4013 to track this effort. I found it a bit hard to explain it well in text, it's probably easier to chat about it next week.