📊 Update faostat data

[pabloarosado]

Main changes

• Added a multi_merge function to owid.catalog (that properly works with Table object).
• Adapted all faostat scripts and steps to the new ETL conventions and the new metadata.
• Removed datasets that were not used and caused technical issues, namely "World Census of Agriculture" (wcad) and "Energy use" (gn).
• Changed country names following FAOSTAT changes (e.g. "Netherlands" -> "Netherlands (Kingdom of the)").
• Removed datasets that do not exist any longer in FAOSTAT, namely ef, el, and ep. I have checked that all variables used in charts from these datasets can (in principle) be replaced by the analogous ones from rf, rl, and rfn, respectively (but will need to confirm once grapher datasets exist).
• In qcl (and therefore in the food explorer) I renamed Flax fibre -> Flax, raw or retted. This follows the change that FAOSTAT did too, by which item 773 (Flax fibre) disappeared from qcl, being replaced by 771 (Flax, raw or retted).
  • Item 773 can now still be found in other datasets, e.g. qi, and, its definition is "Flax, processed but not spun", with description: "Broken, scutched, hackled etc. but not spun. Traditionally, FAO has used this commodity to identify production in its raw state; in reality, the primary agricultural product is the commodity 01929.01 (Flax, raw or retted) which can either be used for the production of fibre or for other purposes (Unofficial definition)".
  • Item 771 now in qcl is defined as "Flax, raw or retted", with description: "Flax Straw, spp. Linum usitatissimum. Flax is cultivated for seed as well as for fibre. The fibre is obtained from the stem of the plant. Data are reported in terms of straw. (Unofficial definition)".
• Manually fixed some spurious values, and removed anomalies that are no longer present in the data.
• Many other small fixes in different steps.

TO-DO for Pablo R (in separate PRs):

• To be able to see the new data in the global food explorer, merge 📊 Update global food explorer owid-content#39
• Update all crop yield data and explorer.
• Improve all FAOSTAT metadata (even if it is currently using the new fields, the metadata is still very messy for some indicators).
• Update all steps using FAOSTAT, and archive old FAOSTAT steps (since they use old functions and metadata).
• After merging, chart-sync staging-update-faostat-data -> production.

For reviewers

There is no need to do a thoroughly review this monster PR. I'd suggest:

• @Marigold could you please review the changes to owid.catalog? Also, note that I temporarily removed some of the fixes that you made to etl/steps/data/garden/faostat/2024-03-14/faostat_fbsc.py to control for memory use. I did that because the concatenate function was not propagating metadata. ETL did not complain in staging, but we may need to adapt this function to work with tables (with metadata propagation). If you think we should do that before merging, let me know and I'll look into it.
• @spoonerf feel free to have a quick look at some parts of the code as you see fit, and maybe at some of the charts (just scroll through the admin to see the latest changes). Please let me know if you find anything odd (apart from old metadata, and bad map brackets, which is something I should fix soon).

Thanks a lot!!

[Marigold]

PR looks good. If it works without memory optimisations then go ahead and merge it. We'll see what it does in production.

EDIT: I checked tb, and there are only two object columns with sizes around 2gb. That's not that much, but perhaps converting them to categories wouldn't cause problems with metadata?

ipdb> p tb.fao_unit_short_name.memory_usage(deep=True) / 2**20
1631.690894126892
ipdb> p tb.fao_element.memory_usage(deep=True) / 2**20
2025.320728302002

[spoonerf]

Hey Pablo!

This looks great - what an absolute monster of a dataset, it never ceases to amaze me!

I'll write some thoughts on the charts down here, but generally they look great:

• Maybe this would come under the 'old metadata' you mention? Many data pages seem to have quite machinery titles - would it be possible to set a variable.presentation.title_public for these? e.g. http://staging-site-update-faostat-data/grapher/minimum-requirement-calories?tab=map & http://staging-site-update-faostat-data/grapher/cropland-area & http://staging-site-update-faostat-data/grapher/food-loss-postharvest-by-region but there are many more.

• Many variables like 'Number of pigs' could have their number of decimal places set to 0.

• Perhaps use a different color scale here, a red one could be more appropriate.

• Very minor point but the xmin here might as well go down to 0

• The second 'fruit' here could be removed I think.