📊 Update faostat data (#2416)

* Adapt create new snapshots script * Fix create_new_snapshots * Add snapshots and meadow steps, and remove wcad * Remove faostat_gn and fix issues * Fix meadow faostat_sdgb * Update docs * Add garden steps and minor improvements on meadow * Fix issues with item code changes in faostat_fs * Add new flag, and increase number of characters for item codes of faostat_sdgb * Remove unused excluded country, fix new element mismatch between fbsh and fbs, and replace assertions with log errors * Fix issue with categorical columns in script that updates custom metadata * Update custom metadata files * Add element amendments for fbsc * Fix various issues * Remove anomaly fix on cocoa beans, which is no longer in the data * Fix issues with region aggregates in fbsc * Explicitly use regions, population and income groups datasets * Fix anomaly of spinach area harvested in China * Fix errors on missing units * Minor fixes * Fix missing metadata in latest faostat_qv, big refactor * Fix changed item in qcl and global food explorer * Add grapher steps (WIP) * Use tables instead of dataframes * Use tables instead of dataframes in fbsc and qcl steps * Use tables instead of dataframes in garden metadata step * Improve metadata handling of meadow steps * Ensure origins are properly propagated * Fix various issues to ensure origins are propagated * Fix missing origins * Raise warning if domains do not exist any longer in FAOSTAT, and remove those * Update docs * 🐛 retry on SSLError * Remove steps that do not exist anymore in FAOSTAT * Ensure origins are propagated * Fix changed country names * Avoid repetitive warning on missing units * Check discrepant item name definitions and improve warning messages and documentation * Improve warning messages * Inspect remaining warnings and improve warning handling * Add garden and grapher step of additional variables * Adapt garden and grapher steps of additional variables * Add multi_merge function to owid catalog * Update metadata of additional variables dataset * Improve metadata of additional variables dataset * Improve function that finds dataset id from name * Improve script to submit faostat chart revisions * Fix errors in script that creates chart revisions * Delete unused faostat script * Fix bad format of description_key * Improve snapshot descriptions * Update custom metadata files and documentation * Improve metadata --------- Co-authored-by: Marigold <[email protected]>
owid · Mar 25, 2024 · 32fff70 · 32fff70
1 parent 42f193c
commit 32fff70
Show file tree

Hide file tree

Showing 124 changed files with 9,878 additions and 355 deletions.
diff --git a/apps/backport/datasync/datasync.py b/apps/backport/datasync/datasync.py
@@ -3,7 +3,7 @@
 from typing import Any, Dict
 
 import structlog
-from botocore.exceptions import EndpointConnectionError
+from botocore.exceptions import EndpointConnectionError, SSLError
 from owid.catalog import s3_utils
 from tenacity import Retrying
 from tenacity.retry import retry_if_exception_type
@@ -32,7 +32,7 @@ def upload_gzip_dict(d: Dict[str, Any], s3_path: str, private: bool = False) ->
     for attempt in Retrying(
         wait=wait_exponential(min=5, max=100),
         stop=stop_after_attempt(7),
-        retry=retry_if_exception_type(EndpointConnectionError),
+        retry=retry_if_exception_type((EndpointConnectionError, SSLError)),
     ):
         with attempt:
             client.put_object(

diff --git a/dag/faostat.yml b/dag/faostat.yml
@@ -323,3 +323,300 @@ steps:
   #
   data://grapher/faostat/2023-06-12/additional_variables:
     - data://garden/faostat/2023-06-12/additional_variables
+  #
+  # FAOSTAT meadow steps for version 2024-03-14
+  #
+  data://meadow/faostat/2024-03-14/faostat_cahd:
+    - snapshot://faostat/2024-03-14/faostat_cahd.zip
+  data://meadow/faostat/2024-03-14/faostat_ei:
+    - snapshot://faostat/2024-03-14/faostat_ei.zip
+  data://meadow/faostat/2024-03-14/faostat_ek:
+    - snapshot://faostat/2024-03-14/faostat_ek.zip
+  data://meadow/faostat/2024-03-14/faostat_emn:
+    - snapshot://faostat/2024-03-14/faostat_emn.zip
+  data://meadow/faostat/2024-03-14/faostat_esb:
+    - snapshot://faostat/2024-03-14/faostat_esb.zip
+  data://meadow/faostat/2024-03-14/faostat_fa:
+    - snapshot://faostat/2024-03-14/faostat_fa.zip
+  data://meadow/faostat/2024-03-14/faostat_fbs:
+    - snapshot://faostat/2024-03-14/faostat_fbs.zip
+  data://meadow/faostat/2024-03-14/faostat_fbsh:
+    - snapshot://faostat/2024-03-14/faostat_fbsh.zip
+  data://meadow/faostat/2024-03-14/faostat_fo:
+    - snapshot://faostat/2024-03-14/faostat_fo.zip
+  data://meadow/faostat/2024-03-14/faostat_fs:
+    - snapshot://faostat/2024-03-14/faostat_fs.zip
+  data://meadow/faostat/2024-03-14/faostat_ic:
+    - snapshot://faostat/2024-03-14/faostat_ic.zip
+  data://meadow/faostat/2024-03-14/faostat_lc:
+    - snapshot://faostat/2024-03-14/faostat_lc.zip
+  data://meadow/faostat/2024-03-14/faostat_metadata:
+    - snapshot://faostat/2024-03-14/faostat_metadata.json
+  data://meadow/faostat/2024-03-14/faostat_qcl:
+    - snapshot://faostat/2024-03-14/faostat_qcl.zip
+  data://meadow/faostat/2024-03-14/faostat_qi:
+    - snapshot://faostat/2024-03-14/faostat_qi.zip
+  data://meadow/faostat/2024-03-14/faostat_qv:
+    - snapshot://faostat/2024-03-14/faostat_qv.zip
+  data://meadow/faostat/2024-03-14/faostat_rfb:
+    - snapshot://faostat/2024-03-14/faostat_rfb.zip
+  data://meadow/faostat/2024-03-14/faostat_rfn:
+    - snapshot://faostat/2024-03-14/faostat_rfn.zip
+  data://meadow/faostat/2024-03-14/faostat_rl:
+    - snapshot://faostat/2024-03-14/faostat_rl.zip
+  data://meadow/faostat/2024-03-14/faostat_rp:
+    - snapshot://faostat/2024-03-14/faostat_rp.zip
+  data://meadow/faostat/2024-03-14/faostat_rt:
+    - snapshot://faostat/2024-03-14/faostat_rt.zip
+  data://meadow/faostat/2024-03-14/faostat_scl:
+    - snapshot://faostat/2024-03-14/faostat_scl.zip
+  data://meadow/faostat/2024-03-14/faostat_sdgb:
+    - snapshot://faostat/2024-03-14/faostat_sdgb.zip
+  data://meadow/faostat/2024-03-14/faostat_tcl:
+    - snapshot://faostat/2024-03-14/faostat_tcl.zip
+  data://meadow/faostat/2024-03-14/faostat_ti:
+    - snapshot://faostat/2024-03-14/faostat_ti.zip
+  #
+  # FAOSTAT garden steps for version 2024-03-14
+  #
+  data://garden/faostat/2024-03-14/faostat_cahd:
+    - data://meadow/faostat/2024-03-14/faostat_cahd
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_ei:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_ei
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_ek:
+    - data://meadow/faostat/2024-03-14/faostat_ek
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_emn:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://garden/wb/2024-03-11/income_groups
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://meadow/faostat/2024-03-14/faostat_emn
+  data://garden/faostat/2024-03-14/faostat_esb:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_esb
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_fa:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://meadow/faostat/2024-03-14/faostat_fa
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_fbsc:
+    - data://meadow/faostat/2024-03-14/faostat_fbsh
+    - data://meadow/faostat/2024-03-14/faostat_fbs
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_fo:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_fo
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_food_explorer:
+    - data://garden/faostat/2024-03-14/faostat_qcl
+    - data://garden/faostat/2024-03-14/faostat_fbsc
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_fs:
+    - data://meadow/faostat/2024-03-14/faostat_fs
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_ic:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://meadow/faostat/2024-03-14/faostat_ic
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_lc:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+    - data://meadow/faostat/2024-03-14/faostat_lc
+  data://garden/faostat/2024-03-14/faostat_metadata:
+    - data://meadow/faostat/2024-03-14/faostat_rt
+    - data://meadow/faostat/2024-03-14/faostat_scl
+    - data://meadow/faostat/2024-03-14/faostat_sdgb
+    - data://meadow/faostat/2024-03-14/faostat_qv
+    - data://meadow/faostat/2024-03-14/faostat_emn
+    - data://meadow/faostat/2024-03-14/faostat_ek
+    - data://meadow/faostat/2024-03-14/faostat_qcl
+    - data://meadow/faostat/2024-03-14/faostat_fbsh
+    - data://meadow/faostat/2024-03-14/faostat_tcl
+    - data://meadow/faostat/2024-03-14/faostat_fa
+    - data://meadow/faostat/2024-03-14/faostat_fo
+    - data://meadow/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_fs
+    - data://meadow/faostat/2024-03-14/faostat_ei
+    - data://meadow/faostat/2024-03-14/faostat_rl
+    - data://meadow/faostat/2024-03-14/faostat_ic
+    - data://meadow/faostat/2024-03-14/faostat_qi
+    - data://meadow/faostat/2024-03-14/faostat_rfn
+    - data://meadow/faostat/2024-03-14/faostat_rfb
+    - data://meadow/faostat/2024-03-14/faostat_esb
+    - data://meadow/faostat/2024-03-14/faostat_lc
+    - data://meadow/faostat/2024-03-14/faostat_rp
+    - data://meadow/faostat/2024-03-14/faostat_cahd
+    - data://meadow/faostat/2024-03-14/faostat_fbs
+    - data://meadow/faostat/2024-03-14/faostat_ti
+  data://garden/faostat/2024-03-14/faostat_qcl:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://meadow/faostat/2024-03-14/faostat_qcl
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_qi:
+    - data://meadow/faostat/2024-03-14/faostat_qi
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_qv:
+    - data://meadow/faostat/2024-03-14/faostat_qv
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_rfb:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_rfb
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_rfn:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://meadow/faostat/2024-03-14/faostat_rfn
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_rl:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_rl
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_rp:
+    - data://meadow/faostat/2024-03-14/faostat_rp
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_rt:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_rt
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_scl:
+    - data://meadow/faostat/2024-03-14/faostat_scl
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://garden/wb/2024-03-11/income_groups
+  data://garden/faostat/2024-03-14/faostat_sdgb:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://meadow/faostat/2024-03-14/faostat_sdgb
+    - data://garden/wb/2024-03-11/income_groups
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+  data://garden/faostat/2024-03-14/faostat_tcl:
+    - data://meadow/faostat/2024-03-14/faostat_tcl
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/wb/2024-03-11/income_groups
+    - data://garden/regions/2023-01-01/regions
+  data://garden/faostat/2024-03-14/faostat_ti:
+    - data://garden/faostat/2024-03-14/faostat_metadata
+    - data://grapher/demography/2023-03-31/population
+    - data://garden/regions/2023-01-01/regions
+    - data://meadow/faostat/2024-03-14/faostat_ti
+    - data://garden/wb/2024-03-11/income_groups
+  #
+  # FAOSTAT grapher steps for version 2024-03-14
+  #
+  data://grapher/faostat/2024-03-14/faostat_cahd:
+    - data://garden/faostat/2024-03-14/faostat_cahd
+  data://grapher/faostat/2024-03-14/faostat_ei:
+    - data://garden/faostat/2024-03-14/faostat_ei
+  data://grapher/faostat/2024-03-14/faostat_ek:
+    - data://garden/faostat/2024-03-14/faostat_ek
+  data://grapher/faostat/2024-03-14/faostat_emn:
+    - data://garden/faostat/2024-03-14/faostat_emn
+  data://grapher/faostat/2024-03-14/faostat_esb:
+    - data://garden/faostat/2024-03-14/faostat_esb
+  data://grapher/faostat/2024-03-14/faostat_fa:
+    - data://garden/faostat/2024-03-14/faostat_fa
+  data://grapher/faostat/2024-03-14/faostat_fbsc:
+    - data://garden/faostat/2024-03-14/faostat_fbsc
+  data://grapher/faostat/2024-03-14/faostat_fo:
+    - data://garden/faostat/2024-03-14/faostat_fo
+  data://grapher/faostat/2024-03-14/faostat_fs:
+    - data://garden/faostat/2024-03-14/faostat_fs
+  data://grapher/faostat/2024-03-14/faostat_ic:
+    - data://garden/faostat/2024-03-14/faostat_ic
+  data://grapher/faostat/2024-03-14/faostat_lc:
+    - data://garden/faostat/2024-03-14/faostat_lc
+  data://grapher/faostat/2024-03-14/faostat_qcl:
+    - data://garden/faostat/2024-03-14/faostat_qcl
+  data://grapher/faostat/2024-03-14/faostat_qi:
+    - data://garden/faostat/2024-03-14/faostat_qi
+  data://grapher/faostat/2024-03-14/faostat_qv:
+    - data://garden/faostat/2024-03-14/faostat_qv
+  data://grapher/faostat/2024-03-14/faostat_rfb:
+    - data://garden/faostat/2024-03-14/faostat_rfb
+  data://grapher/faostat/2024-03-14/faostat_rfn:
+    - data://garden/faostat/2024-03-14/faostat_rfn
+  data://grapher/faostat/2024-03-14/faostat_rl:
+    - data://garden/faostat/2024-03-14/faostat_rl
+  data://grapher/faostat/2024-03-14/faostat_rp:
+    - data://garden/faostat/2024-03-14/faostat_rp
+  data://grapher/faostat/2024-03-14/faostat_rt:
+    - data://garden/faostat/2024-03-14/faostat_rt
+  data://grapher/faostat/2024-03-14/faostat_scl:
+    - data://garden/faostat/2024-03-14/faostat_scl
+  data://grapher/faostat/2024-03-14/faostat_sdgb:
+    - data://garden/faostat/2024-03-14/faostat_sdgb
+  data://grapher/faostat/2024-03-14/faostat_tcl:
+    - data://garden/faostat/2024-03-14/faostat_tcl
+  data://grapher/faostat/2024-03-14/faostat_ti:
+    - data://garden/faostat/2024-03-14/faostat_ti
+  #
+  # FAOSTAT food explorer step
+  #
+  data://explorers/faostat/latest/food_explorer:
+    - data://garden/faostat/2024-03-14/faostat_food_explorer
+  #
+  # FAOSTAT garden step for additional variables
+  #
+  data://garden/faostat/2024-03-14/additional_variables:
+    - data://garden/faostat/2024-03-14/faostat_rl
+    - data://garden/faostat/2024-03-14/faostat_qi
+    - data://garden/faostat/2024-03-14/faostat_qcl
+    - data://garden/faostat/2024-03-14/faostat_sdgb
+    - data://garden/faostat/2024-03-14/faostat_fbsc
+    - data://garden/faostat/2024-03-14/faostat_rfn
+  #
+  # FAOSTAT grapher step for additional variables
+  #
+  data://grapher/faostat/2024-03-14/additional_variables:
+    - data://garden/faostat/2024-03-14/additional_variables