Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean data source keys in config #92

Open
crispy-wonton opened this issue Dec 5, 2024 · 0 comments
Open

Clean data source keys in config #92

crispy-wonton opened this issue Dec 5, 2024 · 0 comments

Comments

@crispy-wonton
Copy link
Collaborator

crispy-wonton commented Dec 5, 2024

We should clean the data source keys in the config as it is getting messy and confusing to find things.

We have to do it carefully to ensure that all the places where the config is called is updated with the appropriate keys.

We also will have to update the config/README config key column.

Here is @helloaidank 's suggestion:

data_source:
  # ONS Postcode Directory
  gb_ons_postcode_dir:
    url: "https://www.arcgis.com/sharing/rest/content/items/487a5ba62c8b4da08f01eb3c08e304f6/data" # Aug 2023 data
    file_path: "Data/ONSPD_AUG_2023_UK.csv" # Aug 2023 data
    s3_path: "s3://asf-heat-pump-suitability/source_data/ONSPD_AUG_2023_UK.csv"

  # Census Data
  census:
    EW_tenure: "s3://asf-heat-pump-suitability/source_data/2021census_2023Mar_tenure_E_W.csv"
    S_tenure: "s3://asf-heat-pump-suitability/source_data/2022_Scotlands_census_tenure_S.csv"
    EW_number_of_rooms: "s3://asf-heat-pump-suitability/source_data/2021census_Mar2023_number_of_rooms_E_W.csv"
    EW_number_of_households: "s3://asf-heat-pump-suitability/source_data/2021_vMar2023_census_numberofhouseholds_EW.csv"
    EW_land_area: "s3://asf-heat-pump-suitability/source_data/2021_vMar2021_census_landareaKM_EW.csv"
    EW_accommodation_type: "s3://asf-heat-pump-suitability/source_data/2021census_Mar2023_accommodation_type_E_W.csv"
    S_accommodation_type: "s3://asf-heat-pump-suitability/source_data/2022_Scotlands_census_accommodation_type_S.csv"

  # Dwelling Age Data
  dwelling_age:
    EW_cdrc: "s3://asf-heat-pump-suitability/source_data/2015cdrc_dwelling_ages_E_W.csv"
    EW_cdrc_la: "s3://asf-heat-pump-suitability/source_data_minor_edits/2015cdrc_dwelling_ages_E_W_per_la_02.parquet"

  # Garden Space Access
  garden_space_access:
    GB_ons: "s3://asf-heat-pump-suitability/source_data/ONS_Apr2020_access_to_garden_space.xlsx"

  # UPRN Data
  uprn:
    GB_osopen: "s3://asf-heat-pump-suitability/source_data/osopenuprn_202405_csv.zip"

  # Local Authority Districts Boundaries
  lad_bounds:
    UK_ons: "s3://asf-heat-pump-suitability/source_data/Local_Authority_Districts_December_2023_Boundaries_UK_BFE_-2600600853110041429/LAD_DEC_2023_UK_BFE.shp"

  # Inspire Land Extent
  inspire_land_extent:
    EW: "s3://asf-heat-pump-suitability/source_data/inspire_ew/"
    S: "s3://asf-heat-pump-suitability/source_data/inspire_scotland/"

  # Microsoft Building Footprint Links
  microsoft_building_footprint:
    global: "https://minedbuildings.z5.web.core.windows.net/global-buildings/dataset-links.csv"

  # Conservation Areas
  conservation_areas:
    E_historic_england: "s3://asf-heat-pump-suitability/source_data/Aug2024_historic_england_conservation_areas_E.geojson"
    W_welsh_gov: "s3://asf-heat-pump-suitability/source_data/2022_welsh_gov_building_conservation_areas_W.gpkg"

  # Off Gas Grid
  off_gas_grid:
    UK_spa: "s3://asf-heat-pump-suitability/source_data/2024_vMar2024_SPA_offgaspostcode_UK.xlsx"

  # Listed Buildings
  listed_buildings:
    E_historic_england: "s3://asf-heat-pump-suitability/source_data/Jun2024_vJul2024_HistoricEngland_listedbuilding_E.gpkg"
    W_cadw: "s3://asf-heat-pump-suitability/source_data/May2024_vMay2024_Cadw_listedbuilding_W.gpkg"
    S_scottish_gov: "s3://asf-heat-pump-suitability/source_data/lb_scotland/Listed_Buildings.shp"

  # LSOA to LAD Lookup
  lsoa_lad_lookup:
    EW_ons: "s3://asf-heat-pump-suitability/source_data/2021_vApr2023_ons_lsoa_to_lad_lookup_EW.csv"

  # LSOA Boundaries
  lsoa_bounds:
    EW: "s3://asf-heat-pump-suitability/source_data/Lower_layer_Super_Output_Areas_2021_EW_BFE_V9_-9107090204806789093/LSOA_2021_EW_BFE_V9.shp"

  # Points of Interest Locations
  poi_locations:
    UK: "s3://asf-heat-pump-suitability/source_data/poi_uk.gpkg"

  # Grid Operators Data
  grid_operators:
    E_ENW:
      dfes_primaries: "s3://asf-heat-pump-suitability/source_data/grid_operators/ENW/dfes-2023-primary-data0.parquet"
      ndp_headroom: "s3://asf-heat-pump-suitability/source_data/grid_operators/ENW/ndp-pry-bsp-headroom.parquet"
      ndp_voronoi: "s3://asf-heat-pump-suitability/source_data/grid_operators/ENW/ndp-pry-voronoi.parquet"
    E_NPg:
      heatmap: "s3://asf-heat-pump-suitability/source_data/grid_operators/NPg/heatmapdemanddata.parquet"
      ndp_demand: "s3://asf-heat-pump-suitability/source_data/grid_operators/NPg/npg_ndp_demand_headroom.parquet"
    S_SPEN:
      spd_substations: "s3://asf-heat-pump-suitability/source_data/grid_operators/SPEN/distributed-generation-sp-distribution-heat-maps-spd-primary-substations.parquet"
      spd_polygons: "s3://asf-heat-pump-suitability/source_data/grid_operators/SPEN/ndp-spd-primary-substation-polygons.parquet"
    W_SPEN:
      spm_substations: "s3://asf-heat-pump-suitability/source_data/grid_operators/SPEN/distributed-generation-sp-manweb-heat-maps-spm-primary-substations.parquet"
      spm_polygons: "s3://asf-heat-pump-suitability/source_data/grid_operators/SPEN/ndp-spm-primary-group-polygons.parquet"
    ES_SSEN:
      demand: "s3://asf-heat-pump-suitability/source_data/grid_operators/SSEN/demand-heat-map-update-feb-24.xlsx - PS.csv"
    S_SSEN:
      sepd_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/SSEN/SEPD Primary Sub Boundaries"
    E_SSEN:
      shepd_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/SSEN/SHEPD Primary Sub Boundaries"
    E_UKPN:
      primaries: "s3://asf-heat-pump-suitability/source_data/grid_operators/UKPN/ukpn_primary_postcode_area.parquet"
    EW_WPD:
      capacity: "s3://asf-heat-pump-suitability/source_data/grid_operators/WPD/wpd-network-capacity-map.csv"
    E_WPD:
      east_midlands_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/WPD/east-midlands-primary.gpkg"
      south_west_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/WPD/south-west-primary.gpkg"
      west_midlands_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/WPD/west-midlands-primary.gpkg"
    W_WPD:
      south_wales_bounds: "s3://asf-heat-pump-suitability/source_data/grid_operators/WPD/south-wales-primary.gpkg"

  # World Heritage Sites
  world_heritage_sites:
    S_historic_environment_scotland: "s3://asf-heat-pump-suitability/source_data/WHS/World_Heritage_Sites.shp"

  # Inspire URLs
  inspire_urls:
    S_ros: "https://ros.locationcentre.co.uk/inspire/"
    EW: "https://use-land-property-data.service.gov.uk/datasets/inspire/download"

  # Scottish Government Data Zone Boundaries
  scottish_gov_dz_boundaries:
    S: "s3://asf-heat-pump-suitability/source_data/2014_Scottish_Government_DataZoneBoundaries_2011_S/SG_DataZone_Bdry_2011.shp"

  # NRS Scotland Dwellings
  nrs_scotland_dwellings:
    S: "s3://asf-heat-pump-suitability/source_data/June2024_NRScotland_households_and_dwellings_S.xlsx"

Originally posted by @helloaidank in #90 (comment)

@crispy-wonton crispy-wonton changed the title clean data source keys in config Clean data source keys in config Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant