Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: check for previously downloaded files #4

Open
sjvrijn opened this issue Jul 3, 2023 · 1 comment
Open

Suggestion: check for previously downloaded files #4

sjvrijn opened this issue Jul 3, 2023 · 1 comment
Assignees

Comments

@sjvrijn
Copy link
Contributor

sjvrijn commented Jul 3, 2023

The run.get_extra_downloads options in the scenario files feel more mandatory than 'extra'.
It also feels error-prone to manually activating these options for the first run and deactivate them afterwards.

One simple alternative could be to always try to download the files, and simply skip the download if they already exist. This would automatically update the set of downloaded files if the date range ever changes. An example implementation:

def download_cds_weather_quantity(
      quantity, year, month, days, hours, area, download_folder):
    '''...'''

>   file_name = f'{download_folder}/{quantity}_{year}_{month}.grib'
>   if os.path.exists(file_name):
>       return  # skip file if already downloaded

    quantity_download = cdsapi.Client()
    quantity_download.retrieve(
        'reanalysis-era5-land',
        {
            'variable': quantity,
            'year': year,
            'month': month,
            'day': days,
            'time': hours,
            'area': area,
            'format': 'grib'
        },
        filename
    )

Note: This does not take changes in the area into account, so that would need to be checked and invalidated separately.

@Omar-Usmani
Copy link
Collaborator

The area check needs to be thought out

@Omar-Usmani Omar-Usmani self-assigned this Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants