Strange issue with subsetting *any* model?

[wx-andrew]

Hi! I have used herbie for quite some time and I recently installed the latest build out of herbie on my work laptop, went to subset model files and its not working.
Below is a very simple example:

I get what appears to be a very positive response:

However, in the command line I see:

and the file is no where to be seen:

On top of that, I am not sure if it is related but I am unable to download from nomads:

To be very clear, I know that its only subsetting that is the issue because when I remove "searchString", the download IS successful.
I am sure that the issue is very obvious unfortunately but I suppose it'll be nice slice of humble pie! :)

Any help would be greatly appreciated!

Thanks :)

[blaylockbk]

Hi @wx-andrew, I'm glad to hear you use Herbie. Sorry the latest release is causing trouble. Thanks for letting me know!

1. Subset files not saving

I haven't run into this issue while testing on my Windows machine. I just tried this and didn't have any issue

from herbie import Herbie
H = Herbie('2022-10-26 12:00', model='nam', fxx=0)
H.download(":(TMP|DPT):.* mb", verbose=True, overwrite=True)

It looks like you are having a permissions issue with the Access Denied error in your console. I'm 70% certain this isn't a Herbie issue. Check your permissions for the directory it is saving to, or see if you can save the files in another directory (use the save_dir= argument to change where the file is saved).

2. Herbie can't find NAM on NOMADS

Yes, there does seem to be a problem with getting NAM data from NOMADS. I tried

H = Herbie('2022-10-26', model='nam', priority="nomads")

It found the IDX file, but not the GRIB2 file!

This is funny because when I look at the sources URL

H.SOURCES

OUT:
{'nomads': 'https://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.20221026/nam.t00z.conusnest.hiresf00.tm00.grib2'}

That URL actually exists

import requests
requests.head(H.SOURCES['nomads']).ok

OUT:
True

I'll have to investigate this and see what's up with NOMADS. It looks like this is true for the HRRR model.

I wonder if something changed on the NOMADS server how downloads are handled... This seems to be true. Herbie will only download the file if the "Content-Length" is greater than 1 MB.

url = H.SOURCES['nomads']
head = requests.head(url)
head.raw.info()["Content-Length"]  # <---- This equals 20! That is not right.

Content Header for AWS file
---------------------------
HTTPHeaderDict({'x-amz-id-2': '1QIYl80KE5iJg9Q3gWc3+VByNTJY+fKW7s3L8lScE1dtMqJ1CWjyNXhvP8rPoZbuGZR15uTytxI=', 'x-amz-request-id': '0Z2DJAB0NZPZ5R1F', 'Date': 'Thu, 27 Oct 2022 03:20:40 GMT', 'Last-Modified': 'Wed, 26 Oct 2022 00:52:12 GMT', 'ETag': '"621755d01adba5de20eb677a00844954"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '135147651'})

Content Header for NOMADS file
------------------------------
HTTPHeaderDict({'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Last-Modified': 'Wed, 26 Oct 2022 00:51:14 GMT', 'ETag': '"4dfcc781-80e3083-5ebe56bf2be1f"', 'Accept-Ranges': 'bytes', 'Cache-Control': 'max-age=14400', 'Expires': 'Thu, 27 Oct 2022 07:16:17 GMT', 'Content-Type': 'text/plain; charset=UTF-8', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Thu, 27 Oct 2022 03:20:27 GMT', 'Content-Length': '20', 'Connection': 'keep-alive', 'Strict-Transport-Security': 'max-age=31536000 ; preload'})

The difference is the 'Content-Type': 'binary/octet-stream' versus 'Content-Type': 'text/plain; charset=UTF-8' which is causing the problem.

[blaylockbk]

FYI: I essentially turned of the "content-length" check in affdbda

[wx-andrew]

@blaylockbk ,
Thanks for getting back in touch! Apologies for the delayed response.

I investigated any sort of permissions issue since that would make sense given it is a work laptop however it appears permissions aren't the issue.
To fully cross this off the checklist I edited the config file and changed the directory to my downloads folder and created a fresh test. Unfortunately, no luck. Here is the entire output of the first try after creating the new directory. My apologies if the text in the image is too small.

The console still shows Access Denied as well.

Below is a snapshot of the permissions window for your convenience as well.

I wish I had more of an inkling to what could be the issue but unfortunately I'm quite lost with this issue since I am not aware of what's fully going on under the hood.

Also, I downloaded the latest build, nomads is now working again (however still not actually saving)! Thanks for investigating that and the information you provided.

I will continue to tinker with it to see if I can get something working and will write back if I find something. In the meantime, do you have any suggestions on some possible hot fixes on my end? Later today, I am going to rebuild herbie on my personal desktop (I haven't used it on my desktop since some of your first iterations!) and see if I experience the same issue.

Thanks for all your help and for dealing with my cluelessness! :)

[amotl]

Hi there.

Maybe I've hit a similar issue last night on HRRR, see #100 (comment). I think the downloaded files were about 800 kB in size, and apparently not decodable: grib_ls croaked with ERROR: unreadable message.

With kind regards,
Andreas.