execution error Not a gzipped file (b've')

[wanyingw193]

Hi,

I'm getting a problem with the 2021 version. When I try to run ReEDS 2021, all_year_load.py and LDC_prep.py return the error that load.csv.gz is not gzipped in gamslog.txt (I remove the checkerrorlevel in a_wirtedata.gms). Can I get some suggestions to fix this error? The error has been pasted below. Thank you so much!

==================
--- . a_writedata.gms(63) 2 Mb
--- call python C:\ReEDS_OpenAccess-2021\input_processing\all_year_load.py C:\ReEDS_OpenAccess-2021\ 0 default EPREFERENCE_Baseflex 0 none C:\ReEDS_OpenAccess-2021\runs\test
data inpu successfully
Traceback (most recent call last):
File "C:\ReEDS_OpenAccess-2021\input_processing\all_year_load.py", line 104, in
index_col=0, float_precision='round_trip')
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 948, in init
self._make_engine(self.engine)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 2010, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 537, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 711, in pandas._libs.parsers.TextReader._get_header
File "pandas_libs\parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas_libs\parsers.pyx", line 2034, in pandas._libs.parsers.raise_parser_error
File "C:\GAMS\win64\30.3\GMSPython\lib_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "C:\GAMS\win64\30.3\GMSPython\lib\gzip.py", line 463, in read
if not self._read_gzip_header():
File "C:\GAMS\win64\30.3\GMSPython\lib\gzip.py", line 411, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b've')

--- . a_writedata.gms(69) 2 Mb
--- call python C:\ReEDS_OpenAccess-2021\input_processing\LDC_prep.py -i C:\ReEDS_OpenAccess-2021\ -o C:\ReEDS_OpenAccess-2021\runs\test4_ercot_seq\inputs_case\ -l default -
Starting LDC_prep.py
Traceback (most recent call last):
File "C:\ReEDS_OpenAccess-2021\input_processing\LDC_prep.py", line 598, in
hourly_prep(**args)
File "C:\ReEDS_OpenAccess-2021\input_processing\LDC_prep.py", line 169, in hourly_prep
index_col=0, float_precision='round_trip')
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 948, in init
self._make_engine(self.engine)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\GAMS\win64\30.3\GMSPython\Lib\site-packages\pandas\io\parsers.py", line 2010, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 537, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 711, in pandas._libs.parsers.TextReader._get_header
File "pandas_libs\parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas_libs\parsers.pyx", line 2034, in pandas._libs.parsers.raise_parser_error
File "C:\GAMS\win64\30.3\GMSPython\lib_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "C:\GAMS\win64\30.3\GMSPython\lib\gzip.py", line 463, in read
if not self._read_gzip_header():
File "C:\GAMS\win64\30.3\GMSPython\lib\gzip.py", line 411, in _read_gzip_header
raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b've')

[wesleyjcole]

Would you check the your csv.gz files in inputs/variability/multi_year to make sure they are all reasonably-sized files? We have often found that an error with csv.gz files is from an improper installation of git lfs, so that these files never really get downloaded from the repository.

[astronobri]

@vcelloho That worked, thank you!!

[balasubramaniansa]

I have the same issue. I'm running in Windows. But I don't have gz files. I have attached the screen shot of that folder. Can you pls help?

[wesleyjcole]

I am seeing a load.csv.gz file. Other files appear to be there in h5 format. This folder looks the same as the one on my local Windows machine, so I'm not sure what the issue is.

[balasubramaniansa]

These are the errors I am getting in the run.
``FileNotFoundError: File C:\Users\bs43384\Documents\ReEDS_OpenAccess\postprocessing\retail_rate_module....\runs\v20230217_181515_ref_seq\inputs_case\ccmult.h5 does not exist`
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\postprocessing\retail_rate_module\..\..\runs\v20230217_181515_ref_seq\inputs_case\ccmult.csv.gz'

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bs43384\\Documents\\ReEDS_OpenAccess\\postprocessing\\retail_rate_module\\..\\..\\runs\\v20230217_181515_ref_seq\\inputs_case\\ccmult.pkl'

[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_upv_reduced.csv'
[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_wind-ons_reduced.csv'
[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_wind-ofs_reduced.csv'
[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_upv_reduced.csv'
[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_wind-ons_reduced.csv'
[Errno 2] No such file or directory: 'C:\Users\bs43384\Documents\ReEDS_OpenAccess\runs\v20230217_181515_ref_seq\outputs\df_sc_out_wind-ofs_reduced.csv'```

I have attached the screen shot of the same.

[wesleyjcole]

This is an error with the postprocessing rather than with the input processing. It looks like things ran correctly, except that the retail rate module did not work. As long as you don't need the retail rate outputs, you should be fine to continue forward.