Error converting dd to csv

[SamRWest]

I'm getting this exception when converting the austimes DD files to CSVs:

...
File "xl2times\utils\dd_to_csv.py", line 147, in save_data_with_headers
    raise ValueError(
ValueError: Mismatched number of columns for param UC_DYNBND between data (2) and mapping (3)

For values:

>>> columns
['UC_N', 'LIM', 'VALUE']
>>> row
('UC_non-decreasing EE penetration', 'UP')

Relevant DD file section is:

SET UC_DYNBND

/
'UC_non-decreasing EE penetration'.UP
/;

And I found a seemingly relevant comment on line 77-79:

   if data[index].startswith("SET"):
            # See https://www.gams.com/latest/docs/UG_SetDefinition.html
            # This can only parse a subset of the allowed representations

So I'm guessing this error is because parsing isn't yet implemented for all the GAMS Set types (and this is one of them), does that sound right?
Any idea what's the right general approach to fixing this?

[olejandro]

No, UC_DYNBND is probably just misclassified in times-info.json. Could you try changing gams-cat from parameter to md-set?

[SamRWest]

Yep, that's got it sorted, thanks! I'll roll that into my next PR (a few other dd_to_csv bugfixes)
Glad we had this chat :)

[olejandro]

Great! Sounds like xl2times is about to produce the numbers for correct and additional rows for AusTIMES. Would be very curious to know how much off we are. :-)

[SamRWest]

Yep it sure is, and I'm also very curious :)
xl2times now converts the whole model (in one region) in about a minute (used to take over an hour). That's a big milestone :)

I'm floating the idea internally of sharing the (currently private) AusTIMES DD files with you guys for future CI benchmarking in here too. Would that be useful?

Might be worth a quick meeting to talk about our respective broader aims and stuff as well, if you guys are available sometime?

[olejandro]

That's quite a speed up. Exciting! 🚀
Having AusTIMES in benchmarks would definitely be very useful. The more model variety we can get the better. At least until we manage to get a good coverage with the unit tests.
Having a quick meeting sounds good! Was thinking about it as well. :-) Will reach out by email.

[SamRWest]

Sounds good. I'm at sam dot west at csiro dot au (no .com). And I'm at UTC+11 (sydney time).

[olejandro]

Thanks! Picked up your email from the git log. :-) My email should be in your inbox soon.

[olejandro]

Addressed in #230