Possible bug in new local datasets scaling / persistence

[ploh]

When trying to migrate a Paddepoel scenario (ID 607984), the new scenario based on a new derived dataset leads to lots of differences in gquery results compared to the old one - even without moving any of the user_values to init. inputs.

I will post the differences and instructions for reproducing my results shortly.

cc @ChaelKruip @antw @grdw @AlexanderWirtz

[ploh]

I have uploaded a csv file with the 528 differences in gquery results.

If you want to reproduce that locally, checkout branch local-datasets-migration_framework of etengine and run

rake scaling:convert SCENARIO=607984

there (the task assumes you have etsource checked out at ../etsource).

To experiment with different user_values / init. inputs, change the generated file tmp/migrate607984.yml accordingly and enter check in the pry session.

The "funny" thing is that even with identical user_values old vs new scenario and empty init. inputs, a lot of gqueries produce different results. That probably means the new scaling still differs from the old scaling in some respects.
I am going to analyze the results to find out where exactly they differ.

[ploh]

After some more testing I found that the differences almost disappear (apart from the number_of_cars not being scaled in the derived dataset based scenario), if I change the old scenario's scaling to not disable agriculture and industry sectors.

Although the has_industry and has_agriculture flags are set to false in the derived dataset, these sectors are not disabled in the new scenario. I created the issue quintel/atlas#90 for this.

I am not sure though, whether this problem can be fixed in Atlas alone or some "zeroing" of nodes needs to happen in ETEngine as well.

[ploh]

After some more testing I found that the differences almost disappear

Correction: For this I also removed all the user_values from the old as well as the new scenario. If I leave them in and only enable agriculture and industry, I still get 277 differences.

[ploh]

I still get 277 differences.

csv with differences

That is with: identical user_values old vs new, no init. inputs, agriculture/industry enabled in old and new

This implies that there is probably some other problem (except the non-disabled sectors) with the new scaling. And this occurs even without migrating user_values to init. inputs at all.

[ploh]

This implies that there is probably some other problem...

As an example I removed all user_values except

residences_roof_surface_available_for_pv: 0.00466567345892605

from old and new. Then the gquery etflex_households_solar_pv_installed returns 2.9150015478022 in the old scenario and 1.81109924970021 in the new scenario - the results are equal without the one remaining user_value.
I do not yet know how this is caused...

[ploh]

I do not yet know how this is caused...

OK, after some (or actually quite a lot) more investigating, I found a bug in the input caching that is causing this. I filed it as quintel/etengine#910

[ploh]

@ChaelKruip @grdw @antw After the bugs (quintel/atlas#90, quintel/etengine#910) are fixed, the above procedure should be re-tried and it should be checked that there are no gquery result differences remaining - that is, before moving any user_values to init. inputs. Only if this works, work on migrating Paddepoel should continue.

I am handing this over to you guys, now. But I will gladly answer questions and explain my findings or the migration rake task in more detail.

[grdw]

@ChaelKruip can this be closed?