Improve profile merging for nested dicts #9
Conversation
vpavlin
force-pushed
the
feature/user-spark-configuration
branch
8 times, most recently
from
372a34c to
3dfe467
Compare
vpavlin
force-pushed
the
feature/user-spark-configuration
branch
from
3dfe467 to
9a4744b
Compare
vpavlin
changed the title
| # lists can be only appended | ||
| if isinstance(b, list): | ||
| # merge lists | ||
| a = list(set(a + b)) |
There was a problem hiding this comment.
This will prevent you from implementing list of dicts in the user profiles since default dictionaries aren't hashable
|
Is the expectation that 2 valid user profiles (userA and userB) can be merged so the merged profile has the combined data from userA and userB? Or is userB's profile the source of truth so that it overwrites anything that exists in userA? |
|
It's complicated:) I would say generally the latter - we have priority list of sources..you could probably mathematically put it as i.e.start from empty profile, add globals as it has the least priority and sets the base, then layer any other applicable profile and then the user speicific stuff The problem with lists is that you don't want to duplicate values, which is why I am using sets. As we control the structure of the singleuser profiles I am not sure if we need to care about list of dicts, but it would be a nice to have, but I am not sure how to achieve it:) I a m open to ideas |
|
That makes sense now. I was looking at the changes in isolation and the merge method was restricting the yaml structure without any high level context. It works for the current structure with one edge case where a list/dict could contain stale data since the original destination data isn't being pruned. |
No description provided.