Would users ever require a different hub model-output directory structure?

[annakrystalli]

Currently we specify the partitioning (directory structure) of the model-output directory only in the documentation and assume it is is fixed.

This obviously makes things simple for hubUtils. I am wondering however whether there would be any situation where this structure might not meet user needs and whether an option for flexibility, i.e. the user to be able to set a different partitioning structure, would be required?

[elray1]

The concepts of teams/models and rounds are central to hubs and so I can't imagine a hub objecting to using those. So I don't think we need to support "less" than the above structure.

I could imagine a hub might want to allow for additional levels of file structure, e.g. using further file path information as is standard in arrow data sets to allow for rapid queries on target id or output type variables. For example, a hub might want to specify that files are saved in paths like model-output/team1-modela/target_var=deaths/location=US/<round-id1>.csv to allow for rapid queries on the target variable and location, which are task id variables that we might often want to filter on as part of/before a data load. (Note: if we do support this, might want to link hub maintainers to this comment on considerations for partitioning schemes)

On the data load side, support for this is almost out of the box from apache arrow, right? But it might take some more thought to specify how the file paths should be organized in hub metadata. I would personally file support for this type of thing as a relatively low priority future enhancement so that we can focus on higher priority validation stuff for now.

[annakrystalli]

Thanks @elray1 👍

Makes sense and happy to just leave a record of the discussion here. If we do end up wanting to support something more we can open a proper issue and discuss implementation details there.

For the record, I feel the best place for it would be in the admin.json file as it is a global setting. It will also require specification of expected data types for any partitioning variables encoded by the file structure to work.

[nickreich]

Agree that this should be future enhancement, and that the current proposed structure seems good.

[nickreich]

Noting that at some point in a voices conversation someone had mentioned possibly wanting a less structured model-output directory, like all files just get dropped in there and maybe team/model names are in the filename. Maybe that's another use case that we could keep in mind, although as with the other "more partitioned" example, I think it's less of a leading case than what we have outlined above.

[harryhoch]

I think this is a matter of philosophy of the project. If we state up front that the goal here is not to claim to be exhaustive, but instead to do what seems to be appropriate based on our experience, we can leave an opening for other variants if/when they arise.

That said, this feels like a case where prematurely generalizing without a specific driving use case could lead to difficulty, so I'd say stick with the current structure until something else arises.