You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
! Unsure whether this should live here or in hubAdmin
Background
For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
To reduce the chances of this happening/mitigate the effects, a number of actions have been proposed:
Improve the documentation on this, get admins to think about the issue early on and warn them to avoid changes in data types.
we could also add functionality that could repair any data type discrepancies and update files to conform to a changed schema. This could help admins in a situation where all the above fail and a breaking schema change needs to be introduced.
The text was updated successfully, but these errors were encountered:
Background
For a hub to be successfully accessed as an arrow dataset, column data types should not change from round to round.
Generally many task IDs that are covered by our schema shouldn't change data type in further rounds as that's somewhat fixed by the schema. Custom task IDs however, which are beyond our control, and the output_type_id column have the potential to change and this could indeed cause problems downstream. This is mainly a problem for parquet files (but has a small chance to cause problems in csvs too).
To reduce the chances of this happening/mitigate the effects, a number of actions have been proposed:
output_type_id_datatype
argument across relevantvalidate_*()
fns #91) and consider a property in the schema where hub admins can configure and communicate this setting (Introduce a property to fix theoutput_type_id
column data type across the hub schemas#87).Add model output file schema repair functionality
As a future feature, once we have created functionality to inspect a hub for integrity and the following have all been implemented:
output_type_id
column data type across the hub schemas#87from_config
option tooutput_type_id_datatype
and set as default setting hubData#44output_type_id_datatype
argument across relevantvalidate_*()
fns #91we could also add functionality that could repair any data type discrepancies and update files to conform to a changed schema. This could help admins in a situation where all the above fail and a breaking schema change needs to be introduced.
The text was updated successfully, but these errors were encountered: