Union clearinghouse data with raw data relay VDS data

[ian-r-rose]

Now that we have both the historical data in the "Clearinghouse" schema and ongoing data from the DB96 system, we'll want to union those two datasets into one complete dataset.

This depends on them having the same schema, so they will want to be unpivoted in the same way (cf #230)

[pingpingxiu-DOT-ca-gov]

@pingpingxiu-DOT-ca-gov

[jkarpen]

Taking this out of a sprint for now, dependency on the data relay server stabilization tasks being completed.

[jkarpen]

Per @pingpingxiu-DOT-ca-gov this task may no longer be needed but we should review when Ian returns and decide then.

[ian-r-rose]

This is still needed: if we want to use the data relay work going forward, we'll need to create a combined dataset with the historical data.

[jkarpen]

Per discussion with the group, this is necessary but we will wait until more QA checks have been done on the data (additional checks will be needed after the data is stitched together as well). Putting this in Sprint 2024-22 as a reminder to revisit this and work on it if we are ready at that point. (Aiming for Nov. 1 as a target date).

Note: This is considered medium, but will require follow-on QA/QC checks that will be more work. That will be tracked in a separate issue.

[jkarpen]

Per @pingpingxiu-DOT-ca-gov and @summer-mothwood we want to hold off on this until the evaluation in #423 is completed, as well as a separate issue (TBD, Summer will create this) on how to handle any gaps found. Moving to sprint 2024-23 as a reminder to check back on this and see if the dependencies are ready.