data loading and validation

[nickreich]

I'm trying to make sense of the data loading and validation pipeline. My understanding is that currently we have something where we do something like

hubUtils::hub_con <- connect_hub(hub_path)
a <- dplyr::collect(hub_con)

To get data from a hub. The resulting a object is a tibble.

This system works fine, and there is some "validation" on the data in a already, by virtue of it coming into the hub already and being "collected". But then to make it an actual model_out_df object, we need to then run another function here, like:

hubUtils::as_model_out_tbl(a)

This feels like an unnecessary and potentially confusing step for new users who then want to go on and use the downstream functions that require the model_out_tbl object (e.g. in hubVis or hubEnsembles). I suggest that, without removing the existing dplyr::collect() functionality, which is nice to have out of the box, we add a hubUtils::collect_hub() function that essentially does the following

a <- hub_con %>%
      dplyr::collect() %>%
      hubUtils::as_model_out_tbl()

as a convenience for the typical user.

[annakrystalli]

First I just want to make a couple clarifying points.

We don't really run dplyr::collect() directly on a hub_con object as doing so on a hub connection would return the entire hub dataset, so not likely something to be performed routinely, especially on large hubs.

Rather this is something users use at the end of a dplyr query to be performed on a hub's data to execute said query.

hub_con %>%
    filter(output_type == "quantile", location == "US") %>%
    collect()

The point of all this is to allow the user the flexibility to write more custom queries (that are also quite performant). It means we have no control over what the output of queries are though so there is no way for us to ensure bolting hubUtils::as_model_out_tbl() to the end of it by default wouldn't end up in an error.

In some ways that's why we developed the function. As users are free to interrogate their hubs in any way they can, passing their output through hubUtils::as_model_out_tbl() before engaging with any of the other hubverse workflows ensures expectations of downstream software are met.

Having said all that, if we describe the expectations clearly and warn of the possibility of failure if the output of any preceding queries does not conform to model_out_tbl expectations, I agree that it could indeed be nice to have a hubUtils::collect_hub() fn that work like:

collect_hub <- function(x) {
  x %>%
    dplyr::collect() %>%
    hubUtils::as_model_out_tbl()
}

and is deployed as such:

hub_con %>%
    filter(output_type == "quantile", location == "US") %>%
    collect_hub()

[annakrystalli]

One thing to additionally note. It does mean adding dplyr as an Imports dependency which we have avoided in hubUtils so far (but admittedly do have as a dependency in other hubverse pkgs)

[nickreich]

The additional dependency doesn't feel like a big deal to me.

[nickreich]

I agree with your points and appreciate the generality and flexibility of just using collect(). But as I've come back to the vignettes for hubUtils and hubVis specifically, it feels clumsy to have to be remembering to call the as_model_out_tbl() function. It's like my expectation as an average user of the hubverse is that when I query and collect the data from a hub it should already be in the format that all the other hubverse tools want. Maybe a way to make this more explicit could be to name the new function something like collect_model_out_tbl() so it's really telling you that you are collecting the data as that object. I still imagine that we would show both the use of collect() and collect_model_out_tbl() in the vignette, to showcase both ways of using it, highlighting on the one hand the flexibility of querying the hub_con objects and on the other hand the way in which all of the hubverse packages tie together nicely with the model_out_tbl objects.