Define load_forecasts approach & scope

[annakrystalli]

In the original covidHubUtils load_forecasts(), function filtering was handled internally by the function through explicit arguments mapping onto to task ids and output_types.

load_forecasts(
  models = NULL,
  dates = NULL,
  date_window_size = 0,
  locations = NULL,
  types = NULL,
  targets = NULL,
  source = "zoltar",
  hub_repo_path,
  local_zoltpy_path,
  zoltar_sqlite_file,
  data_processed_subpath = "data-processed/",
  as_of = NULL,
  hub = c("US", "ECDC", "FluSight"),
  verbose = TRUE
)

PROs

• Explicit definition of standard filtering variables
• Compact, concise code

CONs

• All filtering handled internally. If we go ahead with arrow datasets, this will mean constructing dplyr filtering statements internally (certainly doable)
• Additional task ids or task ids that don't follow load_forecasts() argument names will not be handled automatically and will instead need to be handled through ellipses (...) which feels kind of clucky.

Alternative approach

An alternative approach offered by using arrow datasets is that hubUtils could just manage connecting to the hub (i.e. opening it as an arrow dataset) and then users could write their own filtering code to access whatever data they require, much like accessing data from a database connection. e.g.

hub_con <- connect_hub("path_to_hub")
hub_con |>
    filter(target == "wk inc flu hosp",
           origin_date == "2022-10-01",
           type == "mean") |>
    collect()

PROs

• Less work to do in hubUtils
• Ability for users to write more custom flexible queries

CONs

• More verbose code required to access data
• Less guidance for users re: filtering parameters from function arguments themselves. (however, what hubUtils could focus on is functions for presenting useful summaries of filtering variables.)
• We might end up fielding lot's of bug reports/questions related to user query failures
• Unsure this could work for Zoltar

Could you please indicate below which option you prefer.
Feel free to add comments to the discussion to make the case either way.

[kathsherratt]

Thanks so much Anna! I really like the alternative approach. My thoughts were:

• I prefer the slightly neater division of labour to divide the download from the processing code.
• Even with filtering defined in the function call, I usually end up adding filtering/processing steps after loading - I'd rather keep all of this together in my own code.
• We can still provide additional helper functions to signal task ids to filter by etc.
• Generally I prefer to reduce dependencies on tidyverse packages for internal package dev (but not a big concern).

[elray1]

I like the look of the alternative approach in principle, but the way it works internally is not quite clear to me. Here are a few questions and comments:

1. I agree with Kath that I often end up doing extra filtering after data retrieval, but I think we definitely want to support options to do an initial collection on a subset of model output data in a hub. So I think we do need some way to specify "query parameters". I think that is handled by both of your proposals, but just wanted to clarify that.
2. It looks like the filter function takes a hub connection object as input and returns... also a hub connection object, but with the filtering information attached as properties/attributes?
3. If that's right, would we need to provide a method to "unfilter" a hub connection object, analogous to dplyr::group_by and dplyr::ungroup? In a working session, I might initially try a query with a certain filter on and then realize I need to go back and do a query with a less restrictive filter. If I had saved the intermediate connection object with filters applied, I'd want to be able to get back to a connection object without filters.
4. I do think we want to make sure we can support systems like Zoltar eventually. If the plan is to maintain something like a filters property on hub connection objects, we could presumably handle that by grabbing any applied filters and specifying them as query parameters.
5. Just a note that if we do end up going with the first option, I think we'd want to allow for non-predetermined filter variables via ...