clarify missingness of point estimates; add examples for determining output_type_id format

[zkamvar]

This modifies the description of output_type_id for point estimates to clearly state that the entry should be a missing value, either NA in R or None in Python.

This also states that this applies to schemas prior to v4.0.0 (see hubverse-org/schemas#109) and adds the python one-liner as suggested by @MKupperman in #198.

The preview for this PR is at https://hubdocs--197.org.readthedocs.build/en/197/user-guide/model-output.html

[zkamvar]

I would like to get another modeler perspective on this if it sounds clear: @trobacker

[micokoch]

Anna made the technical comments, but from a readability angle, it looks good to me. It even made me realize I should uncapitalize a "parquet" which I had capitalized previously.

[zkamvar]

I've updated the text according to Anna's suggestions and what I found in #198

[elray1]

I have not read all of this carefully. But I noticed Anna's comment above, "I believe also exists in python via pd.NA, is the closest thing to R's NA and I think that's actually what @MKupperman used in his successful exploration (but I might be wrong)."

Just noting that missing values in polars are best/most often/most generally represented using null (which is in practice set by specifying None). Again, haven't read this but just wanted to quickly mention it in case it's relevant to anything here.

[annakrystalli]

It might indeed be relevant @elray1.

What we are interested here is what missing values will serialise in an interoperable way that will be correctly interepreted as a missing value by our software when read in. We will definitely have NAs in our files as we will always have R users writing to csv files so either way polars will need to deal with that. I believe it can though as you would expect the R version of the polars package to understand NAs. There are also arguments when reading in files that allow us to specify null values on read (in both python and R).

Having said all that, it's of course best not to make any assumptions and actually test to ensure our suggestions don't cause any problems to desired downstream functionality. So @elray1, would you be able to add what a successful test of polars functionality requirements would look like here 🙏? #198

[zkamvar]

This PR is intended to clarify that "NA" in the schema actually means missing data and not the character "NA". I've tested these assumptions in #198 (comment) and I think it satisfies this comment:

Having said all that, it's of course best not to make any assumptions and actually test to ensure our suggestions don't cause any problems to desired downstream functionality.

[micokoch]

It looks very good, as far as I can tell. I made some small capitalization suggestions to be consistent with the rest of hubDocs.

[zkamvar]

@annakrystalli, did I address your concerns?

[annakrystalli]

Getting there from my perspective with some really useful and detailed additions!

Made a few corrections and a suggestion for moving a section and cleaning up some possible misconceptions.

[bsweger]

Thanks for addressing the missing data question. I reviewed the text and tried running all of the code, so made some notes about those things.

As for the requested "Python perspective"... after reading these changes, I'm honestly not sure what the desired outputs are. If I'm understanding the write-up correct, there's a distinction between "NA" for something not required and NA/None for something that is optional and missing?

I might need to hop on a zoom to better understand what we're going for

[bsweger]

Thanks for taking the time to make sure I'm understanding the problem space correctly! Based on that, I made a few more wording suggestions for your consideration (feel free to disregard)

[LucieContamin]

Thank you for the update, it looks all clear to me, I just made some minor comments.

[zkamvar]

Thank you @annakrystalli, @bsweger, @LucieContamin, and @micokoch for your comments!

As per @annakrystalli and @LucieContamin's suggestions, I have moved the documentation about the template generation and arrow schema closer to the examples. Specifically, I added the section about the arrow schema inside the example so that users do not have to look far to know what the arrow schema would look like.

As per @bsweger's suggestions (and #198 (comment)), I have attempted to focus the discussion for the modelers on the column type.

[annakrystalli]

I made a couple of minor last minute corrections and added a suggested note about another useful function, hubData::coerce_to_hub_schema().

Other than that, all my comments have been resolved. Thanks @zkamvar !

[annakrystalli]

FYI as a response to functions being relevant to modelers appearing across too many packages, I've made a proposal to re-export them to hubValidations here: hubverse-org/hubValidations#149

Input welcome

[LucieContamin]

Thank you very much, it looks all good to me. I just made 2 small comment, feel free to ignore

[zkamvar]

Thank you all for sticking with me through this!