Adding/persisting galaxy objIDs to qp ancillary data

[alexandergagliano]

I have qp quantiles stored for 1000 galaxies and am trying to link these properties back to the original galaxies. Would it be possible to revise the metadata of the files to include the original object ID of the galaxy? Currently the metadata only includes:

{'pdf_name': array([b'quant_piecewise'], dtype='|S15'),
'pdf_version': array([0]),
'quants': array([[0. , 0.1 , 0.2 , 0.3 , 0.4 , 0.5 , 0.6 , 0.7 , 0.8 ,
0.9 , 0.999, 1. ]])}

[aimalz]

Alternatively, it would be acceptable to add them to Ensemble.ancil. However, if this attribute doesn't already exist, using Ensemble.add_to_ancil() throws a NoneType error rather than creating an ancil dict (nor does it get added to metadata from Ensemble.add_to_metadata()).

in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=xvals, yvals=yvals, check_input=True))
in_pdfs.ancil = {'GALID': in_df['GALID']}

[eacharles]

You should be able to do

in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=xvals, yvals=yvals, check_input=True))
in_pdfs.set_ancil({'GALID': in_df['GALID']})

[eacharles]

Could you check if Ensemble.set_ancil() works, and if it does, close this issue, thanks?

[aimalz]

It doesn't throw an error, but it doesn't actually get added to the metadata and Ensemble.ancil remains None. Including it as a keyword upon instantiation (in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=xvals, yvals=yvals, check_input=True), ancil=any_dictionary)) causes a KeyError. (Apologies if it's not even supposed to be provided as a dictionary argument -- it would help if there were docstrings to specify the types of anticipated inputs.)

[eacharles]

Have you updated tables_io?

…

On Apr 21, 2022, at 12:36 PM, Alex Malz ***@***.***> wrote: It doesn't throw an error, but it doesn't actually get added to the metadata and Ensemble.ancil remains None. Including it as a keyword upon instantiation (in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=xvals, yvals=yvals, check_input=True), ancil=any_dictionary)) causes a KeyError. (Apologies if it's not even supposed to be provided as a dictionary argument -- it would help if there were docstrings to specify the types of anticipated inputs.) — Reply to this email directly, view it on GitHub <#39>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRIGISJKB33UTORSL2FU6TVGGU5DANCNFSM5T4EKBBQ>. You are receiving this because you commented.

[eacharles]

An you are totally welcome to update the doc strings and make a PR.

…

-e

On Apr 21, 2022, at 12:38 PM, Eric Charles ***@***.***> wrote: Have you updated tables_io? > On Apr 21, 2022, at 12:36 PM, Alex Malz ***@***.*** ***@***.***>> wrote: > > > It doesn't throw an error, but it doesn't actually get added to the metadata and Ensemble.ancil remains None. Including it as a keyword upon instantiation (in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=xvals, yvals=yvals, check_input=True), ancil=any_dictionary)) causes a KeyError. (Apologies if it's not even supposed to be provided as a dictionary argument -- it would help if there were docstrings to specify the types of anticipated inputs.) > > — > Reply to this email directly, view it on GitHub <#39>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRIGISJKB33UTORSL2FU6TVGGU5DANCNFSM5T4EKBBQ>. > You are receiving this because you commented. >

[eacharles]

So, the setting, reading and writing of ancillary data is explicitly being testing in the unit tests.
If you look in https://github.com/LSSTDESC/qp/blob/8f6c2c4bd22f9e3061ae5329625b35ef8fa34280/qp/scipy_pdfs.py#L27 ancil is being set for a particular test case.
Then set_ancil is being called when building test pdfs
https://github.com/LSSTDESC/qp/blob/8f6c2c4bd22f9e3061ae5329625b35ef8fa34280/qp/test_funcs.py#L34
And it the persistence loopback test it is checking that the ancillary data matches in the original version and in the written and re-read version.
https://github.com/LSSTDESC/qp/blob/8f6c2c4bd22f9e3061ae5329625b35ef8fa34280/qp/test_funcs.py#L152
I just confirmed that this line of code is being executed when I run the unit tests. Most likely the issue you are seeing is that you have an older version tables_io that wasn't writing the ancillary data.

[aimalz]

@alexandergagliano and I got the same behavior with presumably different versions of qp, but the first way of doing it from the test setup worked for me after updating tables_io, thanks! Should I make a separate issue for including the ancil functionality to one of the demos, or is it too specific to the RAIL use case?

A closely related issue is that ancillary data doesn't get carried over when converting parameterizations, e.g.

in_pdfs = qp.Ensemble(qp.interp, data=dict(xvals=zgrid, yvals=flow_z, check_input=True), ancil=dict(GALID=in_df['GALID']))
out_pdfs = in_pdfs.convert_to(qp.quant_piecewise_gen, quants=quants, check_input=False)
print(out_pdfs.ancil)
> > None

It's not a bug per se, but it is definitely an instance of the code's behavior not matching standard user expectations.

[eacharles]

Well, go ahead and add a couple of lines in ensemble.convert_to to copy over the ancillary data if it exists.

[eacharles]

This should go in the ancillary data, not the metadata.   The metadata is for stuff common to all the galaxies, the ancillary data is per galaxy.On Aug 4, 2023, at 11:00 AM, Alex Gagliano ***@***.***> wrote: I have qp quantiles stored for 1000 galaxies and am trying to link these properties back to the original galaxies. Would it be possible to revise the metadata of the files to include the original object ID of the galaxy? Currently the metadata only includes: {'pdf_name': array([b'quant_piecewise'], dtype='|S15'), 'pdf_version': array([0]), 'quants': array([[0. , 0.1 , 0.2 , 0.3 , 0.4 , 0.5 , 0.6 , 0.7 , 0.8 , 0.9 , 0.999, 1. ]])} —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>