Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failures to get pars reference files in prefetch cause log messages #5251

Open
stscijgbot-jp opened this issue Aug 14, 2020 · 3 comments · May be fixed by spacetelescope/stpipe#135
Open

Failures to get pars reference files in prefetch cause log messages #5251

stscijgbot-jp opened this issue Aug 14, 2020 · 3 comments · May be fixed by spacetelescope/stpipe#135

Comments

@stscijgbot-jp
Copy link
Collaborator

Issue JP-1645 was created on JIRA by Robert Jedrzejewski:

When step pars reference files are requested from CRDS when running a pipeline, log messages are printed if there is no such pars reference file found in CRDS.  Find a way to disable these messages, as there are many and they are distracting.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Jonathan Eisenhamer on JIRA:

PR #⁠5375 has been created. However, from discussions with the CRDS development team, another solution has presented itself. An email has been sent and reproduced below. Commentary welcome.

Greetings,

This note will describe a proposed change in how Step Parameter Reference Files
are stored in CRDS. The changes are basically just organizational; there are no
changes in implementation.

Currently, step parameter reference files are located within the instruments'
imaps and appear under the individual instrument listings. This placement has
revealed a number of issues and potential issues:

  • There is a potential of having a significant number of step parameter reference types, crowding the reference listing for each instrument.
  • Difficulty in handling the sparsely populated domain space. There are and will always be a large number of steps and pipelines that will never have references defined. The main symptom of this is the unnecessary CRDS errors reported by the pipeline. With the current organization, changes would be required to core code to handle.
  • Difficult to determine coverage and difficult to discover what references could be created.
  • Conceptually, step parameter reference files are not instrument-specific, but observing-mode (exposure type) specific.

To resolve these issues, the proposal is to create a new top-level imap,
tentatively called "calpars", that would reside at the same level as the
individual instruments. Conceptually, this category would contain all things
that directly affect how the calibration pipeline runs. Advantages include:

  • One stop listing of all potential and implemented step parameter reference types, without crowding the instrument-specific reference views.
  • As-designed handling of missing references without core code changes. No more spurious error messages.
  • Space for future expansion of similar references, such as association rules.

The CRDS developers would like to hear thoughts and concerns about this proposal.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Jonathan Eisenhamer on JIRA:

Issue resolved by CCD-783.

@jdavies-st
Copy link
Collaborator

jdavies-st commented Jan 18, 2024

This is still an issue, so I'm reopening. For custom steps with no reftypes listed and no parameter reference files in CRDS (they are custom steps), we still get this error message:

$ pip install snowblind
$ strun snowblind jw01335008001_02101_00001_nrs1_jump.fits
2024-01-18 12:29:58,676 - CRDS - ERROR -  Error determining best reference for 'pars-snowblindstep'  =   Unknown reference type 'pars-snowblindstep'
2024-01-18 12:29:58,677 - stpipe.SnowblindStep - INFO - SnowblindStep instance created.
2024-01-18 12:29:58,720 - stpipe.SnowblindStep - INFO - Step SnowblindStep running with args ('jw01335008001_02101_00001_nrs1_jump.fits',).

One could have crds continue to emit ERROR log message, or we could change these to INFO or WARNING. Or we could just filter out the ERROR log messages in stpipe. Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants