JP-3102: NIRSpec combined MOS/FS processing #8467

melanieclarke · 2024-05-07T16:40:19Z

Resolves JP-3102

Closes #7769

Process fixed slits defined in MSA metafiles, alongside standard MOS slits.

Checklist for maintainers

added entry in CHANGES.rst within the relevant release section
updated or added relevant tests
updated relevant documentation
added relevant milestone
added relevant label(s)
ran regression tests, post a link to the Jenkins job below.
How to run regression tests on a PR
Make sure the JIRA ticket is resolved properly

codecov · 2024-05-07T17:14:47Z

Codecov Report

Attention: Patch coverage is 56.80934% with 111 lines in your changes missing coverage. Please review.

Project coverage is 59.50%. Comparing base (dba65ae) to head (38396d5).
Report is 380 commits behind head on master.

Files with missing lines	Patch %	Lines
jwst/pipeline/calwebb_spec2.py	5.33%	71 Missing ⚠️
jwst/master_background/nirspec_utils.py	50.00%	13 Missing ⚠️
jwst/nsclean/nsclean.py	0.00%	9 Missing ⚠️
jwst/master_background/expand_to_2d.py	63.63%	8 Missing ⚠️
jwst/wavecorr/wavecorr.py	82.85%	6 Missing ⚠️
jwst/pipeline/calwebb_spec3.py	0.00%	3 Missing ⚠️
jwst/assign_wcs/nirspec.py	98.61%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8467      +/-   ##
==========================================
+ Coverage   58.63%   59.50%   +0.87%     
==========================================
  Files         391      391              
  Lines       39081    39245     +164     
==========================================
+ Hits        22914    23354     +440     
+ Misses      16167    15891     -276

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ithout MSA file source definition

melanieclarke · 2024-06-10T16:36:34Z

Okay, I've added documentation everywhere I could think of, and I have merged in the wavecorr changes, so I'm going to take this out of draft now.

New regression test run started here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1511/

Testing locally, I see the changes I expected for test_nirspec_mos_spec2, due to rearranging the slit order. After my last update, I now see no changes for test_nirspec_fs_spec2, except some pixel_replace differences that also appear when testing with master. I see no changes for test_nirspec_fs_spec3 or test_nirspec_mos_spec3.

hbushouse

Comments on review so far. Still much more wade through, but don't want to lose these.

CHANGES.rst

docs/jwst/data_products/msa_metadata.rst

docs/jwst/nsclean/main.rst

hbushouse

Wow, what a lot of great work! Especially all the new (extensive) unit tests.

jwst/extract_1d/extract.py

hbushouse · 2024-06-13T18:50:09Z

Looks like you could use one more regtest run, now that truth files have been updated for unrelated changes.

melanieclarke · 2024-06-13T18:52:18Z

Looks like you could use one more regtest run, now that truth files have been updated for unrelated changes.

Good idea - I'll start another one when the system is free.

Started here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1530/

hayescr · 2024-06-13T20:46:08Z

@melanieclarke I'm looking over this in the next couple days and will let you know whether I have any comments.

melanieclarke · 2024-06-14T13:52:43Z

Latest regression test results show expected NIRSpec MOS spec2 changes for reordering the slits, plus a change I previously overlooked for NIRSpec FS master_background correction.

The change is because the input test data is a cal file containing multiple fixed slits all marked POINT. Before this PR, point sources in non-primary FS were partially handled in the master_background step: it expected to find and use a pathloss extension for both point and uniform sources to correct the background, but did not expect or use flat or photom corrections. This partial correction should never be encountered in the wild anymore: non-primary sources are always marked EXTENDED now. Since the source locations can't be determined, none of the point source corrections are relevant, including pathloss.

In this PR, I changed the master_background handling to expect that a point source has all the necessary extensions (flat, pathloss, and photom), or else the background is not corrected for a point source. It will warn, and use the background directly, as if the source were extended. This is the source of the slight change in the regtest: the existing pathloss point corrections in the test data are ignored when computing the background for the non-primary slits.

hayescr

Thanks for the thorough work @melanieclarke! The changes look good and in testing this with some data everything works/looks as I would expect it to.

melanieclarke · 2024-06-18T14:00:03Z

Thanks for testing @hayescr!

hbushouse

Looks great to me!

Co-authored-by: Howard Bushouse <[email protected]>

@melanieclarke

JP-3564: remove most deep copies of NIRSpec WCS objects implements option for polyfit smoothing, save here, remove next commit JP-3581: initial commit of minimal working example use pipeline to handle backgrounds in association instead of step fixed unit tests, improved output filename handling fixed pipeline failures when step skipped added docs attempted fix readthedocs build responding to @melanieclarke and @drlaw1558 comments added support for selfcal files in asn responding to @drlaw1588 additional comments requested variable name change in badpix_selfcal.py responding to @melanieclarke and @braingram comments bugfix to unit test responding to next round of @melanieclarke comments added spaces to log warnings one more round of comments, thanks @melanieclarke fixes in response to @braingram comments updated regtest to include intermediate files split regtest in two responding to @hbushouse comments pin numpy add changelog Update stcal requirement from <1.6.0 to <1.8.0 (spacetelescope#8571) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> JP-3102: NIRSpec combined MOS/FS processing (spacetelescope#8467) Co-authored-by: Howard Bushouse <[email protected]> JP-3659: Fix for output WCS when first data set is empty (spacetelescope#8562) Co-authored-by: Howard Bushouse <[email protected]> remove requirements-max.txt remove end in ami log messages (spacetelescope#8568) Do not set conda version in Jenkinsfile(s) (spacetelescope#8585) JP-3618: Add intermediate LRS slit wcs reference frame (spacetelescope#8475) Co-authored-by: Howard Bushouse <[email protected]> unpin stsci.imagestats (spacetelescope#8577) Co-authored-by: Howard Bushouse <[email protected]> JP-3660: extract1d _coalesce_bounds fix (spacetelescope#8586) Co-authored-by: Timothy Brandt <[email protected]> Co-authored-by: Howard Bushouse <[email protected]> Fixed a missing import statement Fixed incorrect statement order Removed orphan import statement

github-actions bot added assign_wcs extract_1d extract_2d flatfield photom pipeline NIRSPEC testing exp_to_source spec2 pipeline spec3 pipeline wavecorr master_background labels May 7, 2024

melanieclarke force-pushed the nirspec_mos_fs branch from 6d6fc0a to 0e75426 Compare May 8, 2024 16:57

melanieclarke self-assigned this May 8, 2024

melanieclarke force-pushed the nirspec_mos_fs branch from 31897e1 to 8dce641 Compare May 16, 2024 17:22

melanieclarke force-pushed the nirspec_mos_fs branch from ddaf8d7 to d06baa7 Compare May 23, 2024 21:18

github-actions bot added the regression_testing label May 23, 2024

melanieclarke force-pushed the nirspec_mos_fs branch 2 times, most recently from e98b3a7 to 8429797 Compare May 29, 2024 21:32

github-actions bot added the combine_1d label May 29, 2024

melanieclarke force-pushed the nirspec_mos_fs branch from 56dd6b5 to cc14cfe Compare May 30, 2024 14:12

melanieclarke added 7 commits May 30, 2024 14:48

Fixed slits in MSA file working up through extract_2d

30ad631

Fixed slits via MSA file working through spec2, details TBD

4459b01

Updates for handling fixed slit point sources consistently with and w…

bf19678

…ithout MSA file source definition

Separate spec2 extraction by mode

7f75f2b

Allow FS extraction as MOS in spec2

5983fd9

Assign exp type by slit for spec3

4899141

Ignore non-primary fixed slits

7860826

github-actions bot added documentation source type labels Jun 10, 2024

Documentation updates for NIRSpec MOS/FS

fe37e37

melanieclarke marked this pull request as ready for review June 10, 2024 16:36

melanieclarke requested a review from a team as a code owner June 10, 2024 16:36

melanieclarke added 2 commits June 10, 2024 13:11

Use source_name to determine if FS slit comes from MSA file

3165264

Merge branch 'master' into nirspec_mos_fs

2bf2875

hbushouse reviewed Jun 12, 2024

View reviewed changes

CHANGES.rst Show resolved Hide resolved

docs/jwst/data_products/msa_metadata.rst Show resolved Hide resolved

docs/jwst/nsclean/main.rst Show resolved Hide resolved

Merge branch 'master' into nirspec_mos_fs

41ac52e

hbushouse reviewed Jun 13, 2024

View reviewed changes

jwst/extract_1d/extract.py Show resolved Hide resolved

Merge branch 'master' into nirspec_mos_fs

e81b583

hayescr approved these changes Jun 17, 2024

View reviewed changes

Merge branch 'master' into nirspec_mos_fs

2cbe0b1

melanieclarke removed the combine_1d label Jun 18, 2024

Merge branch 'master' into nirspec_mos_fs

38396d5

hbushouse approved these changes Jun 18, 2024

View reviewed changes

hbushouse merged commit 8c27f16 into spacetelescope:master Jun 18, 2024
27 of 28 checks passed

stscijgbot-jp mentioned this pull request Jun 18, 2024

Implement NRS MOS+FS processing #7769

Closed

melanieclarke deleted the nirspec_mos_fs branch June 18, 2024 20:28

t-brandt pushed a commit to t-brandt/jwst that referenced this pull request Jun 25, 2024

JP-3102: NIRSpec combined MOS/FS processing (spacetelescope#8467)

60aab18

Co-authored-by: Howard Bushouse <[email protected]>

braingram mentioned this pull request Jun 25, 2024

JP-3564: NIRSpec deepcopy fix #8587

Closed

8 tasks

melanieclarke mentioned this pull request Jun 26, 2024

JP-3657: Part 1 of a NIRSpec BOTS speedup #8609

Merged

8 tasks

t-brandt pushed a commit to t-brandt/jwst that referenced this pull request Jul 4, 2024

Removed code that had been deleted in spacetelescope#8467

71fd2bd

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JP-3102: NIRSpec combined MOS/FS processing #8467

JP-3102: NIRSpec combined MOS/FS processing #8467

melanieclarke commented May 7, 2024 •

edited by hbushouse

Loading

codecov bot commented May 7, 2024 •

edited

Loading

melanieclarke commented Jun 10, 2024 •

edited

Loading

hbushouse left a comment

hbushouse left a comment

hbushouse commented Jun 13, 2024

melanieclarke commented Jun 13, 2024 •

edited

Loading

hayescr commented Jun 13, 2024

melanieclarke commented Jun 14, 2024

hayescr left a comment

melanieclarke commented Jun 18, 2024

hbushouse left a comment

JP-3102: NIRSpec combined MOS/FS processing #8467

JP-3102: NIRSpec combined MOS/FS processing #8467

Conversation

melanieclarke commented May 7, 2024 • edited by hbushouse Loading

codecov bot commented May 7, 2024 • edited Loading

Codecov Report

melanieclarke commented Jun 10, 2024 • edited Loading

hbushouse left a comment

Choose a reason for hiding this comment

hbushouse left a comment

Choose a reason for hiding this comment

hbushouse commented Jun 13, 2024

melanieclarke commented Jun 13, 2024 • edited Loading

hayescr commented Jun 13, 2024

melanieclarke commented Jun 14, 2024

hayescr left a comment

Choose a reason for hiding this comment

melanieclarke commented Jun 18, 2024

hbushouse left a comment

Choose a reason for hiding this comment

melanieclarke commented May 7, 2024 •

edited by hbushouse

Loading

codecov bot commented May 7, 2024 •

edited

Loading

melanieclarke commented Jun 10, 2024 •

edited

Loading

melanieclarke commented Jun 13, 2024 •

edited

Loading