Proposed enhancement of extract1d step for NIRISS SOSS data

[stscijgbot-jp]

Issue JP-3588 was created on JIRA by Joseph Filippazzo:

BACKGROUND: For NIRISS SOSS data, the extract1d step of the pipeline uses (by default) the ATOCA algorithm, which is an optimal spectral extraction routine that disentangles the signals from the overlapping order 1 and order 2 traces. This routine is very sensitive to the wavelength value assigned to each pixel by the ‘wavemap' reference file and the location of the trace centers assigned by the ‘spectrace’ reference file. 
 
PROBLEM: These reference files assume that the pupil wheel for the GR700XD grism is in its nominal position, dispersing the spectra onto the same detector pixels each time. However, the SOSS team’s analysis has shown that SOSS observations exhibit a range of pupil wheel positions (recorded as the PWCPOS value in the FITS header) that do not align with the nominal positions represented in the CRDS reference files.
 
SOLUTION: To account for this misalignment when reducing SOSS data, the SOSS team has developed the PASTASOSS Python package (https://github.com/spacetelescope/pastasoss) that transforms the pixel wavelength values and trace positions to reflect those seen in on-sky observations at a variety of PWCPOS values. We have created a pull request of the jwst pipeline (#8397) which incorporates this package to solve our problem. It reads in the PWCPOS value from an input JWST DataModel and saves the appropriate ‘wavemap’ and ‘spectrace’ reference files on the fly to perform the optimal extraction. It then updates the locations of the corrected reference files in the DataModel before performing the 1D spectral extraction with the ATOCA algorithm. All input requirements and output data products remain unaffected by this change and the ‘override_wavemap' and ‘override_spectrace’ keyword arguments still function as intended. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Thanks Joseph Filippazzo ; let's discuss at the JP meeting next week.  One quick comment though from looking at the pull request- generally the pipeline has tried to avoid introducing new dependencies, so importing directly from pastasoss would probably not be ideal.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Hi David! We understand, but we have worked very hard on PASTASOSS and we are convinced is the best long-term solution possible for SOSS needs. PASTASOSS is also an STScI-supported tool.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Hi Joseph Filippazzo! I'm back from leave and reasonably up to date on this project - do you want to have a chat about your current progress and plans? I took a look at the slides you sent, and I have a suggestion or two on implementation.

[stscijgbot-jp]

Comment by David Law on JIRA:

Just wanted to check in on this ticket quickly after the JP meeting today.  How is this looking for 11.1?  I gather that there's still some work to be done on the software side, but is this just final polishing or are there still any larger issues to be worked out?

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

David Law I think we are on track to meet the deadline for 11.1 but Tyler Pauly should definitely weigh in. I don't think there are any roadblocks or large issues to be worked out but the SOSS team will want to extensively test it since it will affect all x1d SOSS data and is a pretty significant change to the extract_1d step. Though we are introducing a new reference file along with the changes (https://jira.stsci.edu/browse/CRDS-848), these updates actually simplify the code and take a lot of the guesswork (complicated and brittle fitting routine) out of the spectral extraction algorithm.

[stscijgbot-jp]

Comment by Rosa Diaz on JIRA:

Just to confirm: when you say in the description "...It reads in the PWCPOS value from an input JWST DataModel and saves the appropriate ‘wavemap’ and ‘spectrace’ reference files on the fly to perform the optimal extraction." you mean 

"... it derives the appropriate ‘wavemap’ and ‘spectrace’ "

meaning, there are no appropriate ‘wavemap’ and ‘spectrace’ reference files in CRDS that this new step needs, only the reference file "pastasoss," correct?

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

Originally it would actually write out a wavemap and spectrace reference file on the fly, save it locally, and then change the reference file path in the pipeline. But that is not how we ended up implementing it.

Now it doesn't use the spectrace and wavemap reference files at all anymore. The arrays formerly pulled from those are now generated from the coefficients in the pastasoss reference file and passed to the extract_1d step.

Extract_1d for SOSS data used to require the spectrace, specprofile, speckernel, and wavemap reference files. Now it only requires the pastasoss, specprofile, and speckernel reference files.

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

By the way, Tyler Pauly pointed us to the PR that has the changes and the SOSS team is currently testing it. We should be finished shortly!

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

In a review of the PR and regression test results, we noted that current output includes order 3 spectra, while the pastasoss correction does not currently have support for this order. We heard from Aarynn Carter:

"Regarding the loss of Order 3, this is something we are aware of and are comfortable with given the other advantages that PASTASOSS provides. Tyler [B.] will be revising the Github version of PASTASOSS to provide Order 3 (and potentially 4) traces over the next few months (targeting the next JP ticket prioritization), at which point we can edit the new pastasoss reference file and re-enable the capability."

[stscijgbot-jp]

Comment by Joseph Filippazzo on JIRA:

I did some tests for the proposed changes with NIRISS SOSS data and the results look good to the SOSS team.

I reduced the SUBSTRIP256 data 'jw01539167001_03101_00001-seg001_uncal.fits' through the DETECTOR1 and SPEC2 pipelines (plots attached) using the changes for build 11.1. The new implementation of the extract1d routine that uses the new pastasoss reference file 'jwst_niriss_pastasoss_0001.asdf' fixes the erroneously shifted wavelength arrays seen the current build.

I also tested it on the SUBSTRIP96 data 'jw03596002001_03101_00001-seg001_uncal.fits' to see if the 10px offset seen in the extraction aperture region was fixed and it is. It also improves the spectral extraction of SUBSTRIP96 data at the detector edges by removing the old ATOCA algorithm transform that tries to fit for the trace locations.

We are happy with the results and excited to see them in the next build!

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

Fixed by #8763

[stscijgbot-jp]

Comment by David Law on JIRA:

Rachel Plesha I'm happy to close this ticket based on the NIRISS team's extensive involvement in testing it already; please do so if you concur.