JP-2922: Use new source_id syntax for level-3 products (#8442)

CHANGES.rst

@@ -20,6 +20,10 @@ assign_wcs
 - Move the assigned source position for dedicated NIRSpec MOS background slits from the
   lower left corner of the slit to the middle of the slit. [#8461]
 
+- Updated the routines that load NIRSpec MOS slit and source data from the MSA meta
+  data file to properly handle background and virtual slits, and assign appropriate
+  meta data to them for use downstream. [#8442]
+
 associations
 ------------
 
@@ -29,6 +33,9 @@ associations
 - Match NIRSpec imprint observations to science exposures on mosaic tile location
   and dither pointing, ``MOSTILNO`` and ``DITHPTIN``. [#8410]
 
+- Updated Level3 rules for new handling of NIRSpec MOS source_id formatting when
+  constructing output file names. [#8442]
+
 dark_current
 ------------
 
@@ -45,6 +52,13 @@ documentation
 
 - Added documentation for NIRCam GRISM time series pointing offsets. [#8449]
 
+exp_to_source
+-------------
+
+- Modified slit sorting to use `source_name` as the key, rather than `source_id`,
+  in order to support changes in `source_id` handling for NIRSpec MOS exposures
+  that contain background and virtual slits. [#8442]
+
 extract_1d
 ----------
 
@@ -103,6 +117,10 @@ outlier_detection
   finished, unless save_intermediate_results is True. This PR also addressed
   the _i2d files not being saved in the specified output directory. [#8464]
 
+- Removed the setting of `self.skip = True` when the step gets skipped (due to
+  inappropriate inputs), so that the step still executes when called again
+  while processing a list of multiple sources. [#8442]
+
 - Added tests for changes made in #8464. [#8481]
 
 - Added the option to use a rolling median instead of a simple median
@@ -124,6 +142,11 @@ pipeline
 - Removed unused ``scale_detection`` argument from ``calwebb_tso3``
   pipeline. [#8438]
 
+- Updated the ``calwebb_spec3`` pipeline handling of NIRSpec MOS inputs, to
+  comply with the new scheme for source ("s"), background ("b"), and
+  virtual ("v") slits and the construction of output file names for each
+  type. [#8442]
+
 pixel_replace
 -------------
 

docs/jwst/data_products/file_naming.rst

@@ -42,25 +42,27 @@ Just as for Stage 2, the suffix distinguishes the different file products of Sta
 
 The FITS file naming scheme for Stage 3 "source-based" products is as follows, where items in parentheses are optional:
 
- jw<ppppp>-<AC_ID>_[<"t"TargID | "s"SourceID>](-<"epoch"X>)_<instr>_<optElements>(-<subarray>)_<prodType>(-<ACT_ID>).fits
+ jw<ppppp>-<AC_ID>_[<"t"TargID | [<"s" | "b" | "v">]<SourceID>](-<"epoch"X>)_<instr>_<optElements>(-<subarray>)_<prodType>(-<ACT_ID>).fits
 
 where
 
  - ppppp: Program ID number
  - AC_ID: Association candidate ID
  - TargID: 3-digit Target ID (either TargID or SourceID must be present)
- - SourceID: 5-digit Source ID
+ - SourceID: 9-digit Source ID
  - epochX: The text "epoch" followed by a single digit epoch number (optional)
  - instr: Science instrument name (e.g. 'nircam', 'miri')
  - optElements: A single or hyphen-separated list of optical elements (e.g. filter, grating)
  - subarray: Subarray name (optional)
  - prodType: Product type identifier (e.g. 'i2d', 's3d', 'x1d')
  - ACT_ID: 2-digit activity ID (optional)
 
-An example Stage 3 product FITS file name is:
+Example Stage 3 product FITS file names are:
 
  jw87600-a3001_t001_niriss_f480m-nrm_amiavg.fits
 
+ jw54321-o066_s000123456_nirspec_f170lp_g235m_s2d.fits
+
 Optional Components
 """""""""""""""""""
 
@@ -72,7 +74,14 @@ TargID vs SourceID
 
     For single-target modes, this is the target identifier as defined in the APT proposal.
 
-    For multi-object modes, such as NIRSpec MOS, this will be the slit ID for each object.
+    For multi-object modes, such as NIRSpec MOS, this will be the source ID for each object.
+    Note that the SourceID value is preceded by one of three characters "s", "b", or "v".
+    For most multi-source observation modes, such as Wide Field Slitless Spectroscopy (WFSS) and
+    NIRSpec Fixed-Slit (FS) spectroscopy, the "s" prefix is used to indicate that the data
+    correspond to a defined source. In some NIRSpec MOS observations, however,
+    there can also be "background" and "virtual" sources
+    (see :ref:`NIRSpec MSA slitlets <msa_background_and_virtual_slits>`).
+    These cases use the "b" and "v" SourceID prefixes.
 
 epochX
 

docs/jwst/data_products/msa_metadata.rst

@@ -2,7 +2,7 @@
 
 MSA Metadata File: ``msa``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
-While not containing any actual science data, the NIRSpec MSA metadata file is nonetheless
+While it doesn't contain any actual science data, the NIRSpec MSA metadata file is nonetheless
 a crucial component of calibration processing for NIRSpec MOS exposures.
 It contains all the slitlet, shutter, and source configuration information that's needed
 by the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline to process a MOS exposure.
@@ -128,6 +128,8 @@ where the values of `MSAMETID` and `PATT_NUM` in the science exposure match
 the values of `msa_metdata_id` and `dither_point_index`, respectively, are
 loaded.
 
+Slitlets with a catalog source
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 To better understand the ways in which these metadata are used, it's useful to
 reference a hypothetical example of data within a ``SHUTTER_INFO`` table.
 The table below shows the first 9 rows of a ``SHUTTER_INFO`` table for a MOS exposure
@@ -186,13 +188,46 @@ within each slitlet should be considered the "primary" shutter. This is especial
 important for slitlets that contain extended sources and hence the `source_id` and
 `background` entries may indicate that the source is present in multiple shutters.
 
-When a slitlet is found that has no shutters with a primary source (i.e. no shutters
-having `primary_source` = "Y"), it is classified as a background slitlet and assigned
-a source ID value that's greater than the maximum source ID assigned to other slitlets
-(because such slitlets all have a source ID of zero in the MSA metadata coming from
-the ground system).
-These background slitlets can then be used in :ref:`master background <master_background_step>`
-subtraction.
+.. _msa_background_and_virtual_slits:
+
+Slitlets without a catalog source
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It is possible for users to define slitlets that do not contain a source that's defined
+in the MPT catalog when constructing an MSA configuration for an observation. These
+kinds of slitlets can be used for two purposes. First, slitlets in which all of the
+constituent shutters only contain background can be used to perform "master background"
+subtraction during the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline processing
+(see :ref:`NIRSpec Master Background <nirspec_modes>` for more details).
+Second, slitlets made up of open shutters that may contain signal from some uncataloged
+source in the field can also be created. These are referred to as "virtual" slitlets.
+
+Background and virtual slitlets have unique metadata in the shutter information table.
+The primary defining piece of data is their assigned `source_id` value, because these
+slitlets don't have a corresponding source listed in the source information table.
+During creation with the MPT, all background slitlets are given a `source_id` of zero.
+Virtual slitlets, on the other hand, are assigned *negative* `source_id` values, starting
+with -1 and counting downwards from there (i.e. each virtual slit has a unique negative
+`source_id` value).
+
+During the parsing of shutter information described in the previous section, when a
+slitlet is found that has no shutters with a primary source (i.e. no shutters
+having `primary_source` = "Y"), it is recognized as a background slitlet. In order to
+be able to track multiple background slitlets through the remaining processing, they
+are reassigned a new `source_id` value equal to their `slitlet_id`. Virtual slitlets,
+meanwhile, retain their unique negative `source_id` values throughout processing.
+
+During Stage 3 processing with the :ref:`calwebb_spec3 <calwebb_spec3>` pipeline,
+unique source-based product file names will be created that distinguish data from the
+three different kinds of slitlets: source, background, and virtual. As described in
+:ref:`source-based file names <src_file_names>`, the `SourceID` field of Stage 3
+file names consists of the 9-digit `source_id` number assigned to each MOS slitlet,
+preceded by one of the three characters "s", "b", or "v", which identifies whether
+the data are from a source, background, or virtual slitlet, respectively. Note that,
+as described above, the `source_id` number used here for background slitlets is a
+copy of their `slitlet_id` number. For example, a Stage 3 file name for data taken
+from a virtual slitlet with `source_id` = -42 will look like:
+
+  jw12345-o066_v000000042_nirspec_f170lp_g235m_x1d.fits
 
 
 The SOURCE_INFO Metadata

docs/jwst/exp_to_source/main.rst

@@ -83,8 +83,8 @@ on the meaning of each field in the file names.
 The FITS file naming scheme for the source-based "cal" products follows
 the standard Stage 3 syntax, such as::
 
- jw10006-o010_s00061_nirspec_f170lp-g235m_cal.fits
+ jw10006-o010_s000000061_nirspec_f170lp-g235m_cal.fits
 
-where "s00061" in this example is the source ID.
+where "s000000061" in this example is the source ID.
 See :ref:`source-based file names <src_file_names>` for more details
 on the meaning of each field in this type of file name.

docs/jwst/pipeline/calwebb_spec3.rst

@@ -114,7 +114,7 @@ input exposure-based products. The source-based collections of data are saved in
 intermediate files, one per source/slit. The root names of the source-based files
 contain the source ID as an identifier and use the same "_cal" suffix as the input
 calibrated exposure files. An example source-based file name is
-"jw00042-o001_s0002_niriss_gr150r_f150w_cal.fits", where "s0002" is the source id.
+"jw00042-o001_s00000002_niriss_gr150r_f150w_cal.fits", where "s00000002" is the source id.
 
 The reorganized sets of data are sent to subsequent steps to process and combine
 all the data for one source at a time.