diff --git a/CHANGES.rst b/CHANGES.rst index ab0386aa24..f48ffac22f 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -34,6 +34,9 @@ documentation - Change docs theme to ``sphinx-rtd-theme`` [#8224] +- Reorganized ``jump`` and ``ramp_fitting`` step docs content that's split between + the jwst and stcal repos. [#8253] + emicorr ------- diff --git a/docs/conf.py b/docs/conf.py index 2b94960439..34f736ff61 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -94,6 +94,7 @@ def check_sphinx_version(expected_version): 'matplotlib': ('http://matplotlib.org/', None), 'gwcs': ('https://gwcs.readthedocs.io/en/stable/', None), 'stdatamodels': ('https://stdatamodels.readthedocs.io/en/latest/', None), + 'stcal': ('https://stcal.readthedocs.io/en/latest/', None), } if sys.version_info[0] == 2: diff --git a/docs/jwst/jump/arguments.rst b/docs/jwst/jump/arguments.rst index 0db7c06c3d..954110a4cf 100644 --- a/docs/jwst/jump/arguments.rst +++ b/docs/jwst/jump/arguments.rst @@ -1,7 +1,10 @@ +.. _jump_arguments: + Arguments ========= -The ``jump`` step has 30 optional arguments that can be set by the user: +The ``jump`` step has many optional arguments that can be set by the user. +The details for each are listed below. **Parameters for Baseline Cosmic Ray Jump Detection** diff --git a/docs/jwst/jump/description.rst b/docs/jwst/jump/description.rst index 7ed340bb2c..c79f46c141 100644 --- a/docs/jwst/jump/description.rst +++ b/docs/jwst/jump/description.rst @@ -4,101 +4,34 @@ Description :Class: `jwst.jump.JumpStep` :Alias: jump +This step finds and flags outliers (usually caused by cosmic-ray hits) in +each pixel of an "up the ramp" IR exposure. + Assumptions ----------- -We assume that the ``saturation`` step has already been applied to the input -science exposure, so that saturated values are appropriately flagged in the -input GROUPDQ array. We also assume that steps such as the reference pixel -correction (``refpix``) and non-linearity correction (``linearity``) have been applied, so -that the input data ramps do not have any non-linearities or noise above the modeled Poisson +We assume that the :ref:`saturation ` step has already been applied to +the input exposure, so that saturated ramp groups are appropriately flagged in the +input GROUPDQ array. We also assume that steps such as +:ref:`reference pixel correction ` and +:ref:`non-linearity correction ` have been applied, +so that the input data ramps do not have any non-linearities or noise above the modeled Poisson and read noise due to instrumental effects. The absence of any of these preceding corrections -or residual non-linearities or noise can lead to the false detection of jumps in the ramps, -due to departure from linearity. +or the presence of residual non-linearities and noise can lead to false detection of jumps +in the ramps, due to departure from linearity. The ``jump`` step will automatically skip execution if the input data contain fewer -than 3 groups per integration, because the baseline algorithm requires two first -differences to work. - -Algorithm ---------- -The algorithm for this step is called from the external package ``stcal``, an STScI -effort to unify common calibration processing algorithms for use by multiple observatories. - -This routine detects jumps in an exposure by looking for outliers -in the up-the-ramp signal for each pixel in each integration within -an input exposure. On output, the GROUPDQ array is updated with the DQ flag -"JUMP_DET" to indicate the location of each jump that was found. -In addition, any pixels that have non-positive or NaN values in the gain -reference file will have DQ flags "NO_GAIN_VALUE" and "DO_NOT_USE" set in the -output PIXELDQ array. -The SCI and ERR arrays of the input data are not modified. - -The current implementation uses the two-point difference method described -in `Anderson & Gordon (2011) `_. - -Two-Point Difference Method -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The two-point difference method is applied to each integration as follows: - -#. Compute the first differences for each pixel (the difference between - adjacent groups) -#. Compute the clipped (dropping the largest difference) median of the first differences for each pixel. -#. Use the median to estimate the Poisson noise for each group and combine it - with the read noise to arrive at an estimate of the total expected noise for - each difference. -#. Compute the "difference ratio" as the difference between the first differences - of each group and the median, divided by the expected noise. -#. If the largest "difference ratio" is greater than the rejection threshold, - flag the group corresponding to that ratio as having a jump. -#. If a jump is found in a given pixel, iterate the above steps with the - jump-impacted group excluded, looking for additional lower-level jumps - that still exceed the rejection threshold. -#. Stop iterating on a given pixel when no new jumps are found or only one - difference remains. -#. If the there are only three differences (four groups), the standard median - is used rather than the clipped median. -#. If there are only two differences (three groups), the smallest one is compared to the larger - one and if the larger one is above a threshold, it is flagged as a jump. +than 3 groups per integration, because the baseline algorithm requires at least +two first differences to work. -#. If flagging of the 4 neighbors is requested, then the 4 adjacent pixels will - have ramp jumps flagged in the same group as the central pixel as long as it has - a jump between the min and max requested levels for this option. +Note that the core algorithms for this step are called from the external package +``stcal``, an STScI effort to unify common calibration processing algorithms +for use by multiple observatories. -#. If flagging of groups after a ramp jump is requested, then the groups in the - requested time since a detected ramp jump will be flagged as ramp jumps if - the ramp jump is above the requested threshold. Two thresholds and times are - possible for this option. - -Note that any ramp values flagged as SATURATED in the input GROUPDQ array -are not used in any of the above calculations and hence will never be -marked as containing a jump. - -Multiprocessing -=============== -This step has the option of running in multiprocessing mode. In that mode it will -split the input data cube into a number of row slices based on the number of available -cores on the host computer and the value of the max_cores input parameter. By -default the step runs on a single processor. At the other extreme if max_cores is -set to 'all', it will use all available cores (real and virtual). Testing has shown -a reduction in the elapsed time for the step proportional to the number of real -cores used. Using the virtual cores also reduces the elapsed time but at a slightly -lower rate than the real cores. - -If multiprocessing is requested the input cube will be divided into a number of -slices in the row dimension (with the last slice being slightly larger, if needed). -The slices are then sent to twopoint_difference.py by detect_jumps. After all the -slices have finished processing, detect_jumps assembles the output group_dq cube -from the slices. - -Subarrays -========= -The use of the reference files is flexible. Full-frame reference -files can be used for all science exposures, in which case subarrays will be -extracted from the reference file data to match the science exposure, or -subarray-specific reference files may be used. +:ref:`Algorithm ` +--------------------------------------- Large Events (Snowballs and Showers) -==================================== +------------------------------------ All the detectors on JWST are affected by large cosmic ray events. While these events, in general, affect a large number of pixels, the more distinguishing characteristic is that they are @@ -107,12 +40,40 @@ counts. These excess counts are, in general, below the detection threshold of normal cosmic rays. To constrain the effect of this halo, the jump step will fit ellipses or circles that -enclose the large events and expand the ellipses and circles by the input expansion_factor -and mark them as jump. +enclose the large events and expand the ellipses and circles by the input +`expand_factor` and mark them as jump (see :ref:`jump step arguments ` +for details). -The two types of detectors respond differently. The large events in the near-infrared +The two different types of JWST detectors respond differently. The large events in the near-infrared detectors are almost always circles with a central region that is saturated. The saturated core allows the search for smaller events without false positives. -The MIRI detectors do not, in general, have a saturated center and are only rarely circular. +The mid-IR (MIRI) detectors do not, in general, have a saturated center and are only rarely circular. Thus, we fit the minimum enclosing ellipse and do not require that there are saturated pixels within the ellipse. + +Multiprocessing +--------------- +This step has the option of running in multiprocessing mode. In that mode it will +split the input data cube into a number of row slices based on the number of available +cores on the host computer and the value of the ``max_cores`` input parameter. By +default the step runs on a single processor. At the other extreme, if ``max_cores`` is +set to "all", it will use all available cores (real and virtual). Testing has shown +a reduction in the elapsed time for the step proportional to the number of real +cores used. Using the virtual cores also reduces the elapsed time, but at a slightly +lower rate than the real cores. + +If multiprocessing is requested, the input cube will be divided into a number of +slices in the row dimension (with the last slice being slightly larger, if needed), +and sent for processing in parallel. +In the event the number of cores (and hence slices) selected exceeds the number of +available image rows, the number of slices will be reduced to match the number of rows. +After all the slices have finished processing, the output GROUPDQ cube - containing +the DQ flags for detected jumps - is reassembled from the slices. + +Subarrays +--------- +Full-frame reference files can be used for all science exposures even if the +science exposure was taken in a subarray mode. If so, subarrays will be +extracted from the reference file data to match the science exposure. +Alternatively, subarray-specific reference files, which match the science +exposure, may be used. diff --git a/docs/jwst/ramp_fitting/appendix.rst b/docs/jwst/ramp_fitting/appendix.rst index 1c0dd341c9..84b9bf80e2 100644 --- a/docs/jwst/ramp_fitting/appendix.rst +++ b/docs/jwst/ramp_fitting/appendix.rst @@ -1,7 +1,12 @@ Appendix ======== -The derivation of the segment-specific readnoise variance (:math:`{ var^R_{s} }`) is shown here. This pertains to both the 'conventional' and 'weighted` readnoise variances - the only difference being the number of groups in the segment. This derivation follows the standard procedure to fitting data to a straight line, such as in chapter 15 of Numerical Recipes. The segment-specific variance from read noise corresponds to :math:`{\sigma_b^2}` in section 15.2. +The derivation of the segment-specific readnoise variance (:math:`{ var^R_{s} }`) +is shown here. This pertains to both the 'conventional' and 'weighted' readnoise +variances - the only difference being the number of groups in the segment. +This derivation follows the standard procedure for fitting data to a straight line, +such as in chapter 15 of Numerical Recipes. The segment-specific variance from +read noise corresponds to :math:`{\sigma_b^2}` in section 15.2. For read noise R, weight w = :math:`{1 / R^2}`, which is a constant. @@ -28,7 +33,7 @@ Summations needed: :math:`{\sum_{k=1}^n k^2= n \cdot (n+1) \cdot (2 \cdot n+1) / 6 = n^3/3 + n^2/2 +n/6 }` -The variance from read noise +The variance due to read noise = :math:`{var^R_{s} = S_1 / D = S_1 / (S_1 \cdot S_{xx} - S_x^2)}` @@ -43,4 +48,5 @@ The variance from read noise = :math:`{ \dfrac{12 \cdot R^2} {(n^3 - n) \cdot t^2}}` -This is the equation in the code and in the segment-specific computations section of the Description. +This is the equation used in the step code and listed in the segment-specific computations +section of the step Description. diff --git a/docs/jwst/ramp_fitting/arguments.rst b/docs/jwst/ramp_fitting/arguments.rst index c4e15b7793..d3756f8f0f 100644 --- a/docs/jwst/ramp_fitting/arguments.rst +++ b/docs/jwst/ramp_fitting/arguments.rst @@ -1,6 +1,6 @@ Arguments ========= -The ramp fitting step has three optional arguments that can be set by the user: +The ramp fitting step has the following optional arguments that can be set by the user: * ``--save_opt``: A True/False value that specifies whether to write the optional output product. Default is False. @@ -17,13 +17,13 @@ The ramp fitting step has three optional arguments that can be set by the user: saturated. * ``--maximum_cores``: The number of available cores that will be - used for multi-processing in this step. The default value is '1' which does not use - multi-processing. The other options are either an integer, 'quarter', 'half', and 'all'. + used for multi-processing in this step. The default value is '1', which results in no + multi-processing. Other options are either an integer, 'quarter', 'half', and 'all'. Note that these fractions refer to the total available cores and on most CPUs these include physical and virtual cores. The clock time for the step is reduced almost linearly by the number of physical cores used on all machines. For example, on an Intel CPU with six real cores and six virtual cores, setting maximum_cores to 'half' results in a decrease of a factor of six in the clock time for the step to run. Depending on the system, - the clock time can also decrease even more with maximum_cores is set to 'all'. + the clock time can also decrease even more with maximum_cores set to 'all'. Setting the number of cores to an integer can be useful when running on machines with a large number of cores where the user is limited in how many cores they can use. diff --git a/docs/jwst/ramp_fitting/description.rst b/docs/jwst/ramp_fitting/description.rst index be91967ebd..d4d63862a6 100644 --- a/docs/jwst/ramp_fitting/description.rst +++ b/docs/jwst/ramp_fitting/description.rst @@ -5,337 +5,127 @@ Description :Alias: ramp_fit This step determines the mean count rate, in units of counts per second, for -each pixel by performing a linear fit to the data in the input file. The fit -is done using the "ordinary least squares" method. -The fit is performed independently for each pixel. There can be up to three -output files created by the step: - -#. The primary output file ("rate") contains slope and other results at - each pixel averaged over all integrations in the exposure. -#. The secondary product ("rateints") contains slope and other results for - each integration, stored as data cubes. -#. A third, and optional, output product is also available, containing detailed - fit information for each ramp segment for each pixel. - -The three types of output products are described in more detail below. +each pixel by performing a linear fit to the "up the ramp" data in the input file. +All fitting is done using the "ordinary least squares" (OLS) method. +The fit is performed independently for each pixel. Bad values flagged via +DQ flags are rejected from the fits. + +The input to the step is assumed to be the fully-corrected and flagged 4-D +data resulting from applying all previous steps of the +:ref:`calwebb_detector1 ` pipeline and will nominally be +the output from the :ref:`jump detection ` step. It is in fact +vital that all anomalies such as saturation, non-linearity, and CR jumps +be corrected or appropriately flagged in order to obtain useful results +from ramp fitting. The count rate for each pixel is determined by a linear fit to the -cosmic-ray-free and saturation-free ramp intervals for each pixel. Hereafter -such intervals will be referred to as a "segment." The fitting algorithm uses an -'optimal' weighting scheme, as described by Fixsen et al, PASP, 112, 1350. -Details of the computations are given below. +cosmic-ray-free and saturation-free ramp intervals for each pixel, with any +intervening groups flagged as "DO_NOT_USE" excluded from the fits. Hereafter +such intervals will be referred to as a ramp "segment." The fitting algorithm uses an +'optimal' weighting scheme, as described by +`Fixsen et al 2000 `_. Segments are determined using the 4-D GROUPDQ array of the input data set, under the assumption that the :ref:`saturation detection ` and :ref:`jump detection ` steps have already been applied, in order to flag occurrences of both saturation and cosmic-ray (CR) hits. -Segments are terminated where saturation flags are found. Pixels are processed -simultaneously in blocks using the array-based functionality of numpy. +A ramp segment is a set of contiguous groups that have no non-zero DQ values +assigned. The one exception to this rule is the occurrence of a "JUMP_DET" +(jump detected) flag: a group with this flag will be used as the first group of +the next segment. Any occurences of a "DO_NOT_USE" flag will be excluded from a +segment. When a "SATURATION" flag is found, the segment is terminated at the +preceding group and all subsequent groups are rejected. +Any segment containing only one good group is ignored if there is any other +segment of length greater than one. +Once all segments have been determined, slopes and variances are determined for +each one. + +Pixels are processed simultaneously in blocks using the array-based functionality of numpy. The size of the block depends on the image size and the number of groups per integration. +The main algorithms for this step are called from the external package ``stcal``. +This package is an STScI effort to unify common calibration processing algorithms +for use by multiple observatories. +Therefore the majority of the remainder of this document links to the relevant +sections of information in the ``stcal`` package. +JWST-specific features are described later within this document. + Upon successful completion of this step, the status keyword S_RAMP will be set to "COMPLETE". -Note that the core algorithms for this step are called from the external package -``stcal``, an STScI effort to unify common calibration processing algorithms -for use by multiple observatories. - Multiprocessing --------------- This step has the option of running in multiprocessing mode. In that mode it will -split the input data cube into a number of row slices based on the number of available -cores on the host computer and the value of the max_cores input parameter. By -default the step runs on a single processor. At the other extreme if max_cores is +split the input data into a number of slices based on the number of available +cores on the host computer and the value of the `maximum_cores` step parameter. By +default the step runs on a single processor. At the other extreme, if `maxiumum_cores` is set to 'all', it will use all available cores (real and virtual). Testing has shown a reduction in the elapsed time for the step proportional to the number of real -cores used. Using the virtual cores also reduces the elapsed time but at a slightly +cores used. Using the virtual cores also reduces the elapsed time, but at a slightly lower rate than the real cores. - -Detailed Algorithms -=================== - -Special Cases -------------- -If the input dataset has only a single group in each integration, the count rate -for all unsaturated pixels in that integration will be calculated as the -value of the science data in that group divided by the group time. If the -input dataset has only two groups per integration, the count rate for all -unsaturated pixels in each integration will be calculated using the differences -between the two valid groups of the science data. - -For datasets having more than a single group in each integration, a ramp having -a segment with only a single group is processed differently depending on the -number and size of the other segments in the ramp. If a ramp has only one -segment and that segment contains a single group, the count rate will be calculated -to be the value of the science data in that group divided by the group time. If a ramp -has a segment having a single group, and at least one other segment having more -than one good group, only data from the segment(s) having more than a single -good group will be used to calculate the count rate. - -The data are checked for ramps in which there is good data in the first group, -but all first differences for the ramp are undefined because the remainder of -the groups are either saturated or affected by cosmic rays. For such ramps, -the first differences will be set to equal the data in the first group. The -first difference is used to estimate the slope of the ramp, as explained in the -'segment-specific computations' section below. - -If any input dataset contains ramps saturated in their second group, the count -rates for those pixels in that integration will be calculated as the value -of the science data in the first group divided by the group time. - -MIRI First and Last Frames -++++++++++++++++++++++++++ -The MIRI :ref:`first frame ` correction step flags all pixels in the -first group of each integration, so that those data do not get used in either the jump -detection or ramp fitting steps. -Similarly, the MIRI :ref:`last frame ` correction step flags all pixels -in the last group of each integration. -The ramp fitting will only fit data if there are at least 2 good groups -of data and will log a warning otherwise. - -NIRCam Frame 0 -++++++++++++++ -If the input data contains a frame zero data cube, those data will be used to -estimate a slope for pixels that are saturated in all groups. If all groups in an -integration are flagged as SATURATED for a given pixel, the frame zero data array -is examined to determine whether or not it is also saturated. Saturated elements of +Because the data are sliced based on the number +of rows, if the number of cores requested for multiprocessing is greater than +the number of rows, the number of cores actually used will be no more than the +number of rows. This prevents any additional cores from operating on empty +datasets, which would cause errors during ramp fitting. + +:ref:`Output Products ` +--------------------------------------------------- + +:ref:`Special Cases ` +----------------------------------------------- + +NIRCam Frame Zero +----------------- +The NIRCam instrument has the ability to downlink data resulting from the initial +frame of each integration (known as "frame zero") when on-board frame averaging is +in use for a given exposure. If the frame zero data were downlinked, they will appear +in a 3-D data cube in the raw data products with a label of "ZEROFRAME". +The data from frame zero can be used to recover a slope estimate for a pixel in the +event the pixel saturates already somewhere within the first *group* of an integration. + +If all groups in an integration are flagged as SATURATED for a given pixel, the frame +zero data array is examined to determine whether or not it is also saturated. Saturated elements of the frame zero array are set to zero by the preceding :ref:`saturation ` step in the pipeline. Unsaturated elements will have non-zero values in the -frame zero array. If the frame zero is not saturated, then it's value will be +frame zero array. If frame zero is *not* saturated, then it's value will be divided by the frame time for the exposure in order to compute a slope for the pixel in that integration. This is analagous to the situation in which only the first group -in an integration is unsaturated and used by itself to compute a slope (see above). - -Note that the computation of slopes from either a single group or single frame zero -value is disabled when the step parameter ``suppress_one_group`` is set to ``True``. -In this case the slope value for a pixel with only one good sample will be set to zero. - -All Cases ---------- -For all input datasets, including the special cases described above, arrays for -the primary output (rate) product are computed as follows. - -After computing the slopes for all segments for a given pixel, the final slope is -determined as a weighted average from all segments in all integrations, and is -written as the primary output product. In this output product, the -4-D GROUPDQ from all integrations is collapsed into 2-D, merged -(using a bitwise OR) with the input 2-D PIXELDQ, and stored as a 2-D DQ array. -The 3-D VAR_POISSON and VAR_RNOISE arrays from all integrations are averaged -into corresponding 2-D output arrays. In cases where the median rate -for a pixel is negative, the VAR_POISSON is set to zero, in order to avoid the -unphysical situation of having a negative variance. - -The slope images for each integration are stored as a data cube in a second output data -product (rateints). Each plane of the 3-D SCI, ERR, DQ, VAR_POISSON, and VAR_RNOISE -arrays in this product corresponds to the result for a given integration. In this output -product, the GROUPDQ data for a given integration is collapsed into 2-D, which -is then merged with the input 2-D PIXELDQ to create the output DQ array for each -integration. The 3-D VAR_POISSON and VAR_RNOISE arrays are -calculated by averaging over the fit segments in the corresponding 4-D -variance arrays. - -A third, optional output product is also available and is produced only when -the step parameter 'save_opt' is True (the default is False). This optional -product contains 4-D arrays called SLOPE, SIGSLOPE, YINT, SIGYINT, WEIGHTS, -VAR_POISSON, and VAR_RNOISE that contain the slopes, uncertainties in the -slopes, y-intercept, uncertainty in the y-intercept, fitting weights, the -variance of the slope due to poisson noise only, and the variance of the slope -due to read noise only for each segment of each pixel, respectively. The y-intercept refers -to the result of the fit at an effective exposure time of zero. This product also -contains a 3-D array called PEDESTAL, which gives the signal at zero exposure -time for each pixel, and the 4-D CRMAG array, which contains the magnitude of -each group that was flagged as having a CR hit. By default, the name of this -output file will have the suffix "_fitopt". -In this optional output product, the pedestal array is -calculated for each integration by extrapolating the final slope (the weighted -average of the slopes of all ramp segments in the integration) for each pixel -from its value at the first group to an exposure time of zero. Any pixel that is -saturated on the first group is given a pedestal value of 0. Before compression, -the cosmic ray magnitude array is equivalent to the input SCI array but with the -only nonzero values being those whose pixel locations are flagged in the input -GROUPDQ as cosmic ray hits. The array is compressed, removing all groups in -which all the values are 0 for pixels having at least one group with a non-zero -magnitude. The order of the cosmic rays within the ramp is preserved. - -Slope and Variance Calculations -------------------------------- -Slopes and their variances are calculated for each segment, for each integration, -and for the entire exposure. As defined above, a segment is a set of contiguous -groups where none of the groups is saturated or cosmic-ray impacted. The -appropriate slopes and variances are output to the primary output product, the -integration-specific output product, and the optional output product. The -following is a description of these computations. The notation in the equations -is the following: the type of noise (when appropriate) will appear as the superscript -‘R’, ‘P’, or ‘C’ for readnoise, Poisson noise, or combined, respectively; -and the form of the data will appear as the subscript: ‘s’, ‘i’, ‘o’ for segment, -integration, or overall (for the entire dataset), respectively. - -Optimal Weighting Algorithm -+++++++++++++++++++++++++++ -The slope of each segment is calculated using the least-squares method with optimal -weighting, as described by Fixsen et al. 2000, PASP, 112, 1350; Regan 2007, -JWST-STScI-001212. Optimal weighting determines the relative weighting of each sample -when calculating the least-squares fit to the ramp. When the data have low signal-to-noise -ratio :math:`S`, the data are read noise dominated and equal weighting of samples is the -best approach. In the high signal-to-noise regime, data are Poisson-noise dominated and -the least-squares fit is calculated with the first and last samples. In most practical -cases, the data will fall somewhere in between, where the weighting is scaled between the -two extremes. - -The signal-to-noise ratio :math:`S` used for weighting selection is calculated from the -last sample as: - -.. math:: - S = \frac{data \times gain} { \sqrt{(read\_noise)^2 + (data \times gain) } } \,, - -The weighting for a sample :math:`i` is given as: - -.. math:: - w_i = (i - i_{midpoint})^P \,, - -where :math:`i_{midpoint}` is the the sample number of the midpoint of the sequence, and -:math:`P` is the exponent applied to weights, determined by the value of :math:`S`. Fixsen -et al. 2000 found that defining a small number of P values to apply to values of S was -sufficient; they are given as: - -+-------------------+------------------------+----------+ -| Minimum S | Maximum S | P | -+===================+========================+==========+ -| 0 | 5 | 0 | -+-------------------+------------------------+----------+ -| 5 | 10 | 0.4 | -+-------------------+------------------------+----------+ -| 10 | 20 | 1 | -+-------------------+------------------------+----------+ -| 20 | 50 | 3 | -+-------------------+------------------------+----------+ -| 50 | 100 | 6 | -+-------------------+------------------------+----------+ -| 100 | | 10 | -+-------------------+------------------------+----------+ - -Segment-specific Computations -+++++++++++++++++++++++++++++ -The variance of the slope of a segment due to read noise is: - -.. math:: - var^R_{s} = \frac{12 \ R^2 }{ (ngroups_{s}^3 - ngroups_{s})(tgroup^2) } \,, - -where :math:`R` is the noise in the difference between 2 frames, -:math:`ngroups_{s}` is the number of groups in the segment, and :math:`tgroup` is the group -time in seconds (from the keyword TGROUP). The derivation of this equation is given in -the appendix of this section, at `readnoise variance derivation. `_. +in an integration is unsaturated and used by itself to compute a slope. +Note that the computation of slopes from either a single group or the single frame +zero value is disabled when the step parameter ``suppress_one_group`` is set to ``True``. +In this case the slope value for such a pixel will be set to zero. -The variance of the slope in a segment due to Poisson noise is: +:ref:`Detailed Algorithms ` +------------------------------------------------------------ -.. math:: - var^P_{s} = \frac{ slope_{est} }{ tgroup \times gain\ (ngroups_{s} -1)} \,, +:ref:`Error Propagation ` +------------------------------------------------------- -where :math:`gain` is the gain for the pixel (from the GAIN reference file), -in e/DN. The :math:`slope_{est}` is an overall estimated slope of the pixel, -calculated by taking the median of the first differences of the groups that are -unaffected by saturation and cosmic rays, in all integrations. This is a more -robust estimate of the slope than the segment-specific slope, which may be noisy -for short segments. +:ref:`Data Quality Propagation ` +----------------------------------------------------------- -The combined variance of the slope of a segment is the sum of the variances: +.. _ramp_charge_migration: -.. math:: - var^C_{s} = var^R_{s} + var^P_{s} - - -Integration-specific Computations -+++++++++++++++++++++++++++++++++ -The variance of the slope for an integration due to read noise is: - -.. math:: - var^R_{i} = \frac{1}{ \sum_{s} \frac{1}{ var^R_{s} }} \,, - -where the sum is over all segments in the integration. - -The variance of the slope for an integration due to Poisson noise is: - -.. math:: - var^P_{i} = \frac{1}{ \sum_{s} \frac{1}{ var^P_{s}}} - -The combined variance of the slope for an integration due to both Poisson and read -noise is: - -.. math:: - var^C_{i} = \frac{1}{ \sum_{s} \frac{1}{ var^R_{s} + var^P_{s}}} - -The slope for an integration depends on the slope and the combined variance of each segment's slope: - -.. math:: - slope_{i} = \frac{ \sum_{s}{ \frac{slope_{s}} {var^C_{s}}}} { \sum_{s}{ \frac{1} {var^C_{s}}}} - - -Exposure-level Computations -+++++++++++++++++++++++++++ -The variance of the slope due to read noise depends on a sum over all integrations: - -.. math:: - var^R_{o} = \frac{1}{ \sum_{i} \frac{1}{ var^R_{i}}} - -The variance of the slope due to Poisson noise is: - -.. math:: - var^P_{o} = \frac{1}{ \sum_{i} \frac{1}{ var^P_{i}}} - -The combined variance of the slope is the sum of the variances: - -.. math:: - var^C_{o} = var^R_{o} + var^P_{o} - -The square-root of the combined variance is stored in the ERR array of the primary output. - -The overall slope depends on the slope and the combined variance of the slope of each integration's -segments, so is a sum over integrations and segments: - -.. math:: - slope_{o} = \frac{ \sum_{i,s}{ \frac{slope_{i,s}} {var^C_{i,s}}}} { \sum_{i,s}{ \frac{1} {var^C_{i,s}}}} - - -Variances in Output Products -++++++++++++++++++++++++++++ -If the user requests creation of the optional output product, the variances of -segment-specific slopes due to Poisson noise, :math:`var^P_{s}`, and read noise, -:math:`var^R_{s}`, are stored in the VAR_POISSON and VAR_RNOISE file extensions, -respectively. - -At the integration-level, the variance of the per-integration slope due to -Poisson noise, :math:`var^P_{i}`, is written to the VAR_POISSON extension of the -per-integration ("rateints") product, and the variance of the per-integration slope -due to read noise, :math:`var^R_{i}`, is written to the VAR_RNOISE extension. -The square-root of the combined variance per integration due to both Poisson and -read noise, :math:`var^C_{i}`, is written to the ERR extension. - -For the primary exposure-level ("rate") product, the overall variance in slope due to -Poisson noise, :math:`var^P_{o}`, is stored in the VAR_POISSON extension, the variance -due to read noise, :math:`var^R_{o}`, is stored in the VAR_RNOISE extension, and the -square-root of the combined variance, :math:`var^C_{o}`, is stored in the ERR -extension. - - - -Weighted Readnoise Variance -+++++++++++++++++++++++++++ +Charge Migration Special Case +----------------------------- If the :ref:`charge migration ` step has been performed prior to ramp fitting, any group whose value exceeds the -`signal_threshold` parameter will have been flagged with the CHARGELOSS and DO_NOT_USE -data quality flags. Due to the DO_NOT_USE flags, such groups will be excluded -from the slope calculations. - -However, it is desired to have a readnoise variance that is similar to pixels -unaffected by charge migration, so an additional type of variance will -be calculated, in which the excluded groups mentioned above will be included. -This additional, 'weighted', readnoise variance is used for weighting in the -:ref:`resample ` step later in the pipeline. The 'weighted' -readnoise variance is written to the VAR_RNOISE extension of each of the 3 -output products. - -The original ('conventional') type of readnoise variance described earlier is still used -internally in other variance calculations but, as mentioned above, is no longer written -to the separate variance extension. +``signal_threshold`` parameter value in that step will have been flagged with the +CHARGELOSS and DO_NOT_USE DQ flags. Due to the presence of the DO_NOT_USE flags, +such groups are excluded from all slope calculations. + +It is still desired, however, to have a read noise variance value for such pixels +that is similar to pixels unaffected by charge migration, so an additional type of +variance is calculated, in which the groups flagged with CHARGELOSS are still included, +despite the fact that those groups do not get included in slope calculations. +This version of the readnoise variance is the one stored in the VAR_RNOISE extension +of the various output products from the step, so that it will be the one used later +in the pipeline flow in the :ref:`resample ` step, if that step is +executed using Inverse Variance Map (IVM) weighting in the resampling process. + +The original version of readnoise variance described earlier, where all groups flagged +with DO_NOT_USE are *not* included, is still used internally +in all other calculations involving readnoise variance.