HAP/Catalogs: Sky background is incorrect when image has too many unused pixels

[stscijgbot-hstdp]

Issue HLA-1284 was created on JIRA by Rick White:

Too many unused pixels that are filled with zeros can cause the sky background not to be calculated correctly. Michele explains in the comments that this comes from a part of the code that was intended to handle ACS/SBC images (which are photon counts and can have many zero pixel values). This has the unintended side effect of not computing the background correctly for many non-SBC images.

There are about 3,200 filter images (out of ~65k ACS images and ~90K WFC3 images) for both ACS and WFC3 that have this problem. So it is a moderately rare problem, but it does affect 3-5% of the images.

An example is image hst_11360_b3_wfc3_uvis_f673n_ib6wb3, which is in a visit that has large shifts between some of the filters. As a result, the region covered by the total image is much larger than the region covered by the f673n image. In the total_trl.txt file for that visit, these messages appear for the f673n image:

File: hst_11360_b3_wfc3_uvis_f673n_ib6wb3_drc.fits
Input image contains excessive zero values in the background. Median: 0.0 RMS: 0.058754702733325864
*** User requested the sigma_clip algorithm to determine the background image. ***
Computation of image background complete

The Median value is deliberately set to zero because there are many zero pixels. There is a special test in the code for this case, and it skips the rest of the background processing (including the Background2D calculation) as a result. But those zero pixels appear because most of the image region is blank due to not being covered by the filter.

In this visit, all the filter images have the same problem. Only the total image (which combines all the filters) is able to compute the median value.

I looked at this because I was working on HLA-1240. I do not know whether this could be related to the problems in that ticket, but it might be.

The correct behavior would be to ignore the pixels in the blanked region when counting zero pixels for this test.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White Just a quick note at this time.  Although we advertise there are two methods to compute the background (i.e., sigma-clipped statistics and Background2D), there are actually three methods.  The third method is considered the "unusual" case as it appeared to apply only to SBC data.  However, your investigation reveals a case we had not considered.  For SBC images, many background pixels appears to be identically zero in the "illuminated" portion of the image.  If the percentage of originally zero combined with any NaN values which were reset to zero are greater than a configured percentage, the median is set to zero and the rms of the non-zero pixels within the footprint is computed.  A flag is also set not allowing any further background computation algorithms to be executed.

Your dataset can be used for further analysis of the background determination algorithms!

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena Thanks, that is very enlightening! I understand and am not surprised that the SBC images require some special handling. Those photon-count images have very different properties than all the other detectors.

You understand the code flow much better than I do, but there are a few approaches that could work:

1. If there is a mask array for the image (not derived based on image != 0), that could be used to count the zero pixels in only the illuminated region.
2. If you have control over when NaNs are replaced with zeros, there could be a test before that to count zero pixels.
3. An instrument-specific config parameter could be added to determine whether this zero-fraction is applied or not. That would be set to apply for ACS/SBC and not to apply for all the other instruments.

This does not affect many total images. I find 900 total images with median=0; 885 of those are for ACS/SBC, and only 15 are for other cameras. There are a total of 57,420 visits in my list (from the current public HST cache), so only 0.026% of the total images have incorrect sky values due to this problem.

On the other hand, it affects a lot of filter (non-total) images. It is more common for the filter images because there are a lot of cases like my example, where there are large differences in sky coverage between filters. When you combine the filter images all together to make the total image, there is not much blank space left. But the individual filters can have a lot of space around them.

I find 5,613 filter images with median=0; 1,646 of those are for ACS/SBC, and 3,967 are for other cameras. There are a total of 100,362 filter images in my list, so 4.0% of them have incorrect sky values due to this problem. That is not very rare.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

I updated the title and description based on your comments. I think this version should be more accurate.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White. I have made changes to eliminate conversion of NaN to zero, as well as as limiting some computations to only SBC. It does seem to me that when working with the Filter products that the footprint mask based upon the FilterProduct should be used instead fo the footprint mask based upon the TotalProduct. Thoughts?

Ooops. I see this is your Option 1 below.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White I processed the ACS/UVIS dataset, ib6wb3, which is the subject of this ticket, as well as two SBC datasets, jecn01n and j9zp02, using a modified version of drizzlepac. The datasets are located in /home/mdelapena/ForRick/ZERO under appropriately named subdirectories. The Filter products now use their "own" footprint mask. Also, the special case of identically zero background is now only applicable to SBC. Please take a look at the results. :)

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena I confirmed that the background gets computed correctly for this image now. The catalogs look very good too (partly because the cosmic ray contamination is greatly reduced by omitting single-exposure filters).

The catalogs for hst_11110_02_acs_sbc_total_j9zp02_drz.fits are empty. They were sparse before (1 source in the segment catalog, 10 in the point catalog). But now they do not have any sources. I'm not sure what happened there (have not looked at the trailer file yet).

The catalogs for hst_16261_01_acs_sbc_total_jecn01_drz.fits look pretty good. They have slightly more sources than the previous version. Interestingly, the image looks different (and better) in the new version. I'm not sure what has changed in the image processing, but that could explain why the hst_11110_02_acs_sbc_total_j9zp02_drz.fits catalogs changed too?

Anyway, bottom line: the new uvis catalogs look good, and the background processing is correct. It would be good to figure out why the catalogs got rejected in the hst_11110_02_acs_sbc_total_j9zp02_drz visit. But the other SBC visit looks better than before, so it may not be worth a big effort to track down the issue.