Concatenating MS: Table column 'unknown' error

[kendak333]

I've processed two epochs of data on a field separately using caracal, up to selfcal. I'm trying to concatenate the files so that I can test additional calibration on the combined data before imaging. However when I try to do this using CASA's concat task, I get the following error:

RuntimeError: Table column SIGMA_SPECTRUM is unknown

Looking into CASA documentation, and it does seem to know that SIGMA_SPECTRUM is a valid column name. Got no help via them, so wondering if it's caracal-processing thing?

[KshitijT]

@kendak333 is this standalone casa outside caracal or is this the concat using the transform worker?

[kendak333]

@KshitijT The error is from standalone casa (versions 5.6 and 6.5). When I use the caracal transform worker

transform:
enable: true
split_field:
enable: false
concat:
enable: true
col: corrected

I get a validation fail for the yaml:

transform:
- Key 'col' was not defined. Path: '/transform/concat'

The readthedocs says 'col' is a string, and I've tried with quotes, without quotes, I've tried for 'all' as well.

[kendak333]

Update: the yaml is only validated if I switch 'split_field' to true, but then it complains because it, quite rightly, can't find a gaincal (this is a target-only dataset, which I'm wanting to combine with a different epoch of the same target).

In doing some more sleuthing, it seems there was a change in MeerKAT MS design, from SIGMA_SPECTRUM to WEIGHT_SPECTRUM, which is possibly why standalone CASA is spitting out the error (the two epochs are two years apart). I'll probably have to fix this before I try concat-ing by any method.

[KshitijT]

I am not sure if CASA should complain as long as both MSs have the same columns? In the initial reductions of the datasets (I am assuming with CARACal) did you switch on the specweights field in the prep worker?

[kendak333]

Yes - the extract from the prep worker below:

specweights:
enable: True
mode: uniform
calculate:
statsfile: use_package_meerkat_spec
weightcols: WEIGHT, WEIGHT_SPECTRUM
noisecols: SIGMA, SIGMA_SPECTRUM
apply: True

I've just tried to concatenate two epochs observed during the same project (one day apart), and am getting the same error "SIGMA_SPECTRUM unknown" issue. Checking with a colleague who processed other epochs of the same set but have managed to concat without issue.

[paoloserra]

Hi @kendak333

If I understand correctly, you're interested in concatenation along the time axis, which CARACal does not support. CARACal concatenates along the spectral axis only. See the description at https://caracal.readthedocs.io/en/latest/manual/workers/transform/index.html#concat .

I have a question though. Do you really need to concatenate? In CARACal, you should be able to image N input MS jointly (as long as the target is the same), and thus get a better sky model, which you can use to calibrate the individual MS files. Why do you need to concatenate them?

[kendak333]

@paoloserra Ok thanks - I did notice eventually that the caracal concat won't do time axis.

Regarding the need or not to concatenate, I've made a combined image using both epochs, and the artefacts are more prominent in that than the individual ones. I've had good success with other fields by combining epochs and running the selfcal worker on the combined ms, so was hoping to do the same here. I'll look into what you suggest though.

As an aside for others looking at this thread - the concat apparently isn't working (with just the CASA concat) because the time-averaging is different in the two epochs (2s vs 4s). I have a weird hurdle where, after averaging the 2s to 4s, one scan does have 4s averaging, and the other inexplicably has 3.99s, so it still won't combine! :)

[paoloserra]

Yeah, I've had some pain with such minor metadata differences also wrt the channel width of FITS cubes -- not nice.

But back to the selfcal results, do you have any ideas about why joint imaging gives you worse artefacts? Was that in CARACal?

[kendak333]

It might be the case that they're there in (one of) the single-epoch images but below/near the noise, and the improvement in noise from the combination makes them pop up more starkly, though I'm not sure that'd explain the level I'm seeing - see below. The combined image is from sending the corrected data from both MSs into wsclean, no additional calibration applied.

Note no DD calibration has been applied at any stage so far.

[paoloserra]

I agree that the artefact in the left panel is too bright to be buried under the noise in the middle and right panels.

It almost looks like the left panel shows an image from DATA rather than CORRECTED. That's really weird.

[kendak333]

Definitely imaging CORRECTED_DATA (went and double checked!) - we're seeing this kind of behaviour quite a bit when we image multiple epochs together, which is why we're trying to go the selfcal-after-combining route.

[paoloserra]

Really puzzling. But since this is important, we could definitely add a time concat option in CARACal if you are interested.