GRS1747-312 dataset #20
Comments
|
Lovely stuff! Now you could take the clean component list (run wsclean with -save-source-list, if you haven't already), use crystalball to predict it into a separate column (say, |
|
okay, I will do it. |
|
I've already found a solution for the problem with the source list. I had
to set niter to 1 instead of 0.
…On Wed, 25 Jan 2023, 19:21 Oleg Smirnov, ***@***.***> wrote:
Lovely stuff!
Now you could take the clean component list (run wsclean with
-save-source-list, if you haven't already), use crystalball to predict it
into a separate column (say, SUN_MODEL_DATA), and see if you can't use
QuartiCal to peel it away? @IanHeywood <https://github.com/IanHeywood>
what do you reckon, workable plan?
—
Reply to this email directly, view it on GitHub
<#20 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWYM2BPQJLCFIUD2UUEMS7DWUFOKBANCNFSM6AAAAAAUGRQSSA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Well, you probably want to try some cleaning -- niter 1 will not clean very deeply at all! |
|
The images look great, however I still think the elongated sunspot complex is being aliased into your map at the very top right, and it probably ending up in the selfcal model. This prospect for aliasing is another nasty way the Sun could ruin things, and one I hadn't really considered when we were thinking about this project at the start. @Victoria-Samboco what are your image size and cell size? I agree that cleaning / peeling would be a good test. Maybe consider making a mask for the sunspots to constrain the deconvolution. Model image interpolation with smops might also be a good thing to try, it might be faster than crystalball if you end up with a lot of clean components. |
|
The image size is 6076. Now Im running the image after the Sun peeling step
to see what happens.
…On Thu, Jan 26, 2023 at 2:37 PM IanHeywood ***@***.***> wrote:
The images look great, however I still think the elongated sunspot complex
is being aliased into your map at the very top right, and it probably
ending up in the selfcal model. This prospect for aliasing is another nasty
way the Sun could ruin things, and one I hadn't really considered when we
were thinking about this project at the start.
@Victoria-Samboco <https://github.com/Victoria-Samboco> what are your
image size and cell size?
I agree that cleaning / peeling would be a good test. Maybe consider
making a mask for the sunspots to constrain the deconvolution. Model image
interpolation with smops might also be a good thing to try, it might be
faster than crystalball if you end up with a lot of clean components.
—
Reply to this email directly, view it on GitHub
<#20 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWYM2BKTRTIQMTPDUOF4EALWUJVXDANCNFSM6AAAAAAUGRQSSA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Are you seeing this level of aliasing with the wgridder enabled? |
Well, no. I didn't have the wgridder enabled... |
|
Try with |
Oky |
@landmanbester Here are the images of the Sun before(left) and after(right) -use-wgridder enabled, doesn't seem to be showing any improvements in terms of aliasing. |
|
@Victoria-Samboco the aliasing occurs when the Sun is out of the field of view of the image that you are making but a ghost image of it appears inside the image away from its true position. You can see this in the very first post in this issue, highlighted by the circle, where the sunspot pattern is visible. You can also see the sunspots aliased into the image that you made here:
The different image sizes and pixel sizes will cause the aliased image to move around. @landmanbester note that the image I made at the very top of the thread was using the w-gridder, and the aliasing is still a big problem. I think image/cell sizes and padding parameters need tweaking to remove the aliasing. |
|
That is a bit surprising. The w-gridder should compute the required padding and kernel support to suppress aliases up to the precision it is invoked with so we should report an issue if this is really the case. It may be that the default accuracy in wsclean is not sufficient. Let me see if I can reproduce with pfb-clean. Just to confirm, it's this one /home/ianh/solarkat/1671435077_sdp_l0_1024ch_GRS1747-312.ms on nash right? CORRECTED_DATA column? @IanHeywood can you recall the imaging parameters you used for the first image? |
|
That's the MS, I think the DATA column is the one to use. Here's the wsclean command: |
Oh okay, yes I can see it. But I'm wondering why the ghost image in circle does not appear in the images on top, but it appears in the images in the bottom. Basically the difference between them are the image sizes. I'm thinking that maybe the difference in image size could be a factor affecting the appearance of the ghost image in the circle. Knowing that larger images have more pixels and contain more information, which can result in increased resolution and clarity. In contrast, smaller images have fewer pixels and can result in lower resolution and decreased clarity. Analysing this images I think the ghost image in the circle might be more apparent in the smaller images because they contain less pixels, and therefore less information to obscure the ghost image.
|
|
I have some hope that the aliasing can be removed by upping the gridder accuracy but we'll have to wait for the deconvolution to finish to confirm. Here are MFS dirty images with pfb-clean (left) and wsclean (right)
The wsclean image I got using the command @IanHeywood posted above. I made the pfb-clean image by specifying a 3deg field of view with an oversampling factor of 2 w.r.t. Nyquist and a gridder accuracy of 1e-7. It's maybe a bit difficult to see from the naturally weighted image but even with the much smaller fov I don't see any aliasing. I will post the model when I have it and will also try with identical imaging parameters to confirm that's where it is coming from. @IanHeywood did you say there is a wgridder-accuracy parameter in wsclean now? I don't see it in the version I am using |
|
Allegedly there is, and you even guessed its name correctly :)
https://wsclean.readthedocs.io/en/latest/wgridding.html Did your pfb-clean run match both the image size and the pixel size of the wsclean image? I think changing either of those will change the position of aliased sources in the image. |
No, they didn't and yes the location of the aliased source will depend on them. I figured it would show up somewhere so let's see. I'll try with identical settings after I see the result of the deconvolution. I didn't want to wait for a 32kx32k PSF, the double sized PSF being a requirement of the deconvolution algorithm I am testing |
|
We could also cook up a residual set of visibilities that just includes the Sun (+ Sgr A?) to make these tests easier, save time on the deconvolution to go looking for the aliases. |
|
I would say yes if I wasn't so sure they shouldn't be there if you do the gridding accurately enough. Maybe a good experiment to test what kind of accuracy we need. I actually didn't realise that the aliases showed up in the MFS dirty image already, otherwise I would have just made those from the start. Here is the MFS dirty with the same image and cell sizes
Not exactly equivalent because, despite my best efforts, I can't get the weighting to match up with wsclean's. I don't think there are any aliases in there though. Can you see anything? Here is a side by side
Not sure if that is a hint of a sunspot or a real source but we could always up the gridding accuracy to find out. |
|
That blighter! I suspect we can get rid of them by using a very conservative form of Clark clean. I'm gonna have to give this a go. Might take a while though |
|
Is the Sun the source on top of the image ?
…On Mon, Feb 6, 2023 at 7:42 PM Landman Bester ***@***.***> wrote:
[image: image]
<https://user-images.githubusercontent.com/16836765/217043349-ebcf7046-ebbc-4889-b550-42eb4f8997ed.png>
That blighter! I suspect we can get rid of them by using a very
conservative form of Clark clean. I'm gonna have to give this a go. Might
take a while though
—
Reply to this email directly, view it on GitHub
<#20 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWYM2BOHUO6WKK2F4TK3JKDWWEZW5ANCNFSM6AAAAAAUGRQSSA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Yes, and the other strong source about a third of the way towards the Sun is the Sgr A complex at the Galactic centre. The centre of the solar system and the centre of the galaxy in one glorious but slightly problematic MeerKAT observation! |
|
The wgridder-accuracy is returning an error I'm running it from this recipe step, what could be the cause? |
|
Again there are large gaps in my stimela knowledge, but maybe the version of wsclean it's calling is too old... The w-gridder is available since WSClean version 2.9, and was further improved in speed in versions 2.10 and 3.0. |
|
Do you know which container it's calling (if that is what it's doing)? |
|
I think the |
I'm working on garfunkel and my wsclean version is the version v3.1 |
/home/samboco/lib/stimela/omstimelation/oms-cabs.yml on garfunkel |
Yes i don't think this option is added |
oky, let me check...I updated wsclen to version 3.2 still returning the same error |
|
Can you just run it directly until the parameter is exposed to the stimela config? |
Ok, I don't see the parameter. You can double check by running It's simple enough to add it. Or just run locally for now as @IanHeywood suggests |
samboco@garfunkel:~/solarkat/SUN_IMAGING_STEPS$ wsclean -name /obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -gridder -wgridder-accuracy 1e-6 -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/
+ + + + + + + + + + + + + + + + + + +
+ An exception occured:
+ >>> Invalid gridder requested: '-wgridder-accuracy'
+ + + + + + + + + + + + + + + + + + +It its not working even... (stimela_env) samboco@garfunkel:~/solarkat/SUN_IMAGING_STEPS$ wsclean -name /obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -use-wgridder -wgridder-accuracy 1e-6 -auto-mask 3 -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/ !!! WARNING: Parameter '-use-wgridder' is deprecated and will be removed in a future version of WSClean.
!!! Use parameter '-gridder' instead.
WSClean version 3.2 (2022-10-21)
This software package is released under the GPL version 3.
Author: André Offringa ([email protected]).
+ + + + + + + + + + + + + + + + + + + |
Yes, the |
|
The parameter names have probably changed in the latest version. You can probably find the correct name by looking at the output of |
|
This is an unrelated problem, but I am getting this error in Wsclean that I have not encountered before: |
@Victoria-Samboco did you also enable I have a container that I run stuff on nash with that has a recent version installed if you want to run it using singularity. |
I succeeded to run it directly from the command line using wsclean -name obs2/im1/im1 -data-column DATA -size 16300 16300 -channels-out 8 -niter 100000 -mgain 0.9 -weight briggs -1.0 -scale 1.1asec -join-channels -save-source-list -fit-spectral-pol 4 -auto-threshold 20 -pol I -padding 1.3 -nwlayers-factor 1 -gridder wgridder -wgridder-accuracy 1e-6 -temp-dir temp_dir 1671435077_sdp_l0_1024ch_GRS1747-312-1.ms/Here is the first restored image. It appears with the sunspots being aliased into the image, but now in a different position. These are images before selfcal. |
|
Folks, if the upstream cab is missing that parameter, you can add it yourself in your recipe temporarily, it's as simple as putting: cabs:
wsclean:
inputs:
wgridder-accuracy:
dtype: floatat the top of your recipe, and that's it. @Kincaidr I'm afraid your MS is corrupted. Sometimes that happens when a process writing to it is killed at just the wrong moment. I have never discovered a way to recover from this -- regenerating it is the only way I know. @Victoria-Samboco the aliasing position will of course depend on the image size, if you think carefully about what the FFT does... |
|
Indeed, the imaging parameters are different from mine which explains the moving aliases. The main issue here is that @landmanbester thinks the alising should vanish when running the wgridder at its 1e-6 precision limit but there they are... @Victoria-Samboco can you please check that using |
According to what it says here https://wsclean.readthedocs.io/en/latest/wgridding.html ( |
|
But wsclean doesn't have a |
|
From the message I assume Andre wants us to switch to
|
|
I missed the memo there. So If so then we're stuck with the aliasing with wsclean. |
|
I'm pretty sure there aren't any aliases in the versions I am making with pfb-clean. These use a gridder accuracy of 1e-7 and double precision. I will test with an accuracy of 1e-6 using single precision to see if I can reproduce. There is also a double-precision-accumulation option in ducc, maybe check if that is exposed in wsclean? |
-gridder -wgridder-accuracy as per @Victoria-Samboco's command line is incorrect either way... I've already added it to the recipe, I'ts working now |
|
Well this is bizarre, I can't seem to reproduce the aliasing even with single precision gridding with an accuracy of 1e-4 (no double precision accumulation either) which should match what wsclean is doing. I've run the comparison using multiple robustness values and I consistently see aliases in wsclean's images (except when using natural weighting where the aliases are not evident in either) and no aliases in those produced by pfb-clean (the wgridder in pfb-clean anyway). I even tried switching off the small inversion in wsclean which should really mean we are comparing apples to apples. I suspect something weird is going on inside wsclean. @o-smirnov can you maybe run this past Andre while you have him there at BASP? |
|
Hmmm -- same gridder of course, same gridding kernel? Could you give me two side by side images I can show to Andre? |
|
I'm having issues getting the weighting schemes to match up and since the aliases don't show up using natural weighting it's a bit difficult but these two are pretty close (they are saturated to the same level but not scale matched) Left is wsclean briggs 0 and on the right is pfb-clean with briggs 0.3 and a gridding accuracy of 1e-4 (single precision and no double accumulation). You can see the alias in the south east quadrant in the image on the left |
|
Isn't padding also a factor? |
|
Yeah definitely but that is computed internally by the wgridder as far I know so should be the same for both. Interesting thing is I usually have agreement with wsclean using natural weighting to within the gridding precision but not on this field |
|
Do tests for such agreement include really bright, extended things outside the field? 😭 |
|
Do you have a link for the MS, and is it ok if I ship a copy to Andre? He's keen to test with MeerKAT data anyway. |
|
The MS details and location are here: Let me ask about sharing this more broadly, as it's ThunderKAT LSP data and still in the proprietary period. |
I asked Andre yesterday, and he seems to think wgridder padding is independently controlled again in later versions of wsclean, and said his documentation might be in error. So maybe worth checking @Victoria-Samboco. |
Greetings all
As what was discussed by Prof. @IanHeywood (in #2 ), here are the images of the 1GC and the Self-calibrated data as well as the image of the Sun from this observation. We can see from the Sun image the very active regions, sunspots.
This first image is from prof Ian, from it we can see sidelobes most prominently on the right of the map as well as the Sgr A, the Galactic centre on top of it. Those sources in the circle are not real, they are actually sunspots being aliased into the image.
This are the 1GC and SSelfcal image that I got from the same dataset. Here is not possible to see the sunspots and the Sgr A probably it because of the size of the image it its not piking it, so I'm imaging it again with a bigger size (10240, hope it is enough :)).
This is the image of the Sun from this observation.
The text was updated successfully, but these errors were encountered: