Selfcal of large datasets

Is MIRIAD being a pain? Let us know your experience.

Moderator: Mark.Wieringa

Post Reply
minh25
Posts: 11
Joined: Fri Aug 17, 2012 3:08 am

Selfcal of large datasets

Post by minh25 »

Dear Mark, Jamie et al.

I have 24 days of ATCA 9GHz data for project C2028 and C2670. First 14 days is C2028 (2010). Last 10 is C2670 (2012).

When I uvaver all the data, and go through the usual imaging and selfcal procedure I lose data after the selfcal step.
i.e. I do uvaver, invert, mfclean, selfcal, invert #2, mfclean #2, restor.

After selfcal data seems to be lost - the output from invert #2 has:
Visibilities accepted: 21205981
Theoretical rms noise: 2.804E-05

While the original invert has:
Visibilities accepted: 42911387
Theoretical rms noise: 2.047E-05

This was a bug I found a few years ago in reducing the 5.5 GHz data for this project but I forgot to post about it. I got around it by inverting and selfcaling the 2010 and 2012 data separately, and then inverting the 2010 and 2012 data together after the uv data had been selfcal'ed.

I will use that solution again, unless there's a fix to selfcal that you can do. Using the full uv dataset, with higher SNR model components, would be more ideal for selfcal I'd imagine.

cheers,
Minh
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Selfcal of large datasets

Post by Mark.Wieringa »

Hi Minh,

selfcal can flag data if the resolution of the model is too low to calculate the model on the longest baselines. Maybe combining all the data results in a lower resolution image than imaging either the 2010 or 2012 data? If this is what is happening you can fix it by explicitly setting the cell size to match the highest resolution data when imaging the combined data.
There is also the zero option in selfcal - it avoids flagging but does it by setting the long baseline model to zero if it cannot be calculated. It can be useful in intermediate steps of calibration with a low res model (e.g., peeling distant sources), but would still need a final selfcal at full resolution.

Let me know if this solves your issue.

Cheers,

Mark
minh25
Posts: 11
Joined: Fri Aug 17, 2012 3:08 am

Re: Selfcal of large datasets

Post by minh25 »

Thanks for the quick repsonse Mark.

The synthesized beam is ~2.5 x 1.0 and ~3.4 x 1.1 arcsec, for the 2010 (6A) and 2012 data (6A + 6D), respectively. So there is slightly lower resolution in 2012. The resolution of the combined dataset is a little better than the 2012 data alone.

However I don't think it's a resolution problem. When I go through the imaging and selfcal steps with the 2010 and 2012 data separately no data is "loss" to invert after selfcal (it uses the same number of visibiities before and after selfcal). If it was a resolution issue shouldn't I see this problem in the 2012 reduction? It's only when all the data is combined.

cheers,
Minh
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Selfcal of large datasets

Post by Mark.Wieringa »

Hi Minh,

The configs and resolutions are pretty similar so it seems unlikely this is the problem unless you are very close to Nyqvist sampling (i.e., cell size is too big)
Maybe you could try a test with one day from each year to see if any points get flagged, if so I could try and fetch those days from the archive and see if I can figure out what is happening.
Alternatively you could put some of the reduced data somewhere I can access.

Cheers,

Mark
minh25
Posts: 11
Joined: Fri Aug 17, 2012 3:08 am

Re: Selfcal of large datasets

Post by minh25 »

Hi Mark,

I tried it with one day from 2012 and one from from 2010, as you suggested, and yes the issue arises. Below is the output from
invert #1-> mfclean -> selfcal -> invert #2 -> mfclean #2 -> restor

The number of visibilities accepted in invert #2 are 2326708 vs 3760469 in invert #1.
I checked and this happens whether I use clip in selfcal or not.

I've placed the uvaver'd uv datatset here:
https://dl.dropboxusercontent.com/u/774 ... 1_9000.zip
(Of course this is the uv data before the selfcal)

Note that particular pointing has no bright sources, but the problem occurs in pointings with bright sources also. This pointing has a bright source.
https://dl.dropboxusercontent.com/u/774 ... 0_9000.zip

cheers,
Minh

uvaver vis=../../epoch3/2012-08-18/ecdf_1.9000,../../../C2028_re-reduce/14_jan_2010/ecdf_1.9000 stokes=i out=ecdfs_1.9000.uv
UvAver: version 1.0 17-Nov-08
Applying bandpass corrections to ../../epoch3/2012-08-18/ecdf_1.9000
Applying freq. dependent gain corrections to ../../epoch3/2012-08-18/ecdf_1.9000
Applying freq. dependent leakage correction to ../../epoch3/2012-08-18/ecdf_1.90
Applying bandpass corrections to ../../../C2028_re-reduce/14_jan_2010/ecdf_1.900
Applying freq. dependent gain corrections to ../../../C2028_re-reduce/14_jan_201
Applying freq. dependent leakage correction to ../../../C2028_re-reduce/14_jan_2

invert vis=ecdfs_1.9000.uv map=ecdf_comb_1.imap beam=ecdf_comb_1.ibeam imsize=3000 cell=0.25 robust=0.5 stokes=i options=mfs,systemp,double,sdb,mosaic offset=3:32:20,-27:47:53

invert: Revision 1.21, 2015/10/29 01:31:58 UTC

Reading the visibility data ...
Making MFS images
Visibilities accepted: 3760469
### Warning: Visibilities rejected: 665371
Number of pointings: 1
Mean Frequency(GHz): 8.99
Doing the geometry calculations ...
Sidelobe suppression area is 2048x2048 arcsec
... this corresponds to uniform weighting
Calculating the weights ...
Applying weights and geometry corrections ...
Theoretical rms noise: 6.686E-05
Forming the beam ...
Finished gridding 33% ...
Finished gridding 67% ...
Forming Stokes I image ...
Mosaicing the image ...
Completed 100% !
mfclean map=ecdf_comb_1.imap beam=ecdf_comb_1.ibeam out=ecdf_comb_1.icmp niters=100 region=relcen,box(-1400,-1400,1400,1400)

mfclean: Revision 1.11, 2013/09/11 00:14:07 UTC

Sub-beam size is 197 by 197
### Warning: Sub-beam size is dangerously small
FFTing the beams ...
Calculating the map*beam ...
Calculating patches ...
Starting to iterate ...
Clark Iterations: 3
Residual min,max,rms: -3.080E-04 3.169E-04 6.265E-05
Clark Iterations: 12
Residual min,max,rms: -3.050E-04 2.992E-04 6.253E-05
Clark Iterations: 45
Residual min,max,rms: -2.879E-04 2.840E-04 6.219E-05
Clark Iterations: 100
Residual min,max,rms: -2.744E-04 2.756E-04 6.169E-05
Stopping -- Maximum iterations performed
selfcal vis=ecdfs_1.9000.uv model=ecdf_comb_1.icmp clip=1e-4 interval=5 options=mfs,phase,relax refant=3

selfcal: Revision 1.18, 2015/03/24 04:30:14 UTC

Calculating the model for ecdf_comb_1.icmp
Clipping model when: pixval < 1.00E-04
Accumulating statistics ...
Finding the selfcal solutions ...
Total number of correlations being used: 4425840
Total number of solution intervals: 29
Solution intervals merged together: 5
Rms of the gain phases (degrees): 44.9
Rms deviation of gain from 1: 7.992E-08
Ratio of Actual to Theoretical noise: 0.923
invert vis=ecdfs_1.9000.uv map=ecdf_comb_1.imap2 beam=ecdf_comb_1.ibeam2 imsize=3000 cell=0.25 robust=0.5 stokes=i options=mfs,systemp,double,sdb,mosaic offset=3:32:20,-27:47:53

invert: Revision 1.21, 2015/10/29 01:31:58 UTC

Reading the visibility data ...
Applying gain corrections to ecdfs_1.9000.uv
Making MFS images
Visibilities accepted: 2099132
### Warning: Visibilities rejected: 2326708
Number of pointings: 1
Mean Frequency(GHz): 8.98
Doing the geometry calculations ...
Sidelobe suppression area is 2048x2048 arcsec
... this corresponds to uniform weighting
Calculating the weights ...
Applying weights and geometry corrections ...
Theoretical rms noise: 8.452E-05
Forming the beam ...
Finished gridding 33% ...
Finished gridding 67% ...
Forming Stokes I image ...
Mosaicing the image ...
Completed 100% !
mfclean map=ecdf_comb_1.imap2 beam=ecdf_comb_1.ibeam2 out=ecdf_comb_1.icmp2 niters=100 region=relcen,box(-1400,-1400,1400,1400)

mfclean: Revision 1.11, 2013/09/11 00:14:07 UTC

Sub-beam size is 197 by 197
### Warning: Sub-beam size is dangerously small
FFTing the beams ...
Calculating the map*beam ...
Calculating patches ...
Starting to iterate ...
Clark Iterations: 3
Residual min,max,rms: -4.084E-04 4.002E-04 7.784E-05
Clark Iterations: 14
Residual min,max,rms: -3.683E-04 3.607E-04 7.759E-05
Clark Iterations: 52
Residual min,max,rms: -3.485E-04 3.424E-04 7.682E-05
Clark Iterations: 100
Residual min,max,rms: -3.335E-04 3.287E-04 7.601E-05
Stopping -- Maximum iterations performed
restor model=ecdf_comb_1.icmp2 map=ecdf_comb_1.imap2 beam=ecdf_comb_1.ibeam2 out=ecdf_comb_1.icln2

restor: Revision 1.13, 2015/10/29 01:31:49 UTC

Using Gaussian beam fwhm of 3.271 by 1.056 arcsec.
Position angle: -7.3 degrees.
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Selfcal of large datasets

Post by Mark.Wieringa »

Hi Minh,

An update on this issue: I have found the problem in the code. For concatenated datasets like yours, with a timespan of more than a year, there is a issue with precision of the solution times. Selfcal stores these as single precision offsets from the start time. This works fine for a small time range, but when the time range gets larger than a year, the offsets become inaccurate and no longer line up with the data. The data are then flagged because they have no matching selfcal solution. I'm still testing the solution (things got interrupted by a bout of observing/data reduction and now a 4 day weekend) so I'll let you know when it's ready to try.

Cheers,

Mark
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Selfcal of large datasets

Post by Mark.Wieringa »

Hi Minh,

During my tests I found another problem in the code (another single precision variable used to accumulate time offsets) and after fixing that it runs without flagging any data.
The new version should be available for mirsync tomorrow.
As you mentioned the S/N for this field is not really enough for selfcal - the phase rms reported by selfcal (45 degrees) is a bit too large: for anything above 40 it is dubious if you are actually improving things, ideally you'd like to get this below 30 degrees.

Cheers,

Mark

% ./reduce.csh

invert: Revision 1.21, 2015/10/29 01:31:58 UTC

Reading the visibility data ...
Making MFS images
Visibilities accepted: 3760469
### Warning: Visibilities rejected: 665371
Number of pointings: 1
Mean Frequency(GHz): 8.99
Doing the geometry calculations ...
Sidelobe suppression area is 2048x2048 arcsec
... this corresponds to uniform weighting
Calculating the weights ...
Applying weights and geometry corrections ...
Theoretical rms noise: 6.686E-05
Forming the beam ...
Finished gridding 33% ...
Finished gridding 67% ...
Forming Stokes I image ...
Mosaicing the image ...
Completed 100% !

mfclean: Revision 1.11, 2013/09/11 00:14:07 UTC

Sub-beam size is 197 by 197
### Warning: Sub-beam size is dangerously small
FFTing the beams ...
Calculating the map*beam ...
Calculating patches ...
Starting to iterate ...
Clark Iterations: 3
Residual min,max,rms: -3.080E-04 3.169E-04 6.265E-05
Clark Iterations: 12
Residual min,max,rms: -3.050E-04 2.992E-04 6.253E-05
Clark Iterations: 45
Residual min,max,rms: -2.879E-04 2.840E-04 6.219E-05
Clark Iterations: 100
Residual min,max,rms: -2.744E-04 2.756E-04 6.169E-05
Stopping -- Maximum iterations performed

selfcal: Revision 1.19, 2016/01/28 22:07:15 UTC

Calculating the model for ecdf_comb_1.icmp
Clipping model when: pixval < 1.00E-04
Accumulating statistics ...
Finding the selfcal solutions ...
Total number of correlations being used: 4425840
Total number of solution intervals: 29
Solution intervals merged together: 5
Rms of the gain phases (degrees): 44.9
Rms deviation of gain from 1: 7.992E-08
Ratio of Actual to Theoretical noise: 0.923

invert: Revision 1.21, 2015/10/29 01:31:58 UTC

Reading the visibility data ...
Applying gain corrections to ecdfs_1.9000.uv
Making MFS images
Visibilities accepted: 3760469
### Warning: Visibilities rejected: 665371
Number of pointings: 1
Mean Frequency(GHz): 8.99
Doing the geometry calculations ...
Sidelobe suppression area is 2048x2048 arcsec
... this corresponds to uniform weighting
Calculating the weights ...
Applying weights and geometry corrections ...
Theoretical rms noise: 6.686E-05
Forming the beam ...
Finished gridding 33% ...
Finished gridding 67% ...
Forming Stokes I image ...
Mosaicing the image ...
Completed 100% !

mfclean: Revision 1.11, 2013/09/11 00:14:07 UTC

Sub-beam size is 197 by 197
### Warning: Sub-beam size is dangerously small
FFTing the beams ...
Calculating the map*beam ...
Calculating patches ...
Starting to iterate ...
Clark Iterations: 3
Residual min,max,rms: -3.351E-04 2.998E-04 6.275E-05
Clark Iterations: 13
Residual min,max,rms: -3.098E-04 2.926E-04 6.261E-05
Clark Iterations: 47
Residual min,max,rms: -2.854E-04 2.918E-04 6.220E-05
Clark Iterations: 100
Residual min,max,rms: -2.743E-04 2.703E-04 6.168E-05
Stopping -- Maximum iterations performed

restor: Revision 1.13, 2015/10/29 01:31:49 UTC

Using Gaussian beam fwhm of 3.004 by 1.038 arcsec.
Position angle: -4.3 degrees.
Post Reply