Something wrong with "pgflag"

[taianmoon]

Hi,

I am trying to reduce the raw data of one of our sources at 9GHZ (See https://atcaforum.atnf.csiro.au/viewtop ... f=21&t=270). Following instructions from the ATCA data analysis webpage, I used "pgflag" to do the flagging on this source. However, it seems to take forever in this step (at least ~5 hrs before I gave up) in any of the pgflag of stokes V, Q or U (such as: $ pgflag stokes=i,q,u,v flagpar=8,5,5,3,6,3 "command=<b"), and miriad seemed to pause with the following:

pgflag: Revision 1.29, 2014/10/24 04:16:34 UTC

Applying bandpass corrections to g345.81.9000/
Applying freq. dependent gain corrections to g345.81.9000/
Applying freq. dependent leakage correction to g345.81.9000/


I think something might be wrong with the calibrators or data, but I don't know where to start investigating this problem....

Thanks!

Tai-An

[Mark.Wieringa]

Hi Tai-An,

Yes, that does seems like something is wrong. It should take a similar amount of time as the 5 GHz data.
The first thing to try is to run uvaver on the data to try and apply the calibration. If that takes forever as well, there must be a problem with the calibration files. Try redoing the calibration - making plots along the way to check things look ok.

Another test is to make a copy of the data and then remove all the calibration files (using delhd vis=g345.81.9000/gains and gainsf,bandpass,leakage,leakagef) and try again. If that still fails to run there is something wrong with the data. In that case redo your reduction from atlod onwards and try again.

When I looked at your data last year I was able to make a 9GHz image (and flag the data using pgflag) so it is more likely to be a problem with your data than pgflag.
Let me know if none of this helps, so we can have a closer look.

Cheers,

Mark

[taianmoon]

Hi Mark,

Thanks for the suggestions.

When I tried uvaver on the source data, only a "header" and a "history" files were present in the output, but no any calibration file or visdata...

What are the suggestive ways to make plots and check the calibration? Thinking of blflag and uvplt but don't know the details and how to work on calibration files...

I'm also guessing that I might have misunderstood the steps of the calibration/flagging processes, or misused the calibrators. Here's a quick flowchart of my processes after uvsplit:

pgflag (stoke V) on 0010-401.9000
mfcal on 0010-401.9000
pgflag (stoke V) on 0010-401.9000
pgflag (stoke Q) on 0010-401.9000
pgflag (stoke U) on 0010-401.9000
gpcal on 0010-401.9000
gpcopy from 0010-401.9000 to 0023-263.9000
gpcal on 0023-263.9000
gpboot with vis=0023-263.9000 and cal=0010-401.9000
gpcopy from 0023-263.9000 to g345.81.9000(the source)
gpaver on g345.81.9000
pgflag (stoke V) on g345.81.9000 (and here takes forever...)

I'm not sure if 0010-401.9000 or 0023-263.9000 are the correct calibrators to use, because supposedly the previous one should be the bandpass calibrator and the later one should be the gain/flux calibrator? Is there a way to check this (like a listobs file)?

Thanks a lot!

Tai-An

[Mark.Wieringa]

Hi Tai-An,

Ok, sounds like uvaver failed as well on your data. I think you'll have to go back a few steps in your reduction and find where things stop working.

You can check the calibration with task gpplt to plot the various tables (gain and bandpass mainly). Also use uvplt to plot the calibrated data on the calibrators to see if they look as they should (see users guide: http://www.atnf.csiro.au/computing/software/miriad/userguide/node87.html ), this also shows the flow chart for the reduction process.

For the 2016-06-10 data I used 1934-638 as the bandpass calibrator, a complication is that it is called 'setup', use the following to fix that straight after you run split:
uvputhd vis=setup.5500 hdvar=source varval=1934-638 out=1934-638.5500 (same for 9000 files)
The source 0010-401 is the phase calibrator (I didn't find 0023-263). The gpboot step should use cal=1934-638 and vis=0010-401.

You can use uvindex on the output of atlod to get a listing of the scans, sources and frequencies.

Cheers,

Mark

[taianmoon]

Hi Mark,

It seems that I used the wrong calibrators. I'm trying on G018 now, another source which had fewer problems in the 5GHz data, so presumably I should use 0023-263 as the phase calibrator and 1934-638 as still the bandpass calibrator?

I did flagging and calibration again following the link webpage but the dirty image looks like this, which I think might be problematic:

g018_imap.png (127.72 KiB) Viewed 4829 times

Checking the calibration table and calibrated dataset:

1. uvplt of 1934-638 (axis=real,imag; options=equal,nobase)

1934-638_uvplt.png (11.25 KiB) Viewed 4829 times

2. varplt of 0023-263 (yaxis=chi): Does this look like a good parallactic angle coverage, as suggested in the webpage?

0023-263_parallactic_angle_coverage.png (7.28 KiB) Viewed 4829 times

3. uvflux of 0023-263: to check if the secondary data fits a point souce:
--------------------------------------------------------------------------------
Pol----Theoretic-----Vector Average----------RMS--------Average----RMS Amp---Number
-----------RMS------------(real,imag)---------Scatter---------Amp-------Scatter----Corrs
------ --- -------- -------------------- ------- --------- -------- ------
I.......8.0E-02....2.939E+00..-4.305E-03...3.0E-01...2.967E+00....1.26E-01..12488424
Q......8.0E-02....-1.432E-03...2.059E-04...8.7E-02....1.035E-01....6.63E-02..12488424
U......8.0E-02.....2.697E-03...3.254E-04...9.3E-02....1.066E-01....7.67E-02..12488424
V......8.0E-02....-7.737E-06...1.815E-04...8.3E-02....1.001E-01....6.06E-02..12488424
--------------------------------------------------------------------------------

4. uvplt of 0023-263 (axis=real,imag; options=equal,nobase):


5. gpplt of 0023-263 (yaxis=phase; options=xygains):


6. gpnorm (vis=0023-263; cal=1934-638): got an error message...
GpNorm: version 1.0 28-Sep-09
### Informational: Bandpass tables present
### Fatal Error: GPNORM does not work correctly in this case

I am not sure if the calibration process went wrong from the above plots and the error in gpnorm might reflect some problem in the data? Problem might also come from flagging for which I only used pgflag (this time it worked and didn't take forever ) in between mfcal and gpcal of both calibrators and after gpcopy and gpaver of the source...

Thanks again!

Tai-An

[Mark.Wieringa]

Hi Tai-An,

Yes I think 1934 should be the bandpass calibrator. I haven't looked at the G018 data, but if 0023-263 is the calibrator observed interleaved with G018 then it is probably the intended phase calibrator.

The configuration has very short baselines, they might be responsible for the large positive and negative 'lumps' in the image. Try making the image without the shortest baseline (or do manual flagging on that baseline to make sure there is no RFI left).

1. the plot shows a large arc in stokes I, this indicates phase errors are still present. You should probably shorten the solution interval and improve the flagging.

2. parallactic angle coverage looks fine

4. the arc in stokes I means you need to shorten the solution interval, the 'messy' Q and U probably come from interference

6. gpnorm is not used much these days, I think you can skip this step

So, yes, I think the calibration is not too bad, but there are still some signs of interference. If you can't get rid of it with pgflag easily, you could try blflag axis=chan,amp or axis=time,amp with options=nofqav and yrange=0.2

Cheers,

Mark

[taianmoon]

Hi Mark,

The dirty map looks better after I excluded the shortest baseline, supposedly ant(1)(2).

However, uvplt of both 1934-638 and 0023-263 still show an "arc", similar to those I showed, after I did more flagging and shorten interval=3 (was 5). I am wondering what might be the appropriate interval to use? Besides, I used blflag to do more flagging but I am not confident if the regions I flagged are correct. For example, this is 1934-638 with axis=chan,amp for a baseline and the green polygon is the flagged area (other baselines look similar. Looks not much rfi...?):

1934-638_blflag.png (44.13 KiB) Viewed 4817 times

And 0023-263 (looks like more rfi...?):

0023-263_blflag.png (30.81 KiB) Viewed 4817 times

I also looked at axis=time,amp , but didn't flag much, and they look like:

1934-638:

1934-638_blflag_2.png (7.45 KiB) Viewed 4817 times

0023-263:



One more thing about flagging is that do we do all the flaggings before calibration, or after calibration, or do both? And should I do similar flaggings on the source visibilities before deconvolution, or only flaggings on calibrators are needed?

Thanks again!

Tai-An

[Mark.Wieringa]

Hi Tai-An,

You can make the interval as short as needed to get the phases stable on the calibrators - a uvplt of phase vs time on the calibrators should show very little phase variation - with interval=0.1 the arc should turn into a circle (or ellipse).

In the 1934 plot I can't see much interference, but the band of points is too wide: the bottom white band is how it should look, the points from 0.85 up (on left axis) indicate something is wrong.
I'm guessing this plot is after calibration and the calibration has pushed some points up (from the time/amp plot it seems the be the start of the first scan). Check the phases for that part, it may be decorrelation due to the long solution interval.

I do flagging on 1934 before calibration, then calibrate and apply the bandpass & leakage to the secondary and flag it before calibrating. Then apply all calibration to the source and flag (look at Stokes V to easily spot RFI).

Cheers,

Mark

[taianmoon]

Hi Mark,

I've gone through all the flagging & calibrations and the imaging clean process, and the cleaned map looks like below. Although the map looks not too noisy, only few number of sources are detected (using sfind task, 11 sources are detected), compared to previous 5 GHz map in which 61 sources were detected. I wonder if I flagged too much uv data thus fainter sources vanish...or other reasons cause this...?

g018_9000_170209.png (1.67 MiB) Viewed 4796 times

Some images during the calibration/flagging processes:

1. 1934-638 in blflag before calibration, looks fine so didn't flag any:

1934_638_blflag.png (16.21 KiB) Viewed 4796 times

2. 1934-638 in uvplt after flagging & calibration. Only ~10 points deviate so I guess it's fine:

1934_638_uvplt.png (7.72 KiB) Viewed 4796 times

3. 0023-263 in uvplt after flagging & calibration. Should look fine:


4. Target source in blflag (stokes V) before calibration applied. I flagged points with amp>~0.27 for this case:


All the calibrations were done with interval=0.1.

Thanks again!

Tai-An

[Mark.Wieringa]

Hi Tai-An,

If the image is for 9GHz, you would not expect as many sources, since the field of view is smaller by about a factor 2.7 and most sources have a negative spectral index.
So probably not too much to worry about. The flagging plots look fine now.
The secondary calibrator real/imag plot still looks a bit strange, it should normally have a disk of colour around the origin with the pol q,u and v, and a second disk further along the x axis at the flux of the secondary, yours has some strange structures around 3 Jy. The 1934 plot has more 'wild' points than it should as well. I've added some examples:

1934 calibrated & flagged

1934-638.9000.iquvplot.png (7.86 KiB) Viewed 4796 times

secondary calibrated & flagged

0010-401.9000.iquvplot.png (18.69 KiB) Viewed 4796 times

Cheers,

Mark