Oscillations in 1934 spectrum after bandpass cal

[and460]

Hi guys,

So I'm trying to get a good bandpass cal for some 16cm CABB data, but when I use UVSPEC to look at the spectrum of 1934 afterwards, I'm seeing oscillations in Stokes I at the lower end of the frequency band, typically on all baselines except for those including ant 6. This doesn't seem right to me, but I have been pretty fastidious about iteratively deriving bandpass solutions and flagging, so I was hoping you might be able to tell me if something is wrong, or if this is as good as it gets.

My calibration does the following:

#initial MFCAL
mfcal vis=1934-638.2100 refant=4 interval=5

#initial flagging
pgflag vis=1934-638.2100 stokes=xx,yy device=/xs command=<b
pgflag vis=1934-638.2100 stokes=yy,xx device=/xs command=<b

#BLFLAGing / manual flagging in here

#MFCAL
mfcal vis=1934-638.2100 refant=4 interval=5

#another round of flagging
pgflag vis=1934-638.2100 stokes=xx,yy device=/xs command=<b
pgflag vis=1934-638.2100 stokes=yy,xx device=/xs command=<b

#BLFLAGing / manual flagging in here

#final MFCAL
mfcal vis=1934-638.2100 refant=4 interval=5

#GPCAL
gpcal vis= 1934-638.2100 refant=4 interval=20 nfbin=16 options=xyvary


I then use UVSPEC to take a look at the results:

Task: uvspec
vis = ./1934-638.2100
select =
line =
stokes = i
interval = 10
hann =
offset =
options =
axis = freq,amp
yrange =
device = /cps
nxy =
log =

With options=nobase,avall set, the spectrum looks really smooth as you would expect. However, with options unset so I can look at individual baselines, I typically get plots that look like the following:



I've tried many combinations of MFCAL settings, GPCAL settings, light flagging, heavy flagging, iteratively flagging then solving for bandpass and back again etc. Each time, I get the same result more or less. So my questions are:

1) Is there anything wrong with what I'm seeing here in UVSPEC?
2) If so, is it likely to be RFI, a lack of unflagged data to derive solutions from, or some other problem with how I've run things?

And hints, tips or help would be much appreciated!

Cheers,

Craig.

[and460]

Oops - just read my post again and realised that I had mis-typed the GPCAL interval input - I am actually using a GPCAL interval of 0.1.

[shane]

Hi Craig,

Here's some recent 1934 data which I would consider very close to as good as I could do.
I see similar effects, with baselines 1-2 and 3-4 the most concerning (6A array config).

Shane

[Mark.Wieringa]

Hi Craig,

>1) Is there anything wrong with what I'm seeing here in UVSPEC?
>2) If so, is it likely to be RFI, a lack of unflagged data to derive solutions from, or some other problem with how I've run things?

The area of the spectrum the wiggles appear in is usually heavily affected by RFI, so solutions may be affected - some signs of this are the small spikes on the spectrum. However since the baselines solution is done channel by channel, it is not clear how you would produce smooth wiggles like this.
Another reason for the wiggles could be the large field of view at the low frequencies - fainter sources at the 100 mJy level might be visible in the field, and unless you include these in the calibrator model or subtract them out, they will affect the solutions and residuals. The wiggles seem faster on the longer baselines, which is what you would expect for confusing sources in the field. You should be able to see the wiggles in the time direction as well, if you plot amp-time for a narrow channel window and you have enough time on 1934.

Cheers,

Mark

[and460]

Thanks very much for your reply Mark.

A couple of people I have talked to since posting this have raised the possibility of additional sources in the field being the cause. For example, Emil Lenc said that he did some high dynamic range imaging of the 1934 region a few years back and found some extended sources nearby that might contribute enough flux to cause problems like this.

Anyway, as far as I understand, Miriad does not have the capability to specify a source model for 1934 in order to derive the bandpass. If this is in fact the cause then, I suppose it is a case of just accepting that the bandpass is only good to the sort of level of accuracy that I'm seeing in my plots? Someone has told me that AIPS has the capability to derive bandpass solutions based on a user-inputted source model. On the whole though, how good or otherwise is AIPS for reducing ATCA data? Not being remotely familiar with it, I would think there would be a lot of potential pitfalls and 'gotchas'...?

Cheers,

Craig.

[Mark.Wieringa]

Hi Craig,

Yes, I remember seeing some faint sources in a old 1934 image too. Tell Emil he's still welcome to answer queries on the forum!

No, mfcal just takes a point source model or one of the built in models, this is limiting at low frequencies. MWA is running into this too.
AIPS maybe able to do this, but I think the recommended path beyond Miriad is CASA now. Either way, you may have trouble doing the polarization calibration in these packages.
I have toyed with the idea of allowing more complicated parameterised models in mfcal/gpcal/selfcal, but haven't got around to it yet.


One thing you could try if you have a clean component or parameter model for the confusing sources is to do a first pass mfcal, then subtract off the confusing sources with uvmodel or uvsfit and then run mfcal again. That should at least reduce the wiggles, possibly to below the level where you'd worry about them.

Cheers,

Mark

[len067]

Hi Mark,

Sorry ... I was in China when I saw Craig's email and didn't have my 1934-638 data set with me so that I could formulate a better-informed response A really simple and quick check would be to generate an ideal simulated CABB data set based on the model that I created for the field (which I believe I created with u-v components). If the wiggle appears in the resulting visibility data set then this would definitely be the culprit. I just need to try and find that data set again (it was on one of the backup hard disks I put together before I left CSIRO).

Cheers,

Emil.

[len067]

Hi All,

Just a quick update, I ran some simulations with my difmap models today and I do get similar ripples to what Shane was seeing and on the same baselines. I don't quite have the simulation right yet (it involves a bit of data hacking and so the side-by-side comparison is a bit difficult) but I'll try and post an update in a day or two when I've got time and I can get things running a bit more sensibly. The main result from this though is that it appears that the ripples can be explained by field sources alone i.e. no other weird calibration effects need to be introduced.

Cheers,

Emil.

[len067]

Hi All,

Shane kindly provided me with the 1934-638 CABB data set that he used to make the plots above. What I did was keep everything in the file the same but replaced the visibility data with the "ideal" uv-model data from my deep difmap data set. Since 1934-638 can't be represented over such a wide band with a simple spectral index (which is what my pre-CABB difmap data set used) I replaced this model component with a polynomial fit. The other change I made is that I made all of the other sources in the field have a spectral index of -1.0 - the difmap model had very steep components because it was seeing more of the "primary beam" effect rather than the actual spectral index of the source. If I had used the original spectral index then the field source would have dramatically over-estimated fluxes at the lower end of the band.

Anyway, here are the uvspec plots for the first four baselines. First for the original data that Shane provided:

Real data SED 1

Real1.png (78.75 KiB) Viewed 37483 times

and then for the simulated data set (this has no noise and is perfectly calibrated):

Simulated data SED 1

Sim1.png (77.26 KiB) Viewed 37483 times

Note that the large'ish ripple is seen on baselines 1-2 and 2-3 and approximately of the same magnitude. The effect seems to be worse on the shorter baselines. This makes sense as these baselines will see the majority of the extended emission from the field sources. In the longest baselines there is virtually no wobble in the bandpass.

There are, of course, differences between the real and simulated plots. This is for a number of reasons e.g. the assumption I've made that all sources have a spectral index of -1, RFI in the real data throwing off the calibration, wobble in the real SED throwing off the bandpass calibration, etc, etc. Nonetheless, I think as a proof of concept, it seems that field sources are enough to introduce the observed level of wobble.

Cheers,

Emil.

[len067]

Hi All,

Here is the second set of plots. Real data first:

Real data 2

Real2.png (78.76 KiB) Viewed 37483 times

Simulated data next:

Simulated data 2

Sim2.png (77.22 KiB) Viewed 37483 times

Check out the ripple on baseline 3-4 ... not a bad match!

Cheers,

Emil.