Linmos has trouble mosaicking many pointings together

Is MIRIAD being a pain? Let us know your experience.

Moderator: Mark.Wieringa

abutler
Posts: 11
Joined: Fri Feb 06, 2015 1:28 pm

Linmos has trouble mosaicking many pointings together

Post by abutler »

Hi all,

I am trying to mosaic together a large number of pointings using linmos. I've imaged my field in 7 sub-bands (at central frequencies between 1460-2996 MHz), with each sub-band having 471 individual pointings (for a total of 3297 images). Each image's file size is about 200 MB.

Linmos clearly doesn't like such a large number of files, dealing with only the first 1133 images before it quit. It gave this error message:
### Warning: Output image is too large, clipping input
### Fatal Error: Exhausted all primary beam objects

I take it this is a memory issue? If so, here's one possible solution: can you set up linmos so that there is an option to turn off the primary beam correction? That way, I can mosaic each sub-band with the primary beam correction on, and then use linmos to combine those 7 mosaics with the primary beam correction off. If that's not possible, then I guess I'll have to make my own code that can combine the 7 sub-band mosaics. Or do you have any other ideas?

Thanks,
Andrew
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Linmos has trouble mosaicking many pointings together

Post by Mark.Wieringa »

Hi Andrew,

I don't think it is a memory issue - there is a built in limit of 20000 primary beam objects. This can be increased, but we'd need some indication of what would be enough. From the failure point we might estimate that 3 times that would do the trick. The limit on the image size is harder to avoid - you might need to make overlapping images of 32768*32768 pixels.
What machine/OS are you running this on and what are the inputs you use for linmos?

I think you can turn off the primary beam correction by changing the header keyword pbtype to 'SINGLE'

Cheers,

Mark
abutler
Posts: 11
Joined: Fri Feb 06, 2015 1:28 pm

Re: Linmos has trouble mosaicking many pointings together

Post by abutler »

Hi Mark,

That's interesting that the maximum number of primary beam objects linmos can handle is 20000, and yet it can't handle the 3297 objects I give it.
The limit on the image size is harder to avoid - you might need to make overlapping images of 32768*32768 pixels.
Are you implying here that linmos has a limit to the size of the mosaics it can produce, or is it limited by the total size of the input images that I feed it?

The machine I'm running linmos on is one of the compute nodes on the "pleiades" cluster at ICRAR / UWA. More details on it can be found here: https://confluence.icrar.org/display/PL ... PC+Cluster

I run linmos from the command line, and the inputs for it look like this (fields.txt contains the list of images I want it to mosaic):
linmos in=@fields.txt bw=0.256 out=mosaic_sb.mir > log_linmos.txt

I'll try changing the keyword pbtype to 'SINGLE' and see what happens...

Cheers,
Andrew
abutler
Posts: 11
Joined: Fri Feb 06, 2015 1:28 pm

Re: Linmos has trouble mosaicking many pointings together

Post by abutler »

Hi Mark,

I've realised that linmos actually produces 10 primary beam objects per image (corresponding to 10 frequency intervals across the bandwidth). So, since I feed it 3297 images, the number of primary beam objects it tries to process is 32970, exceeding its maximum.

Would it be possible to increase linmos' limit to 40000 or more objects? This would make it very easy for me to do what I want to do and I would really appreciate it!

Cheers,
Andrew
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Linmos has trouble mosaicking many pointings together

Post by Mark.Wieringa »

Hi Andrew,

sorry for not responding quicker, somehow I got unsubscribed from the Miriad forum after my last message so I never saw your replies.
I've just updated the number of beam objects to 80000, if you do a mirsync tomorrow (or download the new tar files on Friday) you should get a linmos version that can do what you want.

To answer your earlier question - yes the maximum image size (mosaic or otherwise) Miriad can produce is 32768^2. Changing this is not as simple as increasing the maxdim value, various memory allocations will fail because the 32 bit integers used will overflow. If larger images are necessary let me know and I'll put it on the todo list.

Cheers,

Mark
minh25
Posts: 11
Joined: Fri Aug 17, 2012 3:08 am

Re: Linmos has trouble mosaicking many pointings together

Post by minh25 »

Hi Mark, Andrew, et al.

I can't get linmos to run without a pb correction by setting pbtype to SINGLE. That gets set after one run of linmos anyway, but we found second run of linmos will run a second pb corr on top of that.

I tried unsetting pbtype, and then setting telescop to FOO (i.e. something unknown), but a pbcorr is applied anyway. I don't know how linmos chooses the telescop in that case, probably defaults to ATCA.

So there's no easy way to switch off linmos pb correction? I need this to make a simulated/approximate bandwidth smearing image of our full mosaic, given a bandwidth smearing correction for a single pointing. I could use imcomb if it does exactly what linmos does but just no pb correction? Is that what imcomb does Mark, Jamie et al. ? Thanks.

Minh
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Linmos has trouble mosaicking many pointings together

Post by Mark.Wieringa »

Hi Minh,

You're right, setting pbtype to SINGLE (puthd in=imagename/pbtype value=SINGLE) is not always sufficient. If the image has a mosaic table attached, linmos will read the beam from there before checking the pbtype variable. To make sure it sees the pbtype='SINGLE' you've set, you'll need to remove the mosaic table too. You can use delh in=imagename/mostable to do that.

Imcomb is a bit simpler than linmos, it just requires all images to be on the same coordinate grid and combines them based on rms values without any primary beam calculations. It looks like it will fail if the output image would be too large (linmos clips the input to the largest possible size).

Cheers,

Mark
abutler
Posts: 11
Joined: Fri Feb 06, 2015 1:28 pm

Re: Linmos has trouble mosaicking many pointings together

Post by abutler »

Hi Mark,

Thanks for the Miriad update. I've been able to simultaneously feed linmos all 3297 images (which all have the same size) from all 7 sub-bands and successfully make a mosaic (each pointing was stacked using the images from the 7 sub-bands). However, when I plot the ratio of the integrated flux density to peak flux density vs. SNR (see third attached plot), I get a trend away from 1.0 for low SNR, but the data points should be centered around 1.0 and flat for all SNR.

I don’t see this in the same plot made from using the whole 2.0 GHz bandwidth (the data points are centered on 1.0 for all SNR). See the second attached plot - this is generally how we expect this plot to look.

I see the same trend in the mosaics for 1460, 1716, and 1972 MHz (see the first attached plot for 1460). It occurs whether or not the mfs plane exists in the images (i.e. it doesn’t matter what primary beam correction linmos uses). The higher frequency (2228, 2484, 2740, 2996 MHz) mosaics don’t have this feature in the plots.

I’m wondering if you can help me understand what is going on here. Below are the steps I take to image each pointing. The last step is the linmos step.

1. Initial clean:
(a) invert the uv data
(b) mfclean the dirty map to get initial clean model obtained from stopping after 1000 iterations
(c) restor the initial clean model with the clean beam
(d) Use sigest to calculate the rms noise in the restored image from (c)

2. Clean down to 10*rms (twice)
(a) invert the uv data
(b) mfclean to get clean model that corresponds to clean cutoff = 10*rms
(c) restor this clean model with the clean beam
(d) Use sigest to calculate the rms noise in the restored image from (c)
(e) selfcal the uv data using the model from (b) with clip = 3*rms and interval=0.33

3. Clean down to 6*rms
(a) invert the uv data
(b) mfclean to get clean model that corresponds to clean cutoff = 6*rms
(c) restor this clean model with the clean beam
(d) Use sigest to calculate the rms noise in the restored image from (c)

4. Trim restored image (from step 3) to the mfclean region using imsub (taking out the mfs plane as well)

5. Convolve restored and trimmed image (from step 4) with Gaussian beam (fwhm=6.50,5.70 pa=3.09) using convol

6. Mosaic all pointings together (after they've completed steps 1-5) using linmos (with bw=0.256). As we are taking out the mfs plane, we are currently doing an average primary beam correction across the band (but for 1460 MHz, the fractional bandwidth is just ~17.5%).

Am I doing something incorrect? Would a linmos cutoff be helpful here?

Thanks,
Andrew
Attachments
S-ratio_SNR_blobcat_mosaic_2_sb_mc_1460_cvl_sub_small.png
S-ratio_SNR_blobcat_mosaic_2_sb_mc_1460_cvl_sub_small.png (155.05 KiB) Viewed 7804 times
S-ratio_SNR_blobcat_mosaic_2_convol_small.png
S-ratio_SNR_blobcat_mosaic_2_convol_small.png (181.74 KiB) Viewed 7804 times
S-ratio_SNR_blobcat_mosaic_2_sb-comb_mc_small.png
S-ratio_SNR_blobcat_mosaic_2_sb-comb_mc_small.png (199.17 KiB) Viewed 7804 times
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: Linmos has trouble mosaicking many pointings together

Post by Mark.Wieringa »

Hi Andrew,

I have trouble understanding where the effect comes from. You say you don't see it when using the whole 2GHz bandwidth. What is different in the processing of the sub-bands vs the full 2GHz mosaic?

How does the resolution (restoring beam size) of the low frequency images compare with the Gaussian beam you convolve with? The fact that only the low frequencies are affected makes me suspect beam size is involved, but it is hard to explain why the 1460 MHz image is affected when analysed on its own. It could be something to do with beam sidelobes from uncleaned flux showing up at low freq, but smeared out by the convolution at high freq.

Are you able to detect the effect in the individual images or only after the linmos step?
You could try doing some simulations to see if you can pin down where the effect comes in.

Cheers,

Mark
abutler
Posts: 11
Joined: Fri Feb 06, 2015 1:28 pm

Re: Linmos has trouble mosaicking many pointings together

Post by abutler »

Hi Mark,

There isn't much of a difference between the processing of the sub-bands vs. the full 2.0 GHz bandwidth. These are the only differences: in the full-band, the selfcal interval is 5, the Gaussian beam I convolve to is fwhm=5.39,4.21 pa=2.4, and of course the bw parameter in linmos is 2.0. Other than that, there is no difference.

The convolving beam I used to for the sub-bands was fwhm=6.70,5.50 pa=3.09 (geometric mean = 6.07"). I let restor determine its own fit to the dirty beam for each pointing (leaving fwhm and pa unset) before convolving to this beam. The average 1460 MHz beam was fwhm=5.69,4.26 pa=-2.31 (geometric mean=4.92"). So the convolving beam was ~23% bigger than the average beam. This is roughly consist with the way that the curve seems to end up at Sint/Sp=0.8 for the lowest SNR. The effect shows up in individual pointings as well.

We've done a few tests, and we're almost 100% positive the curve in the sub-band Sint/Sp vs. SNR plots is caused by the restoring beam and/or the beam used to convolve the images. If a beam larger than all the individual pointings' beams is used to restor the images, the curve in the Sint/Sp vs. SNR plot will turn downward with decreasing SNR. If a beam smaller than all the individual pointings' beams is used to restor the images, the curve in the Sint/Sp vs. SNR plot will turn upward with decreasing SNR (see attached plot). If anything is convolved (no matter what the restoring beam size), all the data at relatively low SNR will be dragged downward on the Sint/Sp axis (this effect of convolving is what is shown in the plots in my previous post).

Convolving the full-band data does not drag the data points down on the Sint/Sp vs. SNR plot (I think this is what should happen, because when you convolve, you're convolving the sources and the noise, not just the sources - I don't know why this isn't the case for the sub-bands). However, restoring the full-band pointings with a beam larger than the individual pointings' beams does produce the same downward curve that is present in the sub-band Sint/Sp vs. SNR plots.

So, consistent beam sizes are important! I am going to try and taper the individual pointings so that their beam sizes are as similar as possible before I restor or convolve them. That way, the difference between the beam of any individual pointing and the restoring/convolving beam is minimal. Hopefully the difference isn't unacceptably high, and I can rest assured that the resolution of the noise will be very close to the resolution of the sources.

Does this sound like a good course of action?

Cheers,
Andrew
Attachments
S-ratio_SNR_blobcat_mosaic_2_sb_mc_1460_rsb_small.png
S-ratio_SNR_blobcat_mosaic_2_sb_mc_1460_rsb_small.png (163.93 KiB) Viewed 6440 times
Post Reply