Converting Miriad data into MeasurementSet form

[gmigliori]

Hello,

I am trying to import a calibrated ATCA dataset reduced with miriad to CASA MS format but I'm having some issues. I have tried two approaches:
1- using casa 4.3.1 task importmiriad:
CASA> inp importmiriad
CASA> mirfile='j1550-564.4800'
CASA> vis='test.MS'
where j1550-564.4800 is my calibrated miriad dataset. This is the warning message that I get:
importmiriad: Version 4 - Miriad to MS filler (08-May-2014)
Warning: gains table present, but cannot apply them
(and same for bandpass and leakage corrections)

2-first I convert the vis file to a fits file using miriad task fits:
fits in=j1550-564.4800 op=uvout out=test.fits
Applying bandpass corrections to j1550-564.4800
Applying gain corrections to j1550-564.4800
Applying polarization leakage corrections to j1550-564.4800
Polarisations copied: XX,YY,XY,YX.
169584 visibilities copied to 42396 output records.
Writing FITS antenna table
and then I convert the fits file to a MS format with the CASA task importuvfits:
CASA <2>: inp importuvfits
--------> inp(importuvfits)
# importuvfits :: Convert a UVFITS file to a CASA visibility data set
fitsfile = '' # Name of input UV FITS file
vis = '' # Name of output visibility file (MS)
antnamescheme = 'old' # VLA/EVLA/CARMA only; 'new' or 'old'; 'VA04' or '04' for VLA ant 4

CASA <3>: fitsfile='test.fits'

CASA <4>: vis='test.MS'

CASA <5>: go
--------> go()
Executing: importuvfits()

Here, in theory, the fits task should apply the calibration correction, however in the final MS dataset the calibrated data are missing, looking at the table columns:
['UVW', 'FLAG', 'FLAG_CATEGORY', 'WEIGHT', 'SIGMA', 'ANTENNA1', 'ANTENNA2', 'ARRAY_ID', 'DATA_DESC_ID', 'EXPOSURE', 'FEED1', 'FEED2', 'FIELD_ID', 'FLAG_ROW', 'INTERVAL', 'OBSERVATION_ID', 'PROCESSOR_ID', 'SCAN_NUMBER', 'STATE_ID', 'TIME', 'TIME_CENTROID', 'DATA', 'WEIGHT_SPECTRUM']
it seems that the DATA_CORRECTED column has not been created.
I have tried with a different calibrated dataset but the issue is still there.

Do you have any suggestions?
Thank you in advance for your help,
best,
Giulia

[Mark.Wieringa]

Hi Guilia,

For importmiriad, you need to first create a calibrated dataset in miriad (e.g. using uvaver to apply the calibration tables), fits does this on the fly. But in both cases the calibrated data ends up in the DATA column in the MS. The CORRECTED_DATA column is only produced when you apply calibration in CASA (e.g. using applycal).

This shouldn't stop you from imaging the data, or doing selfcalibration, in CASA though.

Cheers,

Mark

[amykimball]

The importmiriad task in CASA versions 4.4 and earlier contain a bug such that they are unable to read in certain kinds of (mixed-frequency) Miriad data.

The issue showed up with a CABB data set containing four IFs from L/S band (1 and 3 in continuum, 2 and 4 in multiple-channel zooms) and two IFs from C/X band (1 and 2, both continuum).
Importmiriad is trying to combine IF 1 from L/S with IF 1 from C/X into a single spw0, and IF 2 from L/S with IF 2 from C/X into a single spw1, without respect to their differing frequencies.
As a result, Importmiriad breaks with an error similar to:
### Error: Narrow Channel changing from 22532 to 4098

There is a workaround:
(1) run Miriad's atlod, including the option "noif" (i.e., do not split by IF)
(2) run Miriad's uvsplit, with option "nosource". (i.e., do not split by source)
(2a) Note: in the case of two identical continuum IF frequencies, e.g., at 2100 MHz, also include the option "ifchain". (i.e., separate all IFs, even if two are duplicates)
(3) The resulting Miriad data sets can now be imported through CASA's importmiriad task.

Mark Wieringa has submitted a CASA fix which is to appear in CASA version 4.5.

[nicholas.seymour]

I have unique case of converting miriad data to measurementset to be read by wsclean.

I have used uvgen to simulate a 2min MWA snapshot dataset:

> uvgen source=p_4Jy.source ant=phase1.ant baseunit=-3.33564 corr=768,1,0,30.72 freq=0.15,0 radec=0,-26.7013 harange=-0.0166666,0.0166666,0.00111111 lat=-26.7013 pbfwhm=50164 system=400 jyperk=160.5 out=ph1_p4Jy_uv

However, I cannot convert to a MS. I tried

(a) using importmiriad on the output of uvgen directly
(b) converting the miriad file to fits with the 'fits' task (so that I can use importuvfits in casa).

In the latter case I get:

[pandemos:science/mwa/weighting] nseymour% fits in=ph1_p4Jy_uv op=uvout out=ph1_p4Jy.fits

fits: Revision 1.32, 2014/09/15 04:39:01 UTC

Polarisations copied: I.
243840 visibilities copied to 243840 output records.
### Warning: Insufficent information for antenna table

I think the issue is that the simulated data does not have the appropriate antenna information. The antenna format is geocentric coordinates in metres.

thanks,

Nick

[Mark.Wieringa]

Hi Nick,

I managed to reproduce the uvgen->importmiriad failure and will have a look at it today. The fits route worked for me, but I don't have your antenna file.
Can you send me the phase1.ant file so I can try this?

Cheers,

Mark

[nicholas.seymour]

Hi Mark,

thanks for your reply and finding the same error.

I paste the antenna file below as I don't seem able to attach it.

cheers,

Nick


-149.785 265.814 377.011
-95.365 270.188 377.285
-88.512 266.021 377.309
-78.698 258.431 377.291
-86.940 248.452 377.339
-102.325 244.088 377.258
-93.346 242.279 377.270
-148.330 220.271 377.214
-132.939 282.611 377.002
-139.867 275.498 377.018
-123.667 284.854 377.050
-92.889 281.358 377.254
-77.835 282.839 377.253
-83.476 277.563 377.300
-76.988 272.637 377.298
-176.006 289.903 376.623
-23.620 201.546 376.805
-23.802 222.353 376.872
-40.489 235.170 376.961
-54.527 237.518 377.123
-64.851 250.813 377.239
-70.075 259.402 377.266
-60.057 263.676 377.180
-50.676 264.546 377.100
-67.990 271.103 377.236
-60.058 271.905 377.204
-70.885 278.818 377.252
-45.540 273.245 377.066
-53.156 276.268 377.090
-22.659 263.694 376.948
-55.097 284.285 377.181
-14.900 273.870 376.910
-55.560 124.801 376.803
177.467 -43.377 375.005
-21.710 077.005 376.351
091.809 -80.565 375.760
153.700 -259.929 376.476
-67.582 -22.361 376.175
-118.037 -39.873 376.054
-87.516 125.197 377.031
-105.281 217.180 377.255
-98.530 230.162 377.260
-81.845 229.337 377.270
-79.541 238.128 377.270
-75.120 247.002 377.255
-71.017 235.929 377.239
-62.567 228.987 377.210
-50.691 221.885 377.078
-195.980 189.795 377.161
-198.194 095.150 377.024
-284.457 -196.181 374.901
-388.152 -44.022 375.372
-452.686 -15.933 375.111
-468.817 185.187 374.752
-364.524 183.098 375.739
-369.608 266.484 375.240
-160.465 573.207 374.773
-128.046 350.494 376.570
-69.771 294.031 377.192
-78.731 297.259 377.196
-103.941 300.914 377.050
-100.172 288.896 377.157
-395.452 371.152 374.515
-263.547 389.274 375.322
326.141 203.608 373.945
173.425 193.595 375.032
-02.515 292.841 376.892
-46.554 287.469 377.115
-56.141 295.922 377.156
-33.705 351.048 376.976
050.127 536.159 376.181
084.880 519.141 376.314
-518.046 632.503 372.604
-489.943 604.568 372.907
-575.557 415.008 373.374
-585.675 -101.530 375.212
-653.467 142.255 374.462
-685.027 212.168 374.236
-1079.598 087.018 377.009
-1026.074 574.527 374.039
-333.758 1660.732 371.068
-227.151 898.129 373.694
-173.717 781.638 374.217
-256.952 892.005 373.492
-195.018 762.893 374.093
-249.647 731.709 373.797
-300.696 704.326 373.514
-889.471 1088.136 370.381
716.393 1405.255 374.602
288.066 618.265 375.134
238.310 768.413 375.805
227.878 819.724 375.748
061.443 815.255 376.001
-58.874 810.530 375.170
035.224 950.242 376.275
001.015 1412.363 373.682
839.943 990.227 372.696
1177.294 858.632 372.122
1313.611 553.372 370.354
556.122 244.485 372.284
571.541 359.210 372.509
488.857 424.980 372.967
528.598 539.194 372.789
381.069 652.864 374.251
1100.945 151.760 369.065
1201.132 -443.250 368.328
498.629 -264.594 373.220
301.193 -48.952 374.618
370.648 -38.415 374.340
544.324 -91.152 372.812
506.793 -03.216 372.803
523.727 075.494 372.613
653.774 -1037.748 371.071
442.990 -835.306 372.251
-58.801 -505.332 372.913
037.065 -405.050 374.034
232.842 -357.891 375.262
257.984 -366.453 375.221
325.688 -318.508 375.029
326.101 -306.307 375.128
-412.457 -409.060 373.969
-141.898 -420.261 373.503
-480.270 -817.324 375.720
-575.533 -863.782 375.522
-674.021 -638.871 375.340
-631.403 -245.107 375.544
-633.237 -169.362 375.616
-568.540 -257.660 375.246

[Mark.Wieringa]

Hi Nick,

there are a few issues:

* it is best to run uvsplit on the output of uvgen before conversion, this removes the 'wide' channels that are not handled well downstream
* you have 128 antennas, this is more than the standard miriad can handle (MAXANT=64), importmiriad in casa has the same limitation
* CASA doesn't like data with Stokes I correlations, so specify stokes=xx or similar in uvgen
* Miriad doesn't know about MWA, so you can't specify telescop='mwa' in uvgen. The default seems to be 'bima'

The 64 antenna limit in Miriad can be fixed by changing this in maxdim and recompiling, doing the same for casa is a lot harder - I can fix the code but generally you have to wait for the next release to get the fix.

So I think your best best is to use the fits conversion route using uvgen with stokes=xx, uvsplit and fits in Miriad. I managed to successfully load this into casa, but the antenna longitudes were set to -99degrees.

I could not reproduce the fits warning message about the antenna table:

fits op=uvout in=ph1_p4Jy_uv.150 out=ph1_p4Jy_uv.150.fits
fits: Revision 1.32, 2014/09/15 04:39:01 UTC

Polarisations copied: XX,YY.
487680 visibilities copied to 243840 output records.
Writing FITS antenna table


Cheers,

Mark