mfcal reading data slowly

[ycendes]

Hi all,

Quick intro, I'm Yvette, a PhD student working with Bryan Gaensler on Supernova 1987A data. I have however run into an issue with MIRIAD running slowly where Bryan doesn't know what's causing the problem either, so I am posting here in hopes that someone may have the solution.

In short, when I run some data sets MIRIAD reads the data extremely slowly (takes a few minutes on my machine, and 30 seconds to run on Bryan's computer (MacPro 3.7 GHz with 32 GB RAM)). File is here for download here and the command I am using to run it is "mfcal vis=0530-727.8512 interval=.1". Does anyone know why it might be going so slowly?

As a second issue perhaps related to the first, some later steps like maxen are also very, very slow, so just skipping this step is not an ideal option. Suggestions appreciated- thanks!

[Mark.Wieringa]

Hi Yvette,

Good question.. It took a bit of digging, but it turned out that for a long observations of old atca continuum data and short solution intervals, mfcal spends most of its time clearing a giant hash table. The size of this table was increased a lot when we moved to CABB. For CABB data the time it takes to read the data tends to mask the effect.

I've updated mfcal and it now runs to completion in a fraction of a second. The new version will be available by mirsync in about 8 hours and a new tar file gets created Thursday nights.

If you want me to look at the maxen issue, please submit a new question with the details to reproduce the problem.

Cheers,

Mark

[ycendes]

Great, thanks so much for looking into it so quickly! I will check it out in a few hours, and give a shout if there are any further issues.

[StacyMader]

I'm seeing the same kind of thing with mfcal running 100% CPU, processing 1921-293 from last month. Upon finding this post, I did a mirsync and the version I currently have is:

MfCal: $Revision: 1.18 $, $Date: 2017/03/06 03:50:09 $

It doesn't seem to matter whether I set interval to 0.1 or 10 minutes. Currently, mfcal has been running for nearly 15 minutes and nothing appears to have changed in the vis directory.

Running on MBP 10.12.1

[ste616]

Hi Stacy,

What other options have you set in mfcal?

[StacyMader]

Hi Jamie,

Only stokes=ii and interval=[0.1|10]

I have copied the file to kaputar:/data/KAPUTAR_1/mad134/1921-293.22237.7.tar.bz2

The data was taken using 1M0.5k, with 16k channels (I just concatennated all zooms).
I should note everything was fine the other day, but today, I'm running into this problem.

[ste616]

Hi Stacy,

This may not be the problem, but I'd try dropping the "stokes=ii" part for now, as it doesn't really make much sense with mfcal (it needs to determine bandpass and gain solutions for all the products that it finds anyway, and doesn't know anything about polarisation).

[StacyMader]

Hi Jamie,

Doesn't seem to make a difference. Looking at the activity monitor the bytes in/out is 0, even after 2 minutes.

[StacyMader]

After ~ 30 minutes and using edge = 2000 and interval = 2, mfcal completed.

I used edge here as there is a ridiculous number of channels, not required

[ste616]

Hi Stacy,

I didn't time it, and I just left it running from Friday afternoon, but mfcal with interval=1 (no other options) worked for me on kaputar on your 1921-293.22237.7 dataset.