mfcal reading data slowly

Is MIRIAD being a pain? Let us know your experience.

Moderator: Mark.Wieringa

ycendes
Posts: 4
Joined: Wed Jan 11, 2017 8:15 am

mfcal reading data slowly

Post by ycendes »

Hi all,

Quick intro, I'm Yvette, a PhD student working with Bryan Gaensler on Supernova 1987A data. I have however run into an issue with MIRIAD running slowly where Bryan doesn't know what's causing the problem either, so I am posting here in hopes that someone may have the solution.

In short, when I run some data sets MIRIAD reads the data extremely slowly (takes a few minutes on my machine, and 30 seconds to run on Bryan's computer (MacPro 3.7 GHz with 32 GB RAM)). File is here for download here and the command I am using to run it is "mfcal vis=0530-727.8512 interval=.1". Does anyone know why it might be going so slowly?

As a second issue perhaps related to the first, some later steps like maxen are also very, very slow, so just skipping this step is not an ideal option. Suggestions appreciated- thanks!
Mark.Wieringa
ATCA Expert
Posts: 297
Joined: Mon Feb 08, 2010 1:37 pm

Re: mfcal reading data slowly

Post by Mark.Wieringa »

Hi Yvette,

Good question.. It took a bit of digging, but it turned out that for a long observations of old atca continuum data and short solution intervals, mfcal spends most of its time clearing a giant hash table. The size of this table was increased a lot when we moved to CABB. For CABB data the time it takes to read the data tends to mask the effect.

I've updated mfcal and it now runs to completion in a fraction of a second. The new version will be available by mirsync in about 8 hours and a new tar file gets created Thursday nights.

If you want me to look at the maxen issue, please submit a new question with the details to reproduce the problem.

Cheers,

Mark
ycendes
Posts: 4
Joined: Wed Jan 11, 2017 8:15 am

Re: mfcal reading data slowly

Post by ycendes »

Great, thanks so much for looking into it so quickly! :) I will check it out in a few hours, and give a shout if there are any further issues.
StacyMader
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Post by StacyMader »

I'm seeing the same kind of thing with mfcal running 100% CPU, processing 1921-293 from last month. Upon finding this post, I did a mirsync and the version I currently have is:

MfCal: $Revision: 1.18 $, $Date: 2017/03/06 03:50:09 $

It doesn't seem to matter whether I set interval to 0.1 or 10 minutes. Currently, mfcal has been running for nearly 15 minutes and nothing appears to have changed in the vis directory.

Running on MBP 10.12.1
ste616
Site Admin
Posts: 220
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Re: mfcal reading data slowly

Post by ste616 »

Hi Stacy,

What other options have you set in mfcal?
cheers
Jamie Stevens
ATCA Senior System Scientist
StacyMader
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Post by StacyMader »

Hi Jamie,

Only stokes=ii and interval=[0.1|10]

I have copied the file to kaputar:/data/KAPUTAR_1/mad134/1921-293.22237.7.tar.bz2

The data was taken using 1M0.5k, with 16k channels (I just concatennated all zooms).
I should note everything was fine the other day, but today, I'm running into this problem.
ste616
Site Admin
Posts: 220
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Re: mfcal reading data slowly

Post by ste616 »

Hi Stacy,

This may not be the problem, but I'd try dropping the "stokes=ii" part for now, as it doesn't really make much sense with mfcal (it needs to determine bandpass and gain solutions for all the products that it finds anyway, and doesn't know anything about polarisation).
cheers
Jamie Stevens
ATCA Senior System Scientist
StacyMader
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Post by StacyMader »

Hi Jamie,

Doesn't seem to make a difference. Looking at the activity monitor the bytes in/out is 0, even after 2 minutes.
StacyMader
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Post by StacyMader »

After ~ 30 minutes and using edge = 2000 and interval = 2, mfcal completed.

I used edge here as there is a ridiculous number of channels, not required :)
ste616
Site Admin
Posts: 220
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Re: mfcal reading data slowly

Post by ste616 »

Hi Stacy,

I didn't time it, and I just left it running from Friday afternoon, but mfcal with interval=1 (no other options) worked for me on kaputar on your 1921-293.22237.7 dataset.
cheers
Jamie Stevens
ATCA Senior System Scientist
Post Reply