mfcal reading data slowly

Is MIRIAD being a pain? Let us know your experience.

Moderator: Mark.Wieringa

mfcal reading data slowly

Postby ycendes » Sat Mar 04, 2017 8:19 am

Hi all,

Quick intro, I'm Yvette, a PhD student working with Bryan Gaensler on Supernova 1987A data. I have however run into an issue with MIRIAD running slowly where Bryan doesn't know what's causing the problem either, so I am posting here in hopes that someone may have the solution.

In short, when I run some data sets MIRIAD reads the data extremely slowly (takes a few minutes on my machine, and 30 seconds to run on Bryan's computer (MacPro 3.7 GHz with 32 GB RAM)). File is here for download here and the command I am using to run it is "mfcal vis=0530-727.8512 interval=.1". Does anyone know why it might be going so slowly?

As a second issue perhaps related to the first, some later steps like maxen are also very, very slow, so just skipping this step is not an ideal option. Suggestions appreciated- thanks!
ycendes
 
Posts: 4
Joined: Wed Jan 11, 2017 8:15 am

Re: mfcal reading data slowly

Postby Mark.Wieringa » Mon Mar 06, 2017 3:20 pm

Hi Yvette,

Good question.. It took a bit of digging, but it turned out that for a long observations of old atca continuum data and short solution intervals, mfcal spends most of its time clearing a giant hash table. The size of this table was increased a lot when we moved to CABB. For CABB data the time it takes to read the data tends to mask the effect.

I've updated mfcal and it now runs to completion in a fraction of a second. The new version will be available by mirsync in about 8 hours and a new tar file gets created Thursday nights.

If you want me to look at the maxen issue, please submit a new question with the details to reproduce the problem.

Cheers,

Mark
Mark.Wieringa
ATCA Expert
 
Posts: 284
Joined: Mon Feb 08, 2010 1:37 pm

Re: mfcal reading data slowly

Postby ycendes » Tue Mar 07, 2017 3:39 am

Great, thanks so much for looking into it so quickly! :) I will check it out in a few hours, and give a shout if there are any further issues.
ycendes
 
Posts: 4
Joined: Wed Jan 11, 2017 8:15 am

Re: mfcal reading data slowly

Postby StacyMader » Fri Apr 21, 2017 4:18 pm

I'm seeing the same kind of thing with mfcal running 100% CPU, processing 1921-293 from last month. Upon finding this post, I did a mirsync and the version I currently have is:

MfCal: $Revision: 1.18 $, $Date: 2017/03/06 03:50:09 $

It doesn't seem to matter whether I set interval to 0.1 or 10 minutes. Currently, mfcal has been running for nearly 15 minutes and nothing appears to have changed in the vis directory.

Running on MBP 10.12.1
StacyMader
 
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Postby ste616 » Fri Apr 21, 2017 4:26 pm

Hi Stacy,

What other options have you set in mfcal?
cheers
Jamie Stevens
ATCA Senior System Scientist
ste616
Site Admin
 
Posts: 217
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Re: mfcal reading data slowly

Postby StacyMader » Fri Apr 21, 2017 4:30 pm

Hi Jamie,

Only stokes=ii and interval=[0.1|10]

I have copied the file to kaputar:/data/KAPUTAR_1/mad134/1921-293.22237.7.tar.bz2

The data was taken using 1M0.5k, with 16k channels (I just concatennated all zooms).
I should note everything was fine the other day, but today, I'm running into this problem.
StacyMader
 
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Postby ste616 » Fri Apr 21, 2017 4:34 pm

Hi Stacy,

This may not be the problem, but I'd try dropping the "stokes=ii" part for now, as it doesn't really make much sense with mfcal (it needs to determine bandpass and gain solutions for all the products that it finds anyway, and doesn't know anything about polarisation).
cheers
Jamie Stevens
ATCA Senior System Scientist
ste616
Site Admin
 
Posts: 217
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Re: mfcal reading data slowly

Postby StacyMader » Fri Apr 21, 2017 4:40 pm

Hi Jamie,

Doesn't seem to make a difference. Looking at the activity monitor the bytes in/out is 0, even after 2 minutes.
StacyMader
 
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Postby StacyMader » Sun Apr 23, 2017 3:04 pm

After ~ 30 minutes and using edge = 2000 and interval = 2, mfcal completed.

I used edge here as there is a ridiculous number of channels, not required :)
StacyMader
 
Posts: 14
Joined: Thu Jan 24, 2013 4:19 pm

Re: mfcal reading data slowly

Postby ste616 » Mon Apr 24, 2017 9:35 am

Hi Stacy,

I didn't time it, and I just left it running from Friday afternoon, but mfcal with interval=1 (no other options) worked for me on kaputar on your 1921-293.22237.7 dataset.
cheers
Jamie Stevens
ATCA Senior System Scientist
ste616
Site Admin
 
Posts: 217
Joined: Thu Feb 04, 2010 3:27 pm
Location: Paul Wild Observatory Narrabri NSW

Next

Return to MIRIAD

Who is online

Users browsing this forum: No registered users and 1 guest

cron