Flagging with mirflag

[len067]

In case anyone is interested I have created a temporary page for mirflag and plotvis over here:

http://www.atnf.csiro.au/people/Emil.Le ... otvis.html


The tools are still in a very preliminary stage and I haven't yet had a chance to fully test them on all kinds of data (mainly as this was done more as a pet project). I have built binaries for the Mac and for Linux but only have only tested these on a limited number of systems. I have also included all of the source code for those with masochistic tendencies but I should point out that the Makefile is rather primitive and would probably require "tuning" to make it work more generically.

Nonetheless, if you do give it a try let me know how it turns out (or doesn't).

[ste616]

Hi Emil,

Just a note about mirflag, and the documentation on the site you linked to. Your site says that libglut is required for plotvis to work, but does not mention that the mirflag binary (for Linux at least) also requires libglut.

Using the source tarball Makefile also requires libglut for mirflag, although a quick tweak of the Makefile can fix this

[ste616]

I thought I'd share a positive experience I've just had while using mirflag.

I was reducing L-band data centred at 1750 MHz, and the first thing to do is calibrate 1934-638. A quick mfcal, gpcal and I got the following uvplt and uvspec outputs (remember, this is after automatic flagging via atlod's birdie,rfiflag options):

before_uvspec_1934-638.1750.png (10.58 KiB) Viewed 110580 times

before_uvplt_1934-638.1750.png (6.03 KiB) Viewed 110581 times

Clearly, this is riddled with narrowband RFI. I used mirflag in the following way:

mirflag vis=1934-638.1750.1 stokes=xx,xx,yy options=amp,medsed,short
mirflag vis=1934-638.1750.1 stokes=yy,xx,yy options=amp,medsed,short

This is amplitude based flagging, remembering to take the median bandpass away as 1934's spectrum has a slope. The command is issued twice, using the xx pol to flag both xx & yy first, then using the yy pol to do the flagging. This is necessary because some RFI is present only in xx or yy, although to be safe, any RFI found in either pol flags data in both polarisations.

After this one round of flagging, an mfcal and gpcal gives:

after_uvspec_1934-638.1750.png (11.4 KiB) Viewed 110581 times

after_uvplt_1934-638.1750.png (17.52 KiB) Viewed 110582 times

An excellent result that took mere seconds to achieve, and is easily scriptable. Indeed mirflag will be going into my pipeline scripts Thanks Emil!

Oh and for those people wondering why the real v imag plot looks like that, remember that 1934's amplitude slope will make the real axis elongated. As you can see the imaginary axis is still quite compact as it should be.

[len067]

Hi Emil,
Using the source tarball Makefile also requires libglut for mirflag, although a quick tweak of the Makefile can fix this

Hi Jamie,

Ah, thanks for picking this up - I'll try to get this fixed ASAP (hopefully tomorrow as I'm in the stacking workshop today).

Cheers,

Emil.

[len067]

Hi Jamie,

I thought I'd share a positive experience I've just had while using mirflag.

Cool! Thanks for giving mirflag a go and posting the before and after effect It's really great to get some feedback (good or bad) as I've only had a limited opportunity to test it to date ... and this is a really nice demonstration.

Just out of interest, do you recall what percentage of the data was flagged?

Cheers,

Emil.

[ste616]

Hi Emil,

The data started off as 85.42% flagged (amazing how little data survives at L band!). Flagging based on xx pol took it to 85.99% (a 0.57% total increase) and flagging again based on yy took it to 86.76% (another 0.77% total increase).

So mirflag is really rather surgical in its excision, which is fantastic

[len067]

Hi Jamie,

Wow! I was expecting a lot more when I looked at the before and after re-im plots. That's very reassuring because as you say it appears that it is quite surgical in its excision and that's a good thing.

Cheers,

Emil.

[len067]

Using the source tarball Makefile also requires libglut for mirflag, although a quick tweak of the Makefile can fix this

Hi Jamie,

I have posted an update to mirflag to (hopefully) remove the dependency to libglut in the linux binary and in the Makefile (I've tentatively added 64 bit linux support based on modifications that Enno Middelberg made to my Makefile but I haven't had a chance to test this out myself). I've also changed my naming convention for the builds so that the file name contains the version number (it also prints this information when plotvis or mirflag are run) - just to make managing future releases a bit easier.

http://www.atnf.csiro.au/people/Emil.Le ... otvis.html

Cheers,

Emil.

[Mark.Wieringa]

On 27/05/2010, at 9:54 AM, Wieringa, Mark (CASS, Narrabri) wrote:

Hi Emil,

your mirflag is gaining supporters, which is good. But..

the current user on kaputar has mirflag running with 18GB of memory allocated, kaputar has 16GB, so it is swapping and slooow...
Does mirflag have any limits on memory use? Should we request a memory upgrade?

Cheers,

Mark

________________________________________


From: Emil Lenc [Emil.Lenc@csiro.au]
Sent: Thursday, May 27, 2010 10:04 AM
To: Wieringa, Mark (CASS, Narrabri)
Cc: Stevens, Jamie (CASS, Narrabri)
Subject: Re: mirflag hogging memory

Hi Mark,

Is this for a single instance of mirflag running? I'm surprised that it is using this much memory. When I run mirflag I only read a single polarisation and only the amplitude of the visibility so it shouldn't really use too much for a single day observation i.e. ~1-2GB for a typical 12 hr observation (unfortunately, because of the way the flagging algorithms work I read all of the single-pol-amplitudes into memory to avoid multiple passes of the data - which would also be quite slow).

Cheers,

Emil.

________________________________________


On 27/05/2010, at 10:08 AM, Stevens, Jamie (CASS, Narrabri) wrote:

Hi Emil,

I'm with Catarina who is running the task. I can confirm that for a 1.3 MB dataset with options:

stokes=xx,xx,yy options=amp,medsed,short

it basically just prints out the task name and hangs while consuming memory. I've noticed myself that old-correlator continuum datasets also generate this behaviour. I can supply datasets that fail in this way if you'd like to try them out.

cheers
Jamie

________________________________________

Hi Jamie,

Ah, OK, that sounds more like a bug, as opposed to it just using a lot of memory - perhaps it is something to do with pre-cabb data ... that's kind of ironic.

Could you please email me the 1.3 MB data set that you had trouble with and I'll look into it - I can't say that I tested the flagger with pre-cabb data much (well, not at all to be honest).

Could we, ahem, move this conversation onto the ATCA forum by any chance?

Cheers,

Emil.

[ste616]

The discussion of the memory consumption bug is now moved to http://atcaforum.atnf.csiro.au/viewtopic.php?f=10&t=29" onclick="window.open(this.href);return false;