User Tools

Site Tools


huibintemaspampipeline

Running the SPAM pipeline

This section (created with significant help from Pratik Dabhade) describes how to run the pipeline on continuum observations at 150, 235, 325 or 610 MHz, with 16 or 32 MHz of bandwidth, obtained with the GMRT software correlator backend (GSB).


Basic pipeline run

Let's assume you have requested and downloaded a single night observation from the GMRT archive in LTA format. You will end up with 3 files:

  • <project name>_<observe date>.lta - The visibility data in LTA format (e.g., 29_043_24aug2015.lta)
  • <project name>_<observe date>.lta##<observation number>.obslog - The operator log file in ascii format; contains useful information about how the observations went.
  • <project name>_<observe date>.lta##<project_name>.FLAGS[.<index>] - The online flag information in ascii format; contains status information of individual antennas, used for flagging

We will assume that the observation contains at least one of the primary calibrators 3C48, 3C147 or 3C286.

First we convert LTA to UVFITS format:

lta_file_name = "./<project name>_<observe date>.lta"
convert_lta_to_uvfits( lta_file_name )

This creates a UVFITS file in the fits subdirectory, having the same name as the LTA file name but with the extension “.UVFITS” added to it.

Then we derive calibration and flagging information from the primary calibrator(s), pick the best one, transfer those informations to all other source visibilities present in the observation, and export each of these pre-calibrated visibility data sets to UVFITS:

uvfits_file_name = "./fits/<project name>_<observe date>.lta.UVFITS"
pre_calibrate_targets( uvfits_file_name, flags_file_name = lta_file_name + "##*.FLAGS*" )

The fits subdirectory in our project directory now contains pre-calibrated visibility data sets per source. The file name convention is as follows: <source name>_<observation reference date>_<polarization(s)>_<sideband>.UVFITS

The main pipeline will take the pre-calibrated visibilities of any source you specify, and produce a series of images with increasingly more flagging and calibrations.

target_uvfits_file_name = "./fits/<source name>_<observation reference date>_<polarization(s)>_<sideband>.UVFITS"
process_target( target_uvfits_file_name )

This may typically take 6-8 times the duration of the actual observation to complete. The good news is that you can walk away and check the results later. All processing output is captured in a log file: ./datfil/spam_<source name>_<start date>_start_time>.log The final image is: ./fits/<source name>.SP2B.PBCOR.FITS

The pipeline run can be summarized by typing:

summarize_spam_log( "./datfil/spam_<source name>_<observation reference date>_<polarization(s)>_<sideband>*.log" )

There's a million options available to make this process work for different / more complicated data sets. Some of the most common situations are discussed below.


Combining multiple LTA files of one observation

Sometimes a single observation has been broken up into multiple LTA files because the data recording was interrupted (e.g., because of a power failure or system reset). Note that this is different from having a single project spread out over multiple observations. The multiple LTA files can be individually converted to UVFITS before combining:

lta_file_name_1 = "./<project name>_<observe date>.lta"
convert_lta_to_uvfits( lta_file_name_1 )
lta_file_name_2 = "./<project name>_<observe date>.lta.1"
convert_lta_to_uvfits( lta_file_name_2 )

Then the resulting UVFITS files can be combined as follows:

uvfits_file_name_1 = "./fits/<project name>_<observe date>.lta.UVFITS"
uvfits_file_name_2 = "./fits/<project name>_<observe date>.lta.1.UVFITS"
uvfits_file_name = "./fits/<project name>_<observe date>.lta.combined.UVFITS"
combine_uv( uvfits_file_name_1, uvfits_file_name_2, uvfits_file_name )

The output UVFITS can then be passed to pre_calibrate_targets() etc.


Combining observations of the same target

Combining observations on the same target from multiple observations is only possible when the frequency setup was exactly the same during the multiple observations. This is (almost?) always the case when a target was observed during a single project spread over multiple nights.

First, convert and pre-calibrate the observat