huibintemaspampipeline
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
huibintemaspampipeline [2017/06/26 22:52] – [Limited bandwidth observations] huibintema | huibintemaspampipeline [2020/10/05 15:49] – huibintema | ||
---|---|---|---|
Line 1: | Line 1: | ||
===== Running the SPAM pipeline ===== | ===== Running the SPAM pipeline ===== | ||
- | This section | + | This section |
----- | ----- | ||
Line 17: | Line 17: | ||
<code python> | <code python> | ||
lta_file_name = " | lta_file_name = " | ||
- | uvfits_file_name = " | + | convert_lta_to_uvfits( lta_file_name ) |
- | convert_lta_to_uvfits( lta_file_name, uvfits_file_name | + | |
</ | </ | ||
+ | This creates a UVFITS file in the fits subdirectory, | ||
Then we derive calibration and flagging information from the primary calibrator(s), | Then we derive calibration and flagging information from the primary calibrator(s), | ||
<code python> | <code python> | ||
+ | uvfits_file_name = " | ||
pre_calibrate_targets( uvfits_file_name, | pre_calibrate_targets( uvfits_file_name, | ||
</ | </ | ||
Line 40: | Line 41: | ||
./ | ./ | ||
- | There' | + | The pipeline run can be summarized by typing: |
+ | <code python> | ||
+ | summarize_spam_log( " | ||
+ | </ | ||
+ | |||
+ | |||
+ | There' | ||
----- | ----- | ||
+ | ==== Combining multiple LTA files of one observation ==== | ||
+ | |||
+ | Sometimes a single observation has been broken up into multiple LTA files because the data recording was interrupted (e.g., because of a power failure or system reset). Note that this is different from having a single project spread out over multiple observations. The multiple LTA files can be individually converted to UVFITS before combining: | ||
+ | <code python> | ||
+ | lta_file_name_1 = " | ||
+ | convert_lta_to_uvfits( lta_file_name_1 ) | ||
+ | lta_file_name_2 = " | ||
+ | convert_lta_to_uvfits( lta_file_name_2 ) | ||
+ | </ | ||
+ | Then the resulting UVFITS files can be combined as follows: | ||
+ | <code python> | ||
+ | uvfits_file_name_1 = " | ||
+ | uvfits_file_name_2 = " | ||
+ | uvfits_file_name = " | ||
+ | combine_uv( uvfits_file_name_1, | ||
+ | </ | ||
+ | The output UVFITS can then be passed to // | ||
+ | |||
+ | ----- | ||
+ | |||
+ | ==== Combining observations of the same target ==== | ||
+ | |||
+ | Combining observations on the same target from multiple observations is only possible when the frequency setup was exactly the same during the multiple observations. This is (almost?) always the case when a target was observed during a single project spread over multiple nights. | ||
+ | |||
+ | First, convert and pre-calibrate the observations per night. Then make sure that the multiple pre-calibrated UVFITS files of the target are all located in the same (fits) directory, and have a common naming scheme. E.g., < | ||
+ | <code python> | ||
+ | uvfits_file_name = " | ||
+ | process_target( ' | ||
+ | </ | ||
+ | This will read in all UVFITS files that match the wildcard query, concatenate them, and process them as one. | ||
+ | |||
+ | ----- | ||
+ | |||
+ | ==== Changing image weights ==== | ||
+ | |||
+ | The user can change some of the imaging parameters (AIPS IMAGR style), including the image weights. By default, the main pipeline uses no UV range cuts, and sets the Briggs robust parameter to -1 to compensate for the broad PSF wings due to the centrally condensed UV coverage. This can be changed as follows: | ||
+ | <code python> | ||
+ | imagr_params = { ' | ||
+ | process_target( target_uvfits_file_name, | ||
+ | </ | ||
+ | |||
+ | ----- | ||
+ | |||
+ | ==== Keeping main pipeline intermediate files ==== | ||
+ | |||
+ | The main pipeline takes the data through various repetitions of calibration, | ||
+ | <code python> | ||
+ | process_target( target_uvfits_file_name, | ||
+ | </ | ||
+ | |||
+ | ----- | ||
==== Dual-frequency observations ==== | ==== Dual-frequency observations ==== | ||
In case of dual-frequency observations, | In case of dual-frequency observations, | ||
<code python> | <code python> | ||
- | convert_lta_to_uvfits( lta_file_name, | + | convert_lta_to_uvfits( lta_file_name, |
- | convert_lta_to_uvfits( lta_file_name, | + | convert_lta_to_uvfits( lta_file_name, |
</ | </ | ||
The two output UVFITS files can then be processed further as usual. Please note that you may have to manually limit the frequency channel range for the 235 MHz observations, | The two output UVFITS files can then be processed further as usual. Please note that you may have to manually limit the frequency channel range for the 235 MHz observations, | ||
Line 65: | Line 123: | ||
gv ./ | gv ./ | ||
</ | </ | ||
- | Select a channel range over which the bandpass phases are well-behaved (linear) for most antennas, | + | Select a channel range over which the bandpass phases are well-behaved (approximately |
+ | {{ : | ||
+ | |||
+ | Note down the lower- and upper-limit of the channel range. Then re-run | ||
<code python> | <code python> | ||
- | channel_range = [ 10, 225 ] # example channel range to keep | + | channel_range = [ 150, 425 ] # example channel range to keep |
pre_calibrate_targets( uvfits_file_name, | pre_calibrate_targets( uvfits_file_name, | ||
</ | </ | ||
Line 73: | Line 134: | ||
----- | ----- | ||
- | ==== Using different | + | ==== Old hardware-correlator observations ==== |
+ | |||
+ | If you downloaded data from cycle 17 or earlier, it is likely correlated using the GMRT Hardware Backend (GHB; a.k.a. the hardware correlator). For frequencies of 325 MHz and higher, the 32 MHz bandwidth is typically split over an upper-side band (USB) and lower-side band (LSB), both captured in separate LTA files (typical extensions are .lta and .ltb). In SPAM, both sideband LTA files need to be pre-processed separately: | ||
+ | <code python> | ||
+ | convert_lta_to_uvfits( lta_file_name ) | ||
+ | convert_lta_to_uvfits( ltb_file_name ) | ||
+ | </ | ||
+ | The next step is run using the // | ||
+ | <code python> | ||
+ | pre_calibrate_targets( uvfits_file_name_lta, | ||
+ | pre_calibrate_targets( uvfits_file_name_ltb, | ||
+ | </ | ||
+ | The resulting UVFITS files for USB and LSB per target can be combined: | ||
+ | <code python> | ||
+ | uvfits_file_name_usb = " | ||
+ | uvfits_file_name_lsb = " | ||
+ | uvfits_file_name = " | ||
+ | combine_usb_lsb( uvfits_file_name_usb, | ||
+ | </ | ||
+ | The output UVFITS file can be processed further in the main pipeline. | ||
+ | |||
+ | Regarding the main pipeline, there are two options that may be relevant to get to better results. The first option related to the situation explained above, where two sidebands (USB and LSB) are joined together to cover 32 MHz of bandwidth. In that case, it may help to turn on an image-based flagging option that treats the joined USB and LSB separately. Reason for this is that the USB and LSB have separate signal chains, and thus there can be system problems that relate only to one of the two sidebands. | ||
+ | <code python> | ||
+ | process_target( target_uvfits_file_name, | ||
+ | </ | ||
+ | |||
+ | The second option is to turn on baseline-based calibration, | ||
+ | <code python> | ||
+ | process_target( target_uvfits_file_name, | ||
+ | </ | ||
+ | |||
+ | ----- | ||
+ | |||
+ | ==== Using different calibration models ==== | ||
By default, a point source model based on the NVSS, WENSS, VLSSr, SUMSS, and MGPS-2 catalogs is used to bootstrap the phase calibration at the start of the main pipeline (// | By default, a point source model based on the NVSS, WENSS, VLSSr, SUMSS, and MGPS-2 catalogs is used to bootstrap the phase calibration at the start of the main pipeline (// | ||
Line 83: | Line 177: | ||
<code python> | <code python> | ||
- | catalog_name = '< | + | catalog_name = "< |
catalog = read_pybdsm_ascii_catalog( catalog_name ) | catalog = read_pybdsm_ascii_catalog( catalog_name ) | ||
source_list = create_source_list_from_catalog( catalog ) | source_list = create_source_list_from_catalog( catalog ) | ||
Line 91: | Line 185: | ||
----- | ----- | ||
+ | |||
+ | ==== EXPERIMENTAL: | ||
+ | |||
+ | SPAM has some options to process uGMRT wideband data. SPAM does not support the processing of large fractional bandwidths (df/ | ||
+ | |||
+ | The first step is to convert LTA to UVFITS format: | ||
+ | <code python> | ||
+ | lta_file_name = " | ||
+ | convert_lta_to_uvfits( lta_file_name ) | ||
+ | </ | ||
+ | Next, we split the UVFITS file into smaller frequency chunks (subbands): | ||
+ | <code python> | ||
+ | uvfits_file_name = " | ||
+ | split_wideband_uvdata( uvfits_file_name ) | ||
+ | </ | ||
+ | The width of the frequency chunks is automatically set to a sensible value. The resulting 4 or 6 UVFITS files are also located in the fits subdirectory and named " | ||
+ | |||
+ | From here, each frequency chunk is processed independently in a similar fashion as a narrow-band GMRT observations, | ||
+ | < | ||
+ | uvfits_file_name = " | ||
+ | reference_frequency = 450.e6 | ||
+ | pre_calibrate_wideband_targets( uvfits_file_name, | ||
+ | </ | ||
+ | Setting a fixed reference frequency ensures that the frequency averaging of all frequency chunks is the same, which is important when jointly imaging the SPAM output visibilities later. Sensible values seem to be: | ||
+ | < | ||
+ | reference_frequency = 450.e6 | ||
+ | reference_frequency = 650.e6 | ||
+ | </ | ||
+ | The pre-calibrated visibilities per target are located in the fits subdirectory and are named per subband. | ||
+ | |||
+ | Next comes the SPAM main pipeline run. This is best done in separate project directories per subband. If possible, use a good, single reference sky model for all runs. For example, this reference model can be obtained from first running SPAM on the narrow-band GMRT (GSB) data that was recorded alongside the uGMRT wideband data, and extracting a sky model from the final SP2B image using PyBDSF. | ||
+ | < | ||
+ | target_uvfits_file_name = " | ||
+ | UVFITS" | ||
+ | catalog_name = "< | ||
+ | catalog = read_pybdsm_ascii_catalog( catalog_name ) | ||
+ | source_list = create_source_list_from_catalog( catalog ) | ||
+ | resolution = 10. # representative resolution of model image in arcsec | ||
+ | process_wideband_target( target_uvfits_file_name, | ||
+ | </ | ||
+ | |||
+ | If all went well, each SPAM pipeline run on a subband yielded a final image and a calibrated visibility data set (.SP2B.CAL.RR.UVFITS). For use in WSClean, the calibrated visibilities all need to be collected in one directory and converted into measurement sets using CASA. Then WSClean can be used to do a final wideband imaging run. Here is an example: | ||
+ | < | ||
+ | wsclean -weight briggs 0 -pol RR -size 5000 5000 -scale 1.5asec -niter 15000 -auto-threshold 0.5 -auto-mask 3 -gain 0.25 -mgain 0.8 -weighting-rank-filter 3 -join-channels -channels-out 6 -j 8 -mem 80 -name SOURCE_UGMRT3 SOURCE_UGMRT3-01.MS SOURCE_UGMRT3-02.MS SOURCE_UGMRT3-03.MS SOURCE_UGMRT3-04.MS SOURCE_UGMRT3-05.MS SOURCE_UGMRT3-06.MS | ||
+ | </ | ||
+ | Here, SOURCE_UGMRT3-0x.MS are the input measurement sets as produced by CASA. Make sure that " | ||
+ | |||
+ | |||
+ | |||
+ | ----- | ||
+ | |||
+ | Feedback: [[huib.intema@curtin.edu.au|Click here]] | ||
huibintemaspampipeline.txt · Last modified: 2020/10/05 17:46 by huibintema