WAPP, Wide Band Arecibo Pulsar Processor

This page is Jeffs notes on the status of the wapp. The wideband arecibo pulsar processor. The WAPP Home Page is too difficult to change.

This page last updated on 6 Oct 2004.

Latest Fixes and changes 6 Oct 04

A new keyword to the pulsar catalog observing has been added:

OFFSETS ddmmss ddmmss units

Latest Fixes and changes 30 Sept 04

• The WAPP header positions are now really there and should be correct.
• The iflo_flip parameter should now be set properly. iflo_flip indicates whether the band is flipped. It considers the entire IFLO system and should work for all receivers and bands. Please check this for me as I cant check every possible combination.
• The catalog observing mode now has a few added keywords.
  • CLEARALFAOFF - this will clear all beam offsets. This will not clear pointing offsets put on the pointing screen. Note these offsets are now separate. This is equivalent to ALFABEAM 0.
  • ALFAPARA source secs offset - compute the paralactic angle of the source at lst + secs, and rotate ALFA to -para + offset. Waits till ALFA is finished moving.
  • ALFABEAM beam - set the pointing corrections to center ALFA on the specified beam.

Latest Fixes and changes 17 Sept 04

    double rfeed_offaz[7];  /* deg az rotated offset all alfa beam to center */
    double rfeed_offza[7];  /* deg za rotated offset all alfa beam to center */
    double prfeed_offaz;  /* deg az offset to center of alfa beam */
    double prfeed_offza;  /* deg za offset to center of alfa beam */
    double alfa_raj[7];   /* hr starting actual ra position of alfa beams */
    double alfa_decj[7];  /* deg starting actual dec position of alfa beams */
    double alfa_az[7];    /* deg starting actual az position of alfa beams */
    double alfa_za[7];    /* deg starting actual za postion of alfa beams */
    double alfa_ang;      /* deg alfa rotation angle  */
    double para_ang;      /* deg paralactic angle of center beam */
    char frontend[24];    /* receiver abbrev name */

    unsigned char prfeed;  /* centered alfa beam */

Also, there is a new observing screen under Pulsar Observing => More => Pulsar Mapping Modes

Included here are the familiar modes as well as two new ones for the PALFA team. Alfa Beam Check and ALFA calibration check.

• Alfa Beam Check
  • point to the central pixel
  • take a WAPP pulsar scan
  • repeat for all outer pixels
  • return to center
• Alfa Calibration Check
  • turn on the 25 Hz cal
  • point the central pixel to source
  • take a short scan of specified length
  • cycle through all the beams
  • return to the center beam
  • move off source specified by beamwidths
  • take a short scan specified by off secs
  • repeat off for N-S-E-W directions
  • back to central pixel
  • take a short scan of specified length
  • cycle through all the beams
  • turn off the cal, and exit

Latest Fixes and changes 2 Apr 04

Today I changed the folding and spectral line FFT algorithms to use the lags+1 points with data[lags] = 0.0. This will give the frequency channel spacing of bw/n points which was assumed by the old pre-september algorithm. I changed the foldfft routine to do this as well.

Just as I did this today Jon Hagen produced a document describing a rigorous fft for this problem. Let me take the time to understand it - so the algorithm may change again...

Latest Fixes and changes 29 Mar 04

• We discovered a detail with with the signal processing in folding mode. Last September I upgraded to FFTW3 from FFTW2. This change caused a slight difference in the frequecy assigned to the channels. The channel spacing before September 2003 was (bandwidth/N) where N is the number of lags. The channel spacing after September is (bandwidth/(N-1)). The latter is more correct (FFTW_REDFT00) because the old version of the fft was duplicating an extra lag. We made this change without realizing the consequences. Its a small change but extremely important when doing pulsar timing. For this reason, I still recommend that users choose to write lags instead of spectra in folding mode.

  The change to ffw3 occured sometime after September 4, 2003 and on or before September 17, 2003. It probably was Sept 17 as that is the date of modification on the first backup that I have with the fftw3 stuff included.

  Thanks to David Nice et. al. for all the help.

• The old bug in folding mode where you had to switch to Search mode to run monitor has been fixed. In other words. You can click on monitor and it will always assume search mode.
• There is mode called ALFA. Selecting this selects the ALFA pulsar mode. Right now data goes into one file per WAPP with both board's data interleaved. This will be changing. Shouldnt effect single pixel observing, but I wanted to mention it in case it does somehow.
• Non ALFA modes now allow 100 MHz Nine level, 2 polarizations for each wapp. This is new and makes Nine level mode alot more attactive. Tell me if you find a combination where Sanity Check fails in this mode.

Latest Fixes and changes 10 Dec 03

• The main CIMA Pulsar button brings up the WAPP screen and the NON Wapp pulsar window is moved to under Pulsar=>More.
• The controls for 32 bit correlator to folding mode are now in place. Choose 32 bit and either LAGS or Spectra. This is now encoded in the header lagformat variable as a bit mask value 8. So for folding mode lagformat can be LAGS == 2 or spectra == 3 or 10 and 11 when 32 bit lags are dumped and folded. This needs to be tested.

Small change to WAPP start and stop controls

11 Nov. I changed the start and stop controls on the WAPP pulsar observing. Now query observation will return a reasonable message. Also power level setting controls get through so IF system levels can be set when the monitor is running. Careful - this also means that levels can be set while observing. I will leave it for now. I can limit this if it is deemed a problem. Please tell me if these fixes have broken something that I havent tested.

Small change to NON-WAPP pulsar iflo setup

Selecting "non wapp iflo setup" now always adjusts power, if1 and if2 after setting up the iflo. This matches what the old GUI did. Capable observers should be checking power levels under IF/LO Control anyway. Observers who dont check the power will now be saved, usually, depending on where the telescope is at the time of selection.

WAPP Pulsar Header Changed

17 September 2003. The WAPP Pulsar header changed. A complete state of the IF/LO system is now included in the the header. Most software should be unaffected as it only adds a number of values to the end of the header. The version number was increased to 6. (The comments are pretty good, but I will document the details later.)

Data reduction programs that hard-code the header size will break. See WAPP Data File Support below for a description of the Linux library routines that I offer to make it easy to generally find header values and data. I'd be happy to help Solaris users or others having trouble.

And another thing to mention. The folding and spectral line modes now use fftw3 (rather than fftw2) to do the lag to spectra signal processing. (You shouldn't notice except its faster.)

Gigabit Ethernet installed on WAPPs

Gigabit ethernet is now installed on all wapps. A 2 Gigabyte file transfered at 17 Mb/sec using the nfs crossmount. It transfered at 32 Mb/sec using ftp. Like I thought, use ftp and not nfs. August 2003

WAPP errors discovered

17 July 2003. We discovered a serious bug. The WAPPs have not been triggering properly as some observers have guessed. We believe this has been the case for the period:
Tuesday 8 July 9:00 am till 16 July at 6:00 PM.

A series of failures occurred.

• the wappserv computer was re-booted and the trigger control software failed to start. This was a left-over bug from the OS upgrade. Now fixed.
• The GUI failed to report the error. This is now fixed.

I apologize to those who lost data. I'm sure makeup time can be found.

I discovered another bug related to the OS upgrade. There was a timing problem in the edt driver. When this occurs part of the correlator hardware configuration would have waited for an entire driver timeout of 10 seconds before the configuration completed. This was intermittent and never seen in the offline diagnostic software. This bug would have caused the delayed start that we saw. Fixed now. If the wapps were triggering correctly then the start times, although delayed, will reflect the correct starting second.

This may be the elusive "network" problem we have been hunting.

WAPP pulsar setup screen changed

14 July 03. The Apply WAPP Setup button was replaced with two buttons. The Sanity Test and Setup IFLO. The Setup IFLO button should be pressed after making changes to the IFLO setup especially the desired frequencies.

Here is what the buttons do:

• Copy WAPP1 to all - This copies the wapp1 setup to all the wapps. It copies everything except the frequencies including the folding parameters under buttons labeled "Board". It only effects the GUI screen and sends nothing to the wapp hardware.
• Sanity Check - this sends the current configuration to the wapps and checks if the given configuration makes sense to the wapps. It check the modes, speeds, and disk space.
• Setup IFLO - This takes the frequencies and sets the iflo system based on an algorithm specified by Chris Salter. It looks at the frequencies and bandwidths and tries to choose the correct iflo path for the setup. The low-level iflo system assumes the interim correlator center frequency of 260MHz and does not support the wapp center frequencies of 250(100MHz bw) and 275 MHz(<=50MHz bw). The Setup IFLO command adjusts the synthesizer frequencies so that the wapp sees the correct band. The aodisplay screen (upper left) will notice this and show the right frequencies. The black and white engineering screens will show 10 or 15 MHz off as they assume 260 center.

  Another detail of the IFLO setup. The Advanced IFLO setup under the "Receiver IF/LO Control" button is a front end to the ifsetup command. If you know the correct low-level ifsetup command, you can enter it here via the GUI and the WAPP Setup IFLO command will remember it. The frequencies entered on the Advanced IFLO setup screen are the frequencies given to the ifsetup command. The WAPP header gets the ones on the WAPP screen.

  Receiver filter and polarization choices. - Also under Receiver IF/LO Control the is the ability to set the receiver parameters. Simply choose the desired receiver filter or pol (if any for the chosen receiver) and press "send L-Wide Setup" or whatever. Due to limitations in the low level system, the command to set the receiver filters needs to be sent after the ifsetup command, so the Setup IF/LO button remembers and then re-issues the filter and polarization commands.

  Thanks for your patience. We have plans to improve this.

• Start Monitor - Go through the motions to start the wapp running and do everything except write the data to disk. Power levels will appear. Snap will function and you can set the attenuators. This is mostly used to set the power levels before starting. Press Auto Attenuation with monitor is running. To stop the Monitor function, Press "Stop Observation".
• Auto Attenuation - When monitor is running press this to auto-set the wapp attenuators. Other iflo system power settings are reached from the Receiver IF/LO screen. Under the "Manual Power Control" you can manually set the IF/LO power levels. There is a bug. The Attn(cmd) defaults to an incorrect value. Reasonable values are -10 to -25. Then the +1db and -1db buttons will do something sensible.
• Set run time - this is the requested observation time for the wapp observation. Pressing the "Set run time" button will pop-up an entry form. The actual number to the right of the button is a pull down with a few default settings.
• Start Observation - assumes the IF/LO is setup ok. Sends all the parameters and starts the wapp observation. The run time should count down as well as the countdown on the aostatus screen.
• Stop Observation - Stops the wapp observation in progress. If you are running a script like Catalog Observing, this will not stop the script. The script will proceeed to its next task and may start the wapp again. To stop the script, press Abort Observation or End Observation on the AO Observer Display.
• Dismiss - Dismiss the screen. Nothing about the wapp state changes.

New features added to Catalog Observing

10 July 03 Two new keywords were added to the third field in catalog observing. "toward" and "cal_toward". These allow you to start the wapp (with or without cal) while slewing to the next source. Also the caltype (which noise tube) and whether you want the 25Hz cal or the continous noise is now an option on the catalog observing GUI. Details below.

WAPP upgrade complete

7July 03, we completed the upgrade of the wapp os to Red Hat 8.0, 2.4.18-14 kernel. The wapps are up and running. Some details for those interested:

• Each wapp now has its own raid unit.
• Wapps 1 and 2 are at raid level 5 with a spare disk. This means that one of the six disks can fail completely without interruption. To achieve this we give up disk write speed (now at 40Mb/sec) and 50% of the space. (1.5 Tb becomes 1.0 Tb)
• Wapps 3 and 4 are smaller, 500Gb and 700 Gb respectively. Both are at raid level 0. If one of the drives fails the whole drive fails.
• wapp is now called wapp1 so the disk naming convention is consistent. (the name wapp is still aliased)
• slogin to login to the wapps using the password. I may install rlogin or something else but for now use slogin and type the password.
• All the wapp2 disks were copied to the one raid.
• all wapps now have the new 200Mb/sec card from EDT. This allows the dual correlator board upgrades.
• We wanted to put gigabit ethernet into all wapps but the cards we have are not compatible with the wapp motherboards. I don't yet understand why but if we install the gigabit card, (Intel Pro 1000 MT) the motherboard (STL2) wont configure the SCSI properly.
• Wapp2 still has gigabit ethernet (fiber).

WAPP demultiplexiing bug found

For 3 level 50 MHz polarization modes (both 16 and 32 bit) there was a bug discovered in the demultiplexing routines. All previous data taken in this mode is suspect. The data had the correct polarization but not from the same correlator dump. fixed 18 June 2003.

WAPP Folding Mode working

Late June 2003 David Nice, Ingrid Stair and others helped me debug the WAPP folding mode. Should be working. Thanks for the help.

WAPP File Name Convention Changed

On 17 June 2003 we changed the wapp file naming convention. The wapp file names are now, /share/wapp11/project_id.source_name.wapp[234].xxmjd.xxxx The file extension is now 4 digits and is global to all wapps and wapp disks. Now all multi-wapp observations will now start with the same file extension number.

Four WAPPS Operational

Four WAPPs are now available for pulsar observing. Four freqencies over a range of bandwith up to 100MHz (each) are available to observers. The four WAPPs are accessed from the CIMA User's interface. The machines are named wapp wapp2 wapp3 and wapp4. The WAPP capabilities are spelled out here.

All four WAPPS now have reasonable disk space and are usable from the CIMA User's interface. Four WAPPs can be used for observing although there probably remain bugs related to things we havent thought to try. Most people who have tried have had some success. It seems that most trouble comes with configuring the IF system.

The old gui had an Auto-Observe feature and two scripts were used to take some calibration data. These are /share/obs4/usr/pulsar/WAPP.CAL and WAPP.PSR. These are now written into the CIMA executive and are callable from the WAPP Pulsar screen->More

• WAPP.CAL - Pulsar Calibration Script
• WAPP.PSR - Pulsar Pointing Calibration Script

These are in. Ramesh has tested them to his satisfaction. The source names are appended with _north _south etc... If you have other auto-observe features/scripts/parameters that you want to include, tell me I will add them for all to use.

For a while they didnt abort properly. Should be fixed now.

Pulsar Calibration Script -

• turns on the 25 hz cal
• move north 4 beam widths
• take 10 seconds of wapp data
• return to on-source
• take 10 seconds of wapp data
• turn the cal off

• turns on the 25 hz cal
• goes to on-source
• take 10 seconds of wapp data
• foreach direction, north east south west
• move 4 beamwidths
• take 10 seconds of wapp data
• done
• goes to on-source
• take 10 seconds of wapp data
• turn the cal off

Here is a observing script that I wrote for Fernando Camillo. Please use it if its suits your needs. This script will take WAPP observations from a list in a command file. It moves the telescope to the given source and then takes wapp data for the given number of seconds. The observing mode looks at the current source name, trys to find it in the command file and starts the observation there. In this way a command file can be started in the middle.

WAPP Pulsar Catalog Observing-

• Put all of your sources into one catalog.
• Make a command file with three columns,
  • source name ( exactly as in the catalog) For restarting, the source names must be unique.
  • observation time in secs.
  • The third column can be
    • nothing
    • adpwr then the wapp attenuators will be automatically adjusted. It will not adjust other iflo system attenuators.
    • calon, turns the receiver noise cal on for the duration of the scan. It also appends _calon to the source name. The GUI allows you to specify which noise source (e.g. hcorcal) and whether you want the 25 Hz cal or continuous.
    • caloff, same as nothing except it appends _caloff to the source name
    • toward, issues the tracking commands but does not wait. Starts observations immediately. Appends _to to the source name to indicate this. Useful for certain calibrations
    • cal_toward, like toward, issues the tracking commands but does not wait. Turns on the cal as specified. Starts observations immediately. Appends _calto to the source name to indicate this. Useful for certain calibrations (Desh specified this feature)
• Bring up the pointing window, and choose your catalog and point at the first source you want to start at in the command file.
• Bring up the window, Pulsar=>WAPP=>More=>Catalog Observing
• Choose the catalog.
• Choose the command file.
• Configure the wapp and check power with monitor, stop monitor.
• Press Start WAPP Pulsar Catalog Observation.
• If you press "End Observation", the observation will stop when finished with the current source. To restart, you will need to select the pointing screen, point at the desired starting source in the list. Then press Start WAPP Pulsar Catalog Observation.

Late January 2003 the wapp11 raid drive failed completely. We reconfigured it with 4 drives. Its smaller but it is working.

There is a bug in the LO configuration that assumes wapps enabled sequentially. If you wanted to enable wapp four only, for example, do the following workaround:

• enable all wapps
• enter the wapp4 frequency into all wapps
• press apply current setup
• disable wapp1 - wapp3
• press monitor or start observation for wapp4 only

Trouble Shooting Suggestions

In Pulsar mode, if you press "Apply Current Setup", and you dont see "sanity check passed" for each selected WAPP, there is a problem. When other obvious things dont work, try restarting the wapp software. There is a button under Pulsar=>WAPP=>More called "Restart all WAPPs". Another thing to try is to login to the afflicted wapp with "rlogin wapp -l wapp". After login, you can restart the wapp software with "start_wapp".

Its almost never the case that having the operator reboot the wapp will help. But there is one case where I know how to hang the driver so that only a reboot will fix it. Really, I'd like to check it first. With the requsite terabye file-system-check that will result, a cold reboot will take as long as calling me.

Improvements to SNAP

I made a few changes to SNAP. Snap now runs on most any linux machine. Here is how to run it on WAPP4.

xhost wapp4
rlogin wapp4 -lwapp
DISPLAY=observer2:0.1
export DISPLAY
cd /share/wappsrc/snap
snap

Snap was slow because we "upgraded" the van vleck correction routines in the lib area so that it used a more rigorous algorithm. I restored the old Andy Dowd algorithm to snap. Dont believe what snap does to the 9-level data. It will give you an overall reasonable filter shape, just dont believe the fine details. Snap still doesnt display properly in folding mode.

Direct Sampling Available

The new WAPP Direct Sampling mode is available but has yet to be independently tested. For the Mode choose Sampler instead of Search or Folding. The Bandwidth, 50 MHz or less is the only respected parameter. It takes data until you tell it to stop. It does this quietly. ( I need to improve this. Maybe this mode needs its own screen. I dunno.) The data will be saved as search data bit-packed depending on the bw:

2 channels,continuous sampling

Direct Sampled Data Format

The direct sampled data is similar to WAPP search data format. It writes multiple files in 2GB chunks. The binary data follows the header. I'm sure improvements to the header will be necessary. Please suggest some. The data structure from Bill Sisk...

• 1-bit packing Bits 15..0 = ChA Sample 8,B-S8,A-S7,B-S7,A-S6,B-S7,A-S5,B-S7,B-S4,A-S3,B-S3,A-S2,B-S2,A-S1,B-S1
• 2-bit packing Bits 15..0 = ChA Sample 4,B-S4,A-S3,B-S3,A-S2,B-S2,A-S1,B-S1
• 4-bit packing Bits 15..0 = ChA Sample 2,B-S2,A-S1,B-S1
• For processing, the data must be converted to an integer without a dc offset. The equation

  p = [2*b - (2^n)-1 ] 
  

  does the conversion, where b is the n-bit data.
  • For 2-bit data

    p = 2*b - 3
    

    b (binary,decimal)	p (decimal)
    11 = 3	3
    10 = 2	1
    01 = 1	-1
    00 = 0	-3

  
  
  • For 4-bit data

    p = 2*b -15
    

    • b = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
    • p = {-15,-13,-11,-9,-7,-5,-3,-1,1,3,5,7,9,11,13,15}

Notes for Pulsar Observing with WAPP

Here are a few notes for observing with the CIMA and pulsar WAPPs.

• run /home/aoui/bin/cima online on observer2
• Choose the receiver. If you let it, it will automatically do a tsys. Thats ok in the cima gui.
• Choose a source and point at it
• Choose pulsar->wapp and set the configuration as you like it (you can save and restore configurations under, More)
• With the screen showing the desired configuration, press "Apply Current Setup"
• Wait. The apply setup takes a few seconds. There will be a flurry of messages. You should see Sanity Check passed in blue for each wapp configured.
• Press Monitor and look at the power levels. If the power is fubar go to Receiver IF/LO control and look at the main IF power levels first. Choose Auto Adjust as needed. You can set the L-Wide filter and polarization here as well.
• Now back to the WAPP pulsar screen. With the system still in monitor mode, choose Auto Attenuation. You should see the power levels to near 1.0 if all is working.
• Press "Stop Obs" to stop the monitor mode
• Press "Run for" and set the integration time
• Press "Start Observation" and wait for your data.
• Alot of observers like to see the aostatus screen when observing remotely.

I must admit I'm not that thrilled with the human interface of this screen. Please be patient. I will be making changes as they become obvious.

All WAPPs now have digital filters

All four WAPPs now have digital filters installed. These are available for pulsar observing. Bandwidths less than 50Mhz are now supported.

Disk Status/Tape Status

wapp disk drives:

• /share/wapp11 - 700 GB Raid

• The mammoth 2 tape drive is out for repair.
• The princeton DLT drive is fubar as well, use the sun tape drives in the mean time.

• /share/wapp21 - 160 GB Segate
• /share/wapp22 - 160 GB Segate
• /share/wapp23 - 160 GB Segate
• /share/wapp24 - formerly wapp12 160 GB Segate
• /share/wapp25 - formerly wapp13 160 GB Segate
• /share/wapp26 - formerly wapp14 160 GB Segate

• /share/wapp31 - 700GB Raid.

• /share/wapp41 - 700GB Raid

WAPP data file support

• Ascii Header - This is the ascii header as defined at the time of the observation in software. This defines the structure of the binary header and this has no defined size. The structure is called WAPP_HEADER and is (hopefully) well commented as to the meaning of each element.
• Null byte - this marks the end of the ascii header and the start of the binary header
• Binary Header - this is a little endian binary header that is defined by the previous ascii C structure WAPP_HEADER.
• Data Dumps - the data is organized into dumps in time as defined by the number of lags dumped, and the number of IFs dumped (1, 2 or 4). For example, if the header indicated 2 ifs and 256 lags at 16 bits (2 bytes) per dump,
  • 512 bytes Dump 0 If 0
  • 512 bytes Dump 0 If 1
  • 512 bytes Dump 1 If 0
  • 512 bytes Dump 1 If 1
  • 512 bytes Dump 2 If 0
  and so on until the end of the file. For the case of folded data its always 4 bytes floating spectrum or lags.
  • (4 byte) * nlags * nbins, Dump 0 If 0
  • (4 byte) * nlags bin 0
  • (4 byte) * nlags bin 1
  • (4 byte) * nlags bin 2
  • (4 byte) * nlags bin 3
  • ...
  • (4 byte) * nlags * nbins, Dump 0 If 1
  • (4 byte) * nlags * nbins, Dump 1 If 0
  • (4 byte) * nlags * nbins, Dump 1 If 1
  • (4 byte) * nlags * nbins, Dump 2 If 0
  • (4 byte) * nlags * nbins, Dump 2 If 1
  and so on.

There are a number of procedures provided that allow the user to look at this data.

• /share/wapp/utliity/wappstats - This procedure is run on any wapp to give an accounting of the disk and file useage. Use this program to find out where your data is.
• /share/wapp/utility/asciidump file - this procedure parses the wapp header and prints the value of each element of the header.
• /share/wapp/utility/wholedump file - this procedure parses the wapp header and prints the value of each element of the header. Then it prints the data in ascii by IF and dump.
• /share/wapp/utility/ndumps file - show the number of data dumps in the file.
• /share/wapp/utility/showplot - this is a generic TCL plotting routine use