Introduction
PUPPI is the Puerto Rico Ultimate Pulsar Processing Instrument, the newest backend at the Arecibo Observatory. It is a clone of GUPPI, the Green Bank Ultimate Pulsar Processing Instrument, and this page is a near-clone of the GUPPI support guide.
The software running on PUPPI to record data is also the same as what runs on GUPPI. The names of various commands described below containing "guppi" remain the same when running on PUPPI.
If you are a GBT user, you would have used GUPPI via Astrid, the GBT telescope control software. At Arecibo, we expect to integrate PUPPI control into CIMA very soon. Until then, you will be running CIMA to point to a source and track, and control PUPPI from the command line to take data. This is easy!
PUPPI Hardware
Analog Components
- IF conditioner - The initial signal path input to PUPPI. This contains amplifiers and attenuators to set input power levels appropriately. There are no software-controllable settings.
- Clock synthesizer - The synthesizer is used for the main PUPPI ADC clock. It should be set to one of the allowed PUPPI bandwidths (800, 200, or 100 MHz). The synth is locked to the site 10 MHz maser reference frequency. The synth frequency is set via software control.
Digital Components
- ADCs - Two CASPER iADC boards, one per polarization. The ADCs are run in interleaved mode (so clock rate = bandwidth), and sample with 8 bits. Each ADC board is attached to an IBOB.
- IBOBs - CASPER single-FPGA boards. For PUPPI, the IBOBs simply read the ADC output and send it to the BEE2 over XAUI connections.
- BEE2 - CASPER FPGA board containing 5 FPGAs. The BEE2 reads sampled data from the IBOB, and splits it into frequency channels using a digital (polyphase) filter bank. In some modes, the spectra are detected and accumulated in the FPGA, while in others the channelized voltage data are output directly. The output is over four 10-gigabit ethernet lines to the PUPPI 10GbE switch.
- Power strip- A software-controllable power strip that can be used to power-cycle the IBOBs and/or BEE2.
- Power-cycling the IBOBs is done routinely whenever the clock frequency is changed (for different bandwidth modes). This takes ~5 seconds.
Computer Hardware
- 10GbE switch 24-port Fujitsu 10 gigabit network switch. The BEE2, puppimaster, and the gpu cluster are all connected to this.
puppimaster
- The main computer used to record PUPPI data and monitor observations.puppimaster
is visible on the main AO network, and login is via your normal AO account.puppimaster
has a 41 TB raid.gpu01
throughgpu09
- The GPU cluster is used to process and record data for coherent dedispersion mode.- Each node contains an NVIDIA GTX 285 graphics card, and has CUDA installed for programming it.
- Each also has a 5.5 TB RAID for data recording, mounted as /data.
- There is only a single user account, called 'gpu'. Access is controlled via SSH keys.
- These machines are not visible on the main AO network, and can only be logged into via
puppimaster
. - Several shared directories (eg. /home/gpu) are NFS-mounted from
puppimaster
.
Software Components
- Controller- Internal software component responsible for getting/setting HW and SW parameters for PUPPI.
- Python-based, written mainly by R. DuPlain and P. Brandt.
- Main server runs on
puppimaster
. Also has processes running on GPU nodes, and the BEE2. - Client programs connect to the controller to get/set parameters. A python-based interpreter (the "guppi prompt") is one such program, useful for debugging and setting PUPPI parameters outside of CIMA.
- This code loads/configures FPGA personalities on the BEE2.
- Code available at github - http://github.com/nrao/guppi-controller
- DAQ software- also called
guppi_daq
, this code is responsible for receiving, processing and recording the data.- C-based, written mainly by P. Demorest and S. Ransom
- Runs on
puppimaster
and on gpu nodes. - Uses shared memory segments for: storing observing parameters ("status shared memory"), and ring buffers for data processing ("databuf shared memory")
- Also available on github - http://github.com/demorest/guppi_daq.
- In addition to these major components, several smaller scripts are always running in the background on
puppimaster
:- supervisord - This daemon is responsible for (re)starting all the other software components.
- gbtstatus_loop - Periodically reads info from the GBT status database and fills it into the guppi_daq status shared memory.
- guppi_gpu_push - Distributes configuration info from
puppimaster
to the GPU nodes.
Observing Modes
PUPPI has two main modes of operation, incoherent filterbank mode, and coherent dedispersion. Each has a large amount of flexibility in terms of number of channels, etc. Baseband recording is also possible but is still under testing.Incoherent Mode
Sometimes this is also called "search mode" or "filterbank mode". This mode is just like a normal spectrometer except the integration times are very short (typically tens of us). In this mode, the spectra are accumulated in the FPGA hardware, then output over the network and recorded to disk onpuppimaster
. Observing instructions for incoherent mode.
The available parameters are:
- Total bandwidth: 100, 200, or 800 MHz
- Number of channels: Any power of two from 32 to 4096.
- Output: 8 bits only.
- Polarizations: Total intensity (summed) or full-Stokes. 4096-channel mode is total intensity only.
- Minimum spectrum integration times:
- For 100 and 200 MHz bandwidth the minimum integration is given by 4 * (# channels) / BW.
- This is a FPGA hardware limit.
- e.g., for 200 MHz, 2048 channels, minimum integration time is 40.96 us.
- In "Fast4K" mode the minimum is 2 * 4096 / BW.
- For 800 MHz, the minimum is 16 * (# channels) / BW. This is a disk write speed limit (max 200 MB/s to the RAIDs on
puppimaster
).
- For 100 and 200 MHz bandwidth the minimum integration is given by 4 * (# channels) / BW.
- Maximum spectrum integration time:
- The accumulation length (T_int * BW / N_chan) should not exceed ~1000 (typically ~ms).
- The 8-bit output values do not have enough dynamic range for this.
- Data format: Search mode PSRFITS.
- There is a real-time folding mode available, but it is hardly ever used now due to the availability of coherent dedispersion mode. An exception is when observing the pulsed noise cal, the data are often folded in real time at the 25 Hz standard cal period.
- Note that in this mode, the true achievable time resolution depends not just on PUPPI setup but also on dispersion smearing. This in turn depends on the pulsar's DM and the observing frequency.
Coherent Mode
For observing known pulsars, a dispersion measure-specific filter can be applied to completely remove dispersion smearing.This is often done for MSP timing. In this mode, voltage data are output from the FPGA hardware to the gpu cluster computers. The GPUs apply the dedispersion filter and either fold or integrate in real time. Observing instructions for coherent mode.- Total bandwidth: 100, 200, or 800 MHz
- Number of channels: Powers of two from 32 to 2048.
- Polarization: Total intensity or full-Stokes.
- Fold mode:
- Folds the data modulo the current pulse period in real time.
- Outputs time-integrated profiles, minimum dump time is ~1 second.
- Output is floating point profiles in PSRFITS format.
- The time (or pulse phase) resolution is given by N_chan / BW.
- Cal mode: Special case of fold mode, folds at a constant 25 Hz period for the cal signal.
- Coherent search mode:
- Coherently-dedispersed spectra are accumulated in time rather than folded.
- Spectrum integration time is subject to the same total data rate limit (200 MB/s max) as incoherent mode.
- Data are re-quantized to 8 bits and output in search-mode PSRFITS format
- Note, for all types of coherent dedispersion observations, each gpu node processes and independently records 1/8 of the total BW. The 8 sub-bands are assembled into a single output file after the observation is over.
Baseband Recording Mode
For some applications, e.g. cyclical spectroscopy of pulsars, raw voltage can be recorded with PUPPI. In practice, the desirable data rate is too high for only one GPU to do the data recording, so data taking uses a relatively small number of very wide channels that get split between GPUs. The user then recombines them to recover the raw voltage data stream. Observing instructions for baseband mode.- Total bandwidth: 100 or 200 MHz
- Number of channels: 32 or 64
Typical Setups
While there are a potentially large number of combinations of BW, # channels, etc available, a few setups get used much more than others:- 800 MHz BW, 2048 channels, 40.96 us, incoherent. This is used for searches at higher frequencies (L-band, S-band).
- "Fast4k" - 100 MHz BW, 4096 channels, 81.92 us, total intensity, incoherent. This is used for low-freq searches (350 MHz; GBNCC, etc).
- 800 MHz BW, 512 channel, coherent. For MSP timing at L-band and
higher.
- Fold mode is usually used for single pulsars (eg NANOGrav timing).
- Search mode is used for multiple pulsars in the same beam (double pulsar, globular clusters).
- 200 MHz BW, 128 channel, coherent. For MSP timing at 820 MHz.
- 327 MHz receiver: 327 MHz center, 100 MHz BW.
- 430 MHz Gregorian receiver: 430 MHz center, 100 MHz BW.
- L-wide receiver: 1380 MHz center, 800 MHz BW.
- S-low receiver, bottom half of band: 1700-1880, 2040-2400 MHz filters, 2030 MHz center, 800 MHz BW.
- S-low receiver, top half of band: 2600-3100 MHz filter, 2030 MHz center, 800 MHz BW.
The puppi_observe
script
Until PUPPI can be controlled from CIMA, puppi_observe
is
the script that an observer runs on puppimaster
in order
to take data with PUPPI.
- Usage:
puppi_observe [options] parfile
- Options:
-h, --help
(Show this help message and exit)--test
(Test mode - don't actually start an observation)-c, --cal
(Take a cal scan)-b BAND, --band=BAND
(Observing band: 327, 430, L, S. Default: L)-f FREQ, --freq=FREQ
(Override default freq)-T LENGTH, --scan-length=LENGTH
(Total scan length in seconds. Default: 90 for cal, 3600 otherwise)-m MODE, --mode=MODE
(Observing mode: COHERENT_CAL, COHERENT_FOLD, SEARCH, CAL, or RAW)-n NCHAN, --nchan=NCHAN
(Override default number of channels)-w BW, --bw=BW
(Override default bandwidth)-p POL, --pol=POL
(Override default polarization)-P PROJID, --projid=PROJID
(Project ID number, e.g. P1234)-I, --onlyI
(Record total intensity only. Default:full-Stokes)
Software tools for PUPPI
- GUPPI environment: this is set up via the bashrc file that is read in when you log in to
puppimaster
. guppi_status
- Run onpuppimaster
or gpu nodes, this shows a text window of the current status shared memory. Look for the observer's name, project id, source name, etc to be set correctly on the screen.guppi_status
gets updated once approximately every second and so things which should normally change at that cadence should be changing (such as LST, AZ, ZA). If you only see about half of this information or if the information is stale, see the troubleshooting section below.guppi_gpu_status
- Run onpuppimaster
, shows an overview of the status of all gpu nodes.guppi_adc_hist
- Run onpuppimaster
, shows a histogram of count values from the PUPPI ADCs.guppi_monitor
- Run onpuppimaster
, shows a real-time bandpass display only for incoherent modes.pav
andpsrplot
- Run onpuppimaster
or gpu nodes to view data file contents. These are data display programs that are part of the PSRCHIVE data analysis software. These will only work on fold-mode or cal-mode files.
Troubleshooting
If PUPPI appears not to be working correctly, there are several things to check, depending on what exactly the issue is.guppi_status
is not updating every second or has old or incorrect looking data in it
The guppi_status
daemon on puppimaster
needs to be restarted. This can only be done by root on puppimaster
via the supervisorctl command. Call Phil or Arun.
PUPPI is dropping a lot of packets in an incoherent mode
If guppi_status or the guppi log files show that a significant fraction of the data are being consistently dropped, there can be several issues.- You may be running low on disk space on
/data
. use "df -h" to check and re-configure with the other data directory "guppi.datadisk" and re-start the scan. - There could be data transfer happening by one or more people (using "rsync" typically) which are affecting the available I/O rate. Use "ps aux | grep rsync" and/or "top" to see if there are any active rsync users, especially those with an associated sshd process as well. Using the ssh-protocol accidentally over the 1gig (as opposed to 10gigE) network for internal AO transfers can easily cause many dropped packets. Also check to see if the processes are "nice'd". If you find this, either call Phil or Arun, or even better, contact the person running the rsync to have them stop.
- There may be people doing significance data analysis using the same
/data
directory that you are writing to. You can either contact the person and have them "nice" their processes or ask Phil or Arun to kill the processes. Note that data processing by 1 or 2 processes, properly niced does not usually cause many dropped packets.
In a coherent observation mode, one or more of the GPU nodes is not working
The first thing you should do is decide if you want to abort your observation or not in order to fix the problem (which may or may not be fixable). If you decide to do it, here are a couple steps to take:- Can you ping the node in question? If so, ssh to it as gpu (e.g.
"ssh -l gpu gpu01"
) and runguppi_status
on the node. If the status looks good, and is updating, there is probably a problem with PUPPI itself or another hung GPU node. You can check for the latter by checking the uptime's (for instance) on all the GPU nodes:- Try
"gpu_broadcast uptime"
or equivalently"gpu_run uptime"
. If any of the uptimes are very short, it could indicate that a node crashed and re-started incorrectly. If that has happened, the guppi_status screen will not be complete or updating properly. You can restart the GUPPI daemons on a node using the command"/home/gpu/gpu_run"
as root. Note that this requires Phil or Arun.
- Try
- If you can't ssh into the node, the node needs to be (properly) rebooted. You can ask the operator to hit the reset switch on the GPU node. When it reboots, ssh to it as gpu and check
guppi_status
to see if it looks OK. - If
guppi_gpu_status
shows one of the nodes in a strange mode, perhaps with one of the statuses as "Unknown" or in a state very different from the other nodes (like "Exiting" when all the others are running), the GPU node may be in a strange hung state. If this happens, the node may need to be rebooted. Call Phil or Arun, or in an emergency, ask the operator to simply hit the reset switch.