Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.atnf.csiro.au/vlbi/dokuwiki/lib/exe/fetch.php/difx/deller_vdif_support.pdf Дата изменения: Wed Dec 7 23:18:05 2011 Дата индексирования: Tue Apr 12 07:26:24 2016 Кодировка: Поисковые слова: п п п п п п п п

VDIF support in DiFX
Adam Deller ASTRON
5th DiFX workshop, Haystack Observatory


VLBI Data Interchange Format
l

l

l

VDIF is a standard for storage of VLBI data Development began in 2008, version 1.0 agreed upon in 2009 See www.vlbi.org/vdif/

Adam Deller

5th DiFX workshop, Haystack Observatory


VLBI Data Interchange Format
l

Driven by unsuitability of existing formats (Mark4, Mark5B, VLBA, PCEVN) for modern backends, recorders, and correlators
l

l

l

These formats were either based upon or inherited tape-based baggage No possibility for independent subband streams Natural frame sizes poorly matched to UDP/ TCP

Adam Deller

5th DiFX workshop, Haystack Observatory


VLBI Data Interchange Format
l

VDIF allows for a number of independent "data threads"
l

Each of these can contain one or more subbands

l

A data thread is comprised of a series of packets, each with a header
l

Generally, packets are sized to fit within an ethernet frame, although this is not required

Adam Deller

5th DiFX workshop, Haystack Observatory


VLBI Data Interchange Format

Adam Deller

5th DiFX workshop, Haystack Observatory


VLBI Data Interchange Format

Adam Deller

5th DiFX workshop, Haystack Observatory


Status of VDIF support in DiFX
l

Mark5access library (developed by Walter Brisken to unpack VLBA, Mark4 and Mark5B format data) extended to handle single-thread VDIF data
l l

Situation very similar to existing formats mark5access simply unpacks - the responsibility for ordering the packets etc rests with the caller, in this case mpifxcorr datastream mark5access

Adam Deller

5th DiFX workshop, Haystack Observatory


Challenges for VDIF in DiFX
1. 2. 3.

Complex-sampled data Out-of-order and/or missing packets Multiple data sources for a single station

Adam Deller

5th DiFX workshop, Haystack Observatory


1. Complex-sampled data
l

Now supported (not all sampling precisions, but the most common)
l

courtesy work by Chris Phillips

l

Required work in correlator depths (unwinding many implicit assumptions of Nyquist sampling of real data)
l

Ultimately, complex data is actually simpler and more natural for mpifxcorr to handle

l

Probably not yet bullet-proof

Adam Deller

5th DiFX workshop, Haystack Observatory


2. Out-of-order/missing packets
l

Need a pre-buffering stage to sort packets and pad missing data
1.

2.

Could make this external to mpifxcorr, and pass the result on via TCP (simple interface from the mpifxcorr point of view) Or, could just add an extra buffer inside the existing Datastream process inside mpifxcorr

l

Closely related to the following problem...

Adam Deller

5th DiFX workshop, Haystack Observatory


3. Single station, multiple sources
l

l

Already supported by mpifxcorr for 5 years! But... Configuring this via vex2difx is not currently possible
l

l

Implicit assumption of a 1-1 match of "datastream" to "station" Unwinding this is a large task

l

Worse - mpifxcorr expects independent data sources for each datastream
l

Here, packets from different threads may be interleaved into one stream - need to share
5th DiFX workshop, Haystack Observatory

Adam Deller


Solutions 1. The ideal
l

A 1-to-many demultiplexer
l

l

Read a single VDIF stream in to a buffer, reorder it and split into N separate streams, send each stream on to a different location Solve both problems in one step

l

Could go even further; why restrict to VDIF format inputs?
l l

On-the-fly reformatting Could vastly simplify the interface to mpifxcorr, and allow frequency division multiplexing for non-VDIF formats
5th DiFX workshop, Haystack Observatory

Adam Deller


Solutions 1. The ideal
l

A 1-to-many demultiplexer
l

l

l

been begun, but virtually untested, Couflar gom veaturercoer;lete,y nestrict to d fro e f en fu thmp wh a r d VDItotfolrmatnitnputs? with vex2difx F al y uni egrated
l l

Read a single VDIF stream in to a buffer, reorder it and split into N separate streams, send each stream on to a different location A preliminary "VDIF server" has Solve both problems in one step

On-the-fly reformatting Could vastly simplify the interface to mpifxcorr, and allow frequency division multiplexing for non-VDIF formats

Adam Deller

5th DiFX workshop, Haystack Observatory


Solutions 2. The distasteful
l

l

Solve the most common problem trying to correlate a station with multiple identical VDIF threads (same # bits, etc) against "normal" stations Easiest - buffer inside Datastream, and re-multiplex back to a single VDIF thread
l l

Oh, the aesthetic pain! But; the infrastructure to deal with the output already exists, so this has been implemented
5th DiFX workshop, Haystack Observatory

Adam Deller


Solutions 2. The distasteful
l

l

Solve the most common problem trying to correlate a station with mumulte lidenticaFVDavathreads a same # ltiplip exedVDIl is IF ilable as ( mode i against "normal" sta is bits, etc)n DiFX-2.0.1 onwards. Ittions Eaintegrated weh vesidefx. aHostrear, , and siest - buffit r in x2di D ta weve m ims ltiplyet extck sivea singld, VDd F t i unot ex ba en to ly testee an I rehe thretad syntax to enable it is clumsy and long-winded.
l l

Oh, the aesthetic pain! But; the infrastructure to deal with the output already exists, so this has been implemented

Adam Deller

5th DiFX workshop, Haystack Observatory


Recap: VDIF status
l

l

l

Single-thread VDIF, both real-sampled and complex-sampled data, is supported in DiFX Multiple-thread data is supported under restrictive circumstances The ideal future would include an uberdata reorderer external to mpifxcorr, but this is not yet on the agenda
l

But this is ultimately the best way forward to flexible, frequency distributed correlation
5th DiFX workshop, Haystack Observatory

Adam Deller