These instructions are for data transfer from Parkes (pam-store-ext, pkvsi1-ext, pkvsi2-ext), ATCA (cavsi1-ext, cavsi2-ext, caxcube-ext), Mopra (caxcube-ext, mpvsi1-ext, mpvsi2-ext) and ASKAP (cira10).

Requirements:

• ssh access to venice.atnf.csiro.au (ATNF Unix account)
• vlbi password
• access to cass-01-per.it.csiro.au
• access to magnus-data.pawsey.org.au

ssh authentication key setup:

In order to use globus GridFTP to push data to Pawsey via your own user account, you will need to have ssh keys set up. On each recorder host that you need to transfer data from (iff each has its own independent home directory), use ssh-keygen to generate your own ssh keys with a passphrase that you can remember. ssh-keygen will prompt for a filename to save to; use a new unique file in the ~/.ssh area, e.g. òÀÜmyname_rsaòÀÝ. Append the public key to your authorized_keys on magnus with ssh-copy-id -i ~/.ssh/myname_rsa.pub user@magnus-data.pawsey.org.au (this will prompt you for your Pawsey account password). After this you should be able to ssh from the recorder host to magnus without needing to enter your password.

Optionally, to manage your keys on the recorder host type ssh-agent bash and ssh-add ~/.ssh/myname_rsa. ssh-add will ask for your passphrase. After this you should be able to log into magnus without being prompted for your passphrase. To avoid having to repeat this every time you log in again, you can use ssh-agent within screen (man screen for info on screen), and re-attach to the screen to continue transfers when you next log in.

After each run (including mini-runs):

• Using the Current Block Schedule on the wiki or web page, check experiment names and telescopes which took part.
• Update the google docs spreadsheet population from the recmon database. Hover the mouse over the A1 cell to view instructions on how to do this.
• Using the above google docs spreadsheet (or recmon disk label view) work out which experiments were recorded where - note some experiments may be recorded on multiple locations.
• For each station, and for each disk array, transfer data to magnus-data (see below). Only transfer from one of At, Pa, Mp at a time (and from one machine).
• ASKAP can be run in parallel with an east coast telescope.

How to transfer data:

• ssh to venice (using your own ident) then to the data host (one of the above machines) as vlbi to push the data over. It's best to run transfers within screen - see above.
• See the notes on how to use globus-url-copy to transfer directories to magnus-data.pawsey.org.au - can create an alias to save time entering all the options [UPDATE]. You can also use parallelisation, e.g. '-p 2', to speed up transfers. (Note that the òÀÜgloPut7T.shòÀÝ script is now obsolete.) Also note the environment variable $GLOBUS_LOCATION should be set. (Add to .mybashrc and source this from .bashrc, if not already there.) Note the -udt option is typically not possible when transferring from the observatories.
• Data at the Pawsey centre takes the structure /scratch/pawsey0169/transfers/{year}/{experiment}/{antenna}
• Note that for dual-DAS recording, transfer DAS1 and DAS2 data into {experiment}/{antenna}/{DASn} directories.
• You can run globus-url-copy a second time (be sure to include the -sync option) to check everything transferred OK.
• For large disk arrays in particular, rather than transferring data for one experiment at a time it is often more efficient to move the data to be transferred to a single directory òÀÜtransferòÀÝ (on the observatory disk), then copy the whole lot to /scratch/pawsey0169/transfers/2016/{antenna} on magnus-data.
• Once the whole lot is transferred, the data can be moved from {antenna}/{experiment} to the correct {experiment}/{antenna} directories on magnus-data. To help with this rearrange, Chris has a script ~cphillips/bin/movedir.sh which works with alias 'movedir' (to be run from within the experiment directory where the data are currently located). To use it, you can add alias movedir='. ~cphillips/bin/run_movedir.sh' to your own ~/.profile or similar, and add Chris' ~/bin directory to your path (add the line export PATH=/home/cphillips/bin:$PATH to your .bashrc).
• Use the script filesum.sh (~cphillips/bin/filesum.sh at Pawsey) on both the copy and the original data at the observatories to compare the two sets of data (check that total data volume, #files, smallest and largest files and first and last are the same). Once this comparison is made, use alias òÀÜmovediròÀÝ on the original data (at the observatories; the alias is already set in the vlbi account) to move it to a directory called òÀÜcheckedòÀÝ.
• Update the google docs spreadsheet to indicate data transfer has occurred.

Note: Data can also be transferred directly to cass-01-per.it.csiro.au

Data recorded by ATNF telescopes for RadioAstron experiments go to the subdirectory transfers/2016/radioastron/{experiment} at Pawsey. The script for moving RadioAstron data to the correct subdirectory structure can be accessed using alias movera='. ~cphillips/bin/run_movera.sh'

The data are transferred to ASC for correlation using the tsunami transfer protocol. Set up the tsunami server and let Dmitri know [TO BE UPDATED].

Q: Are the notes below still relevant?

As root:

> apt-get install build-essential bzip2 autoconf libxml-parser-perl xinetd openssl telnet
> adduser arcs
> mkdir /opt/globus-4.2.1
> chown arcs:arcs /opt/globus-4.2.1

Add the following to the appropriate interface in /etc/network/interfaces

      up route add -host pbstore.ivec.org gw 202.158.222.113
      down route del -host pbstore.ivec.org gw 202.158.222.113
      up route add -host arcs-df.ivec.org gw 202.158.222.113
      down route del -host arcs-df.ivec.org gw 202.158.222.113

Copy .ssh/authorized_keys to ~arcs/.ssh/ from arcs@cavsi2