Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.adass.org/adass/proceedings/adass96/reprints/seamanr1.pdf Дата изменения: Thu Jan 15 01:00:54 1998 Дата индексирования: Tue Oct 2 14:56:24 2012 Кодировка: Поисковые слова: http astrokuban.info astrokuban

Astronomical Data Analysis Software and Systems VI ASP Conference Series, Vol. 125, 1997 Gareth Hunt and H. E. Payne, eds.

WIYN Data Distribution and Archiving
Rob Seaman IRAF Group,1 NOAO,2 PO Box 26732, Tucson, AZ 85726 Ted von Hipp el U. Wisconsin/WIYN Abstract. The NOAO/IRAF Save the Bits archive has b een op erating for over three years at Kitt Peak National Observatory and at the National Solar Observatory's nighttime program. Since that time, the W. M. Keck Observatory and the Cerro Tololo Inter-American Observatory have also adopted the software. These first generation Save the Bits installations rely on Exabyte tap es as the archival medium, typically using pairs of drives to produce duplicate copies of the data for heightened protection against data loss. The upgrade of Save the Bits that is currently in progress to supp ort writable CD-R drives is discussed. In addition to another media option, this expands the role of the package to include data distribution as well as data archiving. Dual CD-R copies are produced as with tap es. One copy is retained for archival purp oses, but the second copy of each nightly CD is released to the appropriate institution as the principle means of data distribution from the telescop e. The four individual institutions are free to handle their copy of the data in any appropriate way, such as by mounting the disks into a jukeb ox as they are received. Both raw and mountain-reduced data are included in random access FITS files on the ISO 9660 CD-ROMs. Planned future improvements include supp ort for DVD format disks. Save the Bits is freely available to outside institutions and is straightforward to install and manage. Hardware requirements are minimal and other storage media should b e straightforward to supp ort.

1.

Introduction

The WIYN Consortium consists of the University of Wisconsin, Indiana University, Yale University and NOAO--the National Optical Astronomy Observatories. The Consortium manages the 3.5 m optical alt-azimuth WIYN telescop e on
1

Image Reduction and Analysis Facility, distributed by the National Optical Astronomy Observatories National Optical Astronomy Observatories, operated by the Asso ciation of Universities for Research in Astronomy, Inc. (AURA) under co operative agreement with the National Science Foundation.

2

306

© Copyright 1997 Astronomical Society of the Pacific. All rights reserved.


WIYN Data Distribution and Archiving

307

Kitt Peak near Tucson, Arizona. The WIYN telescop e supp orts two primary facility instruments, a wide field CCD imager and the Hydra multi-ob ject sp ectrograph. These instruments are mounted at the two Nasmyth foci of the telescop e allowing b oth instruments to b e used throughout a given night. The primary mirror system takes advantage of modern active optics technology through 66 separate actuators that push or pull on the back face of the mirror to maintain the b est p ossible optical figure. A thermal control system maintains the surface of the mirror within 0.2 C of the ambient air temp erature, eliminating mirror seeing, which is caused by turbulence in cool air over a warmer mirror surface. The motivations for a WIYN Archive and Data Distribution System are to enhance the total science output of the telescop e over its lifetime and to ease the data distribution and handling process at all four WIYN institutions. An archive increases the total science output of the telescop e as investigators attempt new problems with old data, and as statistically large samples of different classes of ob jects are accumulated, often from pro jects initially undertaken for a wide variety of purp oses. Indeed, as a large digital imagery and sp ectroscopy library is accumulated, many new uses will b e made of it. One can exp ect not only statistical studies of commonly observed ob jects, but also test reductions and analyses by observers contemplating new pro jects. Currently WIYN data are recorded by the observers on Exabyte or DAT tap e using IRAF tasks. While this procedure does store data, it does not allow for easy recovery or dissemination of the data. Since tap es do not allow random access, since they may not last more than a few years even if carefully stored, and since every observer writes their data to tap e differently and has different styles of logging their observations, the result is a cumb ersome and heterogeneous data dep ository. Even the original investigator must load their data to disks, often rep eatedly, and often they must spread their data across many disks in order to have access to an entire run at one time. The system describ ed here is based on an automatic and homogeneous data storage process using random-access media (CD-ROM). The long-term goal is a robust and easy to use WIYN data archive. The near-term goal is a robust and easy to use data distribution system. The raw data CD-ROMs are mounted in jukeb oxes at the institution of data ownership and are immediately and continuously readable without transferring the data to hard disk. Additionally, CD-ROMs are stable media and the data are in a standard format. Observers relying on this mechanism exclusively will b e resp onsible for verifying the content of the archival media containing their data. Tools will b e provided to check the image headers and display the images directly from the archival media. Additionally, NOAO backs up all data taken at WIYN and with the KPNO telescop es using Save the Bits. This software automatically stages CCD data taken at all Kitt Peak telescop es to a single Sun workstation and its disks, then writes these data as FITS files to Exabyte tap e. 2. System Overview

The physical approach prop osed is to write WIYN data to two CD-ROMs (for redundancy) simultaneously using two CD-ROM writers, mounted on a dedicated Sun workstation using a version of Save the Bits. A separate directory will b e written for each instrument (currently Hydra and the Imager) on each CD-ROM as well as directories for mountain-reduced data separate from the


308

Seaman and von Hipp el

raw data. The data files will b e copied into these directories as individual FITS files. The data will b e staged to a hard disk b efore writing the CD-ROMs, and the resulting CD-ROMs verified against the hard disk data set and against each other. The mountain-wide Save the Bits archive will also still back up WIYN data. At some p oint during each day the (now archival) CD-ROMs will b e ejected from the drives, such that individual CD-ROMs b elong to individual nights, and thus to individual institutions. One copy of each night's data will b e kept at NOAO, and the other copy returned to the institution which owns the data. For the CD-ROMs which return to the Universities, the media can b e mounted in a jukeb ox and the observer will effectively have new disk space with their observations. For the CD-ROMs which are stored at NOAO, the media will b e available as a backup and for rep eated duplication and distribution to the requesting observers. The advantages of CD-ROMs as the archive media are multiple: they are dep endable, easy to store and use, they are random access media, and they are forward compatible with the next generation of DVD "CD-ROMs." Random access is imp ortant for an archive as well as for data distribution purp oses, as it greatly decreases access time and allows someone to recover, copy and distribute data from widely disparate media locations. Forward compatibility also seems assured with the next generation of higher density DVDs as the principal manufacturers have agreed to this. The ma jor disadvantage of CD-ROM as an archival media is that a single disk only holds 650 MB. Based on current rates of data taking, this data quantity is large enough, however, such that one CD-ROM will b e sufficient to hold the data from a single night on 60% of all nights, while three CD-ROMs will only b e needed approximately ten nights a year. The software to p erform the archiving will b e based on Save the Bits, which will b e modified to archive only WIYN data, and to write to CD-ROM writers. Save the Bits will also p erform the function of creating a growing index file from the headers of all archived data. This will allow for simple searching and p ostprocessing programs to b e written. Save the Bits is proven software which has preserved more than one terabyte of data from seven telescop es at Kitt Peak over the last two and a half years, and it is also b eing used at CTIO and at Keck. This software needs only minor adaptations, its use requires minimal interaction from mountain staff, and it already can handle a numb er of contingencies, including system crashes. 3. Daily Maintenance Chores

The archival system requires the following tasks, generally once, at some p oint b etween dawn and mid-afternoon: 1. check the monitor program, bitmon, to see that the archive system wrote a good pair of CD-ROMs, 2. eject, date-stamp, and log the just written CD-ROM pair, and 3. load new blank CD-ROMs into the CD-R writers.


WIYN Data Distribution and Archiving

309

In the event that one or b oth of the CD-ROM disks of the pair were bad, as indicated by the verification passes of Save the Bits, new blank CD-ROMs are inserted into the drives, and that dataset is re-archived from the staging disk. Occasionally two (and less often three) CD-ROM pairs p er night of data will b e required, and the ab ove process will have to b e rep eated a second (or even a third) time during a given 24 hour p eriod. The system can b e allowed to fall b ehind by several CD-ROMs, if necessary, though this is not desirable. At the end of each run, the CD-ROM disks will b e transferred downtown for a final read verification using a separate CD-ROM drive. This will also provide the opp ortunity to read detailed observing run information from each disk which will b e used to generate an informative lab el to b e printed onto each disk using a thermal printer designed to print directly to normal CD media. References NOAO/IRAF Save the Bits archive3 WIYN Consortium4 Exabyte Corp oration5 CD Information Center6 Stinson, D., Ameli, R. & Zaino, N. 1995, Lifetime of KODAK Writable CD and Photo CD Media7 DVD: Inside Story8

3 4 5 6 7 8

http://iraf.noao.edu/pro jects/stb http://www.noao.edu/wiyn/wiyn.html http://www.exabyte.com http://www.cd-info.com http://www.cd-info.com/CDIC/Technology/CD-R/Media/Ko dak.html http://www.sel.sony.com/SEL/consumer/dvd/index.html