Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.stsci.edu/stsci/meetings/lisa3/reprints/murtaghf.pdf Äàòà èçìåíåíèÿ: Fri Sep 11 18:14:02 1998 Äàòà èíäåêñèðîâàíèÿ: Sun Dec 23 02:46:31 2007 Êîäèðîâêà: Ïîèñêîâûå ñëîâà: http www.astronomy.ru forum index.php topic 4644.0.html

Library and Information Services in Astronomy III ASP Conference Series, Vol. 153, 1998 U. Grothkopf, H. Andernach, S. Stevens-Rayburn, and M. Gomez (eds.)

Distributed Information Search and Retrieval for Astronomical Resource Discovery and Data Mining
Fionn Murtagh Observatoire de Strasbourg, Universit´ Louis Pasteur, Strasbourg, e France; Faculty of Informatics, University of Ulster, Londonderry, Northern Ireland Damien Guillaume Observatoire de Strasbourg, Universit´ Louis Pasteur, Strasbourg, France e Abstract. Information search and retrieval has become by nature a distributed task. We look at tools and techniques which are of importance in this area. Current technological evolution can be summarized as the growing stability and cohesiveness of distributed architectures of searchable ob jects. The ob jects themselves are more often than not multimedia, including published articles or grey literature reports, yellow page services, image data, catalogs, presentation and online display materials, and "operations" information such as scheduling and publicly accessible proposal information. The evolution towards distributed architectures, protocols and formats, and the direction of our own work, are focussed on in this paper.

1.

Needs and Existing Solutions in Astronomy

Astronomy is characterized by massive data collections which are often incompatible, from many different sources, and stored in many different ways [1]. The user ­ astronomer or librarian ­ needs access in a common environment. The traditional solution to this issue is to have centralized databases available. This provides the common environment needed, and helps with quality control. Ma jor databases can be considered, such as Simbad [2], NED [3] and LEDA [4] for astronomical ob jects, VizieR [5] for tables or ADS [6] for astronomical articles. Collecting data, storing them with one format in one database, and providing user interfaces, all require very considerable human resources. For the user, problems may arise when accessing data from different databases, because the data on a given sub ject reside on different servers with different metainformation and formatting. These problems are partially overcome by linkages between the data collections. For example, links between Simbad, NED, VizieR and ADS allow retrieval of articles based on an ob ject which was cited. It is also possible to use a more direct protocol with NED and Simbad to retrieve information about an ob ject, where one system can send a request to another and then format the response in an appropriate way. Such peerto-peer linkage is used by different servers, for example, to find all names for 51

© Copyright 1998 Astronomical Society of the Pacific. All rights reserved.


52

F. Murtagh and D. Guillaume

an astronomical ob ject. When such peer-based protocols are used, the overall system is commonly referred to as a federated database. To improve the flexibility and the reliability of HTTP links, a package called the Uniform Link Generator, GLU, has been developed to keep a distributed list of available data, based on their URLs [7]. GLU tags are used to reference data, and these tags are resolved into URLs automatically. A standard for passing parameters to CGI [8] programs (Common Gateway Interface, the common means of providing a Web server with the ability to execute programs to carry out any desired task) for astronomy has been defined, and is called the Astronomical Server URL, ASU [9]. This standard extends a URL which points to a catalog or database to allow access to specified parts of these datasets. Such access requires a program to be run. In the case of a database, this program involves the submission of an SQL (Standard Query Language - the most usual interface language for relational databases) query. Text indexing and retrieval is often supported by public domain toolsets, which may support the Z39.50 information retrieval protocol or some related but different variant of this protocol. Isite and Isearch are part of one such package which is available from the Center for Networked Information Discovery and Retrieval (CNIDR). The context of such indexing and search toolsets is set out in Murtagh [10]. Gateways to Web servers are available for such text-supporting packages, which replace the erstwhile WAIS user interfaces. 2. Metadata - Descriptive and Content Markup

Today's intelligent agents and knowledge discovery programs in astronomy are restricted to a small amount of data, because of the lack or heterogeneity of metadata. These programs need to have data described formally and precisely, ideally in a standardized format, to allow subsequently for statistical processing and cross-correlating of documents. Currently, all the ma jor steps in the evolution towards standardized metadata exist. In particular in the library database area, Dublin Core [11] is widely used, and has been extended with the Warwick Framework [12]. Distributed environmental and governmental information has employed GILS, the Global Information Locator Service [13], which incorporates such metadata standards. The ESPRIT ADDSIA (Access to Distributed Databases for Statistical Information and Analysis) pro ject [14] seeks, among other ob jectives, to define and apply metadata to official and economic statistical micro- and macrodata. With regard to descriptive metadata information, usually a database is in one state and can be improved by changing to the next state. We will look at this process.


Distributed Information Search and Retrieval
STEPS OF METADATA ORGANISATION IN A DATABASE

53

No explicit metadata: the information resides in the minds of the database creators

Metadata described in an informal way (free text)

Metadata formally and explicitly described (tables or another rigorous format)

Homegeneous metadata described in a standard format

Easy and massive knowledge discovery possible with intelligent agents

· The first step is a minimum for any database, because forgetting it could result in the loss of metadata, for example if the creators of the database leave. This risk should not be taken at any price. · The second step, formalizing all metadata information, is the most difficult, but results in better quality and self-consistency for the database. · Deciding on a standard and adopting it can be painful, and will not necessarily be seen as an improvement for a given institute, but is very rewarding for the more general community, and can cater for new developments in the discipline. · Finally, all this work will enable very important developments in knowledge discovery and many other types of data processing. The preceding steps may not be seen as necessary, but without a standardized way of describing metadata the subsequent processing programs can only use a limited amount of homogeneous data, restricted to a given institute for example. Currently, improvements are planned in regard to many databases, but this work is often delayed by other pro jects having results apparently available more quickly. This leads to the question of the importance of metadata in astronomy, and the answer is that it is very important, probably more so than in many other disciplines: astronomical data are varied, but they have numerous links in them


54

F. Murtagh and D. Guillaume

and they are often needed by different geographically-dispersed groups at the same time and for different purposes. So, working on the question of metadata for any computer-available data collection is not a waste of time. A comprehensive format which can be used as a basis for metadata is SGML [15], Standard Generalized Markup Language. SGML allows the definition of markup languages with a Document Type Definition (DTD) file describing the syntax of the language. It enables the creation of appropriate formats in a very flexible way, and facilitates automatic understanding and reading of the SGML files. The main limitation of SGML is that it only describes text files. The advantage is clearly that no special reader is necessary for viewing an SGML file, but on the other hand binary files have the advantage of being compact and are read faster. The usual solution, if binary files are necessary, is to put references to these files in the SGML files. Generally speaking, SGML is especially useful for exchanging a limited amount of data with lots of metadata. Its role is not limited to simply supporting a specific application on a specific platform. In spite of the ASCII limitation, SGML is currently the best format for most data. It is an international standard (ISO 8879), is widely used (a list of companies and organisations using it would be far too extensive to list) and its use continues to spread in all computing activities. SGML is · flexible: it is simple to modify a DTD and the documents using it accordingly. It is also easy to create a new version of a DTD and keep it compatible with the old versions, for example by adding a tag. · rigorous: despite the flexibility, an SGML document must always conform to a DTD, and the validating parsers verify automatically this conformance. · easily readable: anyone understanding a markup language such as HTML can understand the meaning of an SGML document. As it is in text form, it requires no special editor. · personalizable for multi-users: it is possible, for example, to add specific tags in a document which are not defined in the DTD. It is also possible to use different DTDs and to keep global consistency between them. · content-oriented: an SGML document usually concentrates on the content of the text, not on its appearance: for instance, a title would be included in