Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.atnf.csiro.au/nca/Bibliometric.pdf Äàòà èçìåíåíèÿ: Tue Sep 20 07:30:17 2005 Äàòà èíäåêñèðîâàíèÿ: Mon Oct 1 20:23:27 2012 Êîäèðîâêà:

A Bibliometric Analysis of Astronomical Sciences Publications

Bev Biglia and Linda Butler Research Evaluation and Policy Project Research School of Social Sciences The Australian National University

May 2005


CONTENTS
1 INTRODUCTION ............................................................................................................ 3

1.1 1.2
2

Background............................................................................................................. 3 Analyses undertaken ............................................................................................... 3 Coverage of ISI database......................................................................................... 4 Using bibliometrics ................................................................................................. 4 Small Numbers ....................................................................................................... 4 The Research Evaluation and Policy Project Database............................................. 5 Subject classification............................................................................................... 5 Performance measures ............................................................................................ 5 Shares of world publication and citation totals ................................................ 6 Relative Citation Impact .................................................................................. 6 Collaborative Activity...................................................................................... 6 Actual and expected citation rates ................................................................... 6 Most highly cited publications ......................................................................... 7

METHODOLOGY ........................................................................................................... 4

2.1 2.2 2.3 2.4 2.5 2.6

2.6.1 2.6.2 2.6.3 2.6.4 2.6.5
3

PROFI LE OF ASTRONOMI CAL SCI ENCES ................................................................. 8

3.1 3.2 3.3 3.4 3.5

Publication and citations trends ............................................................................... 8 Placement of Australian astronomy publications ..................................................... 9 Analysis of journal impact quartiles ...................................................................... 10 Collaborative Activity ........................................................................................... 11 Institutional Analysis ............................................................................................ 11

Bibliography..................................................................................................................... 13 Appendix 1 ....................................................................................................................... 14 Appendix 2 Additional Astronomy Analysis ..................................................................... 15 Appendix 3 ....................................................................................................................... 18

Acknowledgements
Certain data included herein are derived from the Australian National Citation Report prepared by the Institute for Scientific Information®, Inc. (ISI®), Philadelphia, Pennsylvania, USA: © Copyright Institute for Scientific Information® 2003. All rights reserved.

2


1 Introduction
1.1 Background In 1995 the Research Evaluation and Policy Project (REPP) was commissioned to undertake a bibliometric analysis for Australian Astronomy: Beyond 2000 which covered the period 19811993. In late 2004, REPP was contracted to update that report for the Australian Academy of Science's Decadal Plan for Australian Astronomy, extending the previous analysis and covering in detail the period 1994 to 2003. The analysis is based on the REPP database which contains all Australian publications indexed in the Institute for Scientific Information's (ISI) three main indices: the Science Citation Index, the Social Sciences Citation Index, and the Arts and Humanities Citation Index. 1 .2 Analyses undertaken

The report provides an analysis of Australia's total publication output in journals indexed by ISI and classified to their Astronomy and Astrophysics journal set. The journals in this set are listed in Appendix 1. The analysis seeks to determine: · Australia's share of world publications and citations, using five year overlapping windows; · Australia's citation rate, using five year overlapping windows, compared to the world citation rate; · the journal placement of Australia's astronomy publications; · the distribution of astronomy journals to impact quartiles based on their average citation rate and the changes in Australia's presence in these quartiles over time; and · trends in collaborative activity In addition, an institutional analysis of astronomy publications is presented. This analysis seeks to determine: · the major institutions contributing to Australia's journal output; · where Australia's most highly cited publications come from (i.e. the top 1% and 5% most highly cited Australian publications); and · how the actual citation rates of institutions compare to the rate that would be expected given the journals in which they publish. It is only possible to confidently undertake a comparative analysis for those institutions with a minimum of 100 publications ­ citation analysis based on publication numbers less than this threshold are statistically unreliable because of the skewed nature of the distribution of citations (very few publications are highly cited, and it is these that drive averages). The institutions with more than 100 publications in astronomical sciences in the last 5 years were: The Australian National University Anglo-Australian Observatory Australia Telescope National Facility University of Melbourne University of New South Wales University of Sydney University of Tasmania

3


Four additional institutions that had less than 100 publications, but more than 75, were included in the analysis, but extreme caution needs to be exercised in the interpretation of the data. These are: Australian Defence Force Academy (ADFA) Monash University Swinburne University University of Adelaide It should be noted that the data for ADFA is also included in the total for the University of New South Wales. After a draft analysis was completed and discussed with the project's reference group, an additional time series analysis was undertaken using a reduced journal set. Ten geophysics journals were deleted from the analysis. These journals are marked with an asterisk in Appendix 1. A detailed investigation of the influence of highly cited publications was also undertaken to determine their impact of average citation rates. These two additional analyses are reproduced in Appendix 2.

2 Methodology
2.1 Coverage of ISI database The use of bibliometric analysis as a tool for evaluating research performance cannot be universally applied across all fields of research (Bourke, Butler and Biglia 1996; NBEET 1994). ISI does not comprehensively cover the output of Australian research in a number of fields, such as most humanities and social science disciplines, or engineering and other applied sciences. However, ISI's coverage of the journals in which Australian researchers in astronomical sciences publish is very comprehensive, and data REPP has analysed in a number of previous studies suggests it covers approximately 95% of total output (Butler and Visser 2005). 2 .2 Using bibliometrics

Bibliometric data answer no single evaluative question in their own right. This information must be seen alongside other measures of esteem, performance, visibility and the testimony of expert peers in the activity that is being analysed. 2 .3 Small Numbers

A special note of caution is required concerning the small numbers of publications tabulated in some research fields or subject categories. In the ISI system as a whole, many publications receive no citations at all, and the majority receive less than 5 citations (Garfield 1989). The number achieving a high citation count is extremely small. The very nature of citation practice means that averages can be disproportionately affected by a single highly cited publication. The smaller the number of publications being analysed, the greater the effect such an item will have on the average. Analyses based on any units with less than 100 publications are unreliable. For the bibliometric measures used in this analysis, some citation information based on units with less than 100 publications have been included for comparative purposes (ie. ADFA, Monash University, Swinburne University and University of Adelaide) but are highlighted in the analyses and must be interpreted with extra caution. 4


2.4 The Research Evaluation and Policy Project Database The REPP database was created from data files purchased from the Institute for Scientific Information (ISI) and, at the time this study was commissioned, covered the period 1981 to 2003. It captures all publications with an Australian address in the three major ISI Indices: Science Citation Index (SCI), Social Sciences Citation Index (SSCI) and Arts & Humanities Citation Index (A&HCI). The database also contains the yearly counts of citations to each of the Australian publications. The focus of the database is the departmental and the institutional address(es) given for each publication rather than the names of authors. REPP `cleans' the addresses in the database to enable analysis to be done at different levels of aggregation. `Cleaning' means ensuring that all variations of the same address are identified and that all publications with variants of that address are allocated to the one `standardised address'. The standardised addresses for all Australian universities and other major research institutions are set up in a hierarchical format, from the institution down through faculties or schools to the department, enabling tabulations at these different levels of aggregation to be produced. Addresses in most other sectors are `cleaned' only to the institution level. The REPP database as it now stands contains close to 350,000 Australian publications. Since there is no consensus in the research policy community about the `best' way to count publications and citations, we list a series of options available in the REPP database and used in the tables compiled for this report: · the types of publications counted were limited to research articles, research notes and review articles; · data were compiled on a tape-year basis (i.e. the year its details were entered in ISI's database not the year the item was published); · the institutional analysis was based on data from the 2002 version of the REPP database, as unified addresses were not available for the 2003 version. The analysis therefore focused on 1998-2002, the most recent five year period available; and · the analysis was based on whole publication counts (i.e. where more than one institution collaborated in a publication, each was given a count of 1 for that publication).

2 .5

Subject classification

In this report, publications are classified to `Astronomical Sciences' on the basis of the journal in which they appear, using the journals listed in ISI's subject category `Astronomy and Astrophysics'. A list of journals classified to this category between 1981 and 2003 is given in Appendix 1. 2 .6 Performance measures

This section gives a general description of the standard bibliometric measures used in the study. Notes relevant to particular points of discussion are included throughout the report as required.

5


2.6.1 Shares of world publication and citation totals
Australia's share of all publications indexed by ISI in the astronomy journal set is calculated. The citations these publications have received are also tabulated and the share these represent of the world total is calculated. Tabulations were made using five-year windows, 1981­85 to 1999-03. For each window, the share is calculated on the basis of publications from the specified five-year period and the citations those publications received within the same five-year period. ISI publication and citation totals have been calculated on the same basis. In addition to providing a common basis across time, the use of windows tends to eliminate the sharpest yearly swings in citation per publication rates.

2.6.2 Relative Citation Impact
The relative citation impact (RCI) compares Australia's share of publications with the share of citations these publications achieve. If the field has equivalent shares of publications and citations, it will have an RCI of 1.00. If it achieves a greater share of citations than publications, its RCI would be greater than 1.00. Conversely, if its share of citations falls below its share of publications, then its RCI would be less than 1.00.

2.6.3 Collaborative Activity
The REPP database enables us to distinguish between publications with different types of authorship, viz: Single author Group Intra-institutional National International one mor mor mor mor author e than e than e than e than onl one one one one y (ie no collaboration) author from the same organisational unit author from different organisational units author from different institutions in Australia country listed in the author addresses

The classification is not hierarchical and all collaborations present in a publication will be ascribed to it. For instance, if a publication involves collaboration between a number of Australian institutions as well as an overseas institution, it will be classified as both International and National. As this measure relies on `cleaned' addresses to calculate the level(s) of collaboration involved in a publication, our analysis cannot go past 2002, the last year for which this data is available.

2.6.4 Actual and expected citation rates
In this measure, the term Actual Citation Rate refers to the average number of citations actually achieved by the publications of the unit under study. The term Expected Citation Rate refers to the average number of citations achieved by all publications in the journals carrying the articles of the unit under study. ISI has supplied REPP with average citation rates for all publications in its journals for each year of publication and for each type of publication. The calculations are weighted by the year of publication and the frequency of publication in particular journals. The expected citation rate is a different calculation to ISI's journal impact factor1, though the measures are related.
1

The calculation for ISI's 1996 impact factor is a/b, wh ere:

6


This analysis is depicted graphically, allowing a comparison to be made between actual and expected citation rates. The diagonal on the graph indicates the point at which actual and expected citation rates converge so that, where the unit under study appears above this diagonal line, it is achieving a rate of citation higher than the `world' average for the journals in which it publishes; and where it appears below the line, it is achieving a rate of citation lower than the `world' average for its publications in those journals. Where a vertical line is drawn on a graph, it indicates the citation per publication figure for all publications in all journals in the field under study, that is, the ISI average. In these graphs, the relationship to the diagonal and vertical lines is the salient point to observe. A point above the line and to the right of the ISI average indicates above world performance in high impact journals; a point below the line and to the left indicates below world performance in low impact journals. Between these extremes, there are many possibilities, such as appearing in high impact journals below the world expected rate or appearing in low impact journals but performing at a higher rate of citation than the expected world rate. Judgments about these relationships should be made carefully.

2.6.5 Most highly cited publications
The number of citations required to rank a publication from 1998 to 2002 in the top 1% and 5% of `Astronomical Sciences' publications in Australia was calculated. By extracting details about the publications that fell within these bands from the REPP database, it was possible to identify the institutions with which the authors of these articles were associated.

a = 1996 citations to ar ticles publish ed in th e journal in 1994 and 1995 b = numb er of ar ticles published in th e journal in 1994 and 1 995; The calcu lation for 1996 expected valu es used in this report is x/y, where: x = numb er of citations receiv ed between 1996 and 2000 to all ar ticles published in the journal in 1996 y = numb er of ar ticles published in th e journal in 1996.

7


3 Profile of Astronomical Sciences
3 .1 Publication and citations trends

7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0

1995-99

1996-00

1997-01

1998-02

1987-91

Figure 1: Australia's share of world publications and citations: Astronomical Sciences 198185 to 1999-2003 ( citations, publications, relative citation impact) There has been a marked change in Au the last review was undertaken. In line increased since the early 1990s. This element of the Research Quantum that block-funding for research is distributed stralia's publication output and citation impact since with most other fields (see Butler 2001), output has coincided with the introduction of the publication applies to all universities, whereby a proportion of on the basis of raw publication counts.

In contrast to most other fields, Australia's share of citations to astronomy publications has increased at an even greater rate than its share of the publications themselves. As a consequence, the relative citation impact for astronomy has climbed well above one.

1992-96

1999-03

1994-98

1983-87

1984-88

1985-89

1986-90

1981-85

1982-86

1988-92

1989-93

1990-94

1991-95

1993-97

8


3 .2

Placement of Australian astronomy publications
Journal i mpact*

Table 1: Australian Percentage of Publications in Astronomical Sciences Journals 1994-2003
% Total Publications that have an Aus tralian author(s )
19 0. 0. 4. 2. 4. 2. 9. 2. 1. 1. 4. 0. 1 1 2 3 0 1 0 5 0 1 5 0 1 0 5 0 0 0 0 0 . . . . . . . . . . . . . . . . . . . . 94 0 0 0 8 1 2 4 6 3 1 5 0 8 4 1 0 0 0 0 3 7 1 0 0 7 0 0 0 8 0 5 0 1995 0.0 0.0 3.0 3.6 4.5 3.0 10.5 2.8 0.9 0.9 5.9 0.0 0.0 3.9 0.9 3.1 2.0 0.0 0 0 13 0 0 0 92 0 2 0 6 1 1 18 1 1 .6 .0 .1 .5 .0 .0 .3 .0 .2 .0 .7 .9 .0 .2 .4 .7 1996 0.0 0.0 7.4 3.9 4.0 2.8 8.9 2.0 1.6 2.2 5.6 1.0 0.0 4.2 1.6 3.1 4.4 0.0 0 0 9 0 0 0 76 2 0 3 0 0 0 18 1 0 .0 .0 .1 .9 .0 .0 .2 .9 .5 .7 .0 .0 .9 .2 .6 .0 19 0. 0. 5. 3. 4. 3. 9. 2. 1. 0. 2. 1. 0. 1. 1. 2. 2. 2. 3. 4. 1. 5. 1. 0. 0. 77. 0. 0. 6. 0. 0. 0. 5. 2. 0. 97 0 0 0 7 7 9 7 6 8 0 7 2 0 8 7 5 0 6 8 0 7 4 8 0 0 8 0 0 7 0 0 4 6 7 0 19 0. 11. 2. 4. 7. 5. 10. 6. 2. 1. 5. 0. 0. 1. 4. 2. 3. 3. 1. 3. 1. 12. 4. 0. 53 1 0 0 3 4 0 0 0 3 . . . . . . . . . . 98 0 1 2 2 1 0 0 7 4 1 6 0 0 1 2 0 2 2 7 2 4 0 1 0 6 4 6 0 1 4 9 0 4 6 19 7. 7. 6. 4. 7. 3. 10. 4. 1. 0. 4. 2. 0. 3. 0. 2. 5. 2. 0. 0. 2. 0. 0. 0. 67 1 0 0 4 0 2 0 0 . . . . . . . . . 99 1 1 7 6 0 3 8 0 6 5 0 2 0 3 0 9 2 9 0 0 8 0 0 0 4 8 9 0 5 0 0 0 9 2000 10 3 4 5 3 9 3 2 2 3 0 0 3 0 3 4 1 3 1 2 16 0 0 73 1 0 0 0 2 2 0 2 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 6 3 3 0 4 3 2 3 1 8 0 9 7 7 5 9 5 9 1 7 9 0 7 6 6 0 0 3 3 0 4 9 20 3. 0. 3. 4. 5. 2. 11. 2. 2. 1. 5. 4. 11. 0. 0. 3. 2. 1. 7. 0. 7. 0. 0. 0. 74 0 2 1 0 0 3 0 3 . . . . . . . . . 01 2 0 9 9 3 6 4 1 6 7 4 4 5 0 0 0 3 8 3 0 7 0 6 0 4 0 8 4 0 0 2 0 1 2002 6.3 6.3 8.9 4.6 6.7 2.1 10.6 4.0 2.1 0.4 3.9 1.7 4.7 0 1 5 2 0 0 . . . . . . 0 8 3 1 0 0 Journal
AN N U R EV AST R ON AST R AN N U R EV EAR T H PL SC

37 9 AST R OPHYS J SU PPL S 9 AST R OPHYS J 9 AST R ON J 7 J GEOPHYS R ES-AT M OS 6 M ON N OT R AST R ON SOC 6 J GEOPHYS R ES-SOL EA 5 AST R ON AST R OPHYS 5 IC AR U S 5 J GEOPHYS R ES-OC EAN S 5 PU BL AST R ON SOC J PN 5 AST R OPAR T PHYS 4 AST R ON AST R OPHYS SU P 4 J GEOPHYS R ES-PLAN ET 4 SOL PHYS 4 PU BL AST R ON SOC PAC 4 J GEOPHYS R ES-SPAC E 4 N EW AST R ON 4 SPAC E SC I R EV 3 AN N GEOPHYS-AT M HYDR 2 GEOPHYS AST R O F LU ID 2 PLAN ET SPAC E SC I 2 R EV M EX AST R ON AST R 1 Q J R OY AST R ON SOC 1 PU BL AST R ON SOC AU ST 1 IN T J M OD PHYS D 1 Z N AT U R F OR SC H A 1 EAR T H M OON PLAN ET S 1 OBSER VAT OR Y 0 AST R ON N AC HR 0 ADV SPAC E R ES 0 J AST R OPHYS AST R ON 0 AST R OPHYS SPAC E SC I 0 AST R OPHYS LET T C OM M 0 AST R ON GEOPHYS 0 C HIN ESE AST R ON AST R 0 IAU SYM P 0 C R AC AD SC I IV-PHYS 0
P AST R ON SOC AU ST AN N GEOPHYS-GER M AN Y IN DIAN J R ADIO SPAC E

.3 .9 .8 .4 .8 .8 .6 .5 .5 .2 .1 .0 .8 .5 .5 .2 .2 .2 .0 .2 .5 .1 .0 .8 .7 .7 .6 .4 .3 .7 .7 .6 .5 .5 .5 .4 .1 .0 .0

1 7 6 3 0 2 1 6 2 9 4 5 3 4 1 9 8 8 8 4 3 0 6 1 4 2 4 1 6 9 8 0 6 4 4 2 2 1 0

0.0 2.2 0.0 51 0 0 0 5 0 0 0 2 . . . . . . . . . 3 8 6 0 6 0 5 0 5

Total 2003 Aust Worl d %Aust 0.0 3 154 1.9 6.3 7 155 4.5 2.9 63 1357 4.6 3.8 915 22378 4.1 5.9 262 4779 5.5 3.0 243 7586 3.2 8.9 859 8580 10.0 2.5 134 4315 3.1 2.0 279 14375 1.9 1.0 23 2101 1.1 3.9 155 3396 4.6 2.3 16 1028 1.6 3.6 14 658 2.1 63 2291 2.7 0.4 12 1328 0.9 4.6 57 1975 2.9 2.6 51 1468 3.5 3.2 90 5147 1.7 1.2 9 357 2.5 11.6 27 1230 2.2 13 1015 1.3 0.0 23 316 7.3 0.9 17 1435 1.2 0.0 1 270 0.4 1 71 1.4 44.6 305 467 65.3 0.7 6 740 0.8 0.0 16 1581 1.0 1.3 12 721 1.7 0.0 6 257 2.3 0.6 10 767 1.3 0.8 74 5935 1.2 0.0 7 266 2.6 3.4 79 3981 2.0 9 567 1.6 0.0 0.0 5.0 5 1 217 3 50 5 1 217 362 5226 242 52 310 116 2.3 0.3 4.2 1.2 96.2 1.6 0.9

0.0 5.6 96.2 0.2 5.3 7.6

7.9 2.9

0.0 0.9 2.8

4.2 0.0 2.8 0.0

2.9 0.0 2.3 1.0

0.0 0.0 4.0 0.0 0.6

6.0

0.0

2.1

* world cites/world pubs in the year 2000

The data in Table 1 is pr of average citation rates year, so it was necessar journals for which the change for the journal.

ovided primarily for information. The journals are sorted on the basis for 2000. Obviously the ranking of journals will change from year to y to choose one period as a reference point. There were also three calculation could not be made, most probably because of a name

Over the last 10 year period the journal that had the highest presence of Australian publications was, not surprisingly, Publications of the Astronomical Society of Australia.

9


However, it is interesting to note that Australian authors now appear on less than half the publications carried by the journal, down from over 90 percent in 1995. 3 .3 Analysis of journal impact quartiles

1 9 9 9 -0 3 1 9 9 7 -0 1 1 9 9 5 -9 9 1 9 9 3 -9 7 1 9 9 1 -9 5 1 9 8 9 -9 3 1 9 8 7 -9 1 1 9 8 5 -8 9 1 9 8 3 -8 7 1 9 8 1 -8 5

0%

20% Quartile 1

40% Quartile 2

60% Quartile 3

80% Quartile 4

100%

Figure 2: Quartile distribution, five year windows, 1981-2003 While journals are spread evenly between the four quartiles in this analysis (i.e. 25 percent in each quartile), the distribution for articles is not even. To put the data in Figure 2 into perspective, it should be noted that approximately 45 percent of articles appear in journals in the top quartile. Australian output in astronomy exceeds this benchmark level, and particularly in recent years has done so by a considerable margin. The sudden jump then decline in publications appearing in quartile 4 journals was due to the brief classification of the Proceedings of the Astronomical Society of Australia to this category.

10


3 .4

Collaborative Activity Number 212 279 113 478 1727 Percentage 9 12 5 21 77

Table 2: Level of collaboration in Astronomy, 1998-2002. Sole Author Group Intra-institutional National International

The most recent five year window for which collaboration data is available is 1998-2002. Only the sole author and international figures are directly comparable to our previous analysis, as the method of calculating collaborative activity now enables us to identify all collaborations, not just the `highest' level. Comparing this data to the previous analysis, we note that single author publications have decreased from 19 percent to 9 percent, and internationally co-authored publications have increased from 55 percent to 77 percent. This change is in line with trends exhibited in all other disciplines ­ there is a general trend in all fields to the increasing internationalisation of research and a decline in single authored papers.

3 .5

Institutional Analysis

The results of two analyses that focus on institutional performance in astronomy are presented in Figure 3 and Table 3. In both the table and chart, those institutions where the analysis rests on less than 100 publications are denoted in grey, and extra caution is required in interpreting the data.
16.0 Univ NSW 12.0 Anglo-Aust Obs Univ Adelaide Univ Melbourne Swinburne Univ Australia Monash Univ Aust Tel Natl Fac ADFA Univ Tasmania Univ Sydney ANU

Average actual citations

8.0

4.0

0.0 0.0 4.0

ISI 8.0 Average citations expected 12.0 16.0

Figure 3: Actual and expected citation rates Astronomical Sciences 1998-2002

11


All institutions with a significant output in Astronomy are at or above the world benchmark and most publish in a high impact set of journals. The two institutions that appear to the left of the vertical line have less than 100 publications. In the case of Swinburne University, further analysis alerted us to a particular concern for this institution ­ most of its publications appeared in 2001 and 2002, and hence had little time to attract citations, thus deflating their average and expected citation rates relative to other institutions. We have not investigated whether this is also the case for the University of Adelaide. The Australian National University and the University of New South Wales from other institutions in terms of their citation impact, followed by the Observatory. This performance can also be seen in Table 3, which details Australia's most highly cited publications for the period 1998-2002 are to be Table 3: Most highly cited publications Astronomical Sciences, 1998-2002 Top 1% = 93 cites (22 pubs) Top 5% = 37 cites (108 pubs)
Total Pubs No. The Aus tralian National University Anglo-Aus tralian Observatory University of New South Wales Australia Telescope National Fac ility University of Tas mania University of Melbour ne University of Sydney Monash University University of Adelaide Australian Defence Force Academy Swinburne University 6 2 2 3 1 1 3 0 8 5 8 1 0 9 8 7 7 8 7 5 3 6 2 9 2 4 7 7 2 Pubs in top 1% No. % 13 2 7 2 2 0 2 0 .1 .7 .8 .5 Pubs in top 5% No. % 5 2 2 1 1 4 0 6 3 2 7 7 4 3 2 8 8 7 4 2 1 1 8 5 3 2 .4 .4 .9 .1 .7 .8 .8 .3 .2 .9 .4

clearly stand out Anglo-Australian where authors of found.

1 1 1

1.2 1.3 1.2

In this analysis, jointly-authored publications were allocated to each Australian institution involved, so some double-counting exists. The University of New South Wales has close to three times the expected number of very highly cited publications (top 1% in Australia), and the Australian National University has double the expected number (and accounts for 60 percent of this band of publications). These two universities, together with the AngloAustralian Observatory, were also the most prominent in the second tier of publications.

12


Bibliography P Bourke, L Butler and B Biglia, 1996, Monitoring Research in the Periphery: Australia and the ISI Indices, Monograph Series No. 3., Research Evaluation and Policy Project, Australian National University, Canberra. L Butler, 2001, Monitoring Australia's Scientific Research, Research Evaluation and Policy Project, Australian National University, Canberra, 173pp. L. Butler and M. Visser, 2005, 'Extending citation analysis to non-source items'. Scientometrics, in press. E Garfield, 1989, cited in D Evered and S Harnett (eds), Ciba Foundation Conference: The Evaluation of Scientific Research, John Wiley & Sons, Chichester (UK). National Board of Employment, Education and Training (NBEET), 1994, Quantitative Indicators of Australian Academic Research, Commissioned Report No. 27, AGPS, Canberra.

13


Appendix 1 List of journals in the ISI's Astronomy and Astrophysics journal set.
J O URNAL ACTA ASTRONOM ADV SPACE RES ANN G EOPHYS A-UPPER *ANN G EOPHYS-AT M HYDR ANN G EOPHYS-GERMANY ANN SOC SCI BRUX ANNU REV ASTRON ASTR *ANNU REV EARTH PL SC ASTRON ASTROPHYS ASTRON ASTROPHYS REV ASTRON ASTROPHYS SUP *ASTRON GEOPHYS ASTRON J ASTRON LETT+ ASTRON NACHR ASTRON REP+ ASTRON ZH+ ASTROPART PHYS ASTROPHYS J ASTROPHYS J SUPPL S ASTROPHYS L ETT CO MM ASTROPHYS SPACE SCI B ASTRON I CZECH CELESTIAL MECH CHINESE ASTRON ASTR CR ACAD SCI IV-PHYS CR P HY S *EARTH MOON PL ANETS *GEOPHYS ASTRO FLUID IAU SYMP ICARUS INDIAN J RADIO SPACE *INT J MOD PHYS D J ASTROPHYS ASTRON *J GEO PHYS RES-AT MOS *J GEO PHYS RES-OCEANS *J GEO PHYS RES-PLANET *J GEO PHYS RES-SOL EA *J GEO PHYS RES-SPACE J RO Y ASTRON SOC CAN MON NOT R ASTRO N SOC MOON PLANETS MOON PLANETS NEW ASTRON OBSERVATORY P ASTRON SOC AUST *PLANET SPACE SCI PUBL ASTRON SOC AUST PUBL ASTRON SOC JPN PUBL ASTRON SOC PAC Q J ROY ASTRON SOC REV MEX ASTRON ASTR SKY TEL ESCOPE SOL PHYS SOV ASTRO N L ETT+ SPACE SCI REV VESTN L ENIN U MMA VESTN MOSK U FIZ AS+ VISTAS ASTRON Z NATURFORSCH A FULL_TITLE ACTA ASTRONOMICA ADVANCES IN SPACE RESEARCH ANNAL ES GEO PHYSICAE SERIES A-UPPER AT MOSPHERE AND SPACE SCIENCES ANNAL ES GEO PHYSICAE-ATMOSPHERES HYDRO SPHERES AND SPACE SCIENCES ANNAL ES GEO PHYSICAE ANNAL ES DE L A SOCIETE SCIENTIF IQUE DE BRUXELL ES SERIES 1 ANNUAL REVIEW OF ASTRONO MY AND ASTRO PHYSICS ANNUAL REVIEW OF EARTH AND PL ANET ARY SCIENCES ASTRONO MY & ASTROPHYSICS ASTRONO MY AND ASTRO PHYSICS REVIEW ASTRONO MY & ASTROPHYSICS SUPPLEMENT SERIES ASTRONO MY & GEO PHYSICS ASTRONO MICAL JOURNAL ASTRONO MY LETTERS-A JOURNAL OF ASTRONOMY AND SPACE ASTROPHYSICS ASTRONO MISCHE NACHRICHT EN ASTRONO MY REPORTS ASTRONO MICHESKII ZHURNAL ASTROPARTICLE PHYSICS ASTROPHYSICAL JOURNAL ASTROPHYSICAL JOURNAL SUPPL EMENT SERIES ASTROPHYSICAL LETTERS & COMMUNICAT IONS ASTROPHYSICS AND SPACE SCIENCE BULLET IN OF THE ASTRONOMICAL INSTITUTES OF CZECHO SLOVAKIA CELESTIAL MECHANICS CHINESE ASTRONO MY AND ASTRO PHYSICS COMPT ES RENDUS DE L ACADEMIE DES SCIENCES SERIE IV PHYSIQUE ASTRO PHYSIQUE COMPT ES RENDUS PHYSIQUE EARTH MOON AND PLANETS GEOPHYSICAL AND ASTRO PHYSICAL FLUID DYNAMICS IAU SYMPOSIA ICARUS INDIAN JOURNAL OF RADIO & SPACE PHYSICS INTERNATIONAL JOURNAL OF MO DERN PHYSICS D JOURNAL OF ASTRO PHYSICS AND ASTRONO MY JOURNAL OF G EOPHYSICAL RESEARCH-ATMO SPHERES JOURNAL OF G EOPHYSICAL RESEARCH-OCEANS JOURNAL OF G EOPHYSICAL RESEARCH-PLANETS JOURNAL OF G EOPHYSICAL RESEARCH-SOLID EARTH JOURNAL OF G EOPHYSICAL RESEARCH-SPACE PHYSICS JOURNAL OF THE ROYAL ASTRONO MICAL SOCIET Y OF CANADA MONTHLY NOTICES OF THE ROYAL ASTRONO MICAL SOCIET Y MOON MOON AND THE PL ANET S NEW ASTRONO MY OBSERVATORY PROCEEDINGS ASTRONOMICAL SOCIETY OF AUSTRALIA PLANETARY AND SPACE SCIENCE PUBL ICATIONS OF THE ASTRONO MICAL SOCIETY OF AUSTRAL IA PUBL ICATIONS OF THE ASTRONO MICAL SOCIETY OF JAPAN PUBL ICATIONS OF THE ASTRONO MICAL SOCIETY OF THE PACIF IC QUARTERL Y JOURNAL OF THE ROYAL ASTRONOMICAL SOCIETY REVIST A MEXICANA DE ASTRO NOMIA Y ASTROFISICA SKY AND T ELESCO PE SOLAR PHYSICS SOVIET ASTRONO MY L ETTERS SPACE SCIENCE REVIEW S VESTNIK L ENINGRADSKOGO UNIVERSITET A SERIYA MATEMATIKA MEKHANIKA ASTRONOMIYA VESTNIK MOSKOVSKOGO UNIVERSITETA SERIYA 3 FIZIKA ASTRONOMIYA VISTAS IN ASTRONO MY ZEITSCHRIFT FUR NATURFORSCHUNG SECTION A-A JOURNAL OF PHYSICAL SCIENCES

14


Appendix 2 Additional Astronomy Analysis 1. Removal of geophysics journals A repeat of the analysis shown in Figure 1 above was undertaken after deleting the following geophysics journals: Ann Rev Earth Pl Sc AnnGeophys-Atm Hydr J Geophys Res ­ Atmos Geophys Astro Fluid J Geophys Res - Sol Ea Planet Space Sci J Geophys Res ­ Oceans Int J Mod Phys D J Geophys Res ­ Planet Earth Moon Planets J Geophys Res ­ Space Astron Geophys
8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0
1983-87 1984-88 1981-85 1982-86 1985-89 1991-95 1998-02 1995-99 1986-90 1987-91 1988-92 1989-93 1990-94 1996-00 1997-01 1999-03 1992-96 1993-97 1994-98

Figure 4: Australia's share of world publications and citations, restricted journal set. (


citations,



publications,



relative citation impact)

The reduced journal set made little difference to the general trends, although it is noted that Australia's share of both publications and citations is higher for this set. 2. Investigation of very highly cited publications Citation distributions are extremely skewed ­ this is true for all research disciplines not just astronomy. Because of this skewed nature, average data can be heavily influenced by a small number of very highly cited publications. REPP was asked to look at the distribution of citations for astronomy publications to see how this may affect the interpretation of Australian data. The two figures below show the distribution of publications by the number of citations they receive for two five year windows, taken 10 years apart. In both cases, citation counts are limited to the same five year period covered by the publication counts, and so are directly comparable.

15


1160 334 263 239 148 147 136 131 128 116 112 110 101 99 95 94 91 88 83 82 80 76 75 74 73 72 70 66 65 63 61 59 58 54 53 52 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 1 1 1 3 1 1 2 1 2 3 4 5 2 4 1 2 4 1 3 2 2 3 4 9 2 5 2 4 3 5 3 10 6 9 11 10 9 7 10 11 22 15 21 20 30 16 26 30 34 35 51 49 48 71 58 75

Number of citations

109 119 131 154 183

267

565

0

100

200

300

400

500

600

Num be r of publications

16


97 78 69 58 55 54 52 51 49 47 46 44 43 37 36 35 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

1 1 1 1 1 1 1 3 1 1 1 1 3 1 3 1 1 3 1 2 2 2 1 2 5 7 1 3 7 2 8 6 9 10 12 9 18 12 16 20 21 26 33 36 46 42 56 103 172 405

Number of citations

0

100

200

300

400

500

600

Num be r of publications

Figure 5: Distribution of citations, 1999-2003 1989-93

Figure 6: Distribution of citations,

Comparing these two periods, there are two factors that can be seen in the data which is reflected in the improved performance of Australian astronomy publications: (i) The proportion of publications uncited in the period has decreased from 36% in 1989-93 to 24% in 1999-03. (ii) 1999-03 contains 13 publications with 100 or more citations (and one with more than 1,000), while there are none for 1989-93. In addition, citation rates world-wide have been increasing. In 1989-93 the world average was 4.02, but by 1999-03 had climbed to 6.21. All three factors have an influence on the trends shown in Figure 1 above. I doubt that these factors are unique to Australia, but we cannot analyse other countries to confirm or disprove this.

17


Appendix 3 Data underlying charts in main body of the text (Figure 1)
Tape 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 year -8 5 -8 6 -8 7 -8 8 -8 9 -9 0 -9 1 -9 2 -9 3 -9 4 -9 5 -9 6 -9 7 -9 8 -9 9 -0 0 -0 1 -0 2 -0 3 2 3 3 3 3 3 3 3 3 4 4 4 5 5 5 5 5 5 5 Wo Pu b s 9408 0249 1949 2930 3812 5245 6330 7428 9220 2956 5790 9050 1112 2656 3399 4289 4874 6274 9982 rld 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 3 3 3 Cites 25839 25497 29148 30791 34672 38011 41646 48117 57854 71950 95977 24359 43876 50106 77134 92638 13481 30056 72696 Austra Pu b s 877 1011 1048 1056 1049 1029 977 1043 1121 1206 1344 1529 1662 1819 1962 2051 2149 2247 2324 lia Cites 4168 4764 4853 4653 4767 4835 4579 5096 5514 5743 6388 7556 8828 9843 12814 14416 16865 19289 20732 Share Pu b s Cites 2.98 3.31 3.34 3.80 3.28 3.76 3.21 3.56 3.10 3.54 2.92 3.50 2.69 3.23 2.79 3.44 2.86 3.49 2.81 3.34 2.94 3.26 3.12 3.37 3.25 3.62 3.45 3.94 3.67 4.62 3.78 4.93 3.92 5.38 3.99 5.84 3.87 5.56 RCI 1.11 1.14 1.15 1.11 1.14 1.20 1.20 1.23 1.22 1.19 1.11 1.08 1.11 1.14 1.26 1.30 1.37 1.46 1.44

(Figure 2)
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 0 0 0 0 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 0 0 0 0 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 Quartile 1 596 680 585 553 552 586 526 551 598 683 754 844 950 1126 1389 1376 1388 1374 1435 Quartile 2 116 84 178 185 187 129 183 176 200 176 228 263 291 253 183 258 393 461 459 Quartile 1 2 2 2 2 2 1 2 2 2 2 2 2 2 2 2 2 2 3 35 15 38 74 78 30 90 13 31 26 07 02 52 53 32 24 16 08 15 Quartile 4 30 32 47 44 32 84 78 103 92 321 155 220 169 187 158 193 152 204 215

18


(Table 1)
Jour nal
A CT A A ST RONOM A DV SPA CE RES A NN GEOPHYS-A T M HYDR A NN GEOPHYS-GERM A NY A NNU REV A ST RON A ST R A NNU REV EA RT H PL SC A ST RON A ST ROPHYS A ST RON A ST ROPHYS REV A ST RON A ST ROPHYS SUP A ST RON GEOPHYS A ST RON J A ST RON LET T + A ST RON NA CHR A ST RON REP+ A ST RON Z H+ A ST ROPA RT PHYS A ST ROPHYS J A ST ROPHYS J SUPPL S A ST ROPHYS LET T COM M A ST ROPHYS SPA CE SCI CHINESE A ST RON A ST R CR A CA D SCI IV-PHYS CR PHYS EA RT H M OON PLA NET S GEOPHYS A ST RO FLUID IA U SYM P ICA RUS INDIA N J RA DIO SPA CE INT J M OD PHYS D J A ST ROPHYS A ST RON J GEOPHYS RES-A T M OS J GEOPHYS RES-OCEA NS J GEOPHYS RES-PLA NET J GEOPHYS RES-SOL EA J GEOPHYS RES-SPA CE J ROY A ST RON SOC CA N M ON NOT R A ST RON SOC NEW A ST RON OBSERVA T ORY P A ST RON SOC A UST PLA NET SPA CE SCI PUBL A ST RON SOC A UST PUBL A ST RON SOC JPN PUBL A ST RON SOC PA C Q J ROY A ST RON SOC REV M EX A ST RON A ST R SOL PHYS SPA CE SCI REV VEST N M OSK U FIZ A S+ Z NA T URFORSCH A

Aust

1

1

5 1

1994 1995 1996 1997 1998 1999 2000 20 Wor l d Aust Wor l d Aust Wor l d Aust Wor l d Aust Wor l d Aust Wor l d Aust Wor l d Aust 45 29 22 22 34 38 32 5 661 9 883 3 331 2 450 6 649 8 400 19 826 13 122 136 136 3 177 2 144 4 141 3 146 1 1 14 17 17 16 14 1 14 1 17 14 10 12 2 18 1 14 2 20 6 1198 10 1065 20 1224 24 1346 33 1353 19 1166 27 1223 54 3 6 7 2 4 7 2 4 225 10 257 13 309 8 438 4 363 14 422 10 259 29 3 38 29 1 24 1 7 412 20 449 20 505 21 451 35 490 31 444 27 507 28 175 125 103 117 117 109 96 35 1 53 34 43 4 91 23 4 174 53 87 141 84 123 99 140 31 43 70 58 51 130 59 6 5 1982 78 2141 80 2055 83 2227 54 1298 142 3109 97 2232 127 0 251 4 134 6 81 8 160 2 91 8 119 6 167 5 8 1 59 147 11 3 83 5 259 2 422 7 508 3 193 8 293 1 264 4 423 14 574 17 1 109 75 92 3 48 38 89 87 69 79 23 45 08 73 59 29 43 83 50 61 81 11 47 57 22 63 91 10 64 18 21 59 30 203 39 90 151 16 17 191 155 53 178 1 32 1 7 8 4 62 4 1 2 20 20 2 10 218 44 1168 178 34 11 06 54 29 05 51 2 2 17 30 37 224 255 47 18 66 63 19 69 68 11 25 522 181 69 35 48 96 95 59 00 28 16 109 183 55 12 77 46 74 76 91 165 24 144 263 64 22 87 62 39 50 81 1

01 2002 2003 Wor l d Aust Wor l d Aust Wor l d 21 25 27 401 3 576 6 758 13 103 1 157 3 50 31 1 16 15 18 1 16 1 16 2042 38 1822 38 1936 3 5 5 18 34 23 40 526 37 552 26 443 113 105 109 34 99 1 181 113 106 107 52 2587 127 546 65 129 72 20 173 6 8 4 3 2 2 1 5 1 2 2 7 6 2 67 1 1 8 6 8 7 115 1 2 40 2 6 4 106 11 8 85 2300 123 318 42 21 87 23 18 1688 269 12 3 38 15 10 19 34 5 2 6 5 7 9 0 4 93 3 15 110 2447 104 440 71 9 7 3 30 20 0 6 3 1 1

1 15 2 1 50 23 1 18 21 108 1

2 33 2

5 1

8 1 2 1 4 14 23 1 13

4 2

3 15 2 1 42 22 4 24 16 87 1 1 6 30 5

3 1 1 26 14 15 14 95 1

4 4 6 1 24 8 1 15 9 79 2

3

10 14 1 12

4 3 4 5 5

4 3 1 4 5 6

7 3 1 5 5

1 30 7 2 12 12 73 2

7 2 1 4 4

8 3 3 5

7 3 1 3 4

7 2 1 4 4

19 18 7 11 116 3

7 3 1 3 6

145 10 1666 589 255 708 657 1207 82 20 11 6 13 11 7 5 1 5

55 1 50 1

69 2 1 36 3

62

693 37 112 42 100 160 22 15 191 115 194

20 52 152 89 164 20 92 194 196 180 181

751 53 37 164 63 86 151 13 20 204 50 160

871 58 32 145 56 66 158 16 253 63 159

876 34 22 171 43 92 173 15 170 141 107

842 57 27 111 38 123 154 7 245 160 170

1014 41 14 17 4 13 12 0 3 5 8

1084 32 18 9 7 11 11 0 8 6 4

5 1 1 4 2 3

3 49 1 3

29 2 9

1 28 1 7

1 32 6 3

1 29 3 3

6 1 4

6

5 2

5 2 1

5

9 3 1

5

34 165 136 142

3

26 166 68 172

9 17

28 196 146 118

1

1

4

1


(Figure 3)
Institut ion The Australian N ational Univ Anglo-Australian Observatory Aust Telescope N ational Facility Univ Melbourne Univ New South Wales Univ Sydney Univ Tasmania Aust D efence For ce Academy Monash Univ Swinburne Univ Technology Univ Adelaide Australia World Pubs 607 285 386 109 253 392 112 77 84 82 77 2256 55838 Actual 8676 3061 3276 779 3494 2645 792 613 756 536 598 22109 375669 Expected 5856 2240 3256 773 2030 3042 939 708 748 506 484 17962 375669 Act./Exp 1.5 1.4 1.0 1.0 1.7 0.9 0.8 0.9 1.0 1.1 1.2 1.2 1.0 Av. Exp 9.6 7.9 8.4 7.1 8.0 7.8 8.4 9.2 8.9 6.2 6.3 8.0 6.7 Av./Act 14.3 10.7 8.5 7.1 13.8 6.7 7.1 8.0 9.0 6.5 7.8 9.8 0.0