Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.stsci.edu/hst/acs/documents/isrs/isr0402.pdf Дата изменения: Tue Jan 20 23:01:05 2004 Дата индексирования: Sat Dec 22 23:26:04 2007 Кодировка: Поисковые слова: 47 tuc

Instrument Science Report ACS 2004-02

Lossy Compression of ACS images
Colin Cox, January 20, 2004

ABSTRACT A method of compressing images stored as floating point arrays was proposed several years ago by White and Greenfield. With the increased image sizes encountered in the last few years and the consequent need to distribute large data volumes, the value of applying such a procedure has become more evident. Methods such as this which offer significant compression ratios are lossy and there is always some concern that statistically important information might be discarded. Several astronomical images have been analyzed and, in the examples tested, compression ratios of about six were obtained with no significant information loss.

Introduction
With the large volumes of data that STScI processes it becomes of interest to deal with large images in compressed form. This is of particular value for the delivery of images to external computers where ultimately there is a limit to the available transmission rate. Lossless compression methods on complex images only offer a reduction in volume of 25% to 50% and sometimes even increases the size of the final files. Lossy methods can frequently compress by factors of ten or more, but there is always concern about what is lost. If images are designed only for viewing, compression factors greater than ten can be achieved with no visible difference between an original and a compressed image. For more critical measurement applications such as astrometry and photometry, one is more concerned that mathematical properties be retained. The question is, has anything scientifically important been lost by the compression.

Operated by the Association of Universities for Research in Astronomy, Inc., for the National Aeronautics and Space Administration


Instrument Science Report ACS 2004-02 A compression method has been developed at STScI that specifically addresses the problem of dealing with images expressed as floating point numbers. A description of the method is given in White and Greenfield, 1999 and only a few pertinent features and the performance on several test images will be described here. Consideration is being given to delivering images in compressed form. We would supply alongside, the program which performs the decompression. The same executable is able to perform the compression, so users can store their own FITS data in compressed form and reconstruct it for analysis. A user would always have the option of receiving uncompressed data. A longer term objective is to create analysis programs which operate directly on the compressed files.

Method Outline
The first stage of the algorithm converts the floating point numbers into integer form. Normally the floating point representation is much finer than needed for the application; many of the bits are not significant. For example, raw ACS data start as 16-bit integers and in CALACS processing are mapped into 32-bit real values which incorporate 24 bits in the mantissa. Retaining non-significant bits allows freedom from worry about rounding error possibly corrupting data, but restricts the compressibility. So the first step is to decide how small the step between successive number representations should be. Any physical observation will contain some noise, partly statistical and probably some instrumental. Variations in the true signal that are much smaller than the noise level cannot really be measured and so, at some point, we will not lose information if we do not record these small variations. The White and Greenfield algorithm takes a section of the image and first calculates the difference between successive pixels. The median value of this difference distribution multiplied by 1.483 is used as an estimate of the variance of the noise level. The factor is derived by analogy to a Gaussian distribution with the same median value. This is clearly a good estimate if most of the image area is sky background or if the signal varies slowly across the field. The image is then converted to a scaled integer array so that a selectable number of bits represents the size of this variance. This is the only step which permits any information loss. The successive operations use the Rice algorithm to losslessly compress the integer array and write the result into a FITS table. The original descriptionof the algorithm is given in a Nov 1991 JPL publication but a more easily found source is given in a web page in the references.

Test Images
White and Greenfield describe several tests involving astrometry and photometry on WFPC2 data plus a simulation involving averaging a set of spectra with a low signal-tonoise ratio. In all cases the compression was found to have a negligible effect on the analysis.

2


Instrument Science Report ACS 2004-02 I have chosen several ACS images which are typical observations and should test the limits of the algorithm. Two are crowded stellar fields, one is an image with large areas not being background and the fourth contains many non-stellar objects. The first test was to compress each image, uncompress it and compare the original and reconstituted image. Absolutely no differences were visible even when images were blinked. The next test was to subtract the raw and final images and see what the difference magnitude was and look for any structure. Where an error image was available for an image, the ratio of the difference to the error was studied. If the differences are small compared with the errors, then we are not losing meaningful data by these differences. A more stringent test, which is more closely related to how the data will be used, was to run the program daofind on original and reconstituted images. daofind locates the position of each star using fitting to selectable PSF shapes and calculates a magnitude. In these studies no attempt was made to find the proper zero point; only the comparative results were considered. A third type of test was to use the utility sextractor which identifies such things as galaxies and irregularly shaped objects, again using mathematical properties of the image which could be degraded by the compression procedure.

Stellar fields
The first image is of a section of 47 Tucanae imaged with the HRC. The image is shown in Figure 1. The reconstructed image could be shown for comparison, but visually it is indistinguishable from the original even on a monitor when one can zoom in and manipulate the stretch and contrast. Figure 2 shows the difference image in which can be seen faint streaks. This arises because each line of the image is treated separately and so can have slightly different levels. Figure 3 shows a histogram of the difference image while Figure 4 shows a histogram of the difference divided by the error of the original. The rectangular distribution is to be expected from the first stage of the compression which is essentially a rounding. The difference will just be the shift from a general real number to the nearest scaled integer. The distribution would be more exactly rectangular for a single compressed section. The different sections do have slightly different quantization steps. As can be seen, the differences are always less than a tenth of the original error size. Figure 5 shows a WFC image which has gone through a drizzling procedure which includes the removal of cosmic rays. Again there is no visible or functional difference between this and the reconstructed image. daofind found the same 2882 stars in each image with the same centroids and magnitudes.

3


Instrument Science Report ACS 2004-02 Figure 1: An HRC image of 47 Tucanae

Figure 2: Difference between original and reconstituted image

4


Instrument Science Report ACS 2004-02 Figure 3: Histograms of difference image

Figure 4: Histogram of ratio image

5


Instrument Science Report ACS 2004-02 Figure 5: ACS/WFC Image

6


Instrument Science Report ACS 2004-02 A section of the output summary from daofind is given for each image in the following tables.
XCENTER 3073.693 3467.698 761.542 3822.269 3011.271 3628.487 3058.853 1429.107 1189.949 414.171 1693.390 639.247 2084.321 939.498 569.180 975.938 2870.752 2151.067 843.221 1239.034 YCENTER 2130.559 2131.352 2131.869 2132.088 2139.097 2146.299 2146.628 2150.769 2158.268 2158.919 2162.829 2169.317 2174.465 2186.435 2191.927 2194.536 2195.066 2205.584 2207.317 2209.130 MAG -1.429 -1.248 -1.451 -3.667 -2.408 -2.030 -0.050 -2.378 -2.435 -2.649 -1.127 -0.759 -2.377 -2.075 -0.427 -0.909 -3.415 -4.225 -1.037 -2.461 SHARP 0.645 0.624 0.916 0.705 0.748 0.685 0.873 0.695 0.629 0.670 0.586 0.732 0.701 0.589 0.648 0.950 0.539 0.657 0.762 0.685 SROUND -0.174 -0.145 -0.355 0.191 0.122 0.305 -0.300 0.412 0.256 0.387 0.258 0.465 0.654 0.743 0.291 0.259 0.311 0.488 0.551 0.264 GROUND 0.039 0.024 -0.341 0.150 -0.065 -0.068 -0.561 0.234 0.041 0.226 -0.165 0.073 0.126 0.091 0.064 0.418 0.424 0.278 0.047 -0.084 ID 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520

Table 1. Sample of daofind output derived from standard image

7


Instrument Science Report ACS 2004-02

XCENTER 3073.693 3467.698 761.543 3822.269 3011.271 3628.486 3058.852 1429.107 1189.949 414.171 1693.390 639.248 2084.321 939.498 569.181 975.938 2870.752 2151.067 843.221 1239.034

YCENTER 2130.558 2131.352 2131.869 2132.088 2139.097 2146.300 2146.628 2150.769 2158.268 2158.920 2162.829 2169.316 2174.465 2186.435 2191.927 2194.537 2195.066 2205.584 2207.317 2209.130

MAG -1.429 -1.247 -1.451 -3.667 -2.408 -2.030 -0.049 -2.378 -2.435 -2.649 -1.127 -0.759 -2.377 -2.075 -0.426 -0.909 -3.415 -4.225 -1.038 -2.461

SHARP 0.645 0.624 0.915 0.705 0.748 0.685 0.873 0.695 0.629 0.670 0.586 0.732 0.701 0.589 0.648 0.950 0.539 0.657 0.762 0.685

SROUND -0.174 -0.144 -0.354 0.191 0.122 0.305 -0.301 0.412 0.256 0.387 0.258 0.465 0.654 0.744 0.289 0.260 0.311 0.488 0.551 0.265

GROUND 0.039 0.025 -0.341 0.150 -0.066 -0.068 -0.561 0.234 0.041 0.226 -0.165 0.073 0.125 0.091 0.063 0.418 0.424 0.278 0.047 -0.084

ID 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520

Table 2. Matching sample of daofind output from reconstructed image Occasional differences between matching positions and magnitudes are seen only in the third decimal place. The image of the V838 Monoceros light echo shell, Figure 6, is an example where most of the image contains source illumination and not much is sky background. Nevertheless, the compression works just as well, again causing no visible change in the image and differences much smaller than the signal. Finally, a sample image from the Great Observatories Origins Deep Survey (GOODS) containing a number of distant irregular galaxies was used in conjunction with a source extractor (Sextractor) program. This program generates a catalog of stars and galaxies found in the image along with several descriptive parameters. The catalogs generated from the image shown in Figure 8, before and after the compression/decompression sequence

8


Instrument Science Report ACS 2004-02 were compared. This turns out to be a sterner test of the compression and does result in some differences. Figure 6: Light Echo Image

The program hunts for faint objects at the limit of the useful signal-to-noise ratio and is therefore sensitive to modifications of the noise spectrum caused by the compression and decompression. 1853 matching sources were located in the original and reconstructed images. 29 were found in the original but missed in the reconstructed image while 39 unmatched sources were found in the reconstructed image. The question that must be addressed is, would using the reconstructed image cause any real source to be overlooked as compared with studying the original image? Figure 9 shows a magnified portion of the image with the sources detected in the original image marked by the best fit ellipses. The mean radius of the contour is drawn at twice the full width at half maximum position. Many of these do not appear to be real sources. The additional circular markers indicate which of the marked sources are not detected in the reconstructed image. For clarity, Figure 10 shows the same area with just the second set of markers. Even with the scale set to histogram equalization and the stretch set to maximize the visibility of faint objects, it is not clear that there is really a source at the center of these regions. The catalog output by Sextractor gives the total count of the fitted elliptical source plus an error estimate. Dividing the count by the error provides a sort of signal to noise statistic. The distribution of this statistic as a function of detected magnitude is shown in Figure 7. The objects missed in the reconstructed image are shown as diamond shapes and cluster around 28th magnitude, near the limit of detection for this image.

9


Instrument Science Report ACS 2004-02 Figure 7: Ratio of flux to flux error as a function of magnitude

Figure 8: Section of a GOODS image showing non-stellar objects

10


Instrument Science Report ACS 2004-02

Figure 9: Objects found by Source extractor

Figure 10: Objects missed in reconstructed image

11


Instrument Science Report ACS 2004-02 The brightest one is given as magnitude 26.6 and is the right hand one of the pair of marked sources in upper middle section of Figure 10. It is not clear whether there is a real object at this position or not. So at the extreme lower limit of useful signal-to-noise level, although visually there is no difference between the original and reconstructed image, there are small statistical differences which might occasionally lead to missing a faint object in analysis. A similar test was run retaining four bits of noise data. The compression factor was then close to five but there were still a few sources that behaved in the same way. The image analyzed above was taken from one epoch of GOODS. Later the same region became available but with five epochs included. Each local area containing a potentially missed source was examined. In none of the cases could a source be seen.

Compression Performance
All the images discussed could be compressed by factors ranging from 4.6 to 9.1. From experiments with several more images I find a typical value of around six using three bits to express the noise level. Using only two bits for the noise level on the WFC 47 Tuc image produced a compression factor of 11.
Original Size in MB 10.5 213.5 16.8 67.1 Compressed Size in MB 2.3 23.5 2.8 11.4 Compression Factor 4.6 9.1 6.0 5.9 Compression Time (sec.) 3 47 5 17 Decompress Time (sec.) 2 29 2 10

Image HRC 47 Tuc WFC 47 Tuc Light Echo Galaxies

Table 3. Compression factors and times with 3 bits of noise. The compression rate is about 4 megabytes per second on the above images, with decompression rates being about 50% faster. These rates were measured on a 359 MHz Sparc 60 machine with all files being on a local disk. The rate does not depend on the size of the file. This is to be expected as the method operates on successive portions of the image simply repeating as many times as necessary. The portions used in all these test were single rows of the image. For the same reason I would not expect any great variation in the performance as a function of the computer memory size. The Appendix gives instructions on how to run the program fcomp together with information on how the different components of a FITS file are treated.

12


Instrument Science Report ACS 2004-02

Summary
The compression method described here appears to be suitable for use in supplying astronomical data. Compression factors of about six are obtained by using three bits to encode the noise level. There is some numerical difference between the original image and its reconstituted form, but except for very faint objects at the detection limit of the image, nowhere did this result in any meaningful difference in the analysis.

Acknowledgements
The program executable fcomp and assistance in using it was supplied by Nadezhda Dencheva. I have had useful discussions with Perry Greenfield who supplied me with material and explanations. I particularly wish to thank Swara Ravindranath who provided me with catalogs of the GOODS images generated by the program SExtractor.

References
White, R. L. and Greenfield, P. 1999, A Scheme for Compressing Floating-Point Images, Astronomical Data Analysis Software and Systems VIII, ASP Conference Series Vol 172, 1999 p125. RICE local "28 March 1996", http://www.tass-survey.org/richmond/rice/rice.1.html#alg GOODS, The Great Observatories Origins Deep Survey, http://www.stsci.edu/ftp/science/goods/

13


Instrument Science Report ACS 2004-02

Appendix
Instructions and options for fcomp This is a fits image compression program, developed by the Science Software Branch at STScI. Three kinds of compression algorithms are available: Rice, Gzip and Plio For a detailed description of the format see: http://heasarc.gsfc.nasa.gov/docs/software/fitsio/compression/compress_image.htm l Usage: fcomp [options] infiles Options: -h[elp] : -r[:n] : -p : -g : -s[uffix] prints this page rice compression for all extensions pixel list for all extensions Gzip compression for all extensions : what string to append to main filename body: e.g. fn_cs.fits if suffix_string is `cs' default suffix is `cmp' for compression, `dcm' for decompression -o[utput]

: works only with one file -t : sets the size of the tile to the entire image. The default tilesize (without -t) is one row -t[:n1[xn2[xn3]]] : sets the tile size to be n1xn2xn3 -d : decompresses all extensions

Without any options, all extensions will be compressed with the default options: defaults: compression algorithm: RICE noise bits :4 Working with extensions:

14


Instrument Science Report ACS 2004-02

Pass the `-e' option to compress individual extensions. Multiple `-e' options are acceptible. The syntax is: -e:extension:[compression algorithm]:[noise bits] extension - can be extension name or extension number compression algorithm (optional) - can be `r', `g' or `p' noise bits (optional) - used with Rice algorithm If algorithm or noise bits are not specified, the default ones are used -e:hst will compress all hst type extensions with default values, which currently are HST Defaults: SCI: compression algorithm: RICE noise bits :6 ERR: compression algorithm: RICE noise bits :1 DQ: compression algorithm: PLIO noise bits :4 SAMP: compression algorithm: GZIP noise bits :4 TIME: compression algorithm: GZIP noise bits :4 Use the `-d,' option to decompress individual extensions. Multiple extension names or numbers are acceptible with `,' as a delimiter. The syntax is:

15


Instrument Science Report ACS 2004-02

-d,extension1[,extension2]

: decompresses a particular extension

Examples: 1. To compress all extensions using RICE with noise bits 3, writing the output to a file, called output.cmp.fits: fcomp -r:3 -o output.fits infile 2. To compress all extensions in all fits files in the current directory, using the default options: fcomp * 3. To compress extension 2 with Rice and noise bits 2 and all extensions with extension name `ERR' with GZIP, in a file infile.cmp.fits: fcomp -e:2:r:2 -e:ERR:g infile.fits 4. To compress all files with hst type extensions, using the defaults for hst extensions with a tile 100x100: fcomp -t:100x100 -e:hst * 5. To decompress all extensions in all files in the current directory : fcomp -d * 6. To decompress extensions 5 and all extensions with extension name `SCI' fcomp -d,5,SCI infile

16