Документ взят из кэша поисковой машины. Адрес оригинального документа : http://imaging.cmc.msu.ru/pub/2009.PRIA.Krylov_Nasonov_Ushmaev.SuperRes.en.pdf Дата изменения: Wed Aug 19 23:00:00 2009 Дата индексирования: Mon Oct 1 19:40:31 2012 Кодировка:

VIDEO SUPER-RESOLUTION WITH FAST DECONVOLUTION*
A. Krylov1, A. Nasonov1, O. Ushmaev
2

1

Laboratory of Mathematical Methods of Image Processing, Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, 119991, Russia, Moscow, Leninskie gory, (495) 939-11-29, kryl@cs.msu.ru, nasonov@cs.msu.ru
2

The Institute of Informatics Problems of the Russian Academy of Sciences, 119333, Russia, Moscow, Vavilova, 44, 2, (499) 135-62-60, olegu@biolink.ru

Super-resolution problem is posed as an inverse deconvolution problem. Fast non-iterative superresolution algorithm based on this approach is suggested. Different super-resolution problem statements for the cases of exactly and inexactly known transform operator were considered. Introduction The problem of super-resolution (SR) is to recover a high-resolution image from a set of several degraded lowresolution images. This problem is very important in human surveillance, biometrics, etc. because it can significantly improve image quality. There are two groups of video SR algorithms: learning-based and reconstruction-based. Learning-based algorithms enhance the resolution of a single image using information on the correspondence of sample low and high-resolution images. Reconstruction-based algorithms use only a set of low-resolution images to construct high-resolution image. More detailed introduction into video SR problems is given in [1], [2]. The majority of reconstruction-based algorithms use camera models [3] for downsampling the high-resolution image. The problem is posed as a set of equations

Ak z uk , k 1,2,..., N ,
where z is reconstructed high-resolution image, u
k

(1)

is k-th low-resolution image, Ak is a downsampling

operator which transforms z to u k , N is a number of low-resolution images. The operator Ak can be generally
*

The work was supported by federal target program "Scientific and scientific -pedagogical personnel of innovative Russia" in 2009 2013 and by grant RFBR 09-01-92474-MHKC


represented as Ak z DH

ca m k

FH

a tm

z n [3], where H
ca m

a tm

is atmosphere turbulence effect, Fk is a warping

operator like motion blur or motion deformation, H noise. We model H
a tm

is camera lens blur, D is a decimation operator, n is a

and H

ca m

as a single Gauss filter H , and the operator Ak takes the form

Ak z DFk Hz .

(2)

Warping operator Fk can be calculated, for example, using motion calculation in base points and interpolation in other points [4], [5]. Variational optical flow estimation approaches are also widely used (see [6], [7], [8], [9]). Problem definition We consider the superresolution problem (2) for z and u given on the discrete set {(i, j ) : i, j } .

k

Warping operator Fk is modeled as a set of correspondences between coordinates of points of source and warped image

Fk : ( ~ik j , ~ik, j ) (i, j ) , x, y

( Fk z)(i, j ) z( ~ik j , ~ik, j ) . x, y

The

operator

D

performs

scaling

( Dz)( x, y) z(sx, sy) , where s is the scaling factor. Combination of Fk and D results in

( DFk z)(i, j ) z( ~si,sj , ~si,sj ) z( xik, j , yik, j ) . xk yk xk yk Here we renamed ( ~si, sj , ~si,sj ) as ( xik, j , yik, j ) .

(3)

The image z is defined on discrete set (i, j ) , but the coordinates ( xik, j , yik, j ) are not grid points, so we use operator H for both filtering and interpolation:

Hz( x, y )

(i , j )

z (i, j )e
(i , j )



( x i ) 2 ( y j ) 2 2
2

2

e



( x i ) 2 ( y j ) 2 2

,

(4)

where is chosen in accordance with scale factor s . We use 0.4 s 2 1 . The problem (1) does not have a solution in most cases. We replace it with an error minimization problem

z R arg min Ak z u z
k 1

N

2 k2

,

(5)

where

2

is standard Euclidian norm.

Using the notation (3), operator Ak z (2) takes the form


( Ak z)(i, j ) Hz( xik, j , yik, j ) ,
and the super-resolution problem (5) takes the form
z R arg min Hz( xik, j , yik, j ) uk (i, j ) . z
k 1 ( i , j ) N 2

(6)

By changing multiple indexes with single index, we rewrite the formula (6) and define the problem as

z R arg min Hz( xn , yn ) wn . z
n

2

(7)

The problem (7) is ill-posed, so regularization methods [10] are used:
2 z R arg min Hz( xn , yn ) wn [ z n

z]

(8)

with a stabilizer [ z ] . Iterative method for solving (8) is discussed in [2]. In this paper, a non-iterative algorithm for solving (7) is proposed. Adaptive deconvolution We consider the problem of deconvolution on discrete 1D set for Gauss filter G

G(i)
The discrete convolution looks as

1 i2 exp ( 2 ) . 2 2

v Hz z G ,
v(i ) z ( j )G(i j ) .
j

The problem of deconvolution is to reconstruct z from given convolution result v Hz
z H 1v .

Inverse operator H

1

^ ^^ ^ ^ ^ can be constructed using Fourier transform: v zG , z v / G , and z can be found as a
1

^ convolution of v with inverse Fourier transform of 1 / G . Nevertheless, operator H

is unbounded in the

continuous case. Thus in the discrete case it significantly amplifies noise for a noisy data. To avoid this, we use a finite adaptive filter


z v C ,

z (i) z ( j )ci j , c j c j .
j k

j k

(9)

Coefficients c

j

in (9) are chosen to minimize z v C 2 . Filter length k is chosen in a way to make

deconvolution fast, but precise enough. We use k 3 . In two-dimensional case, we process consequently the rows and the columns of the image. For given super-resolution problem (5), we convolve low-resolution images with Gauss filter and calculate coefficients c j from given set of images. We seek for

{c j } arg min uk uk G C 2 . C
k 1

N

2

(10)

Experiments have shown that adaptive filter (9) does not significantly amplify noise. It depends on given images. If the images are noisy, then filter coefficients are smoothed and noise level does not significantly increase after deconvolution. This also means that regularization term (8) is not necessary because adaptive filter (9) is automatically tuned to noise level. We have compared adaptive filter with unsharp mask z (v v G) v G . Unsharp mask shows practically the same results, but it takes more time to estimate its parameters ( , ) . Problem solution If the points ( xn , yn ) in (7) are grid points, then deconvolution method using adaptive filter can be used. But in general case coordinate values xn , yn are not discrete. So, we use the following algorithm: 1. Calculate the values of Hz at all grid points (i, j ) using Gauss interpolation (4). 2. Perform deconvolution using adaptive filter. In Figure 1, the proposed super-resolution method is illustrated for test video sequence in comparison with other image resampling and super-resolution methods.

a) Single frame interpolated

b) Regularization-based super-

c) The proposed video super-

d) Regularization-based single


using nearest neighbor method

resolution [2]

resolution method (7).

frame image interpolation [11]

Fig. 1. Super-resolution results using 4 input images and scale factor s=4.

Problem discussion The proposed super-resolution method (7) shows very good results if the warping operator Fk is calculated precisely. If it has errors, the solution becomes unstable. To avoid this, we pose the super-resolution problem (1) in the presence of errors as follows:

zR

1N arg mzin Ak z u N k 1

2 k2

.

(11)

We make single-image super-resolution for every image and then calculate an average image. The approach (11) results in blurred image, but without artifacts caused by warping operator errors. This approach is illustrated in Figure 2.

a) The super-resolution method (7)

b) Super-resolution for unstable data (10)

Fig. 2. Super-resolution results for 4 input frames and factor s=4 for inexact warping operator.

Conclusion Fast non-iterative method for image super-resolution has been suggested. The method shows very good results if the warping operator is exactly estimated like in the case of only sub-pixel shifts in the given video sequence. Special version of the super-resolution algorithm has been suggested for the case of inexact warping operator. References
1. S. Borman, Robert L. Stevenson. Super -Resolution from Image Sequences -- A Review // Midwest Symposium on Circuits and Systems. - 1998. - P. 374­378. 2. A.S. Krylov, A.V. Nasonov, D.V. Sorokin. Face image super -resolution from video data with non -uniform illumination // Proc. Int. Conf. Graphicon. -2008. - P. 150­155. 3. S. Farsiu, D. Robinson, M. Elad, P. Milanfar. Fast and Robust Multi-Frame Super-Resolution // IEEE Trans. On Image Processing. - 2004. - Vol. 13, No. 10. - P. 1327­1344.


4. Sung Won Park, Marios Savvides. Breaking the limitation of manifold analysis for super -resolution of facial images // IEEE Int. Conf. on Acoustics, Speech and Signal Processing. - 2007. - Vol. 1. - P. 573­576. 5. Ha V. Le, Guna Seetharaman. A Super-Resolution Imaging Method Based on Dense Subpixel -Accurate Motion Fields // Proceedings of the Third International Works hop on Digital and Computational Video. - 2002, - P. 35­42. 6. B.D. Lucas, T. Kanade. An iterative image registration technique with an application to stereo vision // Proc. of Imaging understanding workshop. - 1981. - P. 121­130. 7. A. Bruhn, J. Weickert, C. Shnorr. Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods // International Journal of Computer Vision. - 2005. - Vol. 61, No. 3. - P. 211­231. 8. J. Weickert, C. Shnorr. Variational Optic Flow Computation with a Spatio -Temporal Smoothness Constraint // J. Math. Im. & Vis. - 2001. - Vol. 14. - P. 245­255. 9. T. Brox, A. Bruhn, N. Papenberg, J. Weickert. High Accuracy Optical Flow Estimation Based on a Theory for Warping // Proc. 8th European Conf. on Comp uter Vision. - 2004. - Vol. 4. - P. 25­36. 10. A.N. Tikhonov, V.Y. Arsenin. Solutions of Ill Posed Problems // WH Winston, Washington DC. - 1977. 11. A.S. Lukin, A.S. Krylov, A.V. Nasonov. Image Interpolation by Super-Resolution // Proc. Int. Conf. Graphicon. - 2006. - P. 239­ 242. Andrey S. Krylov (born 1956), Graduated from the Faculty of Computational Mathematics and

Cybernetics, Lomonosov Moscow State University (MSU). Received the degree of PhD in 1983. Currently an associate professor and head of the Laboratory of Mathematical Methods of Image Processing at the Faculty of Computational Mathematics and Cybernetics, MSU. His main research interests lie in mathematical methods of multimedia data processing.

Nasonov V. Andrey (born 1985), Graduated from the Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University (MSU). Currently a member of a scientific staff of the Laboratory of Mathematical Methods of Image Processing at the Faculty of Computational Mathematics and Cybernetics, MSU. His main research interests lie in variational methods of image processing, inverse and ill-posed problems.

Ushmaev S. Oleg (born 1981), Graduated from the Lomonosov Moscow State University in

2002. He received the advanced degree in computer science from the Russian Academy of Sciences, Institute for Informatics Problems, in 2004 with a thesis on multimodal biometrics. Since then, he has worked as member of the Institute research stuff. His research interests in the area of pattern recognition and image analysis include biometrics, fast image processing and statistic pattern recognition.