Int J Comput Vis (2009) 81: 167–171 DOI 10.1007/s11263-008-0155-3

An Improved FoE Model for Image Deblurring Dahong Xu · Runsheng Wang

Received: 26 June 2008 / Accepted: 30 June 2008 / Published online: 26 July 2008 © Springer Science + Business, Media, LLC 2008

Abstract Image restoration from noisy and blurred image is one of the important tasks in image processing and computer vision systems. In this paper, an improved Fields of Experts model for deconvolution of isotropic Gaussian blur is developed, where edges are preserved in deconvolution by introducing local prior information. The edges with different local background in a blur image are retained since local prior information is adaptively estimated. Experiments indicate that the proposed approach is capable of producing highly accurate solutions and preserving more edge and object boundaries than many other algorithms. Keywords Image deblurring · Fields of experts · Prior model

1 Introduction Image processing and computer vision systems usually operate on noisy and blurred images. The standard model u0 = k ∗ u + n is applicable to a large variety of image degradation processes that are encountered in practice. Image restoration from noisy and blurred image is one of the important tasks in image processing and computer vision systems. Restoring an image and sharpening the features within the image improves the detail and consequently the utility of an image. There are many approaches based on statistics (Demoment 1989), Fourier and/or wavelet transforms (Neelamani et al. 1999), or variational analysis (Chambolle and Lions 1997; D. Xu () · R. Wang ATR Lab, National University of Defense Technology, Chang sha 410073, China e-mail: [email protected]

Rudin et al. 1992) for it. Total Variation (TV) regularization approach and Fields of Experts (FoE) (Roth and Black 2005) model are paid more attention. The TV regularization problems were first introduced into the context of image denoising in the seminal paper (Rudin et al. 1992) by Rudin, Osher and Fatemi, and has become one of the standard techniques known for preserving sharp discontinuities. The standard total variation model recovers an image from a noisy and blurred image by solving the problem: 1 min f (u) ≡ min k ∗ u − u0 2L2 (Ω) + α u u 2

 |∇u|dx

(1)

Ω

where α is a parameter. u0 is the observed (blurred) image and k is the PSF. It is known that TV regularization works effectively for recovering “blocky” images. We are going to recover u with a priori knowledge of the PSF and the image. To devise numerical schemes for (1), let us write down the first order optimality conditions, namely,   ∂f ∇u = k(−x, −y) ∗ (k ∗ u − u0 ) − α∇ · = 0, ∂u |∇u| x ∈ Ω.

(2)

Following this pioneering work, most attention are focused on developing numerical methods for solving the total variation regularization model (1) and (2). Chan and Mulet (1999), Vogel and Oman (1996) proposed a “lagged diffusivity” procedure that solves the following equations (3) for u(n+1) iteratively 

∇u(n+1) ∇· ∇u(n) α



− λK ∗ (Ku(n+1) − u0 ) = 0

(3)

168

Goldforb and Yin (2005) proposed a unified approach for various total variation models including (1) based on second-order cone programming. They model problem (1) as a second-order cone program, which is then solved by modern interior-point methods. Wang et al. (2007) propose a simple algorithmic framework for recovering images from noisy and blurred image based on total variation (TV) regularization. Using a splitting technique, they construct an iterative procedure of alternately solving a pair of easy subproblems associated with an increasing sequence of penalty parameter values. The main computation at each iteration is three Fast Fourier Transforms (FFTs). In practice, the need for prior models of image structure occurs in many machine vision problems including stereo, optical flow, denoising, super-resolution, and there has been great interest in trying to learn this prior from examples. Recently, some image denoising and deblurring techniques take up prior information based on natural image statistics. Since images are very non-Gaussian statistics, high dimensional, continuous signals, learning their distribution presents a tremendous computational challenge. The most successful recent algorithm is the Fields of Experts (FoE) (Roth and Black 2005) model which has shown impressive performance by modeling image statistics with a product of potentials defined on filter outputs. However, we notice that FoE model utilizes globe prior information and is not applied to deconvolution. In this paper, an improved FoE model for deconvolution of isotropic Gaussian blur is developed, where edges are preserved in deconvolution by introducing local prior information. The edges with different local background in a blurred image are retained since local prior information is adaptively estimated.

2 Related Work Two areas of research relate to our problem: image priors and variational methods. Recently, Roth and Black (2005) develop a framework for learning generic, expressive image priors, known as Fields of Experts (FoE), which capture the statistics of natural scenes, for the purpose of inpainting and denoising images. The approach extends traditional Markov Random Field (MRF) models by learning potential functions over extended pixel neighborhoods. However, there is a great deal of research being carried out on this problem in image denoising. One area of interest is that of priors model based image deblurring. In Jalobeanu et al. (2004), the authors propose to use an inhomogeneous Gaussian model for satellite image deblurring. They address the problem of space-varying parameter estimation for a

Int J Comput Vis (2009) 81: 167–171

Gaussian model by using a hybrid approach. They apply the Maximum Likelihood Estimator (MLE) on complete data that are approximated by the result of a wavelet-based deconvolution algorithm. By this way, a robust estimate of the prior model parameters is obtained. In Takeda et al. (2008), the authors extend the use of Locally adaptive kernel regression for deblurring applications. The proposed algorithm takes advantage of an effective and novel image prior that generalizes some of the most popular regularization techniques. Bar et al. (2006), present an integrated framework for simultaneous semi-blind restoration and image segmentation. They present an algorithm, based on functional minimization, that iteratively alternates between segmentation, parametric blur identification and restoration.

3 Image Deblurring Using a Prior 3.1 Products of Experts High-dimensional data often satisfies many different lowdimensional constraints simultaneously. An efficient way to model such data is therefore to multiply together several probability distributions that are expert over each of these low-dimensional spaces—this is the premise of Hinton’s Products of Experts (Hinton 2002). In the PoE framework experts in the model each work on a linear one-dimensional subspace of the data, by projecting the vectorized patch, x, onto a linear component, Ji . The distribution of each expert is modeled by the Student-t distribution (Huang and Mumford 1999). The prior therefore takes the form  N  1 T 2 −αi 1  1 + (Ji x) , p(x) = Z(Θ) 2

αi > 0,

(4)

i=1

where N is the number of experts in the model, and {J1 , . . . , JN , α1 , . . . , αN } = Θ are the model parameters. Z(Θ), the partition function, is a normalization constant that ensures that p(x) integrates to one. Some algorithms for both learning the model parameters and applying the model that do not require Z(Θ) to be known. Inference algorithms dispense with the partition function by writing the probability as a Gibbs distribution: p(x) = exp(−E(x))/Z(Θ)

(5)

and minimizing the energy: EPoE (x) =

N  i=1

  1 T 2 αi log 1 + (Ji x) . 2

(6)

Int J Comput Vis (2009) 81: 167–171

169

3.2 Fields of Experts

the form:

FoE (Roth and Black 2005) is developed in applying PoE model to vision applications. Its aim is to learn a parametric prior over m × m image patches that accounts for the inter-dependence of neighboring image patches. The energy is therefore explicitly dependent on all the vectorized m × m image patches, x1,...,K , in an image, V , thus taking the form: EFoE (V ) =

K  N  k=1 i=1

  1 T 2 αi log 1 + (Ji xk ) . 2

(7)

It is utilized to the whole image. This model precedes PoE model in vision applications since learning at local image, and has shorting since local image sample is randomly selected. 3.3 Proposed Approach Significant progress in low-level vision has been achieved by algorithms that are based on energy minimization. Typically, the algorithm’s output is calculated by minimizing an energy function that is the sum of two terms: a data fidelity term which measure the likelihood of the input image given the output and a prior term which encodes prior assumptions about the output. For low-level vision tasks, for example image restoration, where the output is a natural image, the prior should capture some knowledge about the space of natural images. In this paper, an improved FoE model for deconvolution of isotropic Gaussian blur is developed, which is extended FoE model. We mention that image contents of a blurred image are variant at different location of the image, and image deblurring should considers the image content variance. It is recently regarded by some authors (Takeda et al. 2008), who extend the use of Locally adaptive kernel regression (AKTV) for deblurring applications. Our main improved issues are that Ji , i = 1, . . . , N are adaptively estimated based on spatially local variances at every local image regions, and relative weighted parameters βi , i = 1, . . . , N are adaptively estimated as well. The Proposed Approach therefore takes Fig. 1 Camerman original, blurred and canny edge map image

1 min f (u) = k ∗ u − u0 2 + γ ∇u2 u 2    1 +β αi log 1 + (JiT uk )2 2

(8)

i,k

where the filter Ji is learned from natural images. It should refer to a fact that the first two items in (8) are often used in general reference about deblur approaches, but parameter γ is fixed. The proposed approach appends the third item, and parameters of both second and third items are adaptively estimated together. A preprocessing is needed in the new approach, Edge regions in a blurred image are extracted, and classified into two classes by a threshold of edge region length, that is, plain region classes and varied region classes. Parameters in plain regions are constant. Parameters in varied regions are adaptively varied. The parameters αi as well as the linear filters Ji can be learned from a set of training images. The parameter γ is in inverse proportion to edge length, and the parameter β is in proportional to edge length. In this paper, for the filter Ji learned from images, we introduce (1) tractable lower and upper bounds on the partition function of models based on filter outputs and (2) efficient learning algorithms that do not require any sampling. These results are based on recent results in machine learning that deal with Gaussian potentials. We extend these results to non-Gaussian potentials and derive a novel, basis rotation algorithm for approximating the maximum likelihood filters. These results allow us to (1) rigorously compare the likelihood of different models and (2) calculate high likelihood models of natural image statistics in a matter of minutes. Applying our results to previous models shows that the nonintuitive features are not an artifact of the learning process but rather are capturing robust properties of natural images.

4 Numerical Results In order to investigate the approach performance, we compare the new approach with AKTV algorithm (Takeda et al. 2008). Experimental images are showed in Figs. 1 and 2,

170

Int J Comput Vis (2009) 81: 167–171

Fig. 2 Paper original, blurred and canny edge map image

Fig. 3 The reconstruction of our approach method

Fig. 6 The reconstruction of AKTV method

Clearly, there are many other deblurring algorithms that are not included in our discussion and comparison, but a more exhaustive comparison is beyond the scope of this work.

5 Conclusions

Fig. 4 The reconstruction of AKTV method

Because of the ill-posedness of the nonlinear image deblurring problem equation (1), some form of regularization is necessary In this paper, an improved FoE model for image deblurring is developed by introducing a prior term into deblurring problem. It can obtain highly accurate solutions and preserving more edge and object boundaries than many other algorithms.

References

Fig. 5 The reconstruction of our approach method

respectively. The original clean images is blurred with Gaussian blurring. The quality of restored images are measured by their edges and object boundaries. Figures 3 and 5 are the results by using our approach, respectively. Figures 4 and 6 are the results by using AKTV algorithm, respectively. Experiments indicate that our method is capable of producing highly accurate solutions and preserving more edge and object boundaries than many other algorithms.

Bar, L., Sochen, N., & Kiryati, N. (2006). Semi-blind image restoration via Mumford-Shah regularization. IEEE Transactions on Image Processing, 15(2), 483–493. Chambolle, A., & Lions, P. L. (1997). Image recovery via total variation minimization and related problems. Numerische Mathematik, 76(2), 167–188. Chan, T. F., & Mulet, P. (1999). On the convergence of the lagged diffusivity fixed point method in total variation image restoration. SIAM Journal on Numerical Analysis, 36(2), 354–367. Demoment, G. (1989). Image reconstruction and restoration: Overview of common estimation structures and problems. IEEE Transactions on Acoustics, Speech, & Signal Processing, 37(12), 2024– 2036. Goldforb, D., & Yin, W. (2005). Second-order cone programming methods for total variation-based image restoration. SIAM Journal on Scientific Computing, 27(2), 622–645.

Int J Comput Vis (2009) 81: 167–171 Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771–1800. Huang, J., & Mumford, D. (1999). Statistics of natural images and models. In Proceedings of the CVPR (pp. 1541–1547). Jalobeanu, A., Blanc-Féraud, L., & Zerubia, J. (2004). An adaptive Gaussian model for satellite image deblurring. IEEE Transactions on Image Processing, 13(4). Neelamani, R., Choi, H., & Baraniuk, R. G. (1999). Wavelet-based deconvolution for ill-conditioned systems. In Proceedings of the IEEE ICASSP (Vol. 6, pp. 3241–3244), March 1999. Roth, S., & Black, M. J. (2005). Fields of experts: a framework for learning image priors. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. 860– 867).

171 Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268. Takeda, H., Farsiu, S., & Milanfar, P. (2008). Deblurring using regularized locally-adaptive kernel regression. IEEE Transactions on Image Processing, 14(4), 550–563. Vogel, C. R., & Oman, M. E. (1996). Iterative methods for total variation denoising. SIAM Journal on Scientific Computing, 17(1), 227–238. Wang, Y., Yin, W., & Zhang, Y. (2007). A fast algorithm for image deblurring with total variation regularization (CAAM Technical Report TR07-10). Department of Computational and Applied Mathematics, Rice University, Houston, Texas 77005, June 2007.