sign in sign up
<<
Home , Digital image processing, Sinc filter

ECE 484 Processing Lec 10 - Image Interpolation

Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: [email protected], Ph: x 2346. http://l.web.umkc.edu/lizhu

slides created with WPS Office Linux and EqualX equation editor

Z. Li, ECE 484 Digital Image Processing, 2019 p.1 Outline  Recap of Lec 09  Image Sampling . subsampling . interpolation  Summary

Z. Li, ECE 484 Digital Image Processing, 2019 p.2 BF - kernel reflects range and value changes  Bilateral Filtering process

W Ws Ws*Wc * c

Output input

kernelphotometry kernel bilateral kernel

Z. Li, ECE 484 Digital Image Processing, 2019 p.3 Non Local Means (NLM) Filters  Baudes et al. (2005) use a weighted average based on similarity

. BL filter's spatial kernel is turned off . intensity kernel reflects similarity between self similar patches: j and its neighborhood y(j) is defined as say 5x5 block centered at j.

Z. Li, ECE 484 Digital Image Processing, 2019 p.4 CBF - Cross  Decouple the filter image input, from the images that gives range/value kernel . a guide image I (in a way NLM is a guided filter ) from say short images, while filter input is the long exposure image

Z. Li, ECE 484 Digital Image Processing, 2019 p.5 Guided Filter - Definitions  Motivated by the Cross Bilateral Filter: . Filter input image {pi} . Guide image {Ii} . Filter output {qi}  Output:

. weight W is completely derived from the guidance image I. . Guided image filter output:

where {ak, bk} is totally defined by the neigborhood wk • The filter image output model: qi = pi - ni . The output is the input image subtract some noise

Z. Li, ECE 484 Digital Image Processing, 2019 p.6 Guided Filter Operations  Denoising . when covariance of I and P is small, then GF is in denoising/ mode

 Edge Preserving . when covariance of I and P are large, i.e, very similar, then GF is preserving edge.

Z. Li, ECE 484 Digital Image Processing, 2019 p.7 Outline  Recap of Lec 09  Image Sampling . anti- subsampling . interpolation  Summary

Z. Li, ECE 484 Digital Image Processing, 2019 p.8 Image Scaling  Need to resample images

This image is too big to fit on the screen. How can we reduce it?

How to generate a half- sized version?

Z. Li, ECE 484 Digital Image Processing, 2019 p.9 Image sub-sampling  sub-sample without filtering, what is wrong ?

1/8

1/4

Throw away every other row and column to create a 1/2 size image - called image sub-sampling

Z. Li, ECE 484 Digital Image Processing, 2019 p.10 Image sub-sampling  Aliasing...

1/2 (2x zoom) 1/4 1/8 (4x zoom) Why does this look so crufty?

Z. Li, ECE 484 Digital Image Processing, 2019 p.11 Even worse for synthetic images  Aliasing effect

Z. Li, ECE 484 Digital Image Processing, 2019 p.12 Sampling and the

 Aliasing can arise when you sample a continuous or image . occurs when your sampling rate is not high enough to capture the amount of detail in your image . Can give you the wrong signal/image—an alias . formally, the image contains structure at different scales . called “frequencies” in the Fourier domain . the sampling rate must be high enough to capture the highest frequency in the image  To avoid aliasing: . sampling rate > 2 * max frequency in the image . This minimum sampling rate is called the Nyquist rate

Z. Li, ECE 484 Digital Image Processing, 2019 p.13 Sampling in Time Domain  needs a discrete representation of

• Many signals originate as continuous-time signals, e.g. conventional music or voice • By sampling a continuous-time signal at isolated, equally-spaced points in time, we obtain a sequence of numbers

Ts

n  {…, -2, -1, 0, 1, 2,…} t T Ts is the sampling period. s s(t)

Sampled analog impulse waveform train Z. Li, ECE 484 Digital Image Processing, 2019 p.14 Spectrum of several special functions  Spatial-Frequency signatures

Z. Li, ECE 484 Digital Image Processing, 2019 p.15 Consequence in Freq Domain  Multiplication with sampling train function, is convolving in freq domain • Replicates spectrum of continuous-time signal At offsets that are integer multiples of sampling frequency

• of impulse train where s = 2  fs

• Example Modulation by Modulation by cos(2 cos( t)  t) F() s G() s

  -2f 2f max max s s s s

Z. Li, ECE 484 Digital Image Processing, 2019 p.16 Shannon/Nyquist Sampling Theorem  What is the sampling rate to recover the original continuous signal w/o loss ? • A continuous-time signal x(t) with frequencies no higher than fmax can be reconstructed from its samples x[n] = x(n Ts) if the samples are taken at a rate fs which is greater than 2 fmax.

Nyquist rate = 2 fmax

Nyquist frequency = fs/2.

• What happens if fs = 2fmax?

• Consider a sinusoid sin(2 p fmax t)

Use a sampling period of Ts = 1/fs = 1/2fmax.

Sketch: sinusoid with zeros at t = 0, 1/2fmax, 1/fmax, …

Z. Li, ECE 484 Digital Image Processing, 2019 p.17 Illustration of Sampling Theorem  1-D example

sampling pattern w 1/w

sampled signal

Spatial domain

Z. Li, ECE 484 Digital Image Processing, 2019 p.18 Reconstruction  Recon via convolving with (low pass filtering) . in practice, truncated sinc() function

sinc function w 1/w

reconstructed signal

Spatial domain Frequency domain

Z. Li, ECE 484 Digital Image Processing, 2019 p.19 Aliasing from under-sampled reconstruction  Aliasing effect (a) f(x) and its spectrum (b) sampling train (c) aliasing from under sampling (a) (d) reconstruction via convolving with sinc function (e) aliasing (b)

(c)

(d)

(e) Z. Li, ECE 484 Digital Image Processing, 2019 p.20 Anti-Aliasing Filtering  Apply low pass filtering to limit the band of signal before filtering . ideally need to use sinc filter . infinite taps, not practical . use Gaussian instead or . Low Pass FIR design below

Z. Li, ECE 484 Digital Image Processing, 2019 p.21 Image Anti-Aliasing Filtering  Images are not bandlimited as we would like

 2D sampling effects

Z. Li, ECE 484 Digital Image Processing, 2019 p.22 2D image anti aliasing example  Pre-Low Pass Filter before subsampling

Z. Li, ECE 484 Digital Image Processing, 2019 p.23 Subsampling with Gaussian pre-filtering  Pre LP filtering with Gaussian

G 1/8

G 1/4

Gaussian 1/2 Solution: filter the image, then subsample • Filter size should double for each ½ size reduction. Why?

Z. Li, ECE 484 Digital Image Processing, 2019 p.24 Subsampling with Gaussian pre-filtering  Anti-alias filtered sub-sampling

Gaussian 1/2 G 1/4 G 1/8 Solution: filter the image, then subsample • Filter size should double for each ½ size reduction. Why? • How can we speed this up? Z. Li, ECE 484 Digital Image Processing, 2019 p.25 Aliasing w/o pre filtering  In comparison, w/o anti-aliasing filtering:

1/2 1/4 (2x zoom) 1/8 (4x zoom)

Z. Li, ECE 484 Digital Image Processing, 2019 p.26 Outline  Recap of Lec 09  Image Sampling . anti-aliasing subsampling . interpolation  Summary

Z. Li, ECE 484 Digital Image Processing, 2019 p.27 Image Resampling/Interpolation  Reconstruct the continuous signal with different filters

Z. Li, ECE 484 Digital Image Processing, 2019 p.28 0-th and 1-st Order Interpolation  Simple interpolation schemes

1st order: not differentiable 0-th order: introduced discontinuity

Z. Li, ECE 484 Digital Image Processing, 2019 p.29 Linear Image resampling/interpolation

 So what to do if we don’t know • Answer: guess an approximation • Can be done in a principled way: filtering

1 d = 1 in this example

1 2 2.5 3 4 5

Image reconstruction • Convert to a continuous function

• Reconstruct by :

Z. Li, ECE 484 Digital Image Processing, 2019 p.30 Cubic Interpolation  Linear, Quadratic and Cubic Interpolation

 Cubic is popular, supported in Matlab x = -3:3; y = [-1 -1 -1 0 1 1 1]; xq1 = -3:.01:3; s = spline(x,y,xq1); plot(x,y,'o',xq1,p,'-',xq1,s,'-.') legend('Sample Points','spline','Location','SouthEast')

Z. Li, ECE 484 Digital Image Processing, 2019 p.31 2D Interpolation  Divide & conquer . 1/2 pel position interpolation

Z. Li, ECE 484 Digital Image Processing, 2019 p.32 Bilinear Interpolation  2D linear interpolation

Z. Li, ECE 484 Digital Image Processing, 2019 p.33 Interpolation Results  NN, Bilinear vs Bicubic

Original image: x 10

Nearest-neighbor interpolation Bilinear interpolation Bicubic interpolation

Z. Li, ECE 484 Digital Image Processing, 2019 p.34 DCT derived Interpolation Filter  DCTIF: Elena & Alex Alshina from Samsung DMC: . JVTVC-F247: https://www.itu.int/wftp3/av-arch/jctvc- site/2011_07_F_Torino/JCTVC-F247-v6.zip  Basic Idea: . DCT for signal representation:

. then the signal at fractional pixel location can be derived as,

Z. Li, ECE 484 Digital Image Processing, 2019 p.35 DCTIF Interpolation Filter  DCT Derived interpolation filters (DCTIF, Samsung) . 1/4 pel interpolations

. row first then column

Z. Li, ECE 484 Digital Image Processing, 2019 p.36 DCTIF Filter  Filter derived from DCT transforms . row interpolation

. column interpolation

 Performance . better than bicubicle !

Z. Li, ECE 484 Digital Image Processing, 2019 p.37 Deep Interpolation Filter  High performing solution in HEVC for sub-pel motion estimation . half pixel location images:

. ground truth:

. Network:

Z. Li, ECE 484 Digital Image Processing, 2019 p.38 Summary  Downsampling images: . downsampling is multiplication with the impulse train in time domain . it is convolution with signal spectrum in freq domain . if not sampling at 2fmax, aliasing.  Interpolation . Ideal reconstruction via Sinc function, infinite support not practical . Other form of interpolation filters, best performing for images, DCTIF . Deep learning solution for interpolation: CNN directly learn fractional pel location images.

Z. Li, ECE 484 Digital Image Processing, 2019 p.39