INTERPOLATED HALFTONING, REHALFTONING, AND HALFTONE COMPRESSION Prof. Brian L. Evans [email protected] http://www.ece.utexas.edu/~bevans Collaboration with Dr. Thomas D. Kite and Mr. Niranjan Damera-Venkata Laboratory for Image and Video Engineering The University of Texas at Austin http://anchovy.ece.utexas.edu/ OUTLINE n Introduction to halftoning n Halftoning by error diffusion 4 Linear gain model 4 Modified error diffusion n Interpolated halftoning n Rehalftoning n JBIG2 halftone compression n Conclusions 2 INTRODUCTION: HALFTONING n Was analog, now digital processing n Wordlength reduction for images 4 8-bit to 1-bit for grayscale 4 24-bit RGB to 8-bit for color displays 4 24-bit RGB to CMYK for color printers n Applications 4 Printers 4Digital copiers 4 Liquid crystal displays 4 Video cards n Halftoning methods 4 Screening 4 Error diffusion 4 Direct binary search 4 Hybrid schemes 3 EXAMPLE HALFTONES Original image Direct binary search Clustered dot screen Floyd Steinberg Dispersed dot screen Modified Diffusion 4 FOURIER TRANSFORMS Original image Direct binary search Clustered dot screen Floyd Steinberg Dispersed dot screen Modified Diffusion 5 ERROR DIFFUSION n 2-D delta-sigma modulator n Noise shaping feedback coder 0 + x i; j xi; j y i; j ei; j H z + n Error filter 7 16 5 3 1 16 16 16 n Raster scan order P P P P P P P P P P P P P P P = Past P P F F F P F = Future F F F F F F F F F F F F F F n Serpentine scan also used 6 ERROR DIFFUSION (cont.) n Quantizer 0 0; xi; j < 0:5 y i; j = 0 1; x i; j 0:5 n Governing equations 0 ei; j = y i; j x i; j 0 x i; j = xi; j hi; j ei; j n Non-linearity difficult to analyze n Linearize quantizer [Kite, Evans, Bovik & Sculley 1997] ni; j Noise Path 0 0 K K n i; j + ni; j n i; j n n Signal 0 0 x i; j K K x i; j s s Path n Separate signal and noise paths [Ardalan & Paulos 1987] 7 LINEAR GAIN MODEL n Quantization error correlated with input [Knox 1992] Floyd-Steinberg Jarvis, Judice & Ninke n Least squares fit of quantizer input to output defines signal gain 0 E [jx i; j j] K = s 0 2 2E [x i; j ] n K constant Signal gain: s n K =1 Noise gain: n 8 GAIN MODEL PREDICTIONS n Noise transfer function (NTF) NTF=1Hz 1.7 1.7 1.5 1.5 1 1 Magnitude 0.5 Magnitude 0.5 0 0 1 1 1 1 0.5 0.5 0.5 0.5 Frequency f / f 0 0 Frequency f / f Frequency f / f 0 0 Frequency f / f y N x N y N x N Predicted Measured n Signal transfer function (STF) K s STF = 1+K 1H z s 4 8 6 3 4 2 Magnitude Magnitude 2 1 1 1 1 1 0.5 0.5 0.5 0.5 Frequency f / f 0 0 Frequency f / f Frequency f / f 0 0 Frequency f / f y N x N y N x N Floyd-Steinberg Jarvis et al. 9 Predicting Signal Gain Ks n Predict Ks from error filter as: = − Ks 1.17R 0.2 π ω ω 2 ω ω ∫ X ( 1, 2 ) d 1d 2 = −π R π ω ω 2 ω ω 2 ω ω ∫ X ( 1, 2 ) H ( 1, 2 ) d 1d 2 −π X(ω1,ω2) = Fourier Transform of input to error filter H(ω1,ω2) = Fourier Transform of error filter But X(ω1,ω2) is the error spectrum, so ω ω = − ω ω X ( 1, 2 ) 1 H ( 1, 2 ) 10 MODIFIED ERROR DIFFUSION n Efficient method of adjusting sharpness [Eschbach & Knox 1991] L 0 00 + x i; j x i; j xi; j y i; j ei; j H z + n Equivalent circuit: pre-filter 00 + x i; j xi; j y i; j Gz ei; j H z + Gz=1+L1 H z n L can be chosen to compensate for frequency distortion 11 UNSHARPENED HALFTONES 1 K s = n L If then STF=1 (flat) K s n Accounts for frequency distortion Original image Jarvis halftone Unsharpened halftone Residual 12 INTERPOLATION n Image resizing n Different methods (increasing cost) 4 Nearest neighbor (NN) 4Bilinear (BL) n Nearest neighbor, bilinear methods 4 Low computational cost 4 Artifacts masked by quantization noise in halftone 4 Correct blurring by using modified error diffusion Interpolation Halftoning Original Halftone I(z) F(z) 13 INTERPOLATION n Design L for flat transfer function using linear gain model (L is constant for a given interpolator) n Compute transfer function of interpolation by M − − 1− z M 1− z M I (z) = x x NN − − 1 zx 1 z y − 2 − 2 1− z M 1− z M I (z) = x x BL − − 1 zx 1 z y n Compute signal transfer function K (1+ L(1− H (z))) F(z) = s + − 1 (Ks 1)H (z) n Compute L to flatten the end-to-end transfer function of the system 14 INTERPOLATION RESULTS Nearest neighbor ×2 Bilinear ×2 1.2 1.2 1 1 0.8 0.8 0.6 0.6 0.4 0.4 Magnitude Magnitude 0.2 0.2 0 0 0 0 0.2 1 0.2 1 0.4 0.8 0.4 0.8 0.6 0.6 0.6 0.4 0.6 0.4 0.8 0.2 0.8 0.2 1 1 0 Frequency f / f 0 Frequency f / f Frequency f / f y N Frequency f / f y N x N x N Transfer function Transfer function L = –0.0105 L = 0.340 15 REHALFTONING n Halftone conversion, manipulation n Assume input and output are error diffused halftones 4 Blurring corrected by using modified error diffusion 4 Noise leakage masked by halftoning 4 64 operations per pixel n For a 512 x 512 image 416 RISC MIPS 4 0.4 s on a 167 MHz Ultra-2 workstation Halftoning Inverse halftoning Re-halftoning Halftone Original 16 REHALFTONING (cont.) n Halftone conversion, manipulation n Error diffused halftones n Fixed lowpass inverse halftoning filter, compromise cut-off frequency 4 Noise leakage masked by halftoning 4 Correct blur by modified error diffusion 4 Computationally efficient Original Inverse Mo di ed Halftone image halftone halftone K 1 + L 1 H z K s s Gz 1+K 1H z 1+ K 1H z s s 17 REHALFTONING (cont.) n Use linear gain model to design L for flat response n Use approximation for digital j! 2 1+j! ! =2 frequency: e n Inverse halftoning filter is a simple separable FIR filter n L is computed to flatten the end to end transfer function of the system n We need to know halftoning filter coefficients for this scheme n Improve halftoning results using knowledge of type of halftone being rehalftoned 18 REHALFTONING RESULTS Original image Rehalftone 1.2 1 0.8 0.6 0.4 Magnitude 0.2 0 0 0.2 1 0.4 0.8 0.6 0.6 0.4 0.8 0.2 1 0 Frequency f / f Frequency f / f y N x N Signal transfer function 19 THE JBIG2 STANDARD n Lossy/lossless coding of bi-level text and halftone data Symbol Symbol Memory region dictionary decoder decoding Document Generic refinement Halftone Halftone region Memory dictionary decoder decoding n Scan vs. random mode 20 THE JBIG2 STANDARD (cont.) n Bi-level text coding 4 Hard pattern matching (lossy) 4 Soft pattern matching (lossless or near lossless) may be context based n Halftone coding 4 Direct halftone compression 4Context based halftone coding 4Inverse halftoning and compression of grayscale image n Implications 4 Printers, fax machines and scanners, will need to decode JBIG2 bitstreams 4 Fast decoding may require dedicated hardware and embedded software 4Need for low complexity, low memory solutions 21 PROBLEMS TO BE SOLVED n JBIG2 compression of halftones 4 Compress halftone directly, using a dictionary of patterns, or 4Convert halftone to grayscale (inverse halftoning) and compress grayscale image n Efficient coding of halftone data 4 Fax machines 4Digital archiving, scanning, and copying n Fast algorithms for JBIG2 codec 4 Interpolated halftoning in decoder 4 Rehalftoning in codec 22 PROBLEMS TO BE SOLVED n JBIG2 embedded decoders 4 Low memory requirements 4Low computational complexity 4 High parallelism n Inverse halftoning: a robust solution for lossy coding of halftones 4 Rendering device can use a different halftoning scheme than encoder 4 Multiresolution halftone rendering (archive browsing) 4 High halftone compression ratios (6:1) 4 Quality enhancement if the encoder halftoning method is transmitted n Low-cost embedded implementations 23 CONCLUSIONS n Linear gain model of error diffusion 4 Validate accuracy of quantizer model 4 Link between filter gain and signal gain n Rehalftoning and interpolation 4 Efficient algorithms 4 Impact on emerging JBIG2 standard n Web site for software and papers 4 http://www.ece.utexas.edu/~bevans/ projects/inverseHalftoning.html 24.