Signal Processing, 3(2), April 2009
Total Page:16
File Type:pdf, Size:1020Kb
The Asia-Pacific Signal and Information Processing Association (APSIPA) Distinguished Lecture 2012 Human-vision Friendly Processing fIfor Images an dGhid Graphics Weisi Lin Email: [email protected] . sg School of Computer Engineering Nanyang Technological University Singapore Presentation at National Sun Yat-Sen University, 20 April 2012 • Introduction ialial • Relevant ppyhysiolog ical & ppysycholog ical rrrr phenomena utouto • Basic Computational Modules T TTT • Perceptual Visual Processing & Applications e of e of • Concluding Remarks, Further Discussion & nnn n Possible Future Work utliutli • List of Most Relevant References (for further OOO O reading) 2 3 Current Problem in Visual Processing System Design The Human Visual System (HVS): the ultimate receiver of most processed images, video, and graphics Gap in most systems: tthtarget: human consumpti ti/on/appreci itiation technical design: non-perceptual criteria Perceptual modeling: user-oriented performance booster 4 Traditional Signal Quality Definitions & Measures • MSE (Mean Square Error) •SNR (Signal Noise Ratio) • PSNR (Peak SNR) • QoS (Quality of Service) • or their relatives 5 Possibilities of Perceptual Evaluation • Subjective viewing tests – ITU BT 500 standard, etc. – MOS (mean opinion score) – Shortcomings • Expensive, time consuming • No t su itabl e for au toma tic in-loop processing e.g., encoding, transmission, relaying, etc. • Not always reliable depending on viewers' physical conditions, emotional states, personalidilttl experience, display context • Objective visual quality metrics (VQMs) – MOS prediction – HVS mo de ling • physiology • psychophysics – Difficulties • inadequate understanding of the HVS • difficulty in modeling • computational complexity 6 Perceptual Modeling sensory perceptual emotional domain- specific non-perceptual models perceptual models application-specific perceptual models performance, difficulty, complexity application scopes Core models Enhanced models 7 8 • Introduction ialial • Relevant ppyhysiolog ical & ppysycholog ical rrrr phenomena utouto • Basic Computational Modules T TTT • Perceptual Visual Processing & Applications e of e of • Concluding Remarks, Further Discussion & nnn n Possible Future Work utliutli • List of Most Relevant References (for further OOO O reading) 9 Which square is brighter, A or B? 10 Adelson’s “Checker-shadow illusion” http://web.mit.edu/persci/people/adelson/checkershadow_ illusion.html 11 Visua l P at hways (lateral geniculate nucleus) 12 Human Eyes 13 The HVS: Specialized Processes & Information Differentiation • Cones – fovea on the retina – color vision • S-cones: blue • M-cones: green • L-cones: red – high visual acuity 14 The HVS: Specialized Processes & Information Differentiation (cont’d) • two pathways from the retina to the primary visual cortex (V1) – Magnocellular pathway • majority of the nerve fibres • thin fibres • slow in info transfer • all color and contrast info – Pavocellular pathway • thick fibres • fast in information transfer • all transient, motion related info • not responsive to chromatic info 15 The HVS: Specialized Processes & Information Differentiation (cont’d) • The visual cortex (V1~V4) distinguish – orientation –form – color – motion 16 Eye Movement – spontaneous smooth-pursuit eye movement (SPEM) • tend to track moving objects •reduce t he ret ilinal vel liocity of fi image – other eye movement • natural drift movement – responsible for perception of static images – very slow (0.8–1.5 deg/s) • saccadic movemen t – responsible for rapidly moving objects 17 Visual Attention • Selectivity – selective awareness of the sensory environment – selective responsiveness to visual stimuli • Two types – bottom-up: external stimuli – top-down: task related • Three stages for bottom-up process – pre-attentive • No capacity limitation • all the information can be processed (color, orientation, motion, curvature, size, depth, luster, shape) – Attention • feature integration • competition – Post-attention • improved search efficiency 18 Psychophysical Experiments Weber-Fechner law Contrast Sensitivity Function (CSF) sine-wave gratings DCT bas is funct ions Kelly’ 79 ,83 19 Contrast masking: relationship between maskee and masker dependency on orientations Legge’80 CT: maskee contrast CM: masker contrast facilitation The c hange o f con trast masking with maskee’s frequency 20 • Introduction ialial • Relevant ppyhysiolog ical & ppysycholog ical rrrr phenomena utouto • Basic Computational Modules T TTT • Signal Decomposition e of e of • Just-noticeable Difference (JND) nnn n • Modeling Visual Attention (VA) utliutli • Perceptual Visual Processing & Applications OOO O • Concluding Remarks, Further Discussion & Possible Future Work • List ofMf Most RlRelevant RfReferences (for furt her reading) 21 Temporal Decomposition • Physiological evidence – two main visual pathways – visual cortex • Signal decomposition – Implemented as FIR/IIR filters • sustained (low-pass) channel • transient (band-pass) channel Coefficients for the sustained and transient 9 -tap FIR filters 22 Spatial Decomposition Simoncelli et al.’92 – Filters Gabor, Cortex, wavelets Gaussian/sterrable pyramid – Stimuli : ori ent ati ons, f requenci es 23 Just-noticeable Difference (JND) • JND: the visibility threshold below which any change cannot be detected by the HVS – e.g.,75% of the subjects • difference: not necessarily distortion better not to say “just-noticeable distortion” – Edge enhancement – Removal of flickers – Post-processing • explicit model is helpful • 2 JNDs, 3 JNDs, …can be also determined 24 JND in DCT Subbands GlfliDCTbbdGeneral formula in DCT subbands: n: DCT block index iji,j: DCT subband where : base threshold due to spatial CSF : eltilevation paramet tdtlier due to luminance adaptation, intra-band masking, and inter-band masking, etc. 25 Modeling Spatial CSF parabola equation (Ahumada & Peterson’92): T: visibility threshold due to spatial CSF f: spatial frequency vertex location: threshold at fp: (Van Nes & Bouman’67) Tmin L: the lumnance other parameters: 26 Luminance Adaptation • a simplified model with Weber-Fechner law (Wa tson’93) 0.649 αlum(n) ~ c (n,0,0) where c(n, 0, 0): DC coefficient of the block •more realisti c mod eli ng Jayant & Safranek’93, Chou & Li’95 25 20 15 10 threshold 5 0 0 50 100 150 200 250 grey-level 27 Contrast Masking • Intra-bdband Mas king (Watson’93, Hontsch & Karam’02) where 0<ζ<1 28 Contrast Masking (cont’d) • Inter-band Masking (Tong&Venetsanopoulos’98) Emh(n): energy in MF and HF 29 JND with Pixels Yang, et al .’05 luminance adaptation overlapping effect texture masking 25 20 dd 15 ~ gradient 10 threshol measure: 5 0 0 50 100 150 200 250 grey-level 30 Temporal Masking Effect (Chou & Chen’96) average inter-frame luminance difference increase of maskinggg effect with increase in interframe changes 31 Eye movement model (Da ly’01) retiliinal image vel liocity: v()(n,t)=vI()(n,t)-vE()(n,t) image plane object velocity in retina eye movement velocity if no eye motion were involved 32 Modeling Visual Attention (VA) The VA -- • a result of the million years’ evolution • foveainretinafovea in retina • eye movement Therefore, the HVS-- • seltilective sensor • with limited source - “processing power” - “internal memory” 33 Auto-generation of visual attention map motion face-eye skin color contrast texture Lu, et al, ’05, IEEE T-IP 34 In line with eye tracking Correspondence with fMRI Lee, et al.’07 Overa ll visua l sensit iv ity --modulation of JND by VA --both local & global features various eye trackers 35 Alternative approach to detect bottom-up VA For an image I(x; y), Fourier Transform VA: (Hou & Zhangg)’07) 36 Influence from audio/speech • Integration of “aural attention” • Multimedia: correlation & interaction of difference media • More often than not , presented simultaneously • Examples in joint modelling – Ma, et. al. ’05 – You, et. al. ’07 37 • Introduction ialial • Relevant ppyhysiolog ical & ppysycholog ical rrrr phenomena utouto • Basic Computational Modules T TTT • Perceptual Visual Processing & Applications e of e of • Concluding Remarks, Further Discussion & nnn n Possible Future Work utliutli • List of Most Relevant References (for further OOO O reading) 38 Visual Quality Gauge 1 XY MAE = ∑∑ Δd(x, y) XY xy−=11 a traditional metric fails 1 XY MSE = ∑∑ MAE 2 XY x−11y= A2 PSNR = 10lg MSE major reasons for failure: is noticeable local characteristics not every Δ d ( x , y ) results in distortion receives same attention global characteristics 39 Noti ceabl e Con tr ast C h an ges ⎧ 0 if I(x, y) − I(x, y) ≤ jnd(x, y) ⎪ c(x, y) = ⎨ I(x, y) − I(x, y) ⎪ otherwise ⎩⎪ jnd(x, y) I(x, y) is calculated in a image neighborhood 40 New Visual Quality Metric • Discrimination of c((,y)x,y) + c ne: c increase at non-edge pixels—degradation - c ne: c decrease at non-edge pixels—degradation + c e: c increase at edge —enhancement c-e: c decrease at edge contrast—the worst degradation − + − + D = α1c ne + α 2c ne + α 3c e − α 4c e where α 3 > max(α1 ,α 2 ) >α 4 >0 Lin, et al.’05 • D reduces to the mean absolute error (MAE) measure, if – JND is constant – different contrast changes are not differentiated 41 “…to tell a good picture from a good one…” •Database with demosaiced image •50 images •1524x1012 scenes •9 subjects Better quality than the original image (Longere, et al.’02) (new metric) (due to edge sharpening) 42 Further Test Results Pearson correlation (for accuracy) (with VQEG-I Data) •320 comppqressed video sequences •Both PAL and NTSC format •Different codecs •768