Rochester Institute of Technology RIT Scholar Works Theses 2004 Image sequence restoration by median filtering Shawn R. Jackson Follow this and additional works at: https://scholarworks.rit.edu/theses Recommended Citation Jackson, Shawn R., "Image sequence restoration by median filtering" (2004). Thesis. Rochester Institute of Technology. Accessed from This Thesis is brought to you for free and open access by RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact [email protected]. Image Sequence Restoration by Median Filtering by Shawn R. Jackson A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Engineering Supervised by Dr. Andreas Savakis Department of Computer Engineering Kate Gleason College of Engineering Rochester Institute of Technology Rochester, NY November of 2004 Approved By: Andreas Savakis Dr. Andreas Savakis, Professor and Department Head Primary Advisor - R.l T. Dept. of Computer Engineering Kenneth Hsu Dr. Kenneth Hsu, Professor Secondary Advisor - R.l T. Dept. of Computer Engineering Greg P. Semeraro Dr. Greg Semeraro, Assistant Professor Secondary Advisor - R.l T. Dept. of Computer Engineering Thesis Release Permission Form Rochester Institute of Technology Kate Gleason College of Engineering Title: Image Sequence Restoration by Median Filtering 1, Shawn R. Jackson, hereby grant permission to the WaUace Memorial Library to reproduce my thesis in whole or part. Shawn R. Jackson Shawn R. Jackson 11 Dedication This work is dedicated to Mom, Dad, Nicole, and Kristina. Thanks for pushing me the whole way. 111 Acknowledgements I would like to show my appreciation to Dr. Savakis, Dr. Hsu, and Dr. Semeraro, Thank you for your guidance and insight. I would also like to thank Eastman Kodak Company for providing the necessary tools and hardware. IV Abstract Median filters are non-linear filters that fit in the generic category of order-statistic filters. Median filters are widely used for reducing random defects, commonly characterized by impulse pepper" or "salt and noise in a single image. Motion estimation is the process of estimating the displacement vector between like pixels in the current frame and the reference frame. When dealing with a motion sequence, the motion vectors are the key for operating on corresponding pixels in several frames. This work explores the use of various motion estimation algorithms in combination with various median filter algorithms to provide noise suppression. The results are compared using two sets of metrics: performance-based and objective image quality-based. These results are used to determine the best motion estimation / median filter combination for image sequence restoration. The primary goals of this work are to implement a motion estimation and median filter algorithm in hardware and develop and benchmark a flexible software alternative restoration process. There are two unique median filter algorithms to this work. The first filter is a modification to a single frame adaptive median filter. The modification applied motion compensation and temporal concepts. The other is an adaptive extension to the multi-level (ML3D) filter, called adaptive multi-level (AML3D) filter. The extension provides adaptable filter window sizes to the multiple filter sets that comprise the ML3D filter. The adaptive median filter is capable of filtering an image in 26.88 seconds per frame and results in a PSNR improvement of 5.452dB. The AML3D is capable of filtering an image in 14.73 seconds per frame and results in a PSNR improvement of 6.273dB. The AML3D is a suitable alternative to the other median filters. Table of Contents Chapter 1: Introduction 1 Chapter 2: Techniquesfor Motion Estimation ofImage Sequences 6 2.1 Motion Estimation 9 2.2 Block Matching Methods 9 2.3 Gradient Based Methods 14 2.4 Pel-Recursive Based Methods 20 2.5 Selected Motion Estimation Algorithms 22 2.5.1 Simplified Black's Motion Estimation Algorithm 23 2.5.2 Horn-Schunck Motion Estimation Algorithm 24 Chapter 3: Techniquesfor Spado-Temporal Filtering ofImage Sequences. 30 3.1 Temporal Filtering 31 3.2 Spatio-Temporal Filtering 33 3.3 Median Filtering 34 3.4 Selected Median Filtering Algorithms 37 3.4.1 Motion Compensated Median Filter Algorithm 40 3.4.2 ML3D Filter Algorithm 41 3.4.3 ML3Dex Filter Algorithm 43 3.4.4 Arce Bi-directional Multi-level Median Filter Algorithm 46 3.4.5 Adaptive Median Filter Algorithm 47 3.4.6 Adaptive ML3D Filter Algorithm 50 3.4.7 Summary of Selected Median Filter Algorithms 53 Chapter 4: Software Implementation ofRestoration Process 55 4.1 Software Interaction Diagram 59 Chapter 5: Hardware Implementation ofRestoration Process 65 5.1 Description of Hardware Platform 65 vi 5.1.1 Overview ofWildStar PCI Hardware 65 5.1.2 Diagram ofWildStar PCI Hardware 68 5.2 Detail Investigation of Hardware Algorithms 70 5.2.1 Data Flow Analysis ofHardware Design 70 5.2.2 Detailed Investigation ofPE1 Design 73 5.2.3 Detailed Investigation of PEO Design 76 5.3 Host Software Roles and Responsibilities 81 6.1 Image Sequence Selection and Preparation 84 6.2 Performance and Timing Based Results 88 6.3 Image Quality Based Results 94 6.4 General Conclusions 98 6.5 Path Forward 103 6.6 Summary 106 Bibliography 6 Appendix: Example Restoration Results 112 vn List of Figures Figure 2.1: Sample Motion Between Frames 8 Figure 2.2: Full Search Block Matching Window Size 13 Figure 2.3: OSA with Displacement of 7 and Minimum Distortion Function at (-1, -1) 15 Figure 2.4: Hierarchical Framework for Efficient Motion Estimation 18 Figure 2.5: Aperture Problem 19 Figure 2.6: Windows Area Selection in Frames 25 Figure 3.1: Example 2-Stage MMF Topology with 3 Sub-Filter Windows 37 Figure 3.2: Original Image ofPeppers 38 Figure 3.3: Corrupted Image with 10% Salt and Pepper Noise 38 Figure 3.4: Corrupted Image Filtered with 3x3 Median Filter 39 Figure 3.5: Corrupted Image Filtered with MMF 39 Figure 3.6: Diagram Depicting Motion Compensated Median Filtering 41 Figure 3.7: Diagram Depicting ML3D Filtering 43 Figure 3.8: Diagram Depicting ML3Dex Filtering 45 Figure 3.9: Diagram Depicting the First Stage of Arce Bi-directional MMF 48 Figure 3.10: Diagram Depicting the AML3D Filter 53 Figure 4.1: Software Interaction Diagram 61 Figure 4.2: Software Interaction Diagram Focused on Image Reading and Writing 62 Figure 4.3: Software Interaction Diagram Featuring Motion Estimation Components 63 Figure 4.4: Software Interaction Diagram Featuring Filtering Components 64 Figure 5.1: WildStar PCI Test Board 68 Figure 5.2: WildStar PCI Mezzanine Memory Module 69 Figure 5.3: Block Diagram ofWildStar PCI Connections 69 Figure 5.4: Overall Image Flow and Design Partitioning 72 Figure 5.5: Memory Flow and Processing Element Responsibilities 72 Figure 5.6: Block Diagram ofWildStar PCI PE1 75 Figure 5.7: Register Map for WildStar PCI PE1 75 Figure 5.8: Block Diagram of WildStar PCI PEO 77 Figure 5.9: Register Map for WildStar PCI PEO 78 Figure 6.1: Original Image from 13 Days Sequence 85 Figure 6.2: Original Image from BW Sequence 86 Figure 6.3: Original Image from Martial Sequence 86 Figure 6.4: Original Image from NASA Sequence 87 Figure 6.5: Original Image from Walk Sequence 87 Figure 6.6: More Sample Resultant Images from BW Sequence 99 Vlll Figure 6.7: Time / Quality Plot for Simplified Black's Motion Estimator 102 Figure 6.8: Time /Quality Plot for Horn-Schunck Motion Estimator 102 IX List of Tables Table 6.1: Table ofProperties ofthe 13 Days Image Sequences 85 Table 6.2: Table ofTotal Times for BW Sequence on Hardware Implementation 90 Table 6.3: Table ofTotal Times for BW Sequence on Software Implementation Using Black's Method for Motion Estimation and Adaptive Median Filter 90 Table 6.4: Table ofTotal Times for Averages ofEach Sequence 91 Table 6.5: Table ofAverage Times for Averages of Each Sequence 91 Table 6.6: Table ofRanked Overall Performance for Filter / Motion Estimation Combination 93 Table 6.7: Table ofImage Quality Statistics for BW Image Sequence using Adaptive and AML3D Median Filters using Black's Method for Motion Estimator 95 Table 6.8: Table of Image Quality Improvements for BW Image Sequence using Adaptive and AML3D Median Filters using Black's Method for Motion Estimator 95 Table 6.9: Table of Image Quality Statistics for Grayscale Sequences Averaged 96 Table 6.10: Table of Image Quality Statistics for Color Sequences Averaged 96 Table 6.11: Table of Image Quality Statistics for All Sequences Averaged 97 Table 6.12: Table ofRanked Overall Improvement for Filter / Motion Estimation Combination 98 Glossary AML3D Adaptive ML3D Filter - an adaptive MMF proposed in this work BM Block Matching motion estimation algorithm Code Value Intensity level of a pixel DFD Displaced Frame Difference DMA Direct Memory Access FPGA Field Programmable Gate Array LUT Look-Up Table MAE Mean Absolute Error ML3DEx Extended ML3D MMF MMF Multi-level Median Filter MSE Mean Squared Error Motion Field Set ofmotion vectors for an image Motion Vector Displacement vector from a pixel in one frame to where it is in the next frame PDC Pixel Difference Classification Pixel Picture element RMSE Root Mean Squared Error VHDL VHSIC Hardware Description Language - High level VLSI design language VHSIC Very High Speed Integrated Circuit VLSI Very Large Scale Integration XI Chapter 1 : Introduction As the demand for DVD and digital content increases, the need for digital film restoration also increases. The desire to preserve old films from degradation is one example of the need for efficient digital film restoration. The cellulose nitrate based film used during the silent picture era is highly volatile and susceptible to age degradation.