VARIABLE BLOCK-BASED DEBLOCKING FILTER FOR H.264/AVC

Seung-Ho Shin, Young-Joon Chai, Kyu-Sik Jang, Tae-Yong Kim

GSAIM, Chung-Ang University, Seoul, Korea

ABSTRACT up to 4x4 blocks, it is possible to decrease artifacts on the boundaries between 4x4 blocks. In the The deblocking filter in H.264/AVC is used on low-end process of decreasing the blocking artifacts, however, the terminals to a limited extent due to computational actual images’ edges may erroneously be blurred. And it complexity. In this paper, Variable block-based deblocking may not be used on low-end terminals due to complex filter that efficiently eliminates the blocking artifacts computation and large memory capacity. Despite such occurred during the compression of low-bit rates digital shortcomings, however, the deblocking filter can be said to motion pictures is suggested. Blocking artifacts are plaid be the most essential technology in enhancing the subjective images appear on the block boundaries due to DCT and image quality. The general opinion of the subjective image quantization. In the method proposed in this paper, the quality test proves that there is distinguishing difference in image's spatial correlational characteristics are extracted by the image qualities with and without the deblocking filter using the variable block information of motion used [6]. compensation; the filtering is divided into 4 modes according to the characteristics, and adaptive filtering is In this paper, we suggest a block-based deblocking filter by executed in the divided regions. The proposed deblocking using the variable block information of the motion method eliminates the blocking artifacts, prevents excessive compensation. By using the variable block information and blurring effects, and improves the performance about 40% considering human’s visual characteristics and moving compared with the existing method. picture characteristics, the filtering mode is classified into 4 types to adapt the filter structure. In section 2, the in-loop deblocking filter and variable block-size are introduced. In section 3, 1. INTRODUCTION the deblocking filtering method using variable block information is proposed. The proposed method is verified H.264, using new coding technologies, increases the through experiments in section 4. The conclusions are stated compression rate with the same image quality compared in section 5. with the existing H.263v2 (H.263+) [2] or MPEG-4 Simple Profile [3]. The remarkable characteristics of H.264 include 2. BACKGROUND variable block motion compensation, multiple reference images, 1/4-pixel motion vector accuracy, and in-loop 2.1. In-loop Deblocking Filter deblocking filter [1]. Although such coding technologies are the main functions to increase compression efficiency, the In H.264/AVC, the block distortion is eliminated by using high complexity leads to the unavoidable increase in the adaptive in-loop deblocking filter. The H.264/AVC computational cost. Performance improvement to decrease deblocking filter can be applied to the edges of all 4x4 the complexity and to prevent the quality deterioration is blocks in a except for edges on slice boundaries. necessary to adapt the newly defined techniques into low- In order to apply the filter to each macroblock, the filtered end terminals. In this paper, methods to improve the pixels at the top and on the left of the current macroblock performance of the deblocking filter to enhance the are used, and the luma and chroma components are subjective image quality on low-end/low-bit rates terminals separately processed. Filtering is applied to vertical / are presented. In the H.264 standards, the deblocking filter, horizontal edges of 4x4 blocks in a macroblock, filtering is also called the loop filter, is used to decrease blocking done first from the left to the right vertically and then from artifacts. The blocking artifact, a distortion resulted due to top to bottom on the horizontal edges. For the 16x16 luma the differences in the quantization parameter values between component, it is applied to four 16-pixel edges; for the 8x8 macroblocks, refers to the phenomenon in the motion chroma components, it is applied two 8-pixel edges. The picture images which is appeared with loss of edge clarity strength of the filter depends on the current quantization, the and tone fuzziness in square patterns. Deblocking filtering is coding modes of neighboring blocks and the gradient of necessary to decrease such distortion on the boundaries image samples across the boundary. The filtering process between macroblocks. Since H.264 can segment 16x16

1-4244-1017-7/07/$25.00 ©2007 IEEE 436 ICME 2007 goes through three processes, such as, boundary strength, The deblocking filtering method proposed in this paper can filter decision, and filter implementation [7]. decrease the computational cost through the phase of boundary strength parameter selection and boundary pixel 2.2. Variable block-size Motion Compensation analysis by classifying filtering modes using variable block information used for motion compensation. Generally in The variable block-size motion compensation (VBSMC) motion compensation, variable blocks are divided into technology of H.264 conforms well to the image blocks of 16x16, 16x8, and 8x16 in flat regions or large characteristics or the motion characteristics inside the objects and divided into blocks of 8x8, 8x4, 4x8, and 4x4 in motion by dividing the motion compensation block size complex and fine regions with lots of motion especially on more finely than the existing H.263 or MPEG-2/4. That is, the part with many edge elements. in MPEG-2 the 16x16-pixel fixed-size motion compensation The filtering performance can be improved by executing block is used; in MPEG-4 two kinds of pixel motion filtering using the spatial correlational characteristics compensation blocks, 16x16 and 8x8, are used. Differently between adjacent blocks of divided variable blocks. The in H.264, seven kinds of motion compensation block sizes, human visual system (HVS) is more sensitive to the from the 16x16 pixel to 4x4 pixels, are used to compensate discontinuity of flat and simple regions than complex motions. regions. In flat and simple regions, strong filtering is applied to decrease block distortion phenomenon; in complex and fine regions, weak filtering is applied in order to prevent the edges of actual objects from being blurred [5].

Filtering starts in the vertical direction of the whole macroblock excluding the edges of slice boundaries and then proceeds in the horizontal direction. The filtering is executed based not on the 4x4 blocks within a macroblock, which is the existing method, but on the variable blocks of motion compensation. The pixel values changed in the Fig.1. Variable Block is used for motion compensation in H.264 vertical filtering are reflected in the horizontal filtering. Fig. For the flat regions or in the case the size of objects is large, 2, 3, and 4 are the examples of determining the filtering the motion compensation is executed with large 16x16 mode according to the adjacent block characteristics in the blocks; for complex regions or in the case the sizes of horizontal filtering of 16x16 variable blocks. objects are small, the motion compensation is achieved by small blocks such as 4x4 blocks. In general the smaller the size of the blocks for motion compensation, the better motion compensation results can be obtained. If the size of the blocks gets smaller, however, more searches should be carried out, thus increasing the complexity and the number of motion vectors to transmit. In order to solve such problem in H.264, an adaptive motion compensation which applies the size of blocks selectively according to the image characteristics is carried out [4].

3. VARIABLE BLOCK-BASED DEBLOCKING FILTER Fig.2. Filtering mode (4) decision of 16x16 variable blocks If the screen size of images gets larger, the computational Fig. 2 shows the case that the adjacent blocks of the 16x16 cost of the deblocking filter increases proportionally. The block are selected as 16x16 or 8x16 on the motion filtering method currently adopted in H.264 standard is compensation process. In such a case, assuming that the diverse in the choice of filter coefficient according to image that surrounds the boundaries is a flat and simple adjacent block characteristics, reference picture region, the strongest filtering is applied by assigning characteristics, and I/P/B coding. Since “If” commands are filtering mode (4). In order to eliminate the internal as well used profusely in the selection of filter coefficient, fast as the block boundary’s blocking artifacts, filtering is computation through pipelining cannot be expected in the applied to all edges of 4x4 blocks in the 16x16 block, in the implementation of the actual deblocking filter. As a result 4 boundaries of the luma component and 2 boundaries of many commercial H.264 Decoders tend not to use the each chroma component. The filtering processing is same as deblocking filter for real-time decoding, which will results existing method. in very fatal image deterioration as time passes.

437 mode (2) is assigned. In this case, assuming that blocking artifacts and actual image feature edges coexist, the filtering is applied to the range of 4 pixels, p0, p1, q0, and q1. The filtering formulas are as the following. p1’, p0’, q0’ and q1’ are produced by filtering of p1, p0, q0 and q1.

d = (p0 – q0) / 5. (4)

p1’ = p1 + sign(d) |d|. q1’ = q1 – sign(d) |d|. (5)

p0’ = p0 + 2 sign(d) |d|. q0’ = q0 – 2 sign(d) |d|. (6)

Fig.3. Filtering mode (3) decision of 16x16 variable blocks In such a way, the filtering mode is determined adaptively Fig. 3 shows the case that the adjacent blocks of the 16x16 by searching the adjacent blocks of the 7-type variable block are 16x8 or 8x8. In this case, assuming that the image blocks and the each formula per filtering mode is applied to that surrounds the boundary is the middle region, filtering the pixels on the concerned block boundary at a time. The mode (3) is assigned. Filtering is applied to the 8 pixels on filtering in the vertical direction proceeds following the the concerned block boundaries, p0 ~ p3 and q0 ~ q3. In this same process as the horizontal case. The filtering modes case, 9 tab Finite Impulse Response (FIR) filter with relationship in the horizontal and vertical boundaries is weights (1/16, 1/16, 1/8, 1/8, 1/4, 1/8, 1/8, 1/16, 1/16) is listed in Table 1. Fig. 5 shows the example of filtering mode applied to them. decision when the proposed method was applied.

Table 1. Filtering Mode table 16x16 16x8 8x16 8x8 8x4 4x8 4x4 Block H V H V H V H V H V H V H V 16x16 4 4 3 3 3 4 2 2 2 1 1 2 1 1 16x8 4 3 4 3 3 3 2 2 2 1 1 2 1 1 8x16 3 4 3 3 3 4 2 2 2 1 1 2 1 1 8x8 3 2 3 2 2 2 2 2 2 1 1 2 1 1 8x4 2 2 2 2 2 2 2 1 2 1 1 1 1 1 4x8 1 2 1 2 1 2 1 2 1 1 1 2 1 1 4x4 1 2 1 2 1 2 1 1 1 1 1 1 1 1 (H: horizontal, V: vertical)

Fig.4. Filtering mode (1) decision of 16x16 variable blocks Fig. 4 shows the case that the adjacent blocks of the 16x16 block are 8x4 or 4x4. Assuming that it is the case where the actual edges of features exist in the complex region of the image, the weakest filtering is applied assigning filtering mode (1). The filtering is applied to 2 pixels, p0 and q0. Since the filtering must be done the most finely and carefully, only the two boundary pixels p0 and q0 are designated as the filtering pixels and the pixels of p0’ and q0’ are calculated by applying the filtering formulas as the following. Fig.5. Example of applied filtering mode. d = (3p – 8p + 8q – 3q ) / 16. (1) 1 0 0 1 Since the existing method determines whether the filtering should be applied through the analysis of block boundary d’ = sign(d)Max[0, |d| – Max(0, 2(|d| – QP))]. (2) whole pixels, the computational cost becomes large. But, if the method proposed in this paper is used, the computational P ’ = p + d’. q ’ = q + d’. (3) 0 0 0 0 cost can be reduced by determining whether the filtering should be applied by the variable block-based. When the filtering blocks are 16x16 and 8x16, which are MB modes, and the adjacent blocks are 8x8 or 4x8, which are sub-MB modes, it is assumed to be the middle regions that are more detail than the case of Fig. 3 and filtering

438 4. EXPERIMENTAL RESULTS more the H.264 coding technology is developed, the deblocking filter’s complexity increase proportionally. The In order to measure the performance of the deblocking filter method proposed in this paper enhanced the performance by proposed in this paper, the JM (Joint Model) version 10 [8], executing deblocking filtering selectively using the variable recommended by the H.264 standardization group, was used block information of motion compensation compared with for the experiments. The sequences used in the experiments the existing method. In detail, it is not necessary to analyze were composed of images that contain various the image characteristics (flat regions and complex regions) characteristics. The format was QCIF and the luma and separately for filtering. According to image characteristics, chroma components were sampled 4:2:0. The experiments strong filtering is executed for the flat regions to minimize were conducted by changing the quantization parameter the blocking artifacts, and weak filtering is applied to (QP) in the I and P frames. complex regions in order to maintain the image features as much as possible. The filtering was executed by the variable block-based to decrease computational cost. As a result the filtering speed improved without much deterioration of quality. According to the results, it was verified that the computational cost can be decreased about 40% without major quality degradation.

Acknowledgments. This research was supported by the HNRC-ITRC(Information Technology Research Center, MIC) program and Seoul R&BD program, Korea.

REFERENCES

[1] Draft ITU-T Recommendation and Final Draft International Standard of Joint Specification (ITU-T Rec. H.264/ISO/IEC 14496-10 AVC), Mar. 2003. [2] Draft ITU-T Recommendation H.263, “Video Coding Fig.6. PSNR and Computation Reduction result of the Foreman for low bitrate communication Telecommunication Standardization Sector of International Telecommunication

Union,” Oct.1995. Table 2. Average Results of the Experimental Sequences [3] ISO/IEC 14496-2, “Information technology-coding of audio-visual objects. Part 2: Visual,” Dec.2001. QP container stefan news ¨P C.R ¨P C.R ¨P C.R [4] A. Ahmad, N. Khan, S. Masud, and M.A. 22 –0.21 38.82 –0.23 46.17 –0.09 47.63 Maud, ”Selection of variable block sizes in H.264,” IEEE 26 –0.19 32.99 –0.21 45.88 –0.11 42.20 Int’l Conf. on Acoustics, Speech, and Signal Processing 30 –0.26 27.86 –0.19 42.33 –0.15 39.47 (ICASSP ’04), Vol.3, pp.173-176, May 2004. 34 –0.29 25.16 –0.24 40.10 –0.12 36.17 [5] S.D. Kim, J. Yi, H.M. Kim, and J.B. Ra, “A deblocking (¨P: ¨PSNR(dB), C.R: Computational Reduction(%)) filter with two separate modes in block-based video coding,” IEEE Trans Circuits Syst. Video Technol, vol.9. Fig. 6 shows the detailed result values of Foreman, one of pp.156-160, Feb.1999. the experimental sequences. Table 2 shows the results of [6] Peter List, Anthony Jock, Joni Lainema, Gisle PSNR differences (¨PSNR) and computation reduction of Bjontegaard, and Marta Karczewicz, “Adaptive Deblocking the experimental sequences. According to the total results, Filter,” IEEE Trans Circuits. Syst. Video Technol, vol.13, the average of PSNR reduction was about 0.19 (dB) with No.7, July.2003. almost no change in objective picture quality and the total [7] E.G. Richardson, “H.264 and MPEG-4 Video average processing time decreased by 38.72% compared Compression,” John Wiley & Sons, pp.184-187, Sept.2003. with the existing method. It is judged that the reduction in [8] JVT software JM10.2, May.2006. the count of execution of “If” and “for” statements by 24.87% influenced the reduction in the total processing time.

5. CONCLUSIONS

Due to the complex computation, it is difficult to implement or apply the H.264 deblocking filter to low-end terminals such as wireless communication or mobile phones. Also the

439