A) Objective MV Bypass Will Greatly Reduce Computational Time

MV bypass

1. Goals. a) Objective – MV bypass will greatly reduce computational time. b) Transcoder uses old MVs; intuitively this means there should be a reduction in quality. Goal of this experiment is to measure: 1) The reduction of the computational time. 2) What is the quality loss when MV bypass option is used?

2. Experimental Setup. Three different video files were chosen for this experiment: 1. Fast motion – basketball movie – “Basketball”. 2. Average/middle motion – shamu video, see world – “Shamu”. 3. Slow/little motion – taking person – “Mike”.

 All three movies are captured with a SONY digital video camera, then encoded to *.avi formats, then *.avi files encoded to corresponding *.m2v files with LSX Encoder 3.0.  2 video sets were received as a result: 1) Three videos coded with MV values -- 16 for P pictures and 8 for B pictures. 2) Three videos coded with MV values -- 32 for P pictures and 16 for B pictures. The reason why these MV values were chosen is that computational time for different MV values would be different and thus savings in computational time would be different too.

All three videos have resolutions of 704x480 pixels – NTSC standard and coded at the rate of 10Mbs as progressive video source. Note: The reason why progressive video coding is very important here is that we gain quality using MB option, and in the case of interlaced video coding there is a quality loss (see other technical report). Why it happens: because interlaced coding uses twice as much MVs as progressive video encoding (in progressive only 2 MVs for each MB – frame encoding, in interlaced 4 MVs for each MB – 2 for the top field and 2 for the bottom field – field encoding).

Hardware: Athlon 1GHz, 256MB RAM, 30GB HDD.

3. Assumptions and definitions:  MV – motion vector  MB – macroblock  P32 B16 in the text or (32,16) on the plots mean that P pictures have max MV value length of 32 and B pictures of 16. Same for P16 B8 and (16,8).  There are two MVs for each macroblock in progressive video, and two types of MBs Forward predicted (P and B pictures) and Backward Predicted (only B pictures) Note: In this paper only Forward Predicted MB’s are considered for simplicity.  “m n” and “k p” definitions mean that old/reused MV’s for particular MB where there is m – X component, n – Y component; new calculated MV’s values using motion estimation routine are: k – X component, p – Y component.

4. Movements in the Pictures.

Movement in frames(16,8) /pic.1/ Basketball

100 Shamu 90 80 Mike 70 t n

e 60 3 per. Mov. m

e 50 Avg. v (Basketball) o 40 3 per. Mov. M 30 Avg. (Shamu) 20 3 per. Mov. 10 Avg. (Mike) 0 0 50 100 150 200 250 300 Frames

Movement in frames(32,16) /pic.2/ Basketball 100 90 Shamu 80 70 Mike t

n 60 e 3 per. Mov. m 50 e Avg. v

o 40 (Basketball)

M 3 per. Mov. 30 Avg. (Shamu) 20 3 per. Mov. 10 Avg. (Mike) 0 0 50 100 150 200 250 300 Frames 1) Motivation: Try to measure amount of motion in the frame and between frames in some numerical value between different types of video.

2) What is what:  Axis X represents frames numbers  Axis Y represents some percentage value K, which is calculated in the following way:  only macroblock with same “m n” and “k p” values where chosen m=k, n=p (this situation is called nonIntra match), and this value K is calculated for each frame. 1 M X[i]2  Y[i]2 K   *100% M i0 Max _ MV _ Frame _ value 1) X[i], Y[i] are the X and Y components for the i macroblock respectively. 2) M is the number/subset of macroblocks, which have nonIntra match characteristic 3) Max_MV_frame_value is max MV value for this particular frame: For example P pictures 16, B pictures 8  The idea in this approach was to measure if nonIntra coded MVs tend to maximum value for the MV in the particular frame.

3)Observations:  This approach matches intuitive perception of the amount of motion in the picture as it could be seen on the plot.  It’s true for both encoded sets P16 B8 – pic.1 and P32 B8 – pic.2; only on the pic.1 the differences between motions are more visible due to small MVs values. For example, for “Basketball” movie in pic.1 value K sometimes is close to 100% for some frames, which means that a lot of MVs in this frame have values close to maximum MV value for this frame. But for pic.2 we can see that situation is different value K is around 70-80%, it means that such MV values as P32 B16 were enough to code most movements in the video efficiently – these values have not been exceeded. 5. Impact on SNR.

Bypass SNR Baketbol(16,8) - SNR/Target bit-rate Origina SNR

30 28 26 24 R

N 22 S 20 18 16 14 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

Bypass SNR Basketball(32,16) - SNR/Target bit-rate Origina SNR

30 28 26 24 R

N 22 S 20 18 16 14 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Bypass SNR Shamu(16,8) - SNR/Target bit-rate Origina SNR 28 26 24 22 20 R N

S 18 16 14 12 10 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

Bypass SNR Shamu(32,16) - SNR/Target bit-rate Origina SNR 28 26 24 22 20 R N

S 18 16 14 12 10 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Bypass SNR Mike(16,8) - SNR/Target bit-rate Origina SNR 35 33 31 29 R

N 27 S 25 23 21 19 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

Bypass SNR Mike(32,16) - SNR/Target bit-rate Origina SNR 35 33

31 29 R

N 27 S 25

23 21 19 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate General(16,8) - SNR/Target bit-rate /pic.3/ Basketball - Bypass SNR 34 32 Basketball - Original SNR 30 28 Sham u - Bypass SNR 26

R 24 Sham u - N

S 22 Original SNR 20 Mike - Bypass 18 SNR 16 Mike - Original 14 SNR 12 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

Genera(32,16) - SNR/Target bit-rate /pic.4/ Bas ketball - Bypas s SNR 34 Bas ketball - 32 Original SNR 30 Sham u - 28 Bypas s SNR 26

R 24 Sham u - N Original SNR S 22 20 Mike - 18 Bypas s SNR

16 Mike - 14 Original SNR 12 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate 1) Motivation: To measure the quality loss with different encoding bit rates. Two parameters are considered bypass SNR and original encoding SNR. 2) What is what:  X-axis represents coding bit rates  Y-represents received SNRs for each bit rate.

3) Observations:  As it can be seen from the graph, at the high bit-rates, MV bypass option gives even gain in quality, and only at the low bit-rates, original encoding gives higher quality. Note: We use progressive video. In the case of interlaced video, MV bypass option gives reduction of picture quality. The nature of this can be seen in differences between progressive and interlaced encoding. By itself, interlaced encoding produces more MV vectors than progressive one (frame-field), and as a result, MV errors are highly possible. This observation is practically same for P16 B8 and for P32 B16 video set, as shown in pic. and pic.4. We can tell that length of MVs doesn’t impact (at least in the case of this video set) the quality of the video. And effect of higher quality for MV bypass encoding is the same for P16 B8 and P32 B16 videos.

4) Summary: With progressive video and a small reduction of bit rate, MV bypass option gives an even more accurate coding technique than original encoding.

6. Macroblock types

Basketball(16,8) - Macroblock types/Target bit-rate 45 40 MB type mismatch

s 35 e p

y 30 nonIntra t

k mismatch 25 c o l

o nonIntra r

c 15 match a

M 10 Intra 5 match 0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Basketball(32,16) - Macroblock types/Target bit-rate 45 40 MB type mismatch 35 s e p 30 nonIntra y t mismatch k 25 c o l

o nonIntra r c 15 match a

M 10 Intra 5 match 0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate

Shamu(16,8) - Macroblock types/Target bit-rate 60 MB type 50 mismatch s e p

t nonIntra

k mismatch c

o 30 l b o nonIntra r c 20 match a M 10 Intra match 0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Shamu(32,16) - Macroblock types/Target bit-rate 60 MB type 50 mismatch s e p

y 40 t nonIntra

k mismatch c

o 30 l b o nonIntra r c 20 match a M 10 Intra match

0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate

Mike(16,8) - Macroblock types/Target bit-rate 45 MB type 40 mismatch

s 35 e p

y 30 nonIntra t

k mismatch c 25 o l b 20 o r nonIntra c a 15 match M 10 5 Intra match 0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Mike(32,16) - Macroblock types/Target bit-rate 45

40 MB type mismatch

s 35 e p

y 30 nonIntra t

k mismatch 25 c o l

o nonIntra r

c 15 match a

M 10 Intra match 5

0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate 1) Motivation: There’re different types of macroblocks, which the encoder provides: a) MB type mismatch – “0 0” and “m n” or “0 0” and “m n”, where m, n are not equal to zero. b) nonIntra mismatch – “n m” and “k p”, where n is not equal k, m is not equal to p. Pairs “n m” and “k p” are not equal to zero at the same time. If n=0, than m must not zero or otherwise. The same is true with “k p” if k=0, than p must be not zero or otherwise. c) nonIntra match – “n m” and “k p”, where n=k, m=p, n,m are not equal to zero at the same time: if n=0, then m must be not zero or otherwise. d) Intra match – “0 0” and “0 0”. This present graph measures the percentage of these macroblocks Note: MB type mismatch + nonIntra mismatch + nonIntra match + Intra match = 100%

2) What is what:  Axis X represents different bit rates for encoded video.  Axis Y represents the percentage of each macroblock group for a given bit-rate.

3) Observations: MB type mismatch grows slowly until 6Mb/s bit rate and then rapidly when bit-rates are decreased as it’s intuitively expected. nonIntra match drops slowly when bit rates are decreased until 6Mb/s and then more rapidly. The probable reason is encoder provides more MBs with MV values equal to zero at low bit rates. MBs are blurred in some way (their boundaries). nonIntra mismatch grows slowly until some bit rate (4-2Mbs) and then drops again. The reason probably is that until some bit rate MV values are changed, and than at low bit rates there’re no MVs for that MB, – it’s just not moving, or in other words, it’s so blurred that it doesn’t matter where it’s placed. Intra match – decreases very slowly 2-3%, when bit rates are decreased. That happens probably because the amount of Intra macroblocks stays the same, independent of bit rate. Note: there’s probably a difference between Intra coded MBs and nonIntra coded MB but with MV values equal to zero, because it’s not moving in the sense of Motion Estimation routine.  For both video sets – P16 B8 and P32 B16 -- in general the behavior (change in the percentage via target bit rate) of all 4 types of MBs are the same.  We can see that for “Basketball (32,16)”, the percentage of Intra matched MVs is higher than in “Basketball (16,8)”, which means that in “Basketball (16,8)” more MBs were coded as Intra than predicted with MVs, due to the short max MVs values. Same situation with MB is traceable for “Shamu” video set.  However in “Mike” video set situation with MBs is different – there are no changes in percentage of different types of MBs between (32,16) and (16,8) coding. That means that for slow motion small MV values are enough to “catch”/predict all movements in the video.

4) Summary: Basically Macroblock Types graph correlates with SNR graph. Amount of errors grows – SNR decreases, and when the amount of errors is too large (low bit-rate), original encoding provides a little better quality picture than with bypass option.

7. Actual bit-rates

Basketball(16,8) - Actual bit-rate/Target bit-rate 10000000

Bypass 8000000 bit-rate e t a r

- 6000000 t i

s Original b /

l b bit-rate a u

t 4000000 c A

2000000 Target bit-rate

0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s Shamu(16,8) - Actual bit-rate/Target bit-rate 10000000

Bypass 8000000 bit-rate e t a r -

t 6000000 i

b s Original / l

b bit-rate a

u 4000000 t c A 2000000 Target bit-rate

0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

Mike(16,8)- Actual bit-rate/Target bit-rate 10000000

Bypass 8000000 bit-rate e t a r

- 6000000 t i

s Original b /

l b bit-rate a u

t 4000000 c A

2000000 Target bit- rate

0 0 2000000 4000000 6000000 8000000 10000000 Target bit-rate b/s

1) Motivation: Find out if there are any differences between actual bit-rate with MB option encoding and original encoding. Find out how much we can compress video of this size (704x480), or what is the lowest possible bit rate. 2) What is what:  Axis X represents target bit-rates  Axis Y shows actual/received bit-rates for original and bypass encoding.

3) Observations:  Bypass and original encoding bit-rates are very close to the target one until 2Mb/s point and then they slowly start to differ from target bit-rate.  We can see that for the “Basketball” and “Shamu” video, which we consider as fast motion and as average motion relatively, lowest achievable bit-rate is close to 1.3Mb/s. For “Mike” video -- slow motion one -- actual bit-rate could be reduced till 1Mb/s. That would give us compression ratio of 1/10.  There is no significant differences in “target bit-rate” – “actual bit-rate” behavior between the (16,8) and (32,16) videos. For example: for (32,16) fast motion video, with target bit-rate 1Mb/s, actual bit-rate could be 1.2Mb/s, which is 0.1Mb less than for (16,8) video – the reason is again in MV length -- less MBs predicted and more MBs coded as Intra. Note: As far as differences between (16,8) and (32,16) video sets are small, as it mentioned erlier - plots for (32,16) videos are not presented in this report.

8. Saving computational time:

Speed gain vs coding type 1.8 12 1.6 11 10 d s

n 1.4 9 e o m c

1.2 8 i e t

7 1 r p

e 6 u p

5 d e e m 0.6 4 e a p

r 3 0.4 S F 2 0.2 1 0 0 MV Original Original Original Original Original Original bypass (8,8) (16,8) (32,8) (16,16) (32,16) (32,32) encoding encoding encoding encoding encoding encoding encoding 1) Motivation: Measure computational time savings between different encoding types.

2) What is what:  X-axis represents encoding types with different parameters as MVs length and original/bypass encoding  Left Y axis shows how many frames per second it was possible to achieve given encoding type.  Right Y axis shows the gain in speed comparing to Original (32,32) encoding. Time for Orig. (32,32) was taken as the base for calculation.

MV bypass means that motion bypass option was used during encoding. Duration of encoding thus stays the same for any MV values . Original P(N,M) encoding, where N and M are 8,16 or 32, means that full re-encoding took place, with MV length N pels for P pictures and M pels for B pictures.

3) Observations: As we can see from the plot, MV vector bypass option gives 10 times speed gain comparing to full (32,32) re-encoding, five times comparing to full (32,16) re-encoding.

9.Summary.

MV bypass option provides huge savings in computational time and quality gain for progressive videos.

10. Some questions and technical explanations: Q: How can it happen that the quality is gained due to Bypass option? A: a) As we use LSX encoder to receive first *.m2v files, it might be possible that LSX has more accurate algorithm for MVs calculation than Berkley encoder, so in that case when Transcoder uses old MVs values instead new ones, a quality gain might happen. b) Probably (a) is not reason, same thing happens (Bypass SNR > Original SNR), when video was first re-encoded in Berkley transcoder at 10Mb/s, and than re-encoded again to 8Mb/s.

Q: What is the difference between the pictures which where originally encoded with MV values P16, B8 and P32,B16. A: When encoding routine uses longer MVs values -- intuitively picture quality should be better, as algorithm finds MBs matches which are located farther from the given macroblock, and as far as we use constant bit rate encoder, more bits could be put into DCT coefficients and that would provide higher quality video.