Degree of Loop Assessment in Microvideo
Total Page：16
File Type：pdf, Size：1020Kb
DEGREE OF LOOP ASSESSMENT IN MICROVIDEO Shumpei Sano, Toshihiko Yamasaki, and Kiyoharu Aizawa Department of Information and Communication Engineering, The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan fsano, yamasaki, [email protected] ABSTRACT Table 1. Ratio of loop/non-loop video in 1,022 microvideo This paper presents a degree-of-loop assessment method for on Vine with a tag “#loop”. microvideo clips. Loop video is one of the popular features non-loop video loop video in microvideo, but there are so many non-loop video tagged Number of video 906 116 with “loop” on microvideo services. This is because upload- Ratio 88.6 % 11.4 % ers or spammers also know that loop video is popular and they their official web site, Vine is introduced as “a mobile service want to draw attention from viewers. In this paper, we statisti- 5 cally analyze the scene dynamics of the video by using color, that lets you capture and share short looping video ,” there optical flow, saliency maps, and evaluate the degree-of-loop. are two characteristics in Vine. First is video length shortness. We have collected more than 1,000 video clips from Vine and Max video length in Vine is 6.5 seconds, which is shorter than subjectively evaluated their degree-of-loop. Experimental re- the other similar services. Second is looping. Shared video sults show that our proposed algorithm can classify loop/non- on Vine are automatically played repeatedly, and many loop loop video with 85.7% accuracy and categorize them into five video which seamlessly connect the last and the first frames degree-of-loop categories with 61.5% accuracy. are uploaded with a hash tag as “#loop”. However, if users try to retrieve loop video, search results include many spam Index Terms— microvideo, short video, loop, degree– video which is not loop. According to our preliminary exper- of–loop iment (see Table 1), 89 percent of video clips which included the “#loop” tag were non-loop video because uploaders want 1. INTRODUCTION to get a lot of views and also know people want to watch loop video. 1 Microvideo sharing services launched in 2013 such as Vine , In this paper, therefore, we propose a degree-of-loop 2 3 MixBit and video on Instagram are rapidly growing as a (DoL) assessment method that can classify loop/non-loop new social network service (SNS). Similar to conventional video by analyzing the spatial and temporal statistics of a var- 4 video sharing services like Youtube , one of the user’s main ious kinds of visual features. To the best of our knowledge, interests is how to create or retrieve interesting video. If this paper is the first attempt of loop/non-loop detection. In sorted by the number of views or favorites, only the clips addition, such technology can be applied to user assistance to that have been revealed on the Internet for a long time can be create better loop video. We have collected 1,022 video from retrieved. To solve this problem, a lot of image processing Vine and manually evaluated the DoL score from 1 (strongly based video interestingness analysis have been proposed. non-loop) to 5 (perfect loop). The list of the video URLs and Although such approaches can also be applied to mi- the subjective degree-of-loop score is available on our project crovideo, new techniques dedicatory designed for microvideo page6. Experimental results using our DoL assessment model would also be required. Actually, some users are trying to cre- demonstrated 85.7% accuracy in loop/non-loop classification ate sophisticated and interesting video by taking advantage of and 61.5% accuracy in five-class DoL classification. the shortness. In Vine, for example, loop video which seems This paper is organized as follows. In section 2, related endless by taking advantage of its automatic repeat play func- works are summarized. Section 3 explains our DoL model tion is popular. Vine is one of the popular microvideo sharing using frame distances and saliency trajectories. Experimental services which launched in January 2013 and obtained 40 results are demonstrated in section 4, followed by the con- million users only within seven months. As announced in cluding remarks in section 5. 1https://vine.co/ 2https://mixbit.com/ 3http://instagram.com/ 5https://blog.twitter.com/2013/vine-a-new-way-to-share-video 4http://www.youtube.com/ 6https://www.hal.t.u-tokyo.ac.jp/˜sano/loopvideo/ 978-1-4799-5751-4/14/$31.00 ©2014 IEEE 5182 ICIP 2014 Features 100000 Video loop video non-loop video 1. Adjacent frame distances Feature selection 10000 - RGB - Brightness 1000 - Magnitude of optical flow SVM 100 2. Loopness probability by adjacent frame distance 10 - RGB - Brightness of distances Magnitude 1 1. Loop/non-loop 0 50 100 150 - Magnitude of optical flow classification 3. Continuity of the region of Frames interest 2. Degree-of-loop - Saliency centroids trajectory classification Fig. 2. Example of RGB distances of loop and non-loop video. The last value is last-frame/first-frame distance. Fig. 1. Overview of degree-of-loop assessment. DoL classification is conducted by using a support vector ma- chine (SVM). 2. RELATED WORKS Content based video ranking [1, 2, 3] has been proposed aim- 3.2. Adjacent Frame Distances ing at retrieving interesting video. Irie et al. [1], assumed that interesting video is more edited than low–interestingness In loop video, the first and last frames need to be visually sim- video and proposed a “degree-of-edit” measure by analyzing ilar in order to smoothly connect the last and the first frames. editing clues such as the number of cuts, detecting sound and We measure the distance between the first and the last frames text captions, and so on. Wei et al. [2] proposed a cross- in terms of RGB, brightness, and optical flow as feature val- I reference video reranking method with fused multimodal fea- ues. di;j(x; y) is per pixel distance in index I between the tures. Tian et al. [3] used user’s labeling efforts in video frames fi and fj where pixel’s coordinate is (x; y). RGB jj − jj2 reranking to bridge the semantic gap. Redi et al. [4] focused di;j (x; y) = RGBi(x; y) RGBj(x; y) (1) on microvideo characterization and introduced a ”creativity” dBr(x; y) = jjY (x; y) − Y (x; y)jj2 (2) measure. They analyzed correlations between creativity and i;j i j Opt jj jj2 the audio-visual features such as filmmaking technique fea- di;j (x; y) = OptF lowi;j(x; y) (3) tures or features to model aesthetics. In [4], it is demonstrated Here, RGBi(x; y) is a vector of RGB value in each pixel of that the degree of loop is an important factor to analyze the the frame i. Y (x; y) is the brightness in YUV color space. creativity. However, only one dimensional feature was used 2 jjOptF lowi;j (x; y)jj is the magnitude of optical flow in each for loopness analysis. pixel between the frames i and j calculated using [10]. Let L Looping video generation techniques have also been pre- be the number of framesP and the last-frame/first-frame dis- sented. Various techniques are proposed [5, 6, 7, 8, 9] to form I I tance defined as F = x;y dL;1(x; y) are used as a feature. looping contents, though they are not focused on microvideo. Schodl et al. [5] proposed video texture which construct a graph between similar images in the video and stochastically 3.3. Loopness Probability by Adjacent Frame Distance transited from one clip to another, thus achieving a random The features in Section 3.2 calculates the distance between but continuous sequence. Kwatra et al. [6] generated looping adjacent frames. However, there are also loop video whose video by synthesizing video texture spatially and temporally background changes dynamically and non-loop video which using a graph cut technique. Agarwara et al. [7] and Rav– is dark or whose background is static. Fig. 2 shows an ex- Acha et al. [8] tried to create panoramic video. Liao et al. [9] ample of RGB distances over the frames of loop video and proposed a method to create varying dynamism looping video non-loop video. As shown in Fig. 2, the small last/first-frame by segmenting scene regions based on degree of motion. distance does not always stand for loop video and vice versa. Therefore, we also consider loopness probability by statistical 3. DEGREE OF LOOP ASSESSMENT analysis of the distances over the frames. Let us define a set of distances from the first frame up to the last frame: 3.1. Method Overview I f I j − g G = g (i; i + 1) i = 1; :::; L 1! (4) X Fig. 1 shows an overview of the proposed method. Given I I the input video, three types of features are extracted: adjacent g (i; j) = log 1 + di;j(x; y) (5) frame distances, loopness probabilities by adjacent frame dis- x;y I ≡ I tances, and continuity of the region of interest. Then, after the and gL g (L; 1) is calculated in each video. feature selection, loop/non-loop classification and five-class B bins direction oriented histogram (Opt b; b = 1::B) is 978-1-4799-5751-4/14/$31.00 ©2014 IEEE 5183 ICIP 2014 0.6 Table 2. DoL score variation in 1,022 video. 0.55 DoL score 1 2 3 4 5 0.5 Number of video 506 229 171 77 39 0.45 0.4 also calculated for{ feature extraction.