4th International Conference on Mechanical Materials and Manufacturing Engineering (MMME 2016) Micro-video Data-Acquisition System Design
Zhengzheng Liu, Hai Ji & Sanxing Cao School of Communication University of China, Beijing 100024, China
ABSTRACT: In recent years, more and more studies about micro-video emerged with the increasingly rapid development of micro-video. This Micro-video Data-Acquisition System gives a method that how to obtain the structured data and the unstructured data of the micro video, and the system has got data of micro-video from about twenty Chinese Internet video platforms. Finally we illustrate the micro-video data acquisition process and its key technology through examples. The data acquired by the Micro-video Data-Acquisition System provides a convenient for the micro-videos’ Big Data analysis, It has great significance to the analysis of the micro-videos’ effect, users’ behavior, and the micro-videos’ creation. KEYWORD: Micro-video; data; acquisition; analysis
1 INTRODUCTION
With the development of Internet technology, com- 2 DATA INFORMATION munication technology and terminal equipment tech- 2.1 Data sources nology, High-quality Internet video-transmission has become not only an important communication chan- This text focuses on domestic Internet video platform nel of medium information, but also an Internet ser- for data acquisition, so the system will obtain data on vices with more and more users. The Internet micro- almost all the influential Internet video platforms of videos is widely accepted with its characteristics of China, which are divided into large Internet video extensive content, the integrity of the plot and short platform and professional micro-video internet plat- duration, which meet the living status of people’s form. As shown in Table I. high-pressure life and fragmentation of the time. Meanwhile, Big Data technology has played a cata- lytic role for Internet micro-videos’ development, da- Table I The target Internet platforms ta become an important carrier for users and Internet video platform to mining videos’ transmission effects, Youku, Tudou, Sohu video, Tencent video, iQIYI, Funshion, Ku6, 56, mango guidance videos’ creation, analysis users’ habits. Large Internet TV, Sina Video, Baidu video, HUASHU video platform How to use new technology to get micro-videos’ core TV, Thunder look, Phoenix video, Letv, data, such as playing number, comment number, the CNTV ... number of support and oppose, video introduction, actors and other information. It is an important re- Professional mi- search topic in the field of information technology. cro-video internet V film, CUCTV, Maxtv... Based on the above, this text studies on the key platform technologies of micro-videos’ data-acquisition pro- cess. The system is mainly use the PHP + MySQL + Apache and based on the CodeIgniter framework for 2.2 Data types the back-end support. Users get micro-videos all data Internet video resource page is mainly composing through the system and grasp the status of the videos’ two types data, one is the structured data, which re- spread. On the one hand, the data can help make pre- flect the level of users’ concern and recognition to its diction and decision to the micro-videos’ creation content or not, structured data is typically embodied and propagation, on the other hand, users’ feedback in digital form, such as playing numbers, comment help to improve the system functions. number, support number, oppose numbers, score and
© 2016. The authors - Published by Atlantis Press 350 so on. One is the unstructured data, which reflect the forms. theme of the video content and other relevant infor- 2) Divide the data into different URL queue, ac- mation. There are many forms of unstructured data, cording to different data content acquired (such the majority are in the form of a text message, such as playing number, comment number, support as video title, video introduce, actor and director in- number, oppose number, etc.), and waiting to be formation, video type, release time and so on. The analyzed one by one until the resolution. system will obtain the data types are shown in Table 3) Using the function of file_get_contents () in the II. PHP to parse the URL, extracting the string it contains if the URL is successfully resolved, 4) Use the regular expressions with the function of Table II Data types obtained by the system preg_match( ) and preg_match_all ( ) in PHP to match the string and obtain the data it contains. Playing number, comment number, support Structured data number, oppose numbers, score... 5) Store the date in the database, complete the oper- ation.
Video title, video introduce, actors, Unstructured director, release time, 3.2 Unstructured data acquisition process data Video type, video time-length, area... Unstructured data acquisition flow chart is shown in Figure 2.
3 SYSTEM DESIGN PROCESS Start The system is divided into two parts, namely the structured data acquisition process and the unstruc- tured data acquisition process, corresponding to the Extract URL from database two kinds of data types. URL waiting and parsing 3.1 Structured data acquisition process No
Structured data acquisition flow chart is shown in URL is valid? Figure 1. No Yes Start
Analyzing URL Extract URL from database Yes
URL waiting and parsing Obtain label by class library
No Store data in the database Analyzing URL
Yes End
Regular expressions matching Figure 2 Unstructured data acquisition flow chart
1) Extract URLS of the video resource page corre- Store data in the database sponding with the “status=0” in the data table. 2) URL in the Video Resource Page is waitting to End be analyzed one by one until the resolution. 3) Before analyzing URL, judge the URL is valid or not by cURL. If it is valid, parse it; If not, re- Figure 1 structured data acquisition flow chart turn URL queue to operate the next URL. cURL is a tool to transfer files and data by the URL 1) Extract an URL corresponding with the every grammar rules, which supports the protocol of video’s structured data from Internet video plat-
351 HTTP, FTP, TELNET, and some others. It can To obtain the string above by analyzing the URL accomplish the remote data acquisition and col- http://up.video.iqiyi.com/ugc- lection, simulated login, the docking interface, updown/quud.do?type=2&dataid=231937800. analog Cookies and other functions. Among them,"" up ": 6376" is the video’s support 4) Parse the valid URL by the function of number, "" down ": 1266" is the video’s oppose file_get_html ( ) in Simple HTML DOM parsing number. Then according to the character of playing library, the function of file_get_html ( ) can number data before and after, use the regular expres- parses the page in accordance with the W3C sions combined with the function of preg_match ( ) standard DOM model. in PHP, to match and acquire it. 5) Locate the HTML label and get the information between label pairs by the function of find ( ) in 4.2 Unstructured data acquisition Simple HTML DOM parsing library and the se- lectors like id, class, tag and so on. The internet video resources pages are generally us- 6) Store the string acquired in the corresponding ing HTML language, and its encoding based on the data table. Then update the value of the status to document object model (DOM). The main character- 1 in the data table, which indicates that the video istic is all the useful information is contained in the resource page text message has been completed structured keywords. HTML language has a lot of acquisition. structured keywords, such as
,