Junmei Feng Recommending Multimedia Xiaoyi Feng Liming Deng Information in a Virtual Han School of Electronic and Information Chang’an City Roaming System Northwestern Polytechnical University Xi’an, PR

Jinye Peng* School of Information Science and Abstract Technology Northwest University This article presents a roaming system of Han Chang’an City, with both virtual reality Xi’an, PR China (VR) technology and information recommendation technology. Nowadays, some new research issues in the cultural heritage domain can be achieved with the rapid develop- ment of VR technology. The ancient site of Han Chang’an City, as one of the most valuable and significant cultural heritages in China, attracts more and more attention around the world. To let more people understand Han Chang’an City and reproduce its beauty, in this article, we propose a virtual roaming system combined with informa- tion recommendation technology. First, Unity3D is selected as the three-dimensional platform to design the scenario model of Han Chang’ an City, and the virtual scene is reconstructed with VR technology, according to real historical data; then, the dynamic information recommendation module is designed to recommend hot topic informa- tion and personalized information. The former is obtained through web crawlers, including the latest released news related to Han Chang’an City for users. The latter is generated by the proposed hybrid recommendation algorithm, which combines explicit and implicit feedback. The performance of the proposed algorithm is validated on two datasets. Finally, we show some results of our system test. Our proposed sys- tem is released online now, and users can wander in the scene any time.

1 Introduction

1.1 Archeological Background

As the capital of the Western (206 BC–25 AD), Han Chang’an City was the political, economic, and cultural center for two hundred years, and was also the eastern end of the . Chang’an City was estab- lished as a capital in 202 BC by the first Han Emperor Gaozu. Figure 1 shows the plan of Han Chang’an City. Its area was about 34.39 square kilometers, and the length of the city wall was about 25.1 kilometers with three gates per side. The main buildings in Han Chang’an City included Weiyang , Changle Palace, Mingguang Palace, and North Palace. Weiyang Palace, as the most important palace of the city, was the center of power of the empire and the place where the emperors lived and rested. Han Chang’an City was destroyed during the political upheaval at the end of the in 904 AD, and an archeological site can be found on the site of Han Chang’an City (see Figure 2). Nowadays, the ancient site of Han Chang’an City is located ten kilometers northwest of today’s Xi’an City, Presence, Vol. 26, No. 3, Summer 2017, 322–336 doi:10.1162/PRES_a_00299 ª 2017 by the Massachusetts Institute of Technology *Correspondence to [email protected].

322 PRESENCE: VOLUME 26, NUMBER 3

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 323

Figure 1. Plan of Han Chang’an City. Reprinted with permission from J. Feng, X. Liu, X. Feng, J. Peng, X. Wang, & X. Peng (2016). A new method of virtual Han Chang’an City navigation system. Proceedings of Eighth International Conference on Digital Image Processing, copyright SPIE.

most representative and typical ‘‘national’’ sites in China, and it is still there as a silent witness of the Silk Road. It is too large to be physically reconstructed; hence, the low-budget virtual reconstruction of Han Chang’an City turns out to be a suitable way to play back the glory of the Western Han Dynasty. It is widely agreed that the focus of a cultural heritage exhibition has been a shift from static information to actively involved users. In the past, the exhibition focused more on cultural relics, but now it pays more attention to personalized services for users. According Figure 2. The site of Han Chang’an City. Photo was taken by to users’ preferences and cultural characteristics Chinese photographer Mu Ren. (Benouaret & Lenne, 2016; Wang et al., 2009), they Province, near the confluence of the Wei and Feng rivers, will be provided with digital information that they are China. Moreover, the site is the largest and the most interested in related to cultural heritage, such as images, completely preserved, with the richest relics, one of the text introduction, and video. In this way, users can fully

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 324 PRESENCE: VOLUME 26, NUMBER 3

enjoy their cultural roaming trip. Considering this fact, this version is difficult to share. Although the web version we combine our virtual Chang’an City system with infor- canbereleasedtoshareonthenetwork,wehavenot mation recommendation technology to make our system implemented the connection with the VR glasses. At pres- more convenient and attractive. ent, the interactive method of the web version is to use the mouse and keyboard. To share our work with scholars, sci- entists, and students who prepare to carry out archeologi- 1.2 Virtual Reconstruction cal or historical research, this article is presented based on 1.2.1 VR Technology. VR (Kyriakou, Pan, & the released web version of virtual Han Chang’an City. Chrysanthou, 2016) is a technology that is used to simu- late a real environment, so that users can fully immerse 1.2.2 Data Collection. In order to reproduce the in the scene and interact with the virtual scene. There- beauty of Han Chang’an City, the first step is to collect fore, roaming in the scene is possible. In recent years, data related to the cultural heritage (Albanese, VR technology is widely used in daily life for its simple d’Acierno, Moscato, Persia, & Picariello, 2011) of Han operation and strong immersion, in areas such as educa- Chang’an City. As Han Chang’an City was seriously tion, entertainment (Ramanathan, Rangan, & Vin, damaged, the original geographic location data and the 2002), cultural heritage (Kim, Kesavadas, & Paley, information of historical relics should be collected, and 2014; Champion, 2015), and health (Pfandler, Lazaro- were obtained through online papers related to Han vici, Stefan, Wucherer, & Weigl, 2017). In terms of cul- Chang’an City (Han Chang’an Archeological Team, tural heritage, the display and protection of cultural 2012). These papers compared different maps, including relics can be improved to a new stage with the combina- the map of China, the map of Xi’an, and the aerial map tion of VR technology and network technology. First, of Han Chang’an City, by referring to archeological high precision and permanent preservation of the endan- findings and historical documents, visiting museums in gered heritage resources can be realized. Second, the Xi’an, consulting experts of Han Chang’an Archeologi- exhibition of cultural relics can be separated from geo- cal Team, and field measurement. The plan of Han graphical restrictions, realizing the sharing of resources. Chang’an City (see Figure 1) was drawn in AutoCAD Generally speaking, the use of VR technology can pro- based on the collected data, which was also imported to mote the cultural and trade industry to enter the infor- Unity3D as the buildings’ layout. The next work is mation age faster and realize the modernization of cul- deciding how to build the model of Han Chang’an City. tural relics display and protection. In this article, we choose 3ds Max (3D Studio Max) as Unity3D (Wang et al., 2010) is used as the development the tool of modeling (Shen & Zeng, 2011). platform for VR in our work. It is a cross-platform 3D game engine developed by Unity Technologies. As a fully 1.2.3 Building the Model. As mentioned above, integrated professional game engine, it reduces the diffi- the structure of Han Chang’an City is extremely compli- culty of modeling and improves the efficiency of the devel- cated, and composed of many famed historical architec- opment cycle. In the current VR interaction technology, tural sites such as Weiyang Palace, Changle Palace, the most used method is handle control. The advantages Mingguang Palace, North Palace, Gui Palace, West of this interaction are fast response and high sensitivity. Market, and East Market. As the realistic degree of the In this article, two versions of the virtual Han Chang’an model directly affects the VR experience, modeling is an City project are released, including PC (Personal Com- important part of the virtual system. puter) and Web Player. In the PC version, VR glasses are In this modeling, 3ds Max was used to design sophis- connected to our virtual scene to help users experience the ticated models such as and cultural relics strictly virtual city, and a handle is used to navigate through the based on geometric data, and Unity3D was applied to multimedia information in the scene. One drawback of design easy models like city walls, roads, and trees to cre- this approach is that the released project is very large, and ate a realistic environment. Taking the model of Weiyang

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 325

Figure 3. Screenshots of the models built in 3ds Max: (a) a view of Weiyang Palace, (b) Front Hall, (c) Jiaofang Hall, (d) Han Emperor.

Palace, for example, it was the official center of govern- The structure of our article is organized as follows. ment and consisted of more than 40 halls, such as Front Section 2 describes the interactive technology used in Hall and Jiaofang Hall. We divided the model of the virtual Han Chang’an City roaming system. Section Weiyang Palace into models of halls to model separately 3 presents the proposed dynamic information recom- and saved them as .FBX format. After the models were mendation module, including hot topic recommenda- prepared, we imported the model files into Unity3D tion and personalized recommendation in detail. Section scene and moved to them to the corresponding positions 4 reports several experiments to test the performance of according to the buildings’ layout of our plan. Figure 3 our system and recommendation results of the system. shows the models of Weiyang Palace built in 3ds Max. Section 5 discusses the results and limitations of our sys- Figure 3(a) shows a view of Weiyang Palace, and the tem. Finally, Section 6 gives a high-level description of other three images are the close shot of small palaces in the conclusion and future work. Weiyang Palace. Front Hall was in the middle of Weiyang Palace, which was built with an incense wood rafter. Jiaofang Hall was the residence of the empress. 2 Design of System Interaction Additionally, considering the modeling efficiency, the modeling size should be taken into consideration. If the Human–computer interaction technology (Yang, project involves huge data, the running speed of the sys- Zhang, & Wu, 2009; Yang & Jie, 2014) is an efficient tem will be slowed down. Thus, reducing the size of the way to realize the dialog between people and computers model by minimizing the amount of data is a smart way with the input and output devices. In this article, inter- to optimize the system (Lin & Zhao, 2014). active technology is mainly used in the areas including

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 326 PRESENCE: VOLUME 26, NUMBER 3

gation map window will be displayed in the upper left corner. This function will be detailed in the next section. The function of the button ‘‘Image Info’’ is to display some basic information of palaces in the form of images. ‘‘Instructions’’ gives users guidance on how to roam.

2.2 Navigation Map

In this article, the navigation map (Mousouris & Styliaras, 2014) is different from general two- dimensional maps with fixed sizes, which cannot show the detailed geographic information. As our roaming Figure 4. Original construction drawings for the screenshot of the system is released online, the designed map is a web- roaming mode selection interface. Reprinted with permission from based application combining Gaode map API (Applica- J. Feng, X. Feng, X. Liu, and J. Peng (2016), The Virtual Wandering tion Program Interface) with the Han Chang’an plan. System of Han Chang’an City Based on Information Recommendation. The city plan is added to the Gaode map API according Proceedings of 15th ACM SIGGRAPH Conference on Virtual-Reality to its longitude and latitude just as a layer covering. Continuum and its Applications in Industry; copyright Association for Hence, the higher the zoom level of the map, the more Computing Machinery, Inc. detailed the information displayed. As HTML can directly communicate with Unity3D scene roaming, navigation map, and multimedia infor- (Feng, Liu, et al., 2016), thus, the map can interact with mation display. the virtual scene online. A point mark is added to the map to display the position of the first person controller 2.1 Scene Roaming in the scene. Users cannot only know their specific posi- tion in the virtual Han Chang’an City scene, but also can With respect to roaming, two modes are estab- correspond to the geographical location in Xi’an. Our lished for users to choose: auto roaming and manual previous work (Feng, Feng, Liu, & Peng, 2016) shows roaming; the default mode is manual roaming. After one more details about this navigation map. determines the destination, auto roaming means that the first person controller moves along the path according to 2.3 Multimedia Information the A* search algorithm for pathfinding (Stamford, Khu- Display Module man, Carter, & Ahmadi, 2014). One of the main limita- tions of this mode is that one cannot artificially stop In this VR system, multimedia information (Barto- before reaching the destination. On the other hand, the lini et al., 2013) is obtained through the following inter- user who selects the manual roaming mode can control active ways: image, 3D models of cultural relics, voice the direction of movement and adjust the perspective explanation, and video. through the function keys on the GUI (Graphical User Images are used to show palaces, historical figures, Interface) which is shown in Figure 4. and historical events. In the virtual scene, round icons Figure 4 presents the screenshot of the roaming mode (as shown in Figure 4) are hidden information points; if selection interface. The four buttons in the lower right you click on an icon, the text image will be a pop-up. corner are function keys including ‘‘Panoramic View,’’ The format of the images in our system is BMP and JPG. ‘‘Map,’’ ‘‘Image Info,’’ and ‘‘Instructions.’’ If the button Models of cultural relics are shown in the GUI control of ‘‘Panoramic View’’ is pressed, one can overlook the interface; users can get more detailed information by whole Han Chang’an City scene accompanied by voice rotating or zooming. As shown in Figure 5, the cultural explanation. If one presses the ‘‘Map’’ button, the navi- relic of the green glaze run the beast bottle is displayed

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 327

The multimedia information cannot be displayed actively according to users’ preferences. As the point of information (POI) is scattered throughout the Han Chang’an City area and the amount of POI is huge, it is hard for users to find their preferred infor- mation at a glance. Considering these drawbacks, we propose a dynamic information recommendation module to meet the requirements of information. The module will provide users with the latest released news related to Han Chang’an City and information recommended based Figure 5. Screenshot of cultural relics display interface. on users’ preferences in the virtual roaming system. According to the different characteristics of the in the form of model, images, and text introduction in dynamic recommendation information, we divide them the interactive interface. The reconstructed model is dis- into two categories: hot topic recommendation and played on the leftmost part of the figure. One can adjust personalized recommendation, which are stored in the size or angle as needed. Images taken from different different database tables. The former mainly focuses on angles and text introduction of the cultural relic are recommending the latest news. Meanwhile, the latter is shown on the right. used to recommend personalized information to users Voice explanation is used only in the panoramic view based on their preferences and characteristics of cultural navigation, explaining the origin, development, and fea- signs, which will help users to improve the roaming tures of palaces. Unity3D supports several voice formats, efficiency. and the voice format in the system is MP3. Video is also shown by the GUI control interface and the content of the video is historical events of Western 3.1 Hot Topic Recommendation Han Dynasty. In this project, the video format is MP4. In this article, the items related to hot topics are obtained from Baidu web pages. Concrete steps are 3 Dynamic Information Recommendation listed as follows: First, the number of palaces in Han Chang’an City A conclusion drawn from Section 2.3 is that was confirmed, and the database table for each palace multimedia information can be displayed in the form was established in SQL Server. Fields in the database of images, models of cultural relics, voice explanation, table include sequence number, title, URL, and time, and video in our virtual system. Nevertheless, three which correspond to the line number of the item, the drawbacks still exist in our information interactive description of the hot topic, the web link of the topic, module: and the storage time of the item, respectively. The displayed content such as images, models of Then, a Python program was designed to obtain the cultural relics, and video is preset and cannot be first five records related to each palace, and save the titles updated in a timely manner. and URL of the records to the relevant database table. The multimedia information displayed in the scene The content in the database is updated every day. is limited, considering the running speed of the vir- Finally, the virtual system updates the displayed infor- tual system. Because different users have different mation continually. The recommended information is preferences, it is difficult to meet the information read from the database and varies by the position in the needs of all users. scene during roaming.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 328 PRESENCE: VOLUME 26, NUMBER 3

interests or provide personalized services through ana- lyzing the data of users’ behaviors. Generally speaking, recommender systems help users to obtain the required information from mass data (Bailey, Bailenson, & Casa- santo, 2016). As the multimedia information related to Han Chang’an City is abundant, in this article, a hybrid recommendation method is adopted in our system to help users quickly access their interests in the virtual roaming scene.

Figure 6. Screenshots of recommendation results for hot topic. 3.2.1 Methods. The core part of RSs is called the recommendation algorithm (Adomavicius & Zhang, 2016), which, to a large extent, determines the type and Two arbitrary recommendation results for hot topic the performance of the RSs. In general, the recommen- are intercepted and shown in Figure 6. Figures 6(a) dation algorithm is classified into three major categories: and 6(b) illustrate the recommendation results at two content-based filtering, collaborative filtering, and different positions in a virtual scene. The effect of mov- hybrid filtering. ing the mouse over the first recommended item is shown The content-based filtering approach (Kim, Ha, Lee, in Figures 6(c) and 6(d). Clicking on each item of Jo, & Ei-Saddik, 2011) utilizes a user’s choices made in recommendation information links to the corresponding the past and tries to recommend items that the user rated webpage. highly before. In a content-based recommender system, a major advantage is that new items added to the system 3.2 Personalized Recommendation can be recommended immediately. But the limitation is Personalized recommendation is another type of obvious: only the items that are similar to those already dynamic information. Although the hot topic recommen- rated by the user have the opportunity to be recom- dation provides users with the latest information, the mended and it is difficult to find out the potential inter- focus of different users on Han Chang’an City may be dif- est of the user. ferent as different users have different preferences. Thus, Unlike the content-based filtering method, collabora- to make the virtual Han Chang’an City system more tive filtering (CF) focuses more on similarity among humanized, intelligent, and convenient, a new application users or items, and the recommendations are made based is proposed to make up for the drawbacks of the hot on the ratings of the items given by users. Users with topic recommendation, which combines the virtual similar ratings to the same items are called neighbors. Han Chang’an roaming system with a personalized Once the neighbors of a user are found, the unrated recommendation algorithm to produce personalized items of the user can be predicted through the neigh- recommendations for different users based on their bors, and then, those items with high predicted ratings preferences. will be recommended to the user. The CF approach is In recent years, with the rapid development of net- simple and convenient, and only needs ratings data with- work technology, the amount of information on the out items’ characteristics. The CF recommendation internet has grown exponentially. Recommender systems algorithm is usually classified into two classes: memory- (RSs) (Bobadilla, Ortega, Hernando, & Gutie´rrez, based algorithms and model-based algorithms (Silva, 2013; Lu, Wu, Mao, Wang, & Zhang, 2015) are the Camilo-Junior, Pascoal, & Rosa, 2016). The main most widely used way to overcome the information over- difference is the processing of the ratings. Memory- load problem (Maes, 1995), which have been widely based algorithms (Liu, Hu, Mian, Tian, & Zhu, 2014) used in daily life to recommend information of users’ include user-based collaborative filtering (UBCF) algo-

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 329

rithms and item-based collaborative filtering (IBCF) 3.2.2 Framework. In recommendation algo- algorithms, which predict ratings with the entire data- rithms, the preference of a user usually relies on explicit base based on history rating records. UBCF algorithms ratings on items. A good example is Amazon, which ena- focus on obtaining the neighbors of the target user, bles users to select and assign some stars to the books while for IBCF algorithms, the target is items. For already purchased. However, due to the cold start prob- model-based algorithms, models are built to represent lem, explicit ratings may not always be available. There- the behavior data of the user based on the collected fore, implicit feedback from users should be exploited, ratings. which immediately reflects the preference of a user. Actually, CF is generally believed to be the most used Implicit feedback is obtained by recording a in RSs, especially adopted by e-commerce websites such user’s behavior and it is always an abundant source of as Amazon, Netflix, and Google News. Nevertheless, information. Examples of implicit feedback include two major shortcomings faced by CF are cold start prob- browsing history, purchase record, and the history of lems and data sparsity. Cold start problems refer to new rating data. users or new items added to the system with no rating In 2008, a model of Singular Value Decomposition records, which makes it difficult to obtain recommenda- plus plus (SVDþþ) was proposed in Koren (2008), tions. Data sparsity refers to the insufficient ratings data which integrates explicit ratings and implicit feedback in the rating matrix. into a matrix factorization model to represent the The hybrid filtering approach combines several kinds preference of a user. Each item i is associated with an

of recommendation approaches, using different sources item factor vector qi, and each user u is associated with of knowledge to solve the problems existing in each one a user factor vector pu. An implicit factor vector yj is of these algorithms. added to this model. The model of SVDþþ is In the domain of cultural heritage (Ardissono, Kuflik, defined as: & Petrelli, 2012), several multimedia information inter- 0 1 X 1 active systems have been developed for users to explore T @ 2 A ^rui ¼ l þ bu þ bi þ qi pu þ jjNuðÞ yj ; (1) valuable cultural and historical information. Bartolini j2NuðÞ et al. (2016) proposed a general multimedia recommen- der system, which was able to manage heterogeneous where ^rui indicates the predicted rating of user u on item multimedia data and provide context-aware multimedia i. The parameters of bu and bi denote the deviations of recommendation services for users. In addition, the sys- user u and item i from the overall average rating m tem has been applied to the mobile environment for respectively. NuðÞrecords the implicit feedback data of both outdoor and indoor scenes within the cultural her- user u, which reflects the preference of the user indi- itage area. As for the collaborative filtering proposal, rectly. In order to get better recommendation results, Rossi, Barile, Improta, and Russo (2016) showed a rec- the squared error should theoretically be minimal on the ommender system using a model-based CF method of ratings dataset K: X matrix factorization algorithm to provide personalized 2 2 2 2 minb;p;q;y ðrui ^ruiÞ þ kðbu þ bi þjjpujj artworks paths in a museum. Among the hybrid techni- ðu;iÞ2K X ques, Benouaret and Lenne (2015) proposed a hybrid 2 2 þjjqijj þ yi Þ: (2) recommendation system, which combined a semantic j2N ðuÞ method using ontologies for representing the museum

knowledge and thesauruses with a semantically enhanced Here, rui is the rating made by user u on item i. The goal CF approach. of learning this model is to predict the unrated ratings In this section, we adopt the hybrid recommendation based on the existing dataset. To avoid overfitting, the algorithm to ensure the accuracy of online recommenda- magnitudes of the learned parameters should be regular- tions, which will be introduced in detail below. ized. Thus, the constant l is used to control the extent

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 330 PRESENCE: VOLUME 26, NUMBER 3

feature vector. In the case of the cold start problem, the content-based filtering approach is executed. The user or item is represented by its feature vector, and the similar- ity is measured by cosine (COS) (Liu et al., 2014). The similarity formula between two users simðÞ u; v COS is expressed in Equation 3. In other cases, we adopt the Figure 7. Schematic of the proposed personalized recommendation KNN and SVDþþ algorithm to predict the unrated module. items through the learned model; the implicit feedback in our model is the browsing history of the items. The training of our model is done offline to improve the effi- of regularization. To learn all these parameters, Equation ciency of the online recommendation. The model will be 2 is optimized through a stochastic gradient descent. updated once a new user or a new item comes to better The parameters are constantly modified along the fastest serve users. gradient direction until the squared error is no longer COS ~ru ~rv falling. simðÞ u; v ¼ ; (3) kkru kkrv In this article, to overcome the data sparsity problem, we extend our algorithm from the k-nearest neighbors where u and v represent item u and item v, respectively, (KNN) CF method to the hybrid recommendation algo- ~ru and ~rv are the vectors of item u, and item v. kk repre- rithm combining the KNN and SVDþþ algorithms sents the magnitude of the vector. (Costa & Manzato, 2016; Koren, Bell, & Volinsky, Personalized Recommendations are responsible for: (i) 2009), which achieved personalized recommendations recommending the first k items to the target user (in this in the virtual Han Chang’an City roaming system. article, k is set to 3); (ii) collecting experiences or sugges- Figure 7 presents the schematic of the proposed tions of users and sending them to the database to fur- hybrid recommendation algorithm. Next, we will detail ther optimize and improve the performance of the rec- the process. ommender module. Database is responsible for: (i) storing our multimedia information; (ii) preserving the personal information 3.2.3 Experiments. In this article, the prediction

that users fill in when registering, the ratings of users on formula pu;i is predefined in Equation 4. items, and the behavior data of users (e.g., browsing his- p n ¼ 0 tory, location, and time spent in front of each item); and p ¼ COS ; (4) u;i a p þð1 aÞp otherwise (iii) receiving feedback and experience from users after COS SVDþþ

using the system. where pu;i represents the predicted rating of user u on User Interactions are responsible for collecting the item i. n is the number of ratings made by user u, and personal information, the ratings of users on items, and the parameter a is used to optimize the proposed algo-

the behavior data of users, and saving them to the rithm. a is a positive real number, a 2½0; 1. pCOS and

database. pSVDþþ are predicted ratings of COS and the SVDþþ Prediction is responsible for predicting the ratings of algorithm, respectively. We choose COS as the similarity users on the unrated items. We designed a hybrid recom- calculation method of KNN (Xia, Feng, Peng, & Fan, mendation algorithm, which combines content-based fil- 2015) because of the better performance and simpler tering and CF recommendation methods. When a new computation in the three KNN algorithms listed in user or new item enters the system, the hobbies of the Table 1. Therefore, in the case of the cold start problem, user or the characteristics of the item will be logged to the nearest neighbors are computed based on the feature the database; then, we convert the text information into vectors and cosine algorithm. In other cases, the pro- feature vectors, so each user or item corresponds to a posed algorithm will switch to the CF algorithm, which

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 331

Table 1. Performance Comparison of the Proposed Algorithm and Baselines on the Two Datasets

Movielens 100K Movielens 1M MAE RMSE MAE RMSE

COS 0.751780 0.959676 0.712781 0.912995 PCC 0.758118 0.968110 0.724935 0.919702 WPCC 0.746222 0.958005 0.711507 0.915104 LFM 0.726059 0.930129 0.671195 0.858500 SVD 0.719130 0.920992 0.664380 0.850696 SVDþþ 0.719882 0.920828 0.664902 0.850266 Proposed 0.716241 0.916741 0.662444 0.847203

combines memory-based and model-based recommen- (RMSE) to indicate the accuracy and quality. MAE and dation algorithms to recommend cultural information to RMSE are defined as: users. X 1 MAE ¼ p r : (5) To verify the superiority of the proposed recommen- K u;i u;i dation algorithm, we conducted experimental evaluation u;i sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi on two real datasets and compare with traditional meth- X 1 2 ods of CF (Liu et al., 2014) such as COS, Pearson corre- RMSE ¼ ðpu;i ru;iÞ ; (6) K u;i lation coefficient (PCC), weighted PCC (WPCC), latent factor model (LFM), singular value decomposi- where K is the number of actual ratings of users on

tion (SVD), and SVDþþ, which are widely adopted in items. pu;i and ru;i are the predicted and real ratings of literature. user u on item i, respectively. MAE and RMSE reflect Two datasets of Movielens 100K and Movielens 1M the deviation degree of the predicted ratings from the are employed to evaluate the effectiveness of algorithms. actual ones. RMSE penalizes large deviation more heav- The Movielens 100K dataset contains 100,000 ratings of ily than MAE. For both metrics, lower values correspond 1682 movies made by 943 users, and the MovieLens 1M to better preference. dataset contains 1,000,209 anonymous ratings of 3,706 Before the experimental evaluation, we conducted a movies made by 6,040 users. In both datasets, each user series of experiments on Movielens 100K to determine has rated at least 20 movies. All the rating values are inte- the optimal parameter a. We compared the results of gers in the scale 1 to 5, where 1 shows that the user is the proposed measure under different values of a in not interested in the movie, and 5 means that the user terms of MAE and RMSE varying nearest neighbors. favors the movie very much. Moreover, the data sparsity The number of nearest neighbors was in the range of 10 of the user-item matrix is 93.4% in Movielens 100K and to 200, and the step was 10. a 2½0; 1, and the step was 95.53% in Movielens 1M. To demonstrate the perform- 0.1. The experimental results showed that when ance and measure the prediction accuracy of the pro- a ¼ 0:2, the metrics of MAE and RMSE reached the posed similarity measure, each dataset is divided into minimum within the entire nearest neighbors. So the two parts, 80% randomly selected used for training, and following experiments were carried out under the condi- the remaining 20% used for testing. tion of a ¼ 0:2. In this article, for estimating the performance of our When evaluating the method on the Movielens 100K proposed algorithm, we adopt the metrics of mean and Movielens 1M datasets, the neighbor size of mem- absolute error (MAE) and root-mean-squared error ory-based methods was set to 40, and the dimension of

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 332 PRESENCE: VOLUME 26, NUMBER 3

factors in model-based methods was set to 50. The ex- Table 2. Average Results Scores of the Two Experiments perimental results on two datasets are listed in Table 1. Experiment 1 Experiment 2 Table 1 compares the performance of our proposed algorithm with the other six approaches on two datasets With With in terms of MAE and RMSE. Note that the model-based Without sys. Without rec. recommendation methods perform better than the Average scores 43.8 72.5 69.5 82.3 memory-based methods. This is due to the fact the memory-based methods suffer from the cold start and data sparsity problems. Our method achieves the best of without and with the dynamic information module performance and outperforms other approaches, espe- were listed below. cially for the Movielens 1M dataset, in which the data Table 2 shows the average results of the two experi- sparsity is higher. ments. Note that the degree of understanding of Han Chang’an City can be greatly improved with the help of 4 System Tests and Results the virtual system from the results of Experiment 1. Experiment 2 indicates that users prefer the system with Nowadays, researchers consider not only accuracy the dynamic information recommendation module. and stability when designing a system and they begin to Users can fully immerse in our virtual system to explore pay attention to users’ opinions to further improve the palaces of interest while absorbing knowledge. system to satisfy most users. In this section, we designed several experiments to evaluate the performance of our 4.2 Accuracy system from two aspects: user satisfaction and accuracy, and we also present some recommendation results. Since our virtual Han Chang’an City roaming sys- tem has been released online, users can log in to the system to experience it at any time; meanwhile, their 4.1 User Satisfaction behavior data is recorded in the database. Therefore, the We carried out two experiments to investigate how performance of our system was tested offline through helpful the system was to users. The experiments were two experiments using the data collected online, which designed to demonstrate that the virtual Han Chang’an was compared with the two most used user-based City with the dynamic information recommendation mod- approaches: PCC and Jaccard (Liu et al., 2014). As the ule can improve users’ experience satisfaction. Two groups data in our dataset was sparse, we selected the users’ data of 40 graduate students who never experienced the virtual of 1000 images related to Han Chang’an City to con- system were asked to complete the experiments. duct our experiments. Experiment 1: We designed a questionnaire on knowl- Experiment 1: We chose five groups of users’ brows- edge of Han Chang’an City, which included 10 choice ing history/ratings from our dataset with fixed test users questions and 20 multiple choice questions. The total of 10. The sparsity of the rating data was 60%, 70%, 80%, points were 100. Then we asked one group of 20 stu- 90%, and 98%, respectively. We randomly selected 80% dents to fill out the questionnaire before and after using of the rating data from each group for training and the the system. The average scores of the statistics of without remaining for testing. and with the virtual system are shown in Table 2. Experiment 2: We tested on five groups of datasets Experiment 2: We asked the other group of students with a fixed data sparsity of 80%, and the number of to use the virtual system. Then, they were asked to use users ranged from 10 to 50. We randomly selected 20% scores to express their satisfaction about the system with- of the rating data from each group for testing. out and with the dynamic information module, which Figure 8 compares our proposed algorithm with other were in the range of 1 to 100. The average results scores approaches in terms of MAE and RMSE varying the data

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 333

Figure 8. Comparison between our algorithm with other approaches in terms of MAE and RMSE.

sparsity and the number of test users. Figures 8(a) and itage site of Han Chang’an City, Shaanxi History Mu- 8(b) show that our algorithm outperforms the other two seum, Google Photo Gallery, Baidu Photo Gallery, Flikr, approaches in the whole range of data sparsity. This is and Search Engine. The database is related to the main because the PCC and Jaccard approaches suffer from data culture, history, and attractions of Han Chang’an City. sparsity and the model-based method can overcome this Moreover, it is still increasing constantly. drawback to a certain extent. In Figures 8(c) and 8(d), After a user logs in to the virtual Han Chang’an sys- our system achieves good performance when the test tem, the system will recommend cultural relics based on number is 10, and MAE and RMSE decrease slowly with the preference of the user. Our system enables a user to the increase of the number of test users after the test assign a number of white lights (five white lights just number is greater than 20. This is because similar users below each image) to the recommendation results as rat- are increasing with the increase of the number of testers. ings, and one white light represents one point. The rat- The performance of our system fluctuates in a small ing values in our system are integers ranging from 1 to range, which demonstrates the stability of our system. 5. Rate 1 means the user does not like the recommenda- tion result, and rate 5 means the user likes it very much. Two screenshots of recommendation results of images 4.3 System Recommendation Results to a user are listed in Figure 9. Our multimedia database currently consists of From the figure we can see when the system recom- about 2000 images and 2500 texts coming from the her- mends three images in Figure 9(a), which are a view of

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 334 PRESENCE: VOLUME 26, NUMBER 3

Figure 9. Two screenshots of personalized recommendation results.

Weiyang Palace, the site of Front Hall, and the recon- including PC and Web Player. VR glasses were adapted structed model of Front Hall, respectively, the user rates in the PC version, with which users can have a strong 4 to the site and rates 5 to the model; it intuitively indi- sense of substitution. cates the user is interested in Front Hall. Then, the The dynamic information recommendation module is recommendation results are changed based on users’ designed to make up for the drawbacks of the current feedback and listed in Figure 9(b), which are the images of VR systems, which display limited information. The the text description, the eaves tile, and the site of Front Hall. information recommended by this module includes hot The personalized recommendation module in our vir- topic information and personalized recommendation in- tual roaming system helps users to visit more quickly by formation. The former is obtained through web crawlers providing recommendations to them, which not only providing users with the latest news related to Han improves the user’s browsing efficiency and knowledge, Chang’an City. The proposed hybrid recommendation the system becomes more attractive and interesting at algorithm combing explicit feedback and implicit feed- the same time. back achieves the latter. The performance of the pro- posed algorithm is validated on two datasets, which is 5 Discussion compared with six other approaches. The performance of our system is evaluated from two In this article, our motivation lies in the combina- aspects of user satisfaction and accuracy offline through tion of the dynamic information recommendation mod- four experiments, which prove that our system is stable ule and virtual Han Chang’an City to realize the and the accuracy is high. However, two limitations still intelligent virtual system and provide users with more exist in our system. One is that we have not connected convenience. the VR glasses to the web version successfully, but we The virtual Han Chang’an City scene was recon- are trying. The other is that we have not tested our structed with VR technology in Unity3D. Two versions system online. Next, we will design several evaluation of virtual Han Chang’an City project were released, metrics for testing online.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 Feng et al. 335

6 Conclusion and Future Work University of Ministry of Education of China, number IRT_17R87. In this article, a virtual roaming system of the Han Chang’an City is proposed, with the combination of VR References technology and multimedia information recommenda- tion technology. With the dynamic information recom- Adomavicius, G., & Zhang, J. (2016). Classification, ranking, mendation module, users can receive their interests and and top-k stability of recommendation algorithms. latest information about Han Chang’an City quickly. INFORMS Journal on Computing, 28(1), 129–147. The main contribution of our work can be summed up Albanese, M., d’Acierno, A., Moscato, V., Persia, F., & Picar- in three aspects: iello, A. (2011). A multimedia semantic recommender sys- tem for cultural heritage applications. Proceedings of IEEE We accomplish the reconstruction of virtual Han Fifth International Conference on Semantic Computing, Chang’an City scene in Unity3D, while the models 403–410. of palaces and cultural relics are built in 3ds Max. Ardissono, L., Kuflik, T., & Petrelli, D. (2012). Personaliza- We design the whole scene roaming system, with a tion in cultural heritage: The road travelled and the one multimedia information display module to the vir- ahead. User Modeling and User-Adapted Interaction, tual Han Chang’an City scene and a navigation map 22(1–2), 73–99. to be convenient for users. Bailey, J. O., Bailenson, J. N., & Casasanto, D. (2016). When does virtual embodiment change our minds? Presence: Our virtual roaming system also has the hot topic Teleoperators and Virtual Environments, 25(2), 222–233. recommendation and personalized recommendation Bartolini, I., Moscato, V., Pensa, R. G., Penta, A., Picariello, modules to provide more useful information to users. A., Sansone, C., & Sapino, M. L. (2013). Recommending In order to let more people understand Han Chang’an multimedia objects in cultural heritage applications. Proceed- City in the Western Han Dynasty, the roaming system is ings of International Conference on Image Analysis and Proc- released online. Users only need to download a web essing, 257–267. player plug-in to experience the strongly interactive Bartolini, I., Moscato, V., Pensa, R. G., Penta, A., Picariello, scene. With the virtual roaming system, users can enjoy A., Sansone, C., & Sapino, M. L. (2016). Recommending the free experience any time, any place. multimedia visiting paths in cultural heritage applications. In our future work, we plan to consider three direc- Multimedia Tools and Applications, 75(7), 3813–3842. tions to improve the system. First, the complete cold start Benouaret, I., & Lenne, D. (2015). Combining semantic a problem is still in our system. Inspired by machine learn- nd collaborative recommendations to generate personalized ing, we plan to adopt a framework combining model- museum tours. Proceedings of East European Conference based CF and deep learning to overcome the complete on Advances in Databases and Information Systems, 477–487. cold start and to improve the accuracy of the recommen- Benouaret, I., & Lenne, D. (2016). Personalizing the museum dation. Second, we are prepared to design several evalua- experience through context-aware recommendations. IEEE tion metrics to test our system online. In addition, we International Conference on Systems, Man, and Cybernetics, intend to add several versions of the language to the sys- 743–748. tem. Ultimately, we expect our work can provide some Bobadilla, J., Ortega, F., Hernando, A., & Gutie´rrez, A. help to scholars, scientists, and students who prepare to (2013). Recommender systems survey. Knowledge-Based Sys- carry out archaeological or historical research. tems, 46(1), 109–132. Champion, E. (2015). Defining cultural agents for virtual her- itage environments. Presence: Teleoperators and Virtual Envi- Acknowledgments ronments, 24(3), 179–186. Costa, A. F. D., & Manzato, M. G. (2016). Exploiting multi- This work was supported by the Program for Changjiang modal interactions in recommender systems with ensemble Scholars and the Innovative Research Team in the algorithms. Information Systems, 56(C), 120–132.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021 336 PRESENCE: VOLUME 26, NUMBER 3

Feng, J., Feng, X., Liu, X., & Peng, J. (2016). The virtual wan- Conference on Information, Intelligence, Systems and Applica- dering system of Han Chang’an City based on information tions, 1–6. recommendation. Proceedings of 15th ACM SIGGRAPH Pfandler, M., Lazarovici, M., Stefan, P., Wucherer, P., & Conference on Virtual-Reality Continuum and Its Applica- Weigl, M. (2017). Virtual reality based simulators for tions in Industry, 75–78. spine surgery: A systematic review. Spine Journal, Official Feng, J., Liu, X., Feng, X., Peng, J., Wang, X., & Peng, X. Journal of the North American Spine Society, 17(9), (2016). A new method of virtual Han Chang’an City naviga- 1352–1363. tion system. Proceedings of Eighth International Conference Ramanathan, S., Rangan, P. V., & Vin, H. M. (2002). Inte- on Digital Image Processing, 100335X. grating virtual reality, tele-conferencing, and entertainment Han Chang’An Archeological Team, Institute of Archaeology, into multimedia home computers. IEEE Transactions on & Chinese Academy of Social Sciences (2012). Architectural Consumer Electronics, 38(2), 70–76. Complex VI of Changle Palace, Han Chang’an City in Xi’an. Rossi, S., Barile, F., Improta, D., & Russo, L. (2016). Towards Chinese Archaeology, 12(1), 70–80. a collaborative filtering framework for recommendation in Kim, H. N., Ha, I., Lee, K. S., Jo, G. S., & Ei-Saddik, A. museums. Procedia Computer Science, 98(C), 431–436. (2011). Collaborative user modeling for enhanced content Shen, W., & Zeng, W. (2011). Research of VR modeling tech- filtering in recommender systems. Decision Support Systems, nology based on VRML and 3DSMAX. Proceedings of Inter- 51(4), 772–781. national Conference on Computer Science and Network Tech- Kim, Y. S., Kesavadas, T., & Paley, S. M. (2014). The virtual nology, 487–490. site museum: A multi-purpose, authoritative, and functional Silva, E. Q. D., Camilo-Junior, C. G., Pascoal, L. M. L., & virtual heritage resource. Presence: Teleoperators and Virtual Rosa, T. C. (2016). An evolutionary approach for combining Environments, 15(3), 245–261. results of recommender systems techniques based on collab- Koren, Y. (2008). Factorization meets the neighborhood: A orative filtering. Expert Systems with Applications, 53, multifaceted collaborative filtering model. Proceedings of 204–218. ACM SIGKDD International Conference on Knowledge Dis- Stamford, J., Khuman, A. S., Carter, J., & Ahmadi, S. (2014). covery and Data Mining, 426–434. Pathfinding in partially explored games environments. The Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization Workshop on Computational Intelligence, 1–6. techniques for recommender systems. Computer, 42(8), 30–37. Wang, S., Mao, Z., Zeng, C., Gong, H., Li, S., & Chen, B. Kyriakou, M., Pan, X., & Chrysanthou, Y. (2016). Interaction (2010). A new method of virtual reality based on Unity3D. with virtual crowd in immersive and semi-immersive virtual Proceedings of International Conference on Geoinformatics, reality systems. Computer Animation & Virtual Worlds, 1–5. 1–12. Wang, Y., Aroyo, L., Stash, N., Sambeek, R., Schuurmans, Y., Lin, L., & Zhao, S. (2014). Construction of campus 3D scene Schreiber, G., & Gorgels, P. (2009). Cultivating personal- based on 3ds Max and Unity3D. Proceedings of Technical ized museum tours online and on-site. Interdisciplinary Sci- Conference on Hydraulic Engineering, 107–111. ence Reviews, 34(2–3), 139–153. Liu, H., Hu, Z., Mian, A., Tian, H., & Zhu, X. (2014). Xia, Z., Feng, X., Peng, J., & Fan, J. (2015). Content- A new user similarity model to improve the accuracy of irrelevant tag cleansing via bi-layer clustering and peer collaborative filtering. Knowledge-Based Systems, 56(3), cooperation. Journal of Signal Processing Systems, 81(1), 156–166. 29–44. Lu, J., Wu, D., Mao, D., Wang, W., & Zhang, G. (2015). Rec- Yang, C., Zhang, F., & Wu, Q. (2009). Role navigation of ommender system application developments: A survey. Deci- cultural heritage based on web interactive technology. sion Support Systems, 74(C), 12–32. Proceedings of IEEE International Conference on Maes, P. (1995). Agents that reduce work and information Computer-Aided Industrial Design & Conceptual Design, overload. Readings in Human–Computer Interaction, 37(7), 1864–1868. 811–821. Yang, K., & Jie, J. (2014). The research of making scenic wan- Mousouris, S., & Styliaras, G. (2014). Implementing digital dering system based on Unity 3D. IEEE Workshop on Elec- cultural heritage map. Proceedings of 5th International tronics, Computer and Applications, 237–239.

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/pres_a_00299 by guest on 25 September 2021