Matching Media Contents with User Profiles by Means of the Dempster

Home , Ewan McGregor, Felicity Jones

arXiv:1704.03048v1 [cs.AI] 10 Apr 2017 r h uhr’onadd o eetteve fSyItalia. Sky of view the reflect not i do expressed and opinions own The authors’ capacity. the personal are his in Gaglione utmr eoepr ftemdl nihn h aae of dataset the enriching model, customers ne the new and of contents support part new existing become to Soon, customers to way contents. cus- contents existing other and discovering new the in contents propose on existing to and beyond help customers, also go and They models profitable implementation tomers. these the more of from towards coming use upgrade benefits their The customer the products. foster improve to and means This retention for. paid pa they valuable product as the appreciate would they a that contents by discover available made to contents able of data company. catalog from media vast model large about the a learn provide to to user they is link feedback his/her goal the are ultimate to th The what at them. and user looking access by the them they directly infer contents to ask but preferences, to and not Her interests enjoy. is they contents problem media the and users between stands that data. able about models reason so to central, and become interpret to preferences regard user. and customers building each from of profile way collected for their the Data appropriate reformulate offering. o to content most often chance requiring the is is interests, the change that his/her a has time Such fit the everybody best at that that demand, so contents enjoying catalog of possibilit the new tailoring with industry, of media in business performing t rprisb en fatyeape ial esuggest we common Finally are that example. industry. tasks toy media for modern model a the the of of paper means applications this possible illustr by and In model properties two. reference Dempster-Shits a the the proposing profi between to using Evidence, relationship of user match side that Theory the best build one to to the suggest at other find we the to looking on order enjoy, audienc and establ in they the characteristics goes content content target that the and relationships better the users to analyze between attempt to in requires This contents of offering ICAMR hsatcewspeae racmlse yCi by accomplished or prepared was article This DISCLAIMER. okn tps neatosi sflt epuesto users help to useful is interactions past at Looking relationship the exploit and discover to aims models These of way the changing radically are technologies Digital Abstract acigMdaCnet ihUe Profiles User with Contents Media Matching mi:[email protected] Email: eateto Engineering of Department Temdaidsr sicesnl esnlzn the personalizing increasingly is industry media —The ymaso h eptrSae Theory Dempster-Shafer the of means by nvriyo Sannio of University eeet,Italy Benevento, ug Troiano Luigi .I I. NTRODUCTION optrSineDepartment Science Computer mi:[email protected] Email: hspaper this n veoUniversity Oviedo i´ n Spain Gijón, afer’s rn D´ıaz Irene ished ating tof rt ose ing ies le, ro in w e. n e hsppri raie sflos eto Ipoie some provides II Section follows: o as reminder organized regarding The inference is contents. make paper and to this it users relationship use between a to relationship and build the [11] to [10], (D- Evidence theory) aim of S Theory we Dempster-Shafer’s the [9], on based in model developed model the eainhp oto hmtk h omo nomto fusi this information of exploit form and the take discover them to of Most order relationship. in tech experimented Different been descriptors. have content is objective to The profiles etc. user job, relate members, family demographi gender, regarding age, information such other which description need an th we content addition the fully monitoring In accessed enjoyed by they items, often implicitly item the how or which partially, rate likes, e.g., to or behavior, user stars infor- customer the using Such asking shows. by by and e.g., gathered series be movies, can as mation such items catalog e-commerc other to applied been [7]. has domains it studied later most the although music recommendations movies, topic, music on being focused [6], been as books has and information RS informat to of social related or kinds research gender) The other age, use items (e.g, also features rate demographic can hotels to RS (booked users behavior songs). the users’ heard monitoring asking by by informati implicitly explicitly This or him. acquired to diff be proposed the be can to can respect that with alternatives habits ent and informati profile user requires a the system to regarding recommender order a In task, preferences. its user scoring complish the to by according them alternatives, ranking available regarding suggestion, [5]. recommenders Hybrid Demo- and [4], [3], recommenders recommenders graphic are Collaborative Content-b They including [2], types, diffusion. recommenders different e-commerce in grouped large the generally gained of systems because th Recommender application supporting [1]. and showed platform developing Netflix in streaming once algorithms attention of large potentiality obtained that (RS), tems of acquisition the customers. support and to contents suitable new Predictivenes also process. them self-growing make a models along entities, new olwn h daepoe y[] n oeconcretely more and [8], by explored idea the Following iia oR,w eddt bu srlknsregarding likings user about data need we RS, to Similar a provide to is system recommender a of purpose The sys- recommender of logic core the at are models These mi:[email protected] Email: neatv vLab Tv Interactive ioGaglione Ciro ia,Italy Milan, k Italia Sky niques etc. , ased of s ion. and on. er- eir on on or cs to c- e e f preliminaries regarding D-S Theory; Section III describes the model; Section IV outlines some examples of application; Section V draws conclusions and future directions.

II. PRELIMINARIES The Dempster-Shafer theory, also known as the Theory of Evidence [10], [11], is used as basis for the preference model presented in [9]. In D-S theory, basic probabilities are allocated to subsets, instead of elements, according to the following definitions. Definition 1. A function m : 2Ω −→ [0, 1] over a set Ω is called a basic probability assignment if Fig. 1. The Boolean lattice of item subsets from Ω = {A,B,C,D} with m(∅)=0 and X m(A)=1 focal elements F (Ω) = {C,BC,AD.} [9] A∈2Ω Definition 2. Let Ω be a set, then A ⊆ Ω is a focal element if m(A) > 0. In addition, F (Ω) ⊂ 2Ω represents the set of constant as far as we move to nodes that do not a probability focal elements induced by m. mass assigned to them. As consequence of this property, we can identify regions of connected nodes, each assuming a Definition 3. Let m be a basic probability assignment function specific value of Belief or Plausibility, as illustrated by Fig. 2. over a set Ω. The Belief of A ⊆ Ω induced by m is defined In this example, focal elements are C, BC and AD with the as follows associated basic probability assignments mC , mBC and mAD Bel(A)= X m(B) (1) (assuming mC + mBC + mAD = 1). This leads to identify B⊆A 8 groups in the lattice, each with Belief and Plausibility Definition 4. Let m be a basic probability assignment function depending from a focal subset of F (Ω). Fig. 2 outlines these over a set Ω. The Plausibility of A ⊆ Ω induced by m is regions for both Belief and Plausibility. we can observe how defined as follows all portions of lattice associated to a given value of Belief or Plausibility are connected. Pl(A)= X m(B) (2) B∩A6=∅ The relationship between Plausibility and Belief is given by the following equation: Pl(A)=1 − Bel(A) (3) where A is the complement of A to Ω. When the probability basic assignments are given by different sources, it is possible to combine them. The first and most common combination method is known as the Dempster’s rule, that is defined as follows:

Definition 5. Let m1 and m2 be two basic probability assignments, the joint basic probability assignment is computed as 1 m1 2(A)= m1(B) · m2(C) (4) , 1 − Z X B∩C=A where Z = X m1(B) · m2(C) (5) B∩C=∅ is a measure of conflict between the two basic probability assignment sets. In addition, it is assumed m1,2(∅)=0. Fig. 2. Belief (top) and Plausibility (bottom) regions induced by F (Ω) = Belief and Plausibility are monotonic functions with respect {C,BC,AD} [9] to inclusion. This means that if we consider the lattice of Ω subsets, as shown in Fig. 1, Belief and Plausibility will If we sort the Belief (or Plausibility) values in ascending increase from bottom (Bel(∅)= Pl(∅)=0)to top(Bel(Ω) = order, we get a sequence of levels, each grouping the nodes Pl(Ω) = 1). In particular Belief and Plausibility will be kept into those that are below the level and over the level. For For instance, according to the example in Fig. 1 F (Ω) = {C,BC,AD}, we have Cr(BCD) = {C,BC} = Cr(BC) and Su(ABD) = Su(BD) = Su(AB) = {BC,AD}. It is straightforward that Cr(A) ⊆ Su(A), for all A ⊆ Ω. The core and support represent the basis for computing respectively the Belief and the Plausibility of A. The core and the support are able to group the subsets of Ω into classes of equivalence as (a) (b) the following definition states. Definition 8 (Cr− and Su− Equivalence). Two sets A and B are said to be Cr-equivalent if and only if Cr(A)= Cr(B)= Cr. A Cr-equivalence class is defined as the collection def ECr == {A ⊆ Ω | Cr(A)= Cr} (8) In addition, A and B are Su-equivalent if and only if Su(A)= Su(B) = Su. The Su-equivalence class obtained from this relation. is defined as (c) (d) def ESu == {A ⊆ Ω | Su(A)= Su} (9) Fig. 4(a) provides an example of Cr-equivalence class assuming as core Cr = {BC,C}. Fig. 4(b) shows the Su- equivalence class for the support Su = {BC,AD}.

(e) (f)

(a) (b) Fig. 4. Examples of Cr-equivalence (a) and Su-equivalence (b) classes

As an immediate consequence, if A and B are Cr- (g) (h) equivalent, then Bel(A) = Bel(B), while if they are Su- equivalent, Pl(A)= Pl(B). Fig. 3. Plausibility levels Cr− and Su− equivalence classes perform a partitioning of 2Ω. Thus, each subset X ⊂ Ω can belong only to one equiva- instance, if we assume lence class. Grouping subsets in Cr− and Su−equivalence classes allows (i) to explore the lattice by moving across 0 ≤ mBC ≤ mAD ≤ mBC + mAD ≤ mC classes, instead of exploring the whole item subset space, and ≤ mBC + mC ≤ mAD + mC ≤ mBC + mAD + mC =1 (ii) to choose a representative of each class, so that the list of recommended items is shorter. For instance, we might be we get the situation depicted by Fig. 3 with respect to interested in using the smallest subset within a Cr-equivalence Plausibility. The following definitions enable the concept of class. classes of equivalence among the subsets with respect to Belief As representative of a Cr−equivalence class we can assume or Plausibility and to identify those elements that are most the smallest subset. We call this set Cr−minimal. For instance, representative of the class. for the class {BC,ABC,BCD}, the core is {C,BC} and Definition 6 (Core). Given a subset A ⊆ Ω, the set of focal the Cr−minimal is BC. It is possible to prove that each elements included in A, core of A, is defined as Cr−equivalence class as one single Cr−minimal. Conversely, def for Su−equivalence classes we assume as representative Cr(A) == {B ∈ F (Ω)|B ⊆ A} (6) the largest subset, that we call Su−maximal. Similarly to Definition 7 (Support). Given a subset A ⊆ Ω and the set of Cr−equivalence classes, it is possible to prove that any focal elements (even partially in A), support of A, is defined Su−equivalence class has one single Su−maximal. For exam- as ple, the class {C,AD}, whose support is {ACD, AC, AD}, def Su(A) == {B ∈ F (Ω)|B ∩ A 6= ∅} (7) as ACD as maximal. III. MODEL It is easy to prove that m(∅)=0 and m(Φ) = 1. Assuming that in our example L , some example of masses In the context of our interest we assume I = {I1,...,Im} | | = 15 as the set of items belonging to the content catalog, while assigned to characteristics are given below. 2 U = {U1,U2,...,Un} as the set of users. • Stars: m(De Niro,Bacon,Pitt)= 15 4 Both sets are projected on two feature spaces, respectively • Director: m(Boyle)= 15 5 made of p and q dimensions. The first is referred to the set of • Year: m(1996) = 15 3 characteristics describing the items in I, C = {C1,...,Cp}, • Genre: m(Drama)= 15 while the second to the user profiling P = {P1,...,Pq}. Both If we refer to profiling features, some examples are the spaces are discrete, so that each Ci and Pj can assume a finite following: number of values. 8 • Age: m(30s)= 15 The relationship between items and users is expressed by 5 • Gender: m(F )= 15 a choice matrix, as that shown in Tab. I. The choice matrix 11 • Location: m(IT )= 15 is places side by side to the item characteristics matrix (left 3 • Interests: m(Movies,Books)= 15 side) and to the profile matrix (top). We notice that focal elements of each dimension are given by its unique values, i.e. by rows after removing duplicates. TABLE I For instance, for the ”Director” and ”Year” dimensions, focal STRUCTURE OF DATASET ASSUMED BY THE MODEL elements are given by the set of director names, i.e., Boyle,

Pq p1,q p2,q p3,q ... pn,q Levinson, Scorsese, Howard, Zemeckis, Edwards and Scott,

Pq p1,q p2,q p3,q ... pn,q and by years, i.e., 1985, 1990, 1994, 1995, 1996, 2000, 2015, 2016. Similarly for ”Age”, i.e., 20s, 30s, 40s, ”Location”, i.e., Pq p1,q p2,q p3,q ... pn,q IT, SP, and ”Gender”, i.e., M, F. They are all single-value ...... dimensions. For them, focal elements are singletons. In this Pq p1,q p2,q p3,q ... pn,q case the model becomes additive. For instance, ❍ U • Bel Scorsese, Boyle m Scorsese m Boyle C1 C2 ... C ❍ ... n ( )= ( )+ ( ) p I ❍ 1 2 3 ❍ • Pl(Scorsese, Boyle)= m(Scorsese)+ m(Boyle) X X c1,1 c1,2 ... c1,p 1 ... Instead, the dimensions ”Genre” and ”Stars” are multi- X X c2,1 c2,2 ... c2,p 2 ... value, so their focal elements are not singletons. For instance, X X X c3,1 c3,2 ... c3,p 3 ... Drama, Comedy-Drama and Adventure-Drama-History are ...... three focal elements of ”Genre”. Similarly, Movies-Books, ...... Books, Sport, Music-Sport are focal elements of ”Interests” c 1 c 2 ... c m X X ... m, m, m,p among the profiling features. For multi-value dimensions the model is not additive. As an example, let us consider the belief In general, data points c and p are multi-valued, mean- i,h j,k of Adventure-Comedy-Sci-Fi-Drama. We have, ing that they are represented by sets of values. For instance if Ch is representing the movie cast, ci,h is represented by the Bel(Adventure, Comedy, Sci − Fi,Drama)= list of actors that are featuring in the movie Ii. Similarly, if m(Adventure, Drama, Sci − Fi)+ P is ”interests”, p will list what the user U is interested k j,k j m(Adventure, Sci − Fi)+ in. In other cases they are single-valued, such as in the case of characteristics such as ”director” and ”year” or in the case m(Comedy, Drama)+ of profiling features such as ”age” or ”location”. An example m(Drama)= of this matrix is given in Tab.II. 1 3 1 3 8 + + + = Let us denote with Φ(Ch) the overall set of values assumed 15 15 15 15 15 over the item characteristic Ch, and with Φ(Pk) the overall Conversely, we have set of values for the user profiling feature Pk. They are respectively given in Tab.III and Tab.IV. Pl(Adventure, Comedy, Sci − Fi,Drama)=1 Since here we are interested to use both information regard- because all focal elements of ”Genre” are involved in its ing the item characteristics and the user profiles, we compute computation. for any |L(K)| So far, we considered each dimension in isolation. They m(K)= (10) provide a range for probability P r(K) = [Bel(K),Pl(K)], |L| with K ⊆ Φ, that is a measure of likelihood that a content in where I characterized by K will be enjoyed by the set of users in • K ⊆ Φ, with Φ being the overall set of a given U, if Φ is referred to some item characteristic Ch. Or, if we characteristic Ch or a profiling feature Pk. look at K as referred to some profiling feature Pk, it is the • L ⊆ I × U is the set of preferences (”likes”) likelihood that a user in U will enjoy the catalog of contents • L(K) ⊆ L is the subset of preferences referred to K offered by means of I. TABLE II THE DATASET USED AS EXAMPLE.

Age 30s 30s 20s 40s Gender M F M M Location IT IT SP IT Movies Music Interests Sport Books Books Sport ❍ U Director Year Stars Genre ❍ I ❍❍ 1 2 3 4 Boyle 1996 Ewan McGregor, Ewen Bremner Drama 0 X X X Levinson 1996 Robert De Niro, Kevin Bacon, Brad Pitt Crime, Drama, Thriller 1 X X Scorsese 2015 Robert De Niro, Leonardo DiCaprio, Brad Pitt Short, Comedy 2 X Scorsese 1990 Robert De Niro, Ray Liotta, Joe Pesci Biography, Crime, Drama 3 X Boyle 2000 Leonardo DiCaprio Adventure, Drama, Romance 4 X Howard 1995 Tom Hanks, Kevin Bacon Adventure, Drama, History 5 X X Zemeckis 1994 Tom Hanks Comedy, Drama 6 X Zemeckis 1985 Michael J. Fox, Christopher Lloyd Adventure, Sci-Fi 7 X Edwards 2016 Felicity Jones, Diego Luna Adventure, Sci-Fi 8 X X Scott 2015 Matt Damon Adventure, Drama, Sci-Fi 9 X

TABLE III Thus, in order to perform a combination of K1 and K2 we OVERALLSETSOFITEMCHARACTERISTICS need to look at L(K1) and L(K2). In the case of conjunction Director Year Actors Genre K = K1 ⊙ K2, we have L(K)= L(K1) ∩ L(K2), so that Boyle 1996 Ewan McGregor Crime |L(K1) ∩ L(K2)| Levinson 2015 Ewen Bremner Drama m(K)= (11) Scorsese 1990 Ray Liotta Thriller |L| Howard 2000 Robert De Niro Short Zemeckis 1995 Kevin Bacon Comedy For instance, if K1 is Zemeckis and K2 is Adventure-Sci-Fi, Edwards 1994 Brad Pitt Biography we have that L(K1) is made of preferences at rows 6 and 7, Scott 1985 Leonardo DiCaprio Adventure 2016 Joe Pesci Romance while L(K2) at rows 7 and 8, so that L(K1)∩L(K2) is made 1 Ray Liotta History only of row 7, and m(K)= 15 . Once we have focal elements Tom Hanks Sci-Fi for the conjunction of the ”Director” and ”Genre”, we can Michael J. Fox Christopher Lloyd compute the belief and plausibility over the conjunction of Felicity Jones the two. For instance, Diego Luna 1 Matt Damon Bel(Zemeckis⊙Drama)= m(Zemeckis⊙Drama)= 15 TABLE IV and OVERALLSETSOFUSERPROFILINGFEATURES 1 Pl(Zemeckis ⊙ Drama)= m(Zemeckis ⊙ Drama)= Age Gender Location Interests 15 20s M IT Books 30s F SP Movies The meaning of P r(Zemeckis⊙Drama) is the likelihood 40s SP Sport that a drama directed by Zemeckis will be enjoyed given L Music and users in U, that is exactly 1 over 15. The other way of combining two dimensions is by means of disjunction. In this case K = K1 ⊕ K2 and If we would like to look at multiple dimensions we are not |L(K1) ∪ L(K2)| allowed to use the Dempter’s combination rule as described in m(K)= (12) the section above. The main issue is that dimensions belong |L| to different domains, so that the information fusion given For instance, with regard to profiling features, if K1 is 20s by Eq.(4) cannot be performed over comparable sets. This and K2 is Sport, we have problem can be solved when we look at focal elements as 7 representative of preferences over the matrix L. Let K1 and Bel(Sport ⊙ 20s)= K2 two features defined over different dimensions. We can 15 combine the two by means of conjunction or disjunctions, as the conjunction of the two collect rows 0,2,3,4,5,8,9. In this depending on the semantics we associate to the operation. case, both belief and plausibility are larger. IV. APPLICATIONS IN MEDIA INDUSTRY in order to decide the order of two alternatives. It is also possible to choose randomly an alternative when the two The model presented so far can be employed for different cannot be sorted. Finally, it is possible to look at other tasks. We briefly outline some of them below. solutions investigated in the field of partial order theory. Recommendations. The model can be used to suggest a content to a user according to each dimension. For instance, V. CONCLUSIONS chosen the dimension of ”Director”, the system might suggest In this paper, we further investigated a preference model directors that are most likely be of interest for the user. It is based on the Dempster-Shafer theory and its application to also possible to combine different dimensions. For example, media industry. This work is an evolution of what has been ”Genre” and ”Year”. In any case, the inference of preferences done so far by introducing some elements of novelty. Among is performed by looking at users indistinguishably, meaning them the possibility of including the user profile as part of the that profile information is not taken into account. inference, instead of being considered neutrally with respect to Audience targeting. In this case, given a single content we different applications and problems that have been discussed are interested to find user profiles that might be interested to in the section before. There are still some issue to address. The it. For example, given a new movie, the model might estimate most important is referred to scalability of the model. Indeed, how likely could be of interest for each range of age. Also the nature of the D-S theory is inherently combinatorial, so in this case it is possible to combine multiple dimensions that that the search space is exploding by including more elements are user profiling features. For instance, considering multiple within the dimension overall sets Φ. The possibility of defining age ranges, taking into account the different genders. equivalence classes in terms of belief and plausibility is a way Content bundling. This application is aimed to propose a to reduce complexity, but still work has to be done to make this bundle of contents to a group of users, possibly with different solution feasible in practice. In addition, the model presented profiles. This result can be suggested by the model through a here requires to be validated. This can be done by looking combination of dimensions among characteristics and profiling at correspondences between the probability ranges and the features. The process can be led by two different perspectives. frequency of positive voted that are after recorded. In the future The first moves from the bundle of contents and it is aimed we aim to develop further the model in order to include more at identifying a group of users that might be interested in. For complex queries and to solve issues regarding the application instance, given all drama movies in 90s, which users could be of the model in practice with respect to large catalogs and interested to such an offer. But it is also possible to move the audience. other way round: selected a group of users, what is the bundle REFERENCES of contents that might be of their interest. With respect to our [1] C. A. Gomez-Uribe and N. Hunt, “The netflix recommender system: example, given users in the 20s that are interested to books, Algorithms, business value, and innovation,” ACM Trans. Manage. Inf. what is the bundle of contents that could be likely of their Syst., vol. 6, no. 4, pp. 13:1–13:19, Dec. 2015. doi: 10.1145/2843948. interest. [Online]. Available: http://doi.acm.org/10.1145/2843948 [2] J. Salter and N. Antonopoulos, “Cinemascreen recommender agent: Segmentation. This is a generalization of the problem Combining collaborative and content-based filtering,” IEEE Intelligent above. In this case both users and contents are objective of the Systems, vol. 21, no. 1, pp. 35–41, Jan. 2006. doi: 10.1109/MIS.2006.4. [Online]. Available: http://dx.doi.org/10.1109/MIS.2006.4 analysis. We are interested to find clusters of users, contents [3] L. Candillier, F. Meyer, and M. Boullé, “Comparing state-of-the-art and user/contents that maximize the likelihood of preferences collaborative filtering systems,” pp. 548–562, 2007. within the group and minimize the likelihood of preferences [4] M. J. Pazzani, “A framework for collaborative, content-based and between groups. For instance, by looking at our example, we demographic filtering,” Artif. Intell. Rev., vol. 13, no. 5-6, pp. 393– 408, Dec. 1999. doi: 10.1023/A:1006544522159. [Online]. Available: could be interested to see if there are users with different http://dx.doi.org/10.1023/A:1006544522159 profiles that are likely to enjoy the same contents, or if there [5] R. Burke, “Knowledge-based recommender systems,” in Encyclopedia are contents are have similar likelihood to be enjoyed by the of Library and Information Science, vol. 69, A. Kent, Ed. Taylor and Francis, 2000, pp. 180–201. audience, or if there groups of users that are likely to enjoy [6] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez, “Recommender the same group of contents, besides the others. systems survey,” Knowledge-Based Systems, vol. 46, no. 0, pp. 109 – 132, 2013. doi: 10.1016/j.knosys.2013.03.012. [Online]. Available: In all tasks above, it plays a key role the possibility of http://www.sciencedirect.com/science/article/pii/S0950705113001044 comparing and ranking alternatives. However, the D-S theory [7] J. J. Castro-Schez, R. Miguel, D. Vallejo, and L. M. López-López, “A provides only an imprecise probability that ranges between highly adaptive recommender system based on fuzzy logic for {B2C} e-commerce portals,” Expert Systems with Applications, vol. 38, no. 3, the lower bound given by the degree of belief and the upper pp. 2441 – 2454, 2011. doi: 10.1016/j.eswa.2010.08.033 bound given by the degree of plausibility. This issue can be [8] K. Zhang and H. Li, “Fusion-based recommender system,” in Infor- addressed by different approaches. mation Fusion (FUSION), 2010 13th Conference on, July 2010. doi: 10.1109/ICIF.2010.5712091 pp. 1–7. The first approach is to use a degree that is representative [9] L. Troiano, L. J. Rodr´ıuez-Muñiz, and I. D´ıaz, “Discovering user of a range, such as the middle point between belief and preferences using dempster-shafer theory,” Fuzzy Sets and Systems, vol. plausibility. Another possibility is to use only belief degrees 278, pp. 98 – 117, 2015. doi: http://dx.doi.org/10.1016/j.fss.2015.06.004 [10] A. P. Dempster, “Upper and lower probabilities induced by a multivalued (conservative approach) or plausibility (challenging approach). mapping,” Annals of Mathematical Statistics, vol. 38, pp. 325–339, 1967. Another approach could be to randomly choose n pairs from [11] G. Shafer, A Mathematical Theory of Evidence. Princeton: Princeton both ranges and to use the majority or pairwise comparisons University Press, 1976.