1

Generalized Score Distribution Lucjan Janowski, Bogdan Cmiel,´ Krzysztof Rusek, Jakub Nawała, Zhi Li

Abstract—A class of discrete probability distributions contains evidence that GSD describes the answering process in a video distributions with limited support, i.e. possible argument values quality test correctly. We strongly believe that GSD distribu- are limited to a set of numbers (typically consecutive). Examples tion can be useful in different fields of science, since it is a of such data are results from subjective experiments utilizing the Absolute Category Rating (ACR) technique, where possible generalization of the existing and widely used distributions. 1 answers (argument values) are {1, 2, ··· , 5} or typical Likert For the proposed distribution the estimation algorithm is scale {−3, −2, ··· , 3}. An interesting subclass of those distri- provided and analyzed by extensive simulation study. The butions are distributions limited to two parameters: describing analysis of different existing video quality databases allow us the mean value and the spread of the answers, and having no to estimate typical parameters of this model. Those results can more than one change in the probability monotonicity. In this paper we propose a general distribution passing those limitations be used as a prior distribution of the GSD parameters in the called Generalized Score Distribution (GSD). The proposed GSD future analysis such as Bayes estimator. covers all spreads of the answers, from very small, given by In order to validate GSD in the context of subjective the Bernoulli distribution, to the maximum given by a Beta experiments we have used four different databases, with more Binomial distribution. We also show that GSD correctly describes than 1,800 PVSs (Processed Video Sequences, i.e. scored subjective experiments scores from video quality evaluations with probability of 99.7%. A Google Collaboratory website sequences). The analyzed PVSs have different number of votes with implementation of the GSD estimation, simulation, and from typical 24 up to 213. The final score shows that GSD visualization is provided. describes the answers correctly with probability of 99.7%. Index Terms—Quality of Experience, Subject model, Subjective In the next section we describe the related work. Section III experiment, Data analysis, New distribution describes the proposed distribution with the estimation process detailed in Section IV. Section V provides short descriptions I.INTRODUCTION of the data sets used. Section VI presents the results. The last section concludes the paper. UBJECTIVE experiments let us collect users’ opinions S about a specific system. Such experiment can be seen as a measuring device. Any measuring device provides certain II.RELATED WORK precision. In this article we analyze subjects’/testers’/users’ Subject answers analysis is a broad topic considered in nu- answers, in order to better understand the random process merous publications starting from ITU standards to technical generating those answers. The obtained result is general and publications provided by companies. The standard focusing on covers interesting class of discrete distributions with limited the objective metrics evaluation is ITU-T P.1401 [2]. Its main support, and two parameters describing the mean and the goal is to describe correct way of analyzing subjective data. answer’s spread. Similarly, a recent publication [3] describes the problem of Our main goal is to check what is the distribution of correct way of comparing groups within subjective experiment. answers provided by subjects. We limit our considerations to Beyond correctness in the literature we can find interesting popular subjective experiments using discrete five point scale proposals for other than typical, focused on MOS, analysis. {1, 2, ··· , 5}. For this scale, numerous different databases are In [4] a number of different metrics related to user behavior available therefore we compare the theoretical and analytical and service acceptance are described. In [5] a method to find results. Nevertheless, the proposed model is general and in the each answer probability is suggested. More recently similar arXiv:1909.04369v1 [stat.ME] 10 Sep 2019 Appendix B we describe the generalization to M point scale. approach to model each answer probability, instead of simple Knowing the answers’ distribution provides important in- mean (MOS), was proposed in [6]. Other publications are formation about the answering process. Further modeling of focused on modeling Quality of Experience in general e.g. this process can lead to better understanding of experiments [7]. The model we propose in this paper is in line with those involving subjects [1]. The ultimate goal is to make data anal- analysis as the model is discrete. ysis of those experiments more precise. Better understanding Not many publications focus on the answering process, which factors influence the obtained answer allows to remove, especially the precision of the obtained score. An interesting or compensate, those aspects in the data analysis. analysis is shown in [8], where the relation between MOS The main contribution of this paper is to propose a new and the standard deviation of scores is studied. In the paper Generalized Score Distribution (GSD), and provide a strong a unique parameter HSE2, describing the test difficulty was B. Cmiel´ is with the Department of Applied Mathematics, AGH University of Science and Technology, Poland. 1The implementation can be found here https://colab.research.google.com/ L. Janowski, J. Nawała, and K. Rusek are with the Department of Telecom- drive/1ioM4JqUaEA8fHJH9V-0iphHt4C9zmI0i munictions of AGH University of Science and Technology, Poland. e-mail: 2In the original paper the parameter was named SOS but we think that HSE [email protected] (from Hossfeld, Schatz, and Egger) will be a better, unambiguous name. SOS Z. Li is with Netflix is traditionally used to describe standard deviation of scores. 2 proposed. Another interesting proposal analyzing the subjec- In general we can describe the subject answer as a function: tive experiment precision is [9] where different confidence U ψ , (2) intervals are studied. = + The analysis of the subjective experiment leads to proposing where ψ is the true quality and  is an error term. An two subject models in [10] and [11]. Those models are algorithm predicting quality should aim in estimating ψ. Still, discussed in more detail further in the paper. User model the error distribution is important and should be modeled. The allows to extend analysis of the subjective experiments like error term represents precision of ψ estimation. Note that the the one described in [1], [12], or make the existing analysis variance term (represented by ) cannot be too complicated. more precise [13], [14]. The user model is modified in [15], In total, we have four different probabilities. Therefore, we [16], [17] which leads to new results, e.g. showing that the would like to use a model for which the variance is described content has significant influence on both variability and mean by a single parameter. We can then use it to predict the true of the obtained answers [17]. quality. In turn, accuracy of its estimation is denoted by the The new subject model proposed in this paper extends variance parameter. the existing analysis by introducing discrete distribution. The For simplicity, we only consider a single sequence (i.e., previous publications used continuous process with discretiza- PVS) analysis. In general, we consider a model, where an tion and censoring (clipping). As the next section describes, answer for a sequence is a random variable U drawn from a this introduces certain error. Therefore, the proposed discrete distribution: model, not having this error, is a better solution. U ∼ F (ψ, θ), (3) In this paper we do not compare our model to models used in different fields working on quality. It is left as an where ψ is the true quality, θ is a parameter describing answers interesting future research. Comparing to those modes is spread, and F () a cumulative distribution function. However, difficult, since all those models need a different data, which in the text we mark θ differently, depending on the model we do not exists commonly in the video quality community. An are considering. This notation makes the description clearer. example could be food industry using “Panel Check” [18], Our notation convention generally follows guidelines of [19], the method proposed by Pane Check relay strongly on SAM (Statistical Analysis Methods) of VQEG (Video Quality the lack of tied answers. For the five point scale used in video Expert Group) group, described in [25]. Nevertheless, we quality evaluation, small number of tied answers is a strong introduce extensions to selected symbols. assumption. In psychology the signal detection theory (SDT) is used A. Continuous Model in the context of subjective quality scores analysis [20], [21]. The models proposed by [10] and [11] describe a subject Such analysis can be complicated taking into account different answer as a continuous normal distribution with certain mean influencing factors [22]. This is especially an interesting ψ, which is true quality, and standard deviation σ describing connection between QoE and psychology, nevertheless, again the error. Therefore, the subject answer is O ∼ N (ψ, σ). a slightly different data are needed. To use SDT we need a Since a subjective scale is discrete we cannot observe O. To recognition and a quality score, in the video quality analysis convert a continuous answer to a discrete one, a subject makes case only quality scores are available. discretization and censoring (clipping). This process converts It is also worth mentioning that the distribution proposed in continuous random variable O to discrete variable U. We can this paper can be used for modeling an answering in Likert calculate each answer probability (i.e., U distribution), as a scale surveys. For example, in [23] the authors consider the function of ψ and σ by the below equations: impact of the number of response categories on the reliability Z s+0.5 2 1 − (x−ψ) of the measurements. In the model of answers provided in P (U = s) = e 2σ2 (4) equation (1) of [23] one can consider GSD distribution of the s−0.5 2πσ random error to estimate the latent unobserved true score. The for s = {2, 3, 4} and same approach can also be used for example in [24] where Z 1.5 2 the problem of measuring patients experience in hospitals is 1 − (x−ψ) P (U = 1) = e 2σ2 (5) considered. −∞ 2πσ

Z ∞ 2 1 − (x−ψ) P U e 2σ2 (6) III.SUBJECT ANSWERSASA RANDOM VARIABLE ( = 5) = 4.5 2πσ Assuming that a test is conducted using five point discrete Note that the parameters estimated from O and U can result scale the subject answer U has a multinomial distribution given in different ψ and σ values since the support of O is any real by: number and the support of U is set {1, 2, ··· , 5}. Let us mark 5 X by ψo and σo the parameters of O distribution which are our P (U = s) = ps, where ps = 1 (1) input parameters and by ψu and σu the mean and standard s=1 deviation of random variable U. For a given input pair (ψo, σo) Such description of a subject answers distribution is general the comparison with the obtained (ψu, σu) is crucial, since we but has four different parameters, coming from five probabil- interpret the input parameters but the real process will behave ities. according to the output parameters. For the ideal model those 3

5 4 Obtained Discrete Correct Continuous 4 3 ) u U

3 ( ψ 2 V

2 1

1 0 1 2 3 4 5 1 2 3 4 5 ψo ψ

Fig. 1: The difference between ψo (the true quality for contin- Fig. 3: The limitation of V(U) as a function of ψ. Green uous model) and ψu (the true quality for discrete model) for surface shows the limitation for discrete distribution. Red 2 σ = 0.1. The straight line represents an ideal model. The red surface is limitation of σu for the continuous model if the 2 line is the obtained value. discrete distribution limitations are used as an input σo.

5 Obtained in terms of variance, parameters are: 2 Correct 1) P (U = 1) = 0.9,P (U = 2) = 0.1 with σ = 0.09 2) P (U = 1) = 0.975,P (U = 5) = 0.025 with σ2 = 0.39. 4 On the other hand, if ψ = 3 the two most extreme distributions are: 1) P (U = 3) = 1 with σ2 = 0.0 u 3 2 ψ 2) P (U = 1) = 0.5,P (U = 5) = 0.5 with σ = 4. The above examples show that the limited support of the discrete distribution limits the possible value of σ in a 2 complicated way. The full dependency is shown in Fig. 3 and marked by the green surface. More details about the variance limitation are in [8]3. 1 In order to use the continuous model we typically limit 1 2 3 4 5 the parameters of the model to the range of the discrete parameters. This solution works well for ψ , which can be ψo o limited to [1, 5] interval and so the ψu will be limited in 2 Fig. 2: The difference between ψo (the true quality for contin- the same, correct interval. Nevertheless, if we limit σo to the uous model) and ψu (the true quality for discrete model) for interval defined by the green surface in Fig. 3, the obtained 2 σ = 1. The straight line represents an ideal model. The red σu is given by the red surface. As we can see there is no 2 2 line is the obtained value. match between σo and σu. This is a strong limitation of the continuous model and cannot be easily solved. Therefore, we propose the following discrete distribution. two values should be identical. In Fig. 1 and 2 we compared the relation between ψo and ψu for σo = 0.1 and σo = 1 B. Discrete Distribution respectively. An alternative solution to a continuous model is the discrete Both Fig. 1 and 2 show large differences between ψ and o distribution4. By the discrete distribution we understand the ψ . To understand this discrepancy we have to understand u specific discrete distribution for which a probability of each the dissimilarities of a continuous and discrete process. For answer is given by a function. We are looking for a function continuous process the parameters can have any possible value of two parameters, allowing us to specify true quality ψ and so ψo ∈ (−∞, ∞) and σo ∈ [0, ∞). On the other hand, ψu ∈ [1, 5] and the range of σu depends on ψu. This is very 3Note that Fig. 3 in [8] is presented for standard deviation and not variance. different from the continuous model, where those parameters Therefore, it looks different, but represents the same phenomenon. are independent. This dependency is caused by a finite number 4We use the term discrete distribution (instead of discrete model) purposely. When using a continuous distribution we need to use a model to link it of arguments values (i.e., subject answers) of the discrete to discrete values. However, when modeling discrete values by a discrete distributions. For example, if ψ = 1.1 the two most extreme, distribution we can think of using this particular distribution as a model. 4 the answers’ precision/spread σ. Since the discrete distribution ψ = 1.3 ◦× • parameter describing the distribution variance is not the same ψ = 2.6 ◦ × • as the process variance we mark it by ρ and not σ. ψ = 4.3 ◦ × • According to our knowledge there is no distribution de- scribed in literature which is: ψ = 3.0 ◦ × • 0 1 4 • discrete • supported on finite M element set Legend: ◦ Vmin(ψ), ×VBin(ψ), • Vmax(ψ) • for any fixed mean covers the whole spectrum of possible variances Fig. 4: Visualization of the the interval [Vmin(ψ),Vmax(ψ)] • with no more than a single change in the probabilities for different ψ values. The interval [Vmin(ψ),Vmax(ψ)] is the monotonicity. biggest for ψ = 3 and has the form of [Vmin(3),Vmax(3)] = Therefore, we propose a new distribution with two parameters [0, 4]. The red and green intervals are the possible variances, describing mean ψ and variance ρ. The ρ parameter describes smaller and grater than VBin(ψ) respectively. variance for a given ψ. Since the proposed distribution is discrete the problems shown on Fig. 1 and 2 do not exist. The question is how to obtain from this shifted Binomial The proposed distribution utilizes existing discrete dis- distribution a class of distributions that covers the whole tributions (with limited support), but covers more possible interval of variances [V (ψ),V (ψ)] (see Fig. 4). variances. An example of a discrete distribution which is min max We would like to obtain this class by: described by two parameters is Beta Binomial distribution [26]. Nevertheless, the smallest variance of the Beta Binomial • adding only a single normalized parameter ρ ∈ (0, 1], distribution is the Binomial distribution variance. It is a strong • variance of the error to be linearly dependent on ρ, limitation (in the case of subjective experiments for video) • variance is a decreasing function of ρ. since in [9] it is suggested that the Binomial distribution has In this way, we would consider the ρ parameter a confidence the highest possible variance for a correctly conducted subjec- parameter (see Fig. 5). tive experiment. Therefore, we need a different distribution. Let us denote by Hρ the distribution of the error fulfilling 1) GSD Definition: Let us start with the equation describing the above conditions. Since the variance of the error is linearly the answers U for a particular PVS dependent on ρ and decreasing then it has to be equal to U ψ , = + (7) VHρ () = ρVmin(ψ) + (1 − ρ)Vmax(ψ). (9) where  is an error with mean value equals to 0. Since U Using formulas (8) and (9) we can easily calculate belongs to the set {1, 2, ..., 5} then the distribution of  has V ψ to be supported on the set 1 − ψ, 2 − ψ, ..., 5 − ψ. Let us 3 max( ) VHρ () = VBin(ψ) ⇒ ρ = C(ψ) := , consider shifted Binomial distribution for : 4 Vmax(ψ) − Vmin(ψ)  4  ψ − 1k−1 5 − ψ 5−k which gives us the value of ρ corresponding to shifted binomial P ( = k − ψ) = , k − 1 4 4 distribution. We have where k ∈ {1, ..., 5} is an user answer. VHρ () ∈ [Vmin(ψ),VBin(ψ)] ⇔ ρ ∈ [C(ψ), 1], Since the support of this distribution and the mean value are which corresponds to the red coloured interval in Fig. 4, and fixed, we obtain fixed shifted Binomial distribution without any freedom. However, we would like to have a class of VHρ () ∈ [VBin(ψ),Vmax(ψ)] ⇔ ρ ∈ [0,C(ψ)], distributions for  with all possible variances. Let us think about how the set of all possible variances, for all distributions which corresponds to the green coloured interval in Fig. 4. supported on the set 1 − ψ, 2 − ψ, ..., 5 − ψ looks like. For variances bigger than VBin(ψ) (the green coloured Remember that the mean values for such distributions are fixed interval in Fig. 4) we use the Beta Binomial distribution. In to 0. This is why the set of all possible variances depends on ψ. other words, we replace success probability parameter by a random variable with the Beta distribution. Since the mean If we denote by Vmin(ψ),Vmax(ψ) the minimal and maximal possible variance respectively, then value is fixed, we only have one free parameter. It can be reparameterized to the interval (0,C(ψ)). The effect of such V (ψ) = (dψe − ψ)(ψ − bψc), min reparameterization gives us the distribution denoted by Gρ: Vmax(ψ) = (ψ − 1)(5 − ψ),  4  PGρ ( = k − ψ) = × and the interval [Vmin(ψ),Vmax(ψ)] is the set of all possible k − 1   variances. Notice that the interval [Vmin(ψ),Vmax(ψ)] is the ρ(ψ−1) (5−ψ)ρ (10) B 4(C(ψ)−ρ) + k − 1, 4(C(ψ)−ρ) + 5 − k biggest for ψ = 3 which is [Vmin(3),Vmax(3)] = [0, 4], and ,  (ψ−1)ρ (5−ψ)ρ  if ψ is not an integer then Vmin(ψ) > 0. Let us return to the B 4(C(ψ)−ρ) , 4(C(ψ)−ρ) shifted Binomial distribution. It is easy to calculate that its variance is equal to: where ρ ∈ (0,C(ψ)) and k ∈ {1, ..., 5}. Remark 1: Notice that for ρ → 0 the distribution Gρ goes to Vmax(ψ) VBin(ψ) := . (8) a two points distribution supported on {1−ψ, 5−ψ} with the 4 5

biggest possible variance equal to Vmax(ψ). For ρ → C(ψ) 4 ψ = 2.8 or ψ = 3.2 the distribution Gρ goes to the shifted Binomial distribution ψ = 2.1 or ψ = 3.9 with variance equal to VBin(ψ). ψ = 1.7 or ψ = 4.3 For variances smaller than VBin(ψ) (the red coloured inter- 3 val in Fig. 4) we use a mixture technique. Specifically, we ψ = 1.3 or ψ = 4.7 take a mixture of the shifted Binomial distribution with the ψ = 1.1 or ψ = 4.9 ) distribution with the smallest possible variance (two points or  ( 2 one point distribution, depending on ψ). Of course the mixture V parameter has to be reparameterized to the interval [C(ψ), 1]. The effect of such reparameterization gives us the distribution 1 denoted by Fρ:

PFρ ( = k − ψ) = ρ − C(ψ) 0 [1 − |k − ψ|]++ 1 − C(ψ) (11) 0 0.2 0.4 0.6 0.8 1 1 − ρ  4  ψ − 1k−1 5 − ψ 5−k ρ , 1 − C(ψ) k − 1 4 4 Fig. 5: Variances of the error term  where ρ ∈ [C(ψ), 1], [x]+ = max(x, 0) and k ∈ {1, ..., 5}. Remark 2: Notice that for ρ → C(ψ) the distribution Fρ goes to the shifted Binomial distribution with variance equal θ = (ψ, σ) for the continuous model. Let (ui) be a vector to VBin(ψ). For ρ → 1 the distribution Fρ goes to a two points of iid samples from U. or one point distribution (depending on ψ) with the smallest An MLE is defined as: possible variance equal to Vmin(ψ). θˆ = arg max `(u|θ), (12) Finally, we obtain the desired GSD: θ

Hρ = Gρ I(ρ < C(ψ)) + Fρ I(ρ ≥ C(ψ)), where X `(u|θ) = ln Pθ(U = ui) (13) where ρ ∈ (0, 1] is a confidence parameter and I(x) is one if i x is true or 0 if x is false. I(x) can be seen as an if function. The variance in that distribution fulfills equation (9). is the likelihood function of distribution U. Since there is no ` θ In order to describe subjective scores correctly and com- closed form for a maximum of ( ), a numerical optimization pletely, a process covering the whole spectrum of variances is used to maximize the equation (13). The objective func- tion (13) is maximized using RMSProp, a first order gradient- is needed. Our discrete distribution Hρ addresses this claim. Furthermore, we believe the proposed distribution can be based optimizer. Optimization stopping criteria are useful for describing other, more general processes. |∇`(θ)| ≤  From this moment we will assume the following Assumption 1: or the number of steps reached some fixed value – whatever is first. U = ψ + , Since the paper proposes a new distribution, we have to where ψ ∈ [1, 5] is an unknown parameter and  is a random implement the likelihood function from scratch. The logarithm variable with distribution functions Hρ, where ρ ∈ (0, 1] is an of (11) and (10) is implemented using low-level TensorFlow unknown parameter. ops (i.e., operations), which allows us to take advantage of Examples of the U distributions are shown in Fig. 6. Note efficient implementations of special function and automatic that for ψ close to 1 or 5, regardless of ρ, the obtained differentiation for gradient-based optimizers provided by the distributions are similar. It comes from the fact, that the library. Details about the implementation are in Appendix A. maximum spread we can obtain, is limited by a small range of possible variances (see Fig. 3). For ψ closer to 3 the obtained A. Numerical Experiments distributions are clearly different. In order to validate our estimation procedures we drawn data from the proposed distribution and estimate the distribution IV. GSD PARAMETERS ESTIMATION parameters from the sampled data. The simulation considers: Continuous model and discrete distribution described in the • a number of subjects N = {6, 12, 24, 48}, previous sections cannot be used without an accurate and • ψ from 1.05 to 4.95 (with 23 different values), efficient parameter estimation procedure. The most simple • ρ from 0.01 to 0.99 (with 23 different values). approach is to use method of moments. However, in the numer- For each triple (ψ, ρ, and N) the simulation is repeated 30 ical experiments, we use a much better Maximum Likelihood times. Estimator (MLE). The distribution of U is parameterized by For 32 cases we obtain a fast convergence (less than 1,000 a pair θ = (ψ, ρ) in the case of discrete distribution or steps). For 6 points the estimated parameters are clear outliers. 6

1 0.95 1 0.95 0.88 0.88 0.8 ψ = 1.30 0.81 0.8 ψ = 2.10 0.81 0.72 0.72 ) 0.61 ) 0.61 s 0.6 s 0.6 = 0.38 = 0.38 ij ij U U ( 0.4 ( 0.4 P P

0.2 0.2

0 0 1 2 3 4 5 1 2 3 4 5 Score s Score s

1 0.95 1 0.95 0.88 0.88 0.8 ψ = 2.85 0.81 0.8 0.81 ψ = 3.90 0.72 0.72 ) 0.61 ) s 0.6 s 0.6 0.61 = 0.38 = 0.38 ij ij U U ( 0.4 ( 0.4 P P

0.2 0.2

0 0 1 2 3 4 5 1 2 3 4 5 Score s Score s Fig. 6: Distributions of U under Assumption 1 for different ψ and ρ.

We assume those are just numerical errors and for experiment data they are easy to avoid by estimating the parameters more 5 than one time and choosing the result with lower likelihood value. After removing the 6 numerically unstable points we obtain 4 a high accuracy (see Fig. 7 and 8). The true quality ψ is typically in the range ±0.25 and ρ is in the range ±0.1. This ˆ is including simulations with only 6 answers. ψ 3 To better analyze the obtained accuracy Fig. 9 shows the accuracy measured by ψ − ψˆ for different number of subjects. As expected, using a higher number of subjects increases 2 accuracy. We can also see clearly the estimation closer to the edges is less scattered. Another conclusion is the confidence interval for ψ should be close to ±0.2 for N = 6 and ±0.1 1 for N = 24. Of course, more formal analysis is needed to obtain correct confidence interval estimation. This problem is 1 2 3 4 5 left as future work. ψ Similarly to Fig. 9 for ψ we present Fig. 10 for the ρ Fig. 7: Comparison between ψ as a simulation parameter and parameter. ρ describes variance and therefore the number of the estimated value ψˆ for all simulations. subjects influences the obtained results more significantly. For a small number of subjects a strong asymmetry of the results 7

1 viewing environment (in accordance with Recommendation ITU-R BT.500-11) with various display technologies. The display resolution is constant (1920x1080). There are 24 0.8 subjects per each experiment, all of whom produce scores for 168 Processed Video Sequences (PVSs). All experiments 0.6 use two sets of PVSs: specific and common. The common set is shown in all experiments. It contain 24 PVSs. The ˆ ρ remaining 144 PVSs are used in the specific set and do not 0.4 repeat across experiments. Those 144 PVSs are created by applying 15 degradation conditions (i.e. Hypothetical Ref- 0.2 erence Circuits, HRCs) to 9 pristine sequences (i.e. Source Reference Sequences, SRCs). All videos are 10 seconds long 0 and contain various coding and transmission artifacts. The distortions are meant to simulate a digital transmission of 0 0.2 0.4 0.6 0.8 1 video over a communication channel. The variation of the ρ ACR method is used, namely Absolute Category Rating with Hidden Reference (ACR-HR). Fig. 8: Comparison between ρ as a simulation parameter and the estimated value ρˆ for all simulations. B. ITS4S Designed for training no-reference (NR) metrics, the ITS4S data set contains 813 unique video sequences (without audio). is clearly visible. It is a consequence of the limitations of the It characterizes a generic adaptive streaming system for high parameter and the correct estimation process, which cannot definition mobile devices. All sequences are four seconds long, obtain a parameter out if its range. For a large number of no sequence is repeated in a single experiment, the data set subjects it is nearly invisible. Based on the results presented in emphasizes original footage (i.e. as created by professional Fig. 10 the confidence interval for N = 6 is close to 0.2, which producers), 35% of sequences contain no compression, the is 20% of the whole range, for N = 24 the confidence interval remaining 65% contain only simple compression artifacts. is closer to 0.1. Again the formal analysis of the confidence All sequences are presented in 720p resolution (1280x720) interval is left as future work. and frame rate equals to 24. However, some sequences are The implementation of the estimation procedure encoded with lower resolution (e.g. 512x288) and have to be is provided at https://colab.research.google.com/drive/ up-scaled to fit the display resolution (i.e. 720p). 1ioM4JqUaEA8fHJH9V-0iphHt4C9zmI0i. The code allows The subjective scores included in the data set come from to estimate parameters from a sample in a general form of two subjective tests performed by two laboratories in two M possible categories (see Appendix B). Also a simulation different countries. One laboratory uses subjects hired through and visualization of the GSD is provided. a temporary hiring agency. The other collects data using 24 engineering students. V. DATA SETS C. MM2 We use four data sets to check practical distribution of subjects’ answers. Those data sets come from four subjective An audiovisual test performs in 10 different environments tests: (i) VQEG HDTV Phase I [27], (ii) ITS4S [28], (iii) (controlled and public). We treat scores coming from dif- AGH/NTIA [29], [30] and (iv) MM2 [31]. This gives a total of ferent environments as independent. Thus, we assume the 71,212 observations. All the tests utilize the Absolute Category data represents 10 independent experiments. Both the stimuli Rating (ACR) method (in accordance with Recommendation (640x480@30fps audiovisual sequences) and scale are held ITU-T P.910). Additionally, all subjective scores are given constant between the experiments. The stimuli represents a on the following five-level scale: (1) Bad, (2) Poor, (3) Fair, wide range of quality, degradation are introduced solely be (4) Good and (5) Excellent. All four data sets are publicly compression. Importantly, the encoding levels of audio and accessible on CDVL (www.cdvl.org)5. If available, data from video streams are chosen to match one of the three encoding training sessions can be used as well. We do it to make the qualities: (i) high, (ii) medium and (iii) low. analysis as general as possible. Following subsections describe Depending on the experiment: 9 to 34 subjects are used, each data set. subjects spanned all age groups, some experiments use over- lapping pools of subjects. Nevertheless, due to the lack of precise subject labeling, we assume all subjects are indepen- A. VQEG HDTV Phase I dent. VQEG HDTV Phase I is a classical data set containing scores from 6 video-only experiments using a controlled D. AGH/NTIA AGH/NTIA is a video-only subjective test designed to 5Search keywords for all four data sets (in the same order as listed in the text): “vqeghdN” (replace N with a number from 1 to 6 to find videos from examine three issues: (i) the behavior of subjects when re- all 6 experiments), “its4s”, “AGH/NTIA” and “mm2”. peatedly rating the same stimuli, (ii) the suitability of subjects 8

0.2 0.2 N = 6 N = 12

0.1 0.1 ˆ ψ

− 0 0 ψ

−0.1 −0.1

−0.2 −0.2 1 2 3 4 5 1 2 3 4 5 0.2 0.2 N = 24 N = 48

0.1 0.1 ˆ ψ

− 0 0 ψ

−0.1 −0.1

−0.2 −0.2 1 2 3 4 5 1 2 3 4 5 ψ ψ

Fig. 9: Comparison between ψ and the estimation error calculated as ψ − ψˆ for different number of subjects N.

screening techniques and (iii) the influence of source sequence we have discrete data the most natural testing tool is Pearson’s reuse on subjects scores. Depending on the stimuli, there are 3 χ2 test, also called goodness-of-fit test (see [32]). to 6 ratings from the same subject. Scores from 29 subjects are In Section V we describe four different databases. Here we available. One of those subjects is a video quality expert, who are interested in a single PVS (video sequence) analysis, we knows the purpose of the experiment. He tries to replicate his are not taking into account from which database specific PVS previous score for any repeating PVS. In addition, two subjects comes from. The key parameter is a number of PVSs. There perform the test based on intentionally incorrect instructions. are 1,879 PVSs with at least 24 answers each (some PVSs This is done to test subjects’ screening methods. have 213 answers). We have five PVSs with all 24 answers being one. We remove them from the further analysis since for VI.SUBJECTIVE DATA ANALYSIS the continuous model we obtain NaNs. This results in 1,874 In the previous sections we propose the new distribution PVSs available for the analysis. For each PVS we estimate and prove that the estimation method extracts data from a the GSD (or continuous model) parameters and, finally, calcu- simulated sample correctly. In this section we validate if the lated p-value of the goodness-of-fit test. We show results for proposed distribution can be used to model real subjective three cases: (i) the GSD, (ii) the quantized continuous model data. In addition we compare the GSD with the continuous (labelled QNormal) and (iii) the continuous model with its model. parameters estimated by sample mean and standard deviation In order to validate if the distribution fits specific data we (labelled Normal). For each case we obtain 1,874 p-values. have to perform a two-steps procedure. The first step is to Fig. 12 compares p-values distributions for all three cases. estimate the distribution parameters from a sample. The second We use p-values (from multiple hypothesis tests) to test a step is to test a null hypothesis that the sample comes from general hypothesis. The procedure is as follows: for a single estimated distribution. Fig. 11 visualizes the procedure. Since PVS we obtain p-value a, if a > α one can assume that this 9

0.15 0.15 N = 6 N = 12 0.1 0.1

5 · 10−2 5 · 10−2 ˆ ρ − ρ 0 0

−5 · 10−2 −5 · 10−2

−0.1 −0.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0.15 0.15 N = 24 N = 48 0.1 0.1

5 · 10−2 5 · 10−2 ˆ ρ − ρ 0 0

−5 · 10−2 −5 · 10−2

−0.1 −0.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 ρ ρ

Fig. 10: Comparison between ρ and the estimation error calculated as ρ − ρˆ for different number of subjects N.

PVS distribution comes from a distribution, for example GSD. The p-values histogram for the GSD in Fig. 12 shows an This result is true for this particular PVS, but we are interested increasing probability of a given p-value as a function of p- in a general PVS distribution. Thus, we have to run this test for value. As described above, it shows a strong evidence that a large number of PVSs, obtaining a large number of p-values the global hypothesis is true. The probability of observing p- ai. If we repeat this hypothesis testing we know that some of value smaller than 0.05 is 0.035, which is in line with the the tests fails by chance. The exact number depends on the test theory. However, for both continuous models we obtain p- difficulty. If a test is “difficult” it fails with a higher probability value smaller than 0.05 with probability 0.073 for QNormal than an “easier” test. Hypothesis testing is constructed in such and 0.106 for the Normal model. Moreover, for Normal and a way that even for “difficult” cases failure probability does not QNormal in Fig. 12 we can see that probability for the smallest exceed α. Therefore, for PVSs following a specific distribution p-value is clearly higher than for slightly larger p-values. In (i.e., the one assumed in the null hypothesis) we should have: both cases the observed answers are not coming from those distributions. P (Ai < α) ≤ α, (14) We run the following test to formally prove the global hypothesis where A is a random variable describing the obtained p-values i H : P (A < α) ≤ α and α is a significance level. 0 This can be satisfied by various p-values distributions. versus Obviously, if all p-values are close to 1, equation (14) is true. H1 : P (A < α) > α, If Ai has uniform distribution, (14) is also true. Nevertheless, for any increase, beyond what is allowed by the uniform which can be interpreted as a test of H0: the model is distribution, in the probability for small p-values, results in correct versus H1: the model is incorrect. If we denote by rejecting the global hypothesis about the specific distribution. pGSD, pQNormal and pNormal the p-values of the above test for 10

Subjective data ui 400

300 GSD or GSD Normal model 200

Observed Parameters 100 frequences Os (ψ, ρ) or (ψ, σ) 0 Teoretical 400 probabilities ps

300 χ2 test, QNormal goodness of fit 200 count

100 p-value

Fig. 11: Algorithm used to test goodness-of-fit. The pipeline 0 starts at the blue “Subjective data“ block. Its output is a p-value 400 (green box), which we use to verify the null hypothesis, that a sample comes from the assumed distribution (either GSD or 300

the normal model) Normal

200 GSD, QNormal and Normal models respectively, we obtain for α = 0.05: 100 pGSD = 0.997252727, 0 pQNormal = 0.000005612, 0.00 0.25 0.50 0.75 1.00 pNormal = 0.000000000 p-value It clearly shows that for QNormal and Normal models too many p-values are below 0.05 to say those models are prob- able. Fig. 12: p-value obtained for different distributions. The red The above result is very important from the further usage line marks the 0.05 significance level. of the GSD in the context of video quality subjective testing perspective. We demonstrate that subjective answers from different subjective tests are following the GSD distribution. Much more interesting is the ρ distribution. This parameter It means that GSD is the correct way of modeling subjective is limited to (0,1] interval. We can see that Normal distribution scores in video quality experiments. We strongly believe that approximates it with a high accuracy. We do not perform a GSD can be used in different cases of experiments with formal test since we know it would be rejected by the high answers on a Likert scale. probability of ones. The analysis of QQ plot shows that normal distribution with values higher than one truncated to one, with A. A Priori Distributions parameters µρ = 0.86 and σρ = 0.071, can be used to simulate Since we have an evidence that the GSD is the correct subjective scores. way of modeling subjective scores it is important for further analysis to describe the typical parameters of this distribution. VII.CONCLUSION We have two parameters: ψ (see Fig. 13) and ρ (see. Fig. 14). In this paper we propose Generalized Score Distribution The first parameter depends on the experiment setup. In (GSD), which is a general form of a discrete distribution with: a typical situation its distribution (defined by many PVSs finite support, two parameters, and not more than one change utilized in the experiment) should be close to uniform dis- in the probability monotonicity. The distribution parameters tribution. What we can see is a lack of very high qualities are: ψ determining the mean, and ρ determining the spread (close to 5). Another observation is a greater number of high of the answers. We focus on the subjective experiments for quality PVSs than lower quality ones. We theorize it is just videos, where subjects score the quality of video sequences an artifact of the type of database we use. on a five point scale. We analyze 1,874 video sequences from 11

ACKNOWLEDGMENT 150 The authors would like to thank Netflix, Inc. for sponsoring this research. This work was supported by the Polish Ministry of Science and Higher Education with the subvention funds 100 of the Faculty of Computer Science, Electronics and Telecom- munications of AGH University. count 50 REFERENCES [1] T. Hoßfeld, P. E. Heegaard, L. Skorin-Kapov, and M. Varela, “No silver 0 bullet: Qoe metrics, qoe fairness, and user diversity in the context of 1 2 3 4 5 qoe management,” in 2017 Ninth International Conference on Quality ˆ of Multimedia Experience (QoMEX), May 2017, pp. 1–6. ψ [2] International Telecommunication Union, “ITU-T Recommendation P.1401: Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models,” Fig. 13: Distribution of ψˆ for all 1,874 PVSs. Tech. Rep., 2012. [Online]. Available: https://www.itu.int/rec/T-REC-P. 1401-201207-I/en [3] K. Brunnström and M. Barkowsky, “Statistical quality of experience analysis: on planning the sample size and statistical 6 significance testing,” Journal of Electronic Imaging, vol. 27, no. 05, p. 1, 2018. [Online]. Available: https://www.spiedigitallibrary. org/journals/journal-of-electronic-imaging/volume-27/issue-05/053013/ Statistical-quality-of-experience-analysis--on-planning-the-sample/10. 4 1117/1.JEI.27.5.053013.full [4] T. Hossfeld, P. E. Heegaard, M. Varela, and S. Möller, “Formal Definition of QoE Metrics,” pp. 1–23, 2016. [Online]. Available: http://arxiv.org/abs/1607.00321{%}0Ahttp://dx.doi.

density 2 org/10.1007/s41233-016-0002-1 [5] L. Janowski and Z. Papir, “Modeling subjective tests of quality of ex- perience with a generalized linear model,” 2009 International Workshop on Quality of Multimedia Experience, QoMEx 2009, pp. 35–40, 2009. 0 [6] M. Seufert, “Fundamental advantages of considering quality of experi- ence distributions over mean opinion scores,” in 2019 Eleventh Interna- 0.6 0.8 1.0 tional Conference on Quality of Multimedia Experience (QoMEX), June ρˆ 2019, pp. 1–6. [7] M. Fiedler, T. Hossfeld, and P. Tran-Gia, “A generic quantitative relationship between quality of experience and quality of service,” IEEE Network, vol. 24, no. 2, pp. 36–41, 2010. Fig. 14: Distribution of ρˆ for all 1,874 PVSs. Red line shows [8] T. Hoßfeld, R. Schatz, and S. Egger, “SOS: The MOS is not enough!” 2011 3rd International Workshop on Quality of Multimedia Experience, the normal distribution fitted to the obtained data. QoMEX 2011, pp. 131–136, 2011. [9] T. Hossfeld, P. E. Heegaard, M. Varela, and L. Skorin-Kapov, “Confidence Interval Estimators for MOS Values,” 2018. [Online]. Available: http://arxiv.org/abs/1806.01126 publicly available databases, and show with probability of [10] L. Janowski and M. Pinson, “The accuracy of subjects in a quality exper- 99.7%, that subjective answers from those databases are drawn iment: A theoretical subject model,” IEEE Transactions on Multimedia, vol. 17, no. 12, pp. 2210–2224, Dec 2015. from the GSD distribution. This result opens a new way of [11] Z. Li and C. G. Bampis, “Recover Subjective Quality Scores from Noisy analyzing and simulating subjective scores. Measurements,” Data Compression Conference Proceedings, vol. Part The advantage of GSD is that ρ parameter can be used to F127767, pp. 52–61, 2017. [12] A. Kumcu, K. Bombeke, L. Platiša, L. Jovanov, J. Van Looy, and determine the type of the underlining process. With ρ close to W. Philips, “Performance of four subjective video quality assessment 1 we know that the process is similar to Bernoulli distribution, protocols and impact of different rating preprocessing and analysis for ρ > C(ψ) we know it is Beta Binomial distribution. This methods,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 1, pp. 48–63, Feb 2017. information can be used as a diagnostic information: what is [13] J. LI and P. L. CALLET, “Improving the discriminability of standard the spread of the answers. Note that GSD can be easily used subjective quality assessment methods: a case study,” in 2018 Tenth In- outside of the video quality domain where information about ternational Conference on Quality of Multimedia Experience (QoMEX), May 2018, pp. 1–3. the answer spread is important. [14] P. G. Freitas, A. F. Silva, J. A. Redi, and M. C. Q. Farias, “Performance In the future research we would like to generalize the GSD analysis of a video quality ruler methodology for subjective quality distribution by adding more parameters similar to [10], [11]. assessment,” Journal of Electronic Imaging, vol. 27, no. 5, pp. 1 – 10 – 10, 2018. [Online]. Available: https://doi.org/10.1117/1.JEI.27.5.053020 Another interesting direction of research would be using GSD [15] H. Wang, I. Katsavounidis, X. Zhang, C. Yang, and C.-C. J. Kuo, “A user for data other than subjective experiments for video. We would model for jnd-based video quality assessment: theory and applications,” like collaborate with scientist working in different fields: from 2018. [Online]. Available: https://doi.org/10.1117/12.2320813 [16] S. Tasaka, “Bayesian hierarchical regression models for qoe estimation audio and image quality to psychology and sociology. In all and prediction in audiovisual communications,” IEEE Transactions on those fields, a proper modeling of answering process would Multimedia, vol. 19, no. 6, pp. 1195–1208, June 2017. help to gain new insights. Our results obtained for video [17] K. D. Singh, Y. Hadjadj-Aoul, and G. Rubino, “Quality of experience estimation for adaptive http/tcp video streaming using h.264/avc,” in quality subjective testing might be used as a proof of concept 2012 IEEE Consumer Communications and Networking Conference for those other domains. (CCNC), Jan 2012, pp. 127–131. 12

[18] R. Romano, P. B. Brockhoff, M. Hersleth, O. Tomic, and T. Næs, `B(ui|θ) is a logarithm of PGρ ( = ui − ψ) given by (10) and “Correcting for different use of the scale and the need for further `Bin is the likelihood function for Binomial distribution analysis of individual differences in sensory analysis,” Food Quality and Preference, vol. 19, no. 2, pp. 197 – 209, 2008, 8th Sensometrics  4  Meeting. `Bin(u, θ) = log + (u − 1)(log(ψ − 1) − log(4))+ [19] O. Tomic, A. Nilsen, M. Martens, and T. Næs, “Visualization of sensory u − 1 profiling data for performance monitoring,” LWT - Food Science and (5 − u)(log(5 − ψ) − log(4)). Technology, vol. 40, no. 2, pp. 262 – 269, 2007. [20] B. Maniscalco and H. Lau, “A signal detection theoretic approach Notice that the function f is the only place where we do for estimating metacognitive sensitivity from confidence ratings,” Con- sciousness and Cognition, vol. 21, pp. 422–430, 2012. the computations in the probability domain (instead of log [21] S. Fleming and N. D. Daw, “Self-evaluation of decision-making: A gen- probability), however it is build around log(1 + x) having eral bayesian framework for metacognitive computation,” Psychological numerically stable implementation in function log1p. Review, vol. 124, pp. 91–114, 01 2017. [22] J.-R. King and S. Dehaene, “A model of subjective report and objective The logarithm of the newton symbol presents in both `B discrimination as categorical decisions in a vast representational space,” and `Bin can also be efficiently implemented as Philosophical Transactions of The Royal Society B Biological Sciences, vol. 369, p. 20130204, 03 2014. n [23] D. F. Alwin, E. M. Baumgartner, and B. A. Beattie, “Number of response log = log Γ(n+1)−log Γ(k+1)−log Γ(n−k+1), categories and reliability in attitude measurement,” Journal of Survey k Statistics and Methodology, vol. 6, pp. 212–239, 2018. where log Γ is a logarithm of the gamma function, provided [24] D. L. Malott, B. R. Fulton, D. Rigamonti, and S. Myers, “Psychometric testing of a measure of patient experience in Saudi Arabia and the United as a special function lgamma in TensorFlow. Notice that this Arab Emirates,” Journal of Survey Statistics and Methodology, vol. 5, component is commonly omitted from the likelihood because pp. 398–408, 2017. it does not depend on θ. [25] L. Janowski, J. Nawała, W. Robitza, Z. Li, L. Krasula, and K. Rusek, “Notation for subject answer analysis,” 2019. [26] K. R. Coombes, The Beta-Binomial Distribution. [Online]. APPENDIX B Available: https://cran.r-project.org/web/packages/TailRank/vignettes/ GENERALIZATION betabinomial.pdf [27] M. Pinson, F. Speranza, M. Barkowski, V. Baroncini, R. Bitto, S. Borer, The described distribution is aiming {1, 2, ··· , 5} Y. Dhondt, R. Green, L. Janowski, T. Kawano et al., “Report on the scale/support. Here we present a generalization for validation of video quality models for high definition video content,” Video Quality Experts Group, 2010. {1, 2, ··· ,M} scale. [28] M. H. Pinson, “Its4s: A video quality dataset with four-second unre- Let us denote peated scenes,” NTIA/ITS, Tech. Rep. NTIA Technical Memo TM-18- 532, Feb. 2018. Vmin(ψ) = (dψe − ψ)(ψ − bψc), [29] L. Janowski and M. Pinson, “Subject bias: Introducing a theoretical user model,” in 2014 Sixth International Workshop on Quality of Multimedia Vmax(ψ) = (ψ − 1)(M − ψ), Experience (QoMEX), Sep. 2014, pp. 251–256. [30] M. H. Pinson and L. Janowski, “Agh/ntia: A video quality subjective M − 2 V (ψ) C(ψ) = max . test with repeated sequences,” NTIA/ITS, Tech. Rep. NTIA Technical M − 1 V (ψ) − V (ψ) Memo TM-14-505, June 2014. max min [31] M. H. Pinson, L. Janowski, R. Pepion, Q. Huynh-Thu, C. Schmidmer, Let P. Corriveau, A. Younkin, P. Le Callet, M. Barkowsky, and W. Ingram, “The influence of subjects and environment on audiovisual subjective PFρ ( = k − ψ) = tests: An international study,” IEEE Journal of Selected Topics in Signal ρ − C(ψ) 1 − ρ Processing, vol. 6, no. 6, pp. 640–651, Oct 2012. [1 − |k − ψ|]+ + [32] R. E. Walpole, R. H. Myers, S. L. Myers, and K. Ye, Probability & 1 − C(ψ) 1 − C(ψ) (15) statistics for engineers and scientists, 8th ed. Upper Saddle River: M − 1  ψ − 1 k−1 M − ψ M−k Pearson Education, 2007. , k − 1 M − 1 M − 1 where ρ ∈ [C(ψ), 1], k = 1, ..., M and APPENDIX A M − 1 ESTIMATION PG ( = k − ψ) = ρ k − 1  (ψ−1)ρ (M−ψ)ρ  Numerically optimized likelihood function B (M−1)(C(ψ)−ρ) + k − 1, (M−1)(C(ψ)−ρ) + M − k ,  (ψ−1)ρ (M−ψ)ρ  B (M−1)(C(ψ)−ρ) , (M−1)(C(ψ)−ρ) ( (16) X `B(ui|θ), ρ ≥ C(ψ) `(u|θ) = ` u |θ f u , θ , ρ < C ψ , i Bin( i ) + ( i ) ( ) where ρ ∈ (0,C(ψ)), k = 1, ..., M. If we denote by Hρ the distribution function of the noise where then we assume that

Hρ = Gρ I(ρ < C(ψ)) + Fρ I(ρ ≥ C(ψ)), f(ui, θ) = log(1 − ρ) − log(1 − C(ψ))+ where ρ ∈ (0, 1] is a confidence parameter (see Remark 3). (ρ − C(ψ))[1 − |u − ψ|] log(1 + i + ), Remark 3: The variance of the noise is equal to  ui−1  5−ui 4
ψ−1 5−ψ (1 − ρ) ui−1 4 4 VHρ () = ρVmin(ψ) + (1 − ρ)Vmax(ψ). 13

Since the variance of the noise is a decreasing function of ρ ∈ (0, 1] then this parameter has an interpretation as a confidence parameter. From this moment we assume the following Assumption 2: U = ψ + , where ψ ∈ [1,M] is an unknown parameter,  are independent random variables with distribution functions Hρ, where ρ ∈ (0, 1] is an unknown parameter.