13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system? Has Fandango.com fixed the ‘bug’ in its rating system? Uni Lee 2/12/2021 In 2015, an investigation by a data jounralist named Walt Hicky from FiveThirtyEight (https://fivethirtyeight.com/features/fandango-movies-ratings/) revealved that the movie ratings on Fandango is significantly more left-skewed than those of the competitors. After the report was released, Fandango.com announced that it was caused by a bug in the system and that they will fix it shortly. Have they fixed the bug? If they have not, we can assume that the left-skewedness of movie ratings was intentional.
To answer the research question, we will compare Fandango.com’s movie ratings in 2015 to those in 2016 to determine if there has been any change. Data Walt Hicky has made the data available on Github (https://github.com/fivethirtyeight/data/tree/master/fandango). The 2016 data has also been made available on Github (https://github.com/mircealex/Movie_ratings_2016_17) thanks to the Dataquest team.
# Read in data from the repositories f_2015_raw <- read.csv("data/12_fandango/fandango_score_comparison.csv", sep=";") f_2016_raw <- read.csv("data/12_fandango/movie_ratings_16_17.csv", sep=";")
# Select relevant variables f_2015 <- f_2015_raw %>% select(FILM, starts_with("Fandango")) f_2016 <- f_2016_raw %>% select(movie, year, fandango)
head(f_2015)
## FILM Fandango_Stars Fandango_Ratingvalue ## 1 Avengers: Age of Ultron (2015) 5.0 4.5 ## 2 Cinderella (2015) 5.0 4.5 ## 3 Ant-Man (2015) 5.0 4.5 ## 4 Do You Believe? (2015) 5.0 4.5 ## 5 Hot Tub Time Machine 2 (2015) 3.5 3.0 ## 6 The Water Diviner (2015) 4.5 4.0 ## Fandango_votes Fandango_Difference ## 1 14846 0.5 ## 2 12640 0.5 ## 3 12055 0.5 ## 4 1793 0.5 ## 5 1021 0.5 ## 6 397 0.5
head(f_2016)
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 1/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
## movie year fandango ## 1 10 Cloverfield Lane 2016 3.5 ## 2 13 Hours 2016 4.5 ## 3 A Cure for Wellness 2016 3.0 ## 4 A Dog's Purpose 2017 4.5 ## 5 A Hologram for the King 2016 3.0 ## 6 A Monster Calls 2016 4.0
Sampling error To draw general conclusions about the population (all movies that have been rated by Fandango), we have to randomly select samples from the population. Since we are comparing the ratings in 2015 to 2016, we need two datasets that consist of randomly selected samples of movies released in 2015 and 2016.
However, the datasets we have are not randomly selected samples of the population. The datasets were selected purposefully according to criteria that the data collectors used for the purpose of their analysis. This puts us at the risk of introducing large sampling error.
In order to overcome the limitation of the datasets in our hands, we have to come up with a proxy goal for this analysis. Instead of comparing all movies on Fandango, we will compare ratings of popular movies in 2015 to those of popular movies in 2016. Do the ratings of popular movies in 2015 differ from 2016? To compare the two datasets, we have to make sure that the datasets do the following:
Each dataset contains only the movies that were released in that year. Movies were selected based on the same criteria for popularity.
First, we will create two subsets of movies by the release year.
# Although the 2015 dataset does not have a separate column on release year, it is in cluded in the movie title. We can extract this data using stringr package.
only_2015 <- f_2015 %>% mutate(year=str_sub(FILM, -5, -2)) %>% filter(year==2015) # 1 29 movies
# The 2016 dataset conveniently provides the release year column.
only_2016 <- f_2016 %>% filter(year==2016) # 191 movies
Secondly, we have to make sure that both datasets follow the same criteria for popularity. Unfortunately, the two datasets use different definitions of “popular” movies. Whereas the 2015 dataset contains movies with more than 30 votes, it is unclear how the second dataset selected movies.
# 2015 dataset contains only movies whose votes are over 30. nrow(only_2015 %>% filter(Fandango_votes<30))
## [1] 0
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 2/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
# There is no column for votes in the second dataset names(only_2016)
## [1] "movie" "year" "fandango"
How do we know that the second dataset has enough popular movies to be representative of 2016? We can randomly sample 10 movies from this dataset and check manually to see if they have received more than 30 votes.
set.seed(1) sample_n(only_2016, size=10)
## movie year fandango ## 1 Imperium 2016 4.5 ## 2 The Legend of Tarzan 2016 4.5 ## 3 Sausage Party 2016 3.5 ## 4 The Girl with All the Gifts 2016 4.0 ## 5 Elvis & Nixon 2016 3.5 ## 6 Bad Moms 2016 4.5 ## 7 Free State of Jones 2016 4.0 ## 8 Lights Out 2016 4.0 ## 9 Bleed for This 2016 4.0 ## 10 Morgan 2016 3.5
Change in Fandango’s rating system When I went to Fandango.com to check the votes of the randomly selected movies, I noticed that Fandango no longer uses 5-star rating system. Instead, it uses Audience Scores verified by Rotten Tomatoes.
Audience Score: The percentage of users who made a verified movie ticket purchase and rated this 3.5 stars or higher.
To check movies’ popularity, we can use the number of reviews on Rotten Tomatoes. However, this would still be imperfect because we don’t know how comparable the size of reviewers on Fandango is to that of Rotten Tomatoes. In addition, the number of reviews must have increased since the release date (2016) over time.
Despite the limitations, we can conclude that the second dataset represents popular movies in 2016. Analysis Overview of distribution: Kernel density plots For a general assessment, we will draw two kernel density plots of movie ratings in 2015 and 2016.
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 3/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
# Combine two datasets into one table only_2015_2 <- only_2015 %>% rename("movie" = FILM, "fandango" = Fandango_Stars) %>% select(movie, fandango) %>% mutate(year=2015) ratings_full <- rbind(only_2015_2, only_2016) ratings_full$year <- as.factor(ratings_full$year)
# Plot ggplot(ratings_full, aes(x=fandango, color=year)) + geom_density() + labs(title="Distribution of Fandango movie ratings in 2015 - 2016", x="Star Ratings", y="Density") + xlim(0,5) + theme_classic() + theme(plot.title=element_text(hjust=0.5)) + scale_color_manual(values=c("blue", "red")) + geom_vline(xintercept=mean(only_2015_2$fandango), linetype="dashed", color="blue") + geom_vline(xintercept=mean(only_2016$fandango), linetype="dashed", color="red")
In this graph, we can observe that the mean rating decreased in 2016. There are less frequencies for ratings above 4.5. However, the ratings are still heavily left-skewed. Frequency distribution tables For a more granular assessment, we will create frequency tables for each sample.
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 4/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
# Create absolute and relative frequency distributions for each year freq_2015 <- only_2015_2 %>% group_by(fandango) %>% summarise(Freq=n()) %>% mutate(fr eq_pc_2015=round(Freq/nrow(only_2015_2)*100,2))
## Since the 2015 does not have any 2.5 ratings, we will add that information to the table. missing_rating <- c(2.5, 0, 0) freq_2015 <- rbind(freq_2015, missing_rating) %>% arrange(fandango)
freq_2016 <- only_2016 %>% group_by(fandango) %>% summarise(Freq=n()) %>% mutate(fre q_pc_2016=round(Freq/nrow(only_2016)*100,2))
# Put them into a table for comparison rating <- unique(ratings_full$fandango) %>% sort() freq_pc_2015 <- freq_2015$freq_pc_2015 freq_pc_2016 <- freq_2016$freq_pc_2016
freq_table <- tibble(rating, freq_pc_2015, freq_pc_2016)
# Make the table look pretty gt(freq_table) %>% tab_header( title= "Frequncy distribution of ratings in 2015 and 2016" ) %>% cols_label(rating="Rating", freq_pc_2015="2015 (%)", freq_pc_2016="2016(%)") %>% cols_align("center")
Frequncy distribution of ratings in 2015 and 2016
Rating 2015 (%) 2016(%)
2.5 0.00 3.14
3.0 8.53 7.33
3.5 17.83 24.08
4.0 28.68 40.31
4.5 37.98 24.61
5.0 6.98 0.52
In comparison to 2015, the distribution of ratings in 2016 were more spread across ratings. The most frequent ratings was 4.0, half a star down from 2015. Sample Statistics (mean, median, mode) Although the frequency table was useful, sample statistics may help us get a more accurate sense of the data.
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 5/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
# Since R does not have a built-in function for calculating modes, we will use a cust om function. mode <- function(x) { ux <- unique(x) ux[which.max(tabulate(match(x,ux)))] }
statistics <- ratings_full %>% group_by(year) %>% summarise( mean=mean(fandango), median=median(fandango), mode=mode(fandango))
# Visualize the table ## Prepare the table for graphing statistics_long <- statistics %>% pivot_longer(cols=mean:mode, names_to="statistics", values_to="value")
## Graph ggplot(statistics_long, aes(x=statistics, y=value, fill=year)) + geom_bar(stat="identity", position="dodge", alpha=0.85) + theme_classic() + labs(title="Summary statistics of fandango movie ratings 2015-2016", y="Rating in Stars") + theme(axis.title.x=element_blank()) + scale_fill_manual(values=c("blue", "red"))
From the summary statistics, we can see that the mean ratings decreased by 0.2 starts and the mode decreased by 0.5. However, the median remains the same.
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 6/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system? How does the 2016 ratings compare to other rating sites? How significant are the changes in movie ratings on Fandango.com? To answer this question, we will revisit the full dataset provided for 2016. This dataset provides ratings of other websites that have been normalized to a 0-5 scale to enable meaningful comparison.
# Select movies released in 2016 and relevant variables rating_2016 <- f_2016_raw %>% filter(year==2016) %>% select(movie, year, fandango, nr_metascore, nr_imdb, nr_tmeter, nr_audience)
# Prepare the table for graphing rating_2016_long <- rating_2016 %>% pivot_longer(fandango:nr_audience, names_to="comp any", values_to="rating")
comparison<-ggplot(rating_2016_long, aes(x=rating, fill=company, color=company)) + geom_density(stat="density", alpha=0.3) + scale_fill_manual(name="Company", values=c("red", "blue", "yellow", "purple", "gre y")) + scale_color_manual(values=c("red", "blue", "yellow", "purple", "grey")) + theme_minimal() + labs(title="Comparison to Fandango.com's competitors", subtitle="Normalized ratings distribution of 191 popular movies released in 2016", x= "Rating in Stars", y="Density") + theme(legend.position="none")
# Prepare company labels for each company company <- unique(rating_2016_long$company) label <- c("Fandango", "Metacritic", "IMDB", "Tomatometer", "Rotten Tomatoes User") x_pos <- c(4, 2, 2.75, 0.5, 1.5) y_pos <- c(1.05, 0.4, 1.2, 0.2, 0.25)
label_tb <- tibble(company, x_pos, y_pos, label)
# Add company labels to the graph comparison + geom_text(data=label_tb, x=x_pos, y=y_pos, label=label)
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 7/8 13.2.2021 Has Fandango.com fixed the ‘bug’ in its rating system?
The graph above shows that even after the changes that Fandango has made, its ratings are still inflated in comparison to other websites. Conclusion Our analysis show that Fandango.com has indeed fixed the “bug” in its system, albeit to a limited extent. The average movie ratings decreased by 0.2. The decrease in average is likely due to the fact that many movies that were rated as 4.5 were rounded down to 4.0. Despite the change that Fandango has made, its ratings were still inflated in comparison to other websites in 2016.
file:///Users/UniLee/HelloWorld/DataQuest/dataquest_r/12_fandango.html 8/8