Affective Evaluation: Dragalia Lost Alexa Tarrayo, 301225457 Gordon Huang, 301247073 Joelle Lei, 301274701 Introduction Dragalia Lost is an action role playing game (RPG) and is the creation of a joint development team of Nintendo and Cygames, available on IOS and Android. Accompanied by cute, Japanese style visuals and catchy music, Dragalia Lost makes for an exciting experience at first glance. Dragalia Lost features an array of different aspects that are all included in the experience of playing the game, including a combat mode, a gatcha (summoning random members to the party) aspect, castle upgrading, and training and upgrading the members of your party. However, the primary focus of the evaluation team aims to discover users’ reaction to the first fifteen minutes of the game. As mobile gaming is so saturated that most merely download, try, and delete, the evaluation team looks to investigate how captivating Dragalia Lost is by letting users play through the tutorial. The tutorial features mostly combat and story as the game’s primary “pull” and therefore, the following report will focus on this portion of the game. The objective of the evaluation team aims to discover how people would interact with the interface to accomplish the desired goals. In addition, the evaluation team would also provide design recommendations at the end of the report. Methodology Participants Participant A Participant B Participant C Participant D Age 21 25 18 23 Occupation Student Administrator Student Photographer Familiarity Playing 5 1 5 3 Mobile Games (1-5 Rating, 5 is familiar) Gaming on Other PC Desktop None PlayStation 4 Nintendo Consoles PC Desktop Switch (Check all) Nintendo Switch Heard of Dragalia No No Yes No The evaluation team chose participants with varying degrees of experience with mobile games, and gaming in general. Participants were also made sure of to be new players to Dragalia Lost, despite maybe seeing it via promotion of my word of mouth. This provides context for the evaluation team about a user’s potential skill but does not give any prior experience to Dragalia Lost and the tutorial experience. Method Due to the nature of mobile gaming, the evaluation team conducted a series of four studies with each participant. Three out of the four participants conducted the study in a comfortable place in their home, while the remaining fourth participant did their test in an area they found to be where mobile gaming would suit them -- a study lounge at their school. The evaluation process begins with making sure the participant reads and signs the Informed Consent Form. After, a demographics questionnaire is given to provide insight on the participant’s familiarity with mobile gaming, as well as any other background that may assist the evaluation team in understanding the user’s reaction and gameplay. The user is then told about Dragalia Lost briefly, and begins the tutorial, playing for about fifteen minutes. Users were encouraged to play further if they completed the tutorial prior to the fifteen minutes ending. During the participant’s playtime, one of the evaluators records the process from the first-person perspective. The evaluator is only allowed to assist when the participant is truly stuck. After the participants are done with the playthrough, they were immediately asked to answer the IMI questionnaires that asked the participants to rate their thoughts and experiences when they were playing the tutorial. Shortly after the IMI questionnaires took place, a Cued Recall Debrief session is conducted. The participant would then watch their own playthrough under monitor of the evaluator, expressing their thoughts, emotions, and feedback toward their own actions, as well as the gameplay during different stages of the game. The participants were encouraged to talk about their experiences and thoughts freely. If the participants do not know when to speak, the debriefer would guide the participants by asking questions. Data Collection Data collection was as follows -- pre-test, the actual test, and then post-test. As mentioned prior, the before testing each participant, a demographics questionnaire was provided. This enabled the evaluating team some background knowledge about each participant. During the test, data was collected using first-person video recording, where a camera was held over the participant’s shoulder to stimulate that view. Data was also collected from the participant’s IMI questionnaire. The IMI questionnaire assesses the participants’ interest/enjoyment, perceived competence, perceived choice, and pressure and tension while playing through the tutorial of Dragalia Lost. After, during the Cued Recall to Debrief session, the participant’s reflection was recorded, their face and their previous playthrough on screen. Two other evaluators would then conduct a coding of the debrief session, highlighting the player’s commentary and reaction in relation to positives, neutral, and negative statements. To prep the data for analyzation, the evaluators came together and summarize and combine the code together and calculate the inter-rater reliability values to indicate how similar the results from the two evaluators are. Results Participant A: Participant A also had a significant amount of negative comments at 11 - 12. Participant ​ A had positive comments at 6 and neutral at around 3 - 6. The data collected from both coders had a correlation of 0.94. From Participant A’s IMI score, his Interest/Enjoyment was at an average of 5.0 with 1.0 standard deviation. Perceived Competence was at an average of 6.8 with 0.4 standard deviations. Perceived choice with 5.2 average rating and a 0.4 standard deviation. Pressure/Tension at a 0.6 rating and 0.5 standard deviations. Based on the score, there is a significant difference between the average score of his Interest/Enjoyment at 5.0 and his Pressure/Tension of 0.6 average rating. From the results, the evaluation team can determine that Participant A enjoys most part of the playthrough due to his higher interest and enjoyment and low pressure and tension. However, by taking Participant A’s cued recall debrief into considerations, there are some problems that Participant A faced during his playthrough. Although some of his negative comments were connected to one another, it is still important to take those comments into considerations when determining Participant A’s overall experience with the game. Participant B: Out of all the participants, Participant B had the most negative comments determined by ​ the two coders ranging from 12 - 14. In addition, her positive comments were around 3 - 6 and neutral comments were at 8 from both coders. The data collected from both coders had a correlation of 0.94. From Participant B’s IMI evaluation, her Interest/Enjoyment had an average of 3.0 with standard deviation at approximately 1.5. Her Perceived Competence scored an average of 5.2 with the standard deviation at 1.1. Perceived Choice had a 2.4 average and a 1.1 standard deviation. Her pressure/tension had a 2.0 average and a 1.2 standard deviation. From the IMI scores, we can see that her average score for Interest/Enjoyment (3.0) does not have a significant difference towards her pressure/tension score which is at an average of 2.0. Although from Participant B’s IMI scores, her average interest/enjoyment is slightly higher than her pressure/tensions, her scores in the cued recall debrief demonstrates that the game did not provide a positive experience based on the results of her negative comments were significantly higher than her positive comments from both coders. Participant C: For both coders, Participant C’s negative comments range from 7 - 11. In addition, his ​ positive comments were at 6 and neutral comments were around 3 - 5. The data collected from both coders had a correlation of 0.99. From Participant C’s IMI evaluation, his Interest/Enjoyment had an average of 6.7 with 0.8 standard deviations. His Perceived Competence scored an average of 6.2 and 1.1 standard deviations. Perceived Choice had an average of 6 with 0.7 standard deviations. Pressure/Tension had an average of 2.2 and 1.8 standard deviations. From Participant C’s IMI score, we can see that Participant C’s score for Interest/Enjoyment, Perceived Competence, and Perceived Choice ranges from 6 - 6.7 which are significantly higher than his score on pressure/tension which had an average of 2.2. Looking at Participant C’s IMI score, we can see that Participant C’s overall experience with the game was positive with high enjoyment and interest. Although, from the scores that both coders interpreted based on Participant C’s cued recall debrief sessions, Participant C’s score on negative comments was slightly higher than his positive comments. The scores do not provide a significant difference to determine that Participant C had a completely negative impression on the game. Participant D: Participant D’s had the most positive comments ranging from 6 - 9. In addition, his ​ negative comments were at 4 and neutral comments were around 5 - 7. The data collected from both coders had a correlation of 0.99. From Participant D’s IMI score, his Interest/Enjoyment score an average of 4.7 with 1.5 standard deviations. Perceived Competence had an average of 5.6 with 1.3 standard deviations. Perceived Choice had an average of 4.0 with 1.9 standard deviations. Pressure/Tension had an average of 2.2 with 0.8 standard deviations. Based on the score provided, Participant D scores a range of 4.0 - 5.6 with Interest/Enjoyment, Perceived Competence, and Perceived while Pressure/Tension score significantly lower at an average of 2.2. This data led the team to determine that Participant D had an enjoyable experience with the game due to low pressure/tension and high perceived competence and enjoyment.