Affective Evaluation: Dragalia Lost Alexa Tarrayo, 301225457 Gordon Huang, 301247073 Joelle Lei, 301274701

Introduction

Dragalia Lost is an action role playing game (RPG) and is the creation of a joint development team of Nintendo and Cygames, available on IOS and Android. Accompanied by cute, Japanese style visuals and catchy music, Dragalia Lost makes for an exciting experience at first glance. Dragalia Lost features an array of different aspects that are all included in the experience of playing the game, including a combat mode, a gatcha (summoning random members to the party) aspect, castle upgrading, and training and upgrading the members of your party. However, the primary focus of the evaluation team aims to discover users’ reaction to the first fifteen minutes of the game. As mobile gaming is so saturated that most merely download, try, and delete, the evaluation team looks to investigate how captivating Dragalia Lost is by letting users play through the tutorial. The tutorial features mostly combat and story as the game’s primary “pull” and therefore, the following report will focus on this portion of the game. The objective of the evaluation team aims to discover how people would interact with the interface to accomplish the desired goals. In addition, the evaluation team would also provide design recommendations at the end of the report.

Methodology

Participants Participant A Participant B Participant C Participant D

Age 21 25 18 23

Occupation Student Administrator Student Photographer

Familiarity Playing 5 1 5 3 Mobile Games (1-5 Rating, 5 is familiar)

Gaming on Other PC Desktop None PlayStation 4 Nintendo Consoles PC Desktop Switch (Check all) Nintendo Switch

Heard of Dragalia No No Yes No

The evaluation team chose participants with varying degrees of experience with mobile games, and gaming in general. Participants were also made sure of to be new players to Dragalia Lost, despite maybe seeing it via promotion of my word of mouth. This provides context for the evaluation team about a user’s potential skill but does not give any prior experience to Dragalia Lost and the tutorial experience.

Method

Due to the nature of mobile gaming, the evaluation team conducted a series of four studies with each participant. Three out of the four participants conducted the study in a comfortable place in their home, while the remaining fourth participant did their test in an area they found to be where mobile

gaming would suit them -- a study lounge at their school. The evaluation process begins with making sure the participant reads and signs the Informed Consent Form. After, a demographics questionnaire is given to provide insight on the participant’s familiarity with mobile gaming, as well as any other background that may assist the evaluation team in understanding the user’s reaction and gameplay. The user is then told about Dragalia Lost briefly, and begins the tutorial, playing for about fifteen minutes. Users were encouraged to play further if they completed the tutorial prior to the fifteen minutes ending. During the participant’s playtime, one of the evaluators records the process from the first-person perspective. The evaluator is only allowed to assist when the participant is truly stuck. After the participants are done with the playthrough, they were immediately asked to answer the IMI questionnaires that asked the participants to rate their thoughts and experiences when they were playing the tutorial. Shortly after the IMI questionnaires took place, a Cued Recall Debrief session is conducted. The participant would then watch their own playthrough under monitor of the evaluator, expressing their thoughts, emotions, and feedback toward their own actions, as well as the gameplay during different stages of the game. The participants were encouraged to talk about their experiences and thoughts freely. If the participants do not know when to speak, the debriefer would guide the participants by asking questions.

Data Collection

Data collection was as follows -- pre-test, the actual test, and then post-test. As mentioned prior, the before testing each participant, a demographics questionnaire was provided. This enabled the evaluating team some background knowledge about each participant. During the test, data was collected using first-person video recording, where a camera was held over the participant’s shoulder to stimulate that view. Data was also collected from the participant’s IMI questionnaire. The IMI questionnaire assesses the participants’ interest/enjoyment, perceived competence, perceived choice, and pressure and tension while playing through the tutorial of Dragalia Lost. After, during the Cued Recall to Debrief session, the participant’s reflection was recorded, their face and their previous playthrough on screen. Two other evaluators would then conduct a coding of the debrief session, highlighting the player’s commentary and reaction in relation to positives, neutral, and negative statements. To prep the data for analyzation, the evaluators came together and summarize and combine the code together and calculate the inter-rater reliability values to indicate how similar the results from the two evaluators are.

Results

Participant A: Participant A also had a significant amount of negative comments at 11 - 12. Participant ​ A had positive comments at 6 and neutral at around 3 - 6. The data collected from both coders had a correlation of 0.94. From Participant A’s IMI score, his Interest/Enjoyment was at an average of 5.0 with 1.0 standard deviation. Perceived Competence was at an average of 6.8 with 0.4 standard deviations. Perceived choice with 5.2 average rating and a 0.4 standard deviation. Pressure/Tension at a 0.6 rating and 0.5 standard deviations. Based on the score, there is a significant difference between the average score of his Interest/Enjoyment at 5.0 and his Pressure/Tension of 0.6 average rating.

From the results, the evaluation team can determine that Participant A enjoys most part of the playthrough due to his higher interest and enjoyment and low pressure and tension. However, by taking Participant A’s cued recall debrief into considerations, there are some problems that Participant A faced during his playthrough. Although some of his negative comments were connected to one another, it is still important to take those comments into considerations when determining Participant A’s overall experience with the game.

Participant B: Out of all the participants, Participant B had the most negative comments determined by ​ the two coders ranging from 12 - 14. In addition, her positive comments were around 3 - 6 and neutral comments were at 8 from both coders. The data collected from both coders had a correlation of 0.94. From Participant B’s IMI evaluation, her Interest/Enjoyment had an average of 3.0 with standard deviation at approximately 1.5. Her Perceived Competence scored an average of 5.2 with the standard deviation at 1.1. Perceived Choice had a 2.4 average and a 1.1 standard deviation. Her pressure/tension had a 2.0 average and a 1.2 standard deviation. From the IMI scores, we can see that her average score for Interest/Enjoyment (3.0) does not have a significant difference towards her pressure/tension score which is at an average of 2.0. Although from Participant B’s IMI scores, her average interest/enjoyment is slightly higher than her pressure/tensions, her scores in the cued recall debrief demonstrates that the game did not provide a positive experience based on the results of her negative comments were significantly higher than her positive comments from both coders.

Participant C: For both coders, Participant C’s negative comments range from 7 - 11. In addition, his ​ positive comments were at 6 and neutral comments were around 3 - 5. The data collected from both coders had a correlation of 0.99. From Participant C’s IMI evaluation, his Interest/Enjoyment had an average of 6.7 with 0.8 standard deviations. His Perceived Competence scored an average of 6.2 and 1.1 standard deviations. Perceived Choice had an average of 6 with 0.7 standard deviations. Pressure/Tension had an average of 2.2 and 1.8 standard deviations. From Participant C’s IMI score, we can see that Participant C’s score for Interest/Enjoyment, Perceived Competence, and Perceived Choice ranges from 6 - 6.7 which are significantly higher than his score on pressure/tension which had an average of 2.2. Looking at Participant C’s IMI score, we can see that Participant C’s overall experience with the game was positive with high enjoyment and interest. Although, from the scores that both coders interpreted based on Participant C’s cued recall debrief sessions, Participant C’s score on negative comments was slightly higher than his positive comments. The scores do not provide a significant difference to determine that Participant C had a completely negative impression on the game.

Participant D: Participant D’s had the most positive comments ranging from 6 - 9. In addition, his ​ negative comments were at 4 and neutral comments were around 5 - 7. The data collected from both coders had a correlation of 0.99. From Participant D’s IMI score, his Interest/Enjoyment score an average of 4.7 with 1.5 standard deviations. Perceived Competence had an average of 5.6 with 1.3 standard deviations. Perceived Choice had an average of 4.0 with 1.9 standard deviations. Pressure/Tension had an average of 2.2 with 0.8 standard deviations. Based on the score provided, Participant D scores a range of 4.0 - 5.6 with

Interest/Enjoyment, Perceived Competence, and Perceived while Pressure/Tension score significantly lower at an average of 2.2. This data led the team to determine that Participant D had an enjoyable experience with the game due to low pressure/tension and high perceived competence and enjoyment. In addition, the data collected from the cued recall debrief data also suggest that Participant D had more positive comments as opposed to his negative comments.

Discussion and Implications

Positive - Polished visuals help users feel immersed: Participant A states, “the graphics are really nice ​ for a free game,” impressed. Dragalia Lost boasts two large Japanese video-game “supers” -- Nintendo, and Cygames. Their collaboration really shows in the game’s visuals. Dragalia Lost is accompanied by uniform and beautiful character designs, making each playable character and summonable dragon a thrill to collect and level up. Even Participant B, who had the most negative comments coding wise, said, “the [battle] animations were cute,” and “hitting the minions was fun.” This is due to the game’s high fidelity graphics, and art.

Positive - Easy to understand gameplay controls: Participants enjoyed the simplicity of the controls. ​ Participant C summarized their thoughts well when he stated, “they’re smooth and all you had to do was ​ tap and swipe,” furthermore stating that “it’s pretty much what you do in any other app, tap and swipe.” Dragalia Lost made it a point to utilize the mobile phone, making the game playable in portrait mode with one hand. The easy and familiar to use controls made for an easy entry point in keeping players open-minded in playing the rest of the game.

Negative - Heavy, dragged out dialogue: Participants loathed the dialogue in ​ the game. While the dialogue and story narrative were received well in terms of story, with Participant A commenting, “the words seem like comic-y, like an ​ anime, so it’s okay,” the length of the dialogue was lamented. All participants expressed issues with the dialogue, and felt it made them grow bored, and thus ruined their gameplay, and willingness to move forward.

Recommendation: The team suggests that Dragalia Lost implement a way to ​ fix the source of the dragged out dialogue. The team observed that although user taps the screen to advance forward, the dialogue will not change until the voice acting line is finished. This is also based on Participant D’s desire, “I wish that ​ when you tap it wouldn’t just skip the text it would skip the spoken dialogue as well,” further saying, “I didn’t want to press the skip button and miss the entire screen.” This is ​ ​ inconvenient for players that want to enjoy the story, but are fast readers that digest text easily. Therefore, our team suggests a feature where the user can turn off the voice acting to advance forward at their leisure, or for the voice acting to simply fade away when they tap the screen to prompt the next text box.

Negative - Onboarding is repetitive and unclear: Many of the participants found the non-skippable ​ tutorial very boring. Participant C acknowledged the tutorial, saying, “The beginning I could get through ​

it, but after awhile it was unbearable,” later growing frustrated with how the game dragged out the tutorial. This led to a disconnect between what the players wanted, which was more gameplay and less exposition. The tutorial also seemed to be “building up” to something but left the players in the dark as to what they should care about. Participant B discussed this issue when she reached the menu screen (right) with everything locked, saying, “what are the things that the game is trying to guide me to do” and later being annoyed with the tutorial prompts, “why do i need a team? Why do i need to summon? What is the point?”

Recommendation: After taking a closer look at Participant A’s comments, ​ in particular with him saying, “I just want to play the game and not look at this other stuff,” rehauling the game’s onboarding process to feature more gameplay and less forcible direction. The team suggests that the game would benefit from integrating quick, pop-up style tutorial prompts that are weaved in as one plays the game at their own leisure. Due to the game’s story element, a user will have to ​ ​ progress through the story anyway. There is no need for such a long tutorial before letting the player experience the rest of the game’s mechanics. This enables users to find their own connection to the game faster, thus keeping a retained interest.

Negative - Clunky, unresponsive controls: Although the ​ participants enjoyed the game’s simple controls, the execution of said controls were not of the same fidelity as the rest of the game. Most participants wished the easy swipe and tap would work perfectly, but after playing for a few minutes, found flaws, with Participant B stating, “I thought it was just ​ swiping to move,” thinking she needed to do more when she was correct -- all a player needs to do is swipe to move.

Recommendation: An optional, onscreen, joystick like ​ mechanic (featured right, from the game Dragons Nest Mobile) would be helpful for new players still perfecting the controls. An on-screen controller would help new players get used to the control style even further, as well as direct them to learn more advanced techniques, like comboing.

Conclusion

Dragalia Lost is an engaging game, compete with captivating visuals, art, and music. The game does its best to be friendly toward new players by introducing easy to use swipe and tap controls. However, the game tries to put the user into an overwhelming environment too quickly, and thus deters users away from staying connected to the game. With some adjustments to the tutorial to allow more freedom and playful exploration, as well as the addition of more visual controller feedback, Dragalia Lost could accomplish its goal of appealing to new, and veen unfamiliar with mobile, players.

Appendix https://vimeo.com/304127637

Participant A IMI Scoring

Cued Recall Debrief Coding 1

Cued Recall Debrief Cording 2

Interrater Reliability

Participant B IMI Scoring

Cued Recall Debrief Coding 1

Cued Recall Debrief Coding 2

Interrater Reliability

Participant C IMI Scoring

Cued Recall Debrief Coding 1

Cued Recall Debrief Coding 2

Interrater Reliability

Participant D IMI Scoring

Cued Recall Debrief Coding 1

Cued Recall Debrief Coding 2

Interrater Reliability

Consent Forms

References

Dragalia Lost (Version 1.1.2) [Video game]. (2018, October 28). Retrieved November 30, 2018, from

https://itunes.apple.com/us/app/dragalia-lost/id1352230941?mt=8. Screenshots by author. ​

Dragon’s Nest M (Version 1.3.1) [Video game]. (2018, September 6). Retrieved November 30, 2018, from

https://play.google.com/store/apps/details?id=com.dragonnestm.ggplay.koramgame.global&hl=e

n. Screenshots by author. ​