TENETS OF SELECTED PICTUREBOOK SCHOLARSHIP APPLIED TO THE PRACTICE OF THE ADAPTATION OF THE PICTUREBOOK, THE KING HAS GOAT EARS, TO A PICTUREBOOK APP by Cynthia Nugent A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Arts in The Faculty of Graduate and Postdoctoral Studies (Children’s Literature) The University of British Columbia (Vancouver) April, 2016 © Cynthia Nugent, 2016 Abstract This hybrid creative study distills concepts from picturebook scholarship, and writings on audiobooks, sound semiotics, multimodality, and reading as a form of play to create a set of guiding questions for the creation of a story app called The King’s Ears. Three important concepts foreground the inquiry. Firstly, picturebook stories are played multimodally through the combined contributions of words and pictures. Shaping the app’s modes of sound, interaction, and animation to interrelate with the words and pictures, and with each other to tell the story was the central challenge of the app design. The second significant idea that governs this design-based approach is that children read picturebooks differently than adults do. The third principle is that the interactions, sounds, animations, and navigation of the picturebook app should be child-controlled and replayable. From a process of iterative cycles of design, enactment, analysis, and redesign to develop an original media artifact, a framework emerged that can be used to guide the development and assessment of picturebook apps, as well as sharable theories that can inform the work of other designers. ii Preface This dissertation is original, unpublished, independent work by the author, Cynthia Nugent. iii Table of Contents Abstract ................................................................................................................................................. ii Preface.................................................................................................................................................. iii Table of Contents ................................................................................................................................. iv List of Figures ..................................................................................................................................... vii Acknowledgements .............................................................................................................................. ix Dedication ............................................................................................................................................ xi Introduction ......................................................................................................................... 1 1.1 Motivation for the Study ....................................................................................................... 1 1.2 Background ........................................................................................................................... 5 1.2.1 Affordances of the iPad .................................................................................................... 5 1.2.2 Picturebook App Characteristics ....................................................................................... 6 1.3 Purpose of the Study ............................................................................................................. 9 1.4 Significance of the Study .................................................................................................... 11 1.5 Research Questions ............................................................................................................. 12 1.6 Key Terms ........................................................................................................................... 12 Literature Review .............................................................................................................. 19 2.1 How the Modes of Words and Pictures Work Together in Picturebooks ........................... 19 2.2 The Mechanics of Word-Picture Interaction ....................................................................... 22 2.3 How Children Read Picturebooks ....................................................................................... 23 2.4 The Picturebook as Multimodal Text ................................................................................. 25 2.5 Play and Puzzles in the Picturebook ................................................................................... 26 2.6 Animation and Movement in the Picturebook .................................................................... 27 2.7 Sound .................................................................................................................................. 29 2.7.1 Musical Language in the Picturebook ............................................................................. 30 2.7.2 Representing Sound in Picturebook Image, Layout, and Design ................................... 31 2.7.3 Sound in Films and Audiobooks ..................................................................................... 32 2.7.4 Sound in Apps ................................................................................................................. 34 2.8 Interaction in Picturebooks ................................................................................................. 36 iv 2.8.1 Movable Book Interactive Mechanisms Converted to Digital ....................................... 38 2.9 Conclusion .......................................................................................................................... 41 Guiding Design Principles for the Creation of a Picturebook App ................................... 43 3.1 Introduction ......................................................................................................................... 43 3.2 Integrated Modes to Tell Story - Picturebook Theory Applied to the App ........................ 44 3.3 Picturebook Scholarship on How Children Read ............................................................... 55 3.3.1 App Hints ........................................................................................................................ 56 3.3.2 Word Highlighting .......................................................................................................... 58 3.4 Modes Should Be Child-Controlled and Repeatable .......................................................... 59 3.4.1 Repeatable Child-Controlled Animation vs. Embedded Video ...................................... 59 3.5 Conclusion .......................................................................................................................... 60 Adapting The King Has Goat Ears to a Picturebook App ................................................ 63 4.1 About the Codex ................................................................................................................. 63 4.2 Ongoing Research Was Part of the Process ........................................................................ 64 4.2.1 Technical Research ......................................................................................................... 65 4.2.2 Aesthetics vs. File Size ................................................................................................... 66 4.3 Adaptation ........................................................................................................................... 67 4.3.1 Moving from Paper to Screen ......................................................................................... 68 4.3.2 White Screen vs. White Paper ........................................................................................ 69 4.3.3 Finding Sounds ............................................................................................................... 69 4.3.4 Avoiding Text Transience and Redundancy ................................................................... 70 4.3.5 Replacing Text with Sound ............................................................................................. 73 4.4 Designing for Children ....................................................................................................... 74 4.4.1 UI and Navigation ........................................................................................................... 74 Discussion and Future Work ............................................................................................. 78 5.1 Conclusion .......................................................................................................................... 78 5.2 What’s Next? ...................................................................................................................... 80 5.3 Suggestions for Future Research ........................................................................................ 81 5.4 Learning the Hard Way ....................................................................................................... 82 5.5 My Thesis Experience......................................................................................................... 84 Bibliography ........................................................................................................................................86 v Appendices ...........................................................................................................................................93