The ‘E’ in QWERTY: Musical Expression with Old Computer Interfaces Chris Nash Department of Computer Science and Creative Technologies, UWE Bristol Frenchay Campus, Coldharbour Lane, Bristol, BS16 1QY, UK [email protected] ABSTRACT symbolic actions, such as Ctrl-C for Copy, or Ctrl-O for Open). The This paper presents a development of the ubiquitous computer keyboard’s QWERTY layout enables the development of computing keyboard to capture velocity and other continuous musical properties, virtuosity, in the form of motor skill and image schemata [12], as in order to support more expressive interaction with music software. seen in touch typing, and similar to the learning mechanisms that Building on existing ‘virtual piano’ utilities, the device is designed to allow performers more expressive control of musical instruments. [5] provide a richer mechanism for note entry within predominantly non- Indeed, the widespread development of such skill among computer realtime editing tasks, in applications where keyboard interaction is a users has engendered a design and layout that is resistant to change, central component of the user experience (score editors, sequencers, and thus we can expect the QWERTY keyboard to be a fixture in DAWs, trackers, live coding), and in which users draw on virtuosities studios and workstations for the foreseeable future. However, in both music and computing. there is significant scope for adapting and extending the design In the keyboard, additional hardware combines existing scan code within the constraints of existing interaction styles to improve (key press) data with accelerometer readings to create a secondary expressiveness and control in musical scenarios. USB device, using the same cable but visible to software as a By comparison, musical (MIDI) keyboards are accurate and separate USB MIDI device aside existing USB HID functionality. expressive, but optimized for specialist musical, rather than general- This paper presents and evaluates an initial prototype, developed purpose use. Supporting from 25 to 88 discreet, unlabeled, linearly using an Arduino board and inexpensive sensors, and discusses arranged piano keys; the MIDI keyboard’s design and ergonomics design considerations and test findings in musical applications, limit its utility as a computer control device, for tasks other than drawing on user studies of keyboard-mediated music interaction. simple assignable triggers (though [4] offers a fascinating study of Without challenging more established (and expensive) performance the piano as a chording keyboard for text-entry). The MIDI keyboard devices; significant benefits are demonstrated in notation-mediated is designed for expression and creativity rather than efficiency and interaction, where the user’s focus rests with software. productivity, which is notably enabled through sensitivity to velocity and pressure (MIDI Velocity and Aftertouch) [1], and enhanced by passive haptic feedback from fully- or semi-weighted keys. In MIDI, Author Keywords pressure is captured at 7-bit resolution (0 to 127), effectively adding a keyboard input, digital music notation, virtuosity, computer music ‘continuous’ scale of control to the triggering of discrete pitches, significantly increasing the nuance of expressive input, albeit ACM Classification conditional on a concomitant increase in virtuosity. • Human-centered computing~Keyboards • Applied computing~ The goal of this project is to similarly extend the discrete symbolic Sound and music computing • Hardware~Sensors and actuators control offered by the computer keyboard with support for continuous scales of musical control, such as pressure and velocity; 1. INTRODUCTION increasing the expressiveness of the keyboard without compromising In digital music, computer and musical keyboards support different its existing utility, functionality, or ergonomics. Accordingly, the aspects of composition and production, which can partition the envisaged user experience is characterized by fluid mixed-mode creative process, between creativity and productivity phases. [11] interaction with the device and software: symbolic input and The embodied interaction style supported by instruments (including keyboard shortcuts in computer modes, alternating with expressive MIDI devices) encourages musicians to ideate, compose, and musical input in performance mode. Specifically; rather than perform music away from the computer, before capturing (recording) realtime, live musical performance, this research focuses on it in a realtime performance. In the computer, the sequencer/DAW supporting richer input modes and interaction styles for “offline” then draws on keyboard and mouse interaction to edit and refine the notation and music editing, such as those in score editing, musical data, mediated through a visual medium (UI or notation). sequencing, and DAW software. However, generic computer input styles (keyboard, mouse, WIMP, etc.) are not optimized for fluid and expressive music interaction, splitting and shifting focus between the software environment (notation) and a separate hardware input device (instrument). The computer keyboard is powerful and precise with respect to symbolic input, but lacks musical fidelity. Data entry is processed as enumerated scan codes, triggered by roughly 100 discrete keys, corresponding to alphanumeric symbols or functions, with modifiers (Shift, Ctrl, etc.) to expand control to additional layers of symbols or functions. Both data entry and program control are designed in sympathy with written language (e.g. mnemonic shortcuts for non- Figure 1. A Velocity-sensitive Computer Keyboard. 224 The following sections discuss the background, applications, and This virtuosity, often likened to “musical touch-typing”, is widely development of a prototype device that uses an accelerometer to evident in videos1 of tracker users (as studied in [8] and [9]), extend a standard computer keyboard with velocity-sensitivity, as but nonetheless highlights areas for improved ergonomics and well as other dimensions and properties of musical input. Findings expressiveness. For example, the keyboard’s lack of velocity control from tests and user evaluations are presented, and discussed in the makes auditioning and entering melodic passages limited and context of relevant computer music interaction scenarios. cumbersome, requiring manual entry of dynamics. While the virtual piano supports live playback and “key jamming”, the fixed velocity 2. BACKGROUND offers limited expression, yielding a mechanical performance and The virtual piano is a common utility provided in music software, or hindering the ideation of new musical ideas – consequently, available as a standalone virtual MIDI device (e.g. VMPK). A piano influencing the aesthetic of users with less musical experience. Since keyboard layout is tacitly overlaid on the computer keyboard, dynamics can significantly affect timbre, users may also commit to allowing pitch to be triggered at a fixed velocity. A degree of instrumentations prematurely, not knowing the range of the voice. polyphony is supported, but due to shared resources in keyboard While this paper principally explores velocity for musical construction, some key combinations are not possible (the precise applications, the additional degree of freedom and analogue input limitations vary between keyboard models). Mouse interaction is offered by continuous-scale pressure readings have other uses, as often also supported, exploiting 2D visualisations to input both pitch explored in other research. Inspired by music keyboards, Microsoft (key) and velocity (vertical offset of key click), but cumbersome and Research previously developed a prototype pressure-sensitive typing imprecise for rapid, accurate entry, even in short phrases. Virtual keyboard, conceived for use in general-purpose computing [2] – pianos are thus designed for playing simple melodies, phrases, and though applications, in gaming (varying walking speed) and instant monophonic pitch selection, and are designed for convenience in messaging (mapping ‘emotion’ to font size), are only briefly simple tasks (without leaving the computer) or where specialist input discussed and, curiously, no consideration is made of musical use. hardware is not available (e.g. mobile settings, amateur setups). Their design replaces the standard keyboard’s conductive membrane While the mouse is an important tool for novice users and with a custom-manufactured carbon ink printed layer that captures unavoidable in some GUI designs, its single point of focus limits the variable pressure, output as modulated voltage (rather than simply expressive bandwidth and precision of control. In the visual UIs of closing a circuit). The paper does not detail how the pressures / most professional music editing packages (sequencers, musical voltages are integrated with symbolic input or exposed in software or typesetters, DAWs), expert users migrate to keyboard shortcuts and the operating system, nor detail the relative cost and complexity of bimodal keyboard-and-mouse interaction styles, for more efficient components and manufacturing. However, the intricacy of the control of the program and direct manipulation styles of editing (e.g. design contrasts the relatively simple, low-cost, and accessible drag and drop). Such literacy with keyboard shortcuts represents a accelerometer augmentation taken in the following section. form of computer virtuosity, enabled through repeated practice and learning, significantly improving