AUTOMATIC EDITION OF SONGS FOR /FRETS ON FIRE

Ana M. Barbancho, Isabel Barbancho, Lorenzo J. Tardon´ Cristina Urdiales Dep. Ingenier´ıa de Comunicaciones Dep. Tecnolog´ıa Electronica´ Universidad de Malaga´ Universidad de Malaga´ 29071 Malaga,´ Spain 29071 Malaga,´ Spain

ABSTRACT the lower music tones. Also, it has to be considered that the button displayed on the left hand side of the screen (green In this contribution, an automatic system of song edition for button) is the most used in both games, however, the button at Guitar Hero/Frets on Fire is presented. The system performs the far right (magenta button) is the less used, actually, it only three fundamental stages: time analysis, frequency analysis appears in the higher difficulty levels of the game. and button allocation. The temporal analysis of the musical signal is used to obtain the rhythmic pattern of the song. The frequency analysis is perform on the basis of a Mel filter bank because these filters provide an approximation to the human perception of low and high frequency sounds. The allocation of buttons is done using the rhythmic pattern extracted, the Mel filters outputs and, also, taking into account a selectable difficulty level. The distributions of the buttons obtained with the designed system are similar to the ones found in the pre- a) b) determined songs of the games. Index Terms— Guitar game, rhythm pattern, Mel-filters. Fig. 1. a) Guitar Hero, b) Frets on Fire.

In both, Guitar Hero and Frets on Fire, the player selects 1. INTRODUCTION a song from a predetermined list, without any possibility of including his favorite songs even having the audio file of the Guitar Hero is a popular , developed by Red Oc- particular song and the rights to use them. To solve this prob- tane, in which a player uses a guitar-shaped controller with lem, Frets on Fire includes a song editor. This is not enough colored buttons to simulate rock music playing. More than 14 because the usage of the song editor turns out to be very bor- million copies have been sold of this game to date [1]. Frets ing, difficult and it needs a certain level of music knowledge on Fire [2] is an open source clone of Guitar Hero for PC that to obtain an acceptable result. The aim of this paper is to de- uses the keyboard as controller. This kind of games is so in- velop an automatic song editor for Frets on Fire/Guitar Hero teresting that specific controllers has been developed for blind so that the designed system generate from the audio file of any players [3]. song, all the necessary files to play the song in these games Principal screens of Guitar Hero and Frets on Fire are rep- with different difficulty levels. resented in Fig. 1. The goal in these games is the following: The outline of the paper is as follows: in section 2, the the player must press the correct combination of colored but- proposed system is presented, then the three main parts are tons synchronously with the music. The sequence of buttons described in the subsequent subsections devoted to time anal- is strongly related to the perception of tone and to the rhythm ysis (section 2.1), frequency analysis (section 2.2) and button of the music. These guitar games include several difficulty allocation (section 2.3). In section 3, some results are shown 3 4 levels that allow to adjust the number of buttons used ( , and, finally, some conclusions are drawn in section 4. or 5), the number of buttons played simultaneously and the speed at which the buttons have to be pressed. 2. PROPOSED SYSTEM The particular time in which a certain button has to be pressed is determined by the rhythm of the song. The button Frets on Fire requires five files to be available to play each to be pressed will depend, mainly, on the tessitura: the buttons song. These files are: on the right hand side of the screen will be assigned to the • label.png:Thisfile includes the song cover. higher music tones and the buttons on the left hand side to • song.ini:Thistextfile contains the song title, the artist This work was supported by the Ministerio de Educacion´ y Ciencia of name, the cassette color and the scores. the Spanish Government under Project No. TSI2007-61181. • song.ogg: This is the audio file of the song.

978-1-4244-4291-1/09/$25.00 ©2009 IEEE 1186 ICME 2009

Authorized licensed use limited to: Universidad de Malaga. Downloaded on September 25, 2009 at 08:37 from IEEE Xplore. Restrictions apply. • guitar.ogg:Thisfile includes guitar sounds. • Energy peaks separated by less than 0.1s, are removed. • notes.mid:Thisfile contains the information of the se- This is equivalent to restrict the highest beat rate to 600 quence of buttons to be pressed and for how long the beats per minute (bpm). buttons have to be pressed. • The duration of each beat is the smaller time between: the duration of time between two consecutive energy The core of the game is the sequence of buttons to press peaks and the elapsed time between the detection of a such that it is perceived as synchronized with the music. The peak and the window in which the measured energy is system proposed in this paper calculates, automatically and under a certain threshold relative to the peak energy. without any user interaction, the sequence of buttons to play Using these conditions, guitar rests are also taken into from the audio file of any song. account.

INPUT Fig. 3 shows an example of the waveform, measured en- song.wav ergy and peaks selected from the initial passage of the Beat- les’ song ’Love Me Do’, together with its score. In this fig- ure, it can be observed how the selected beats match with the instants in which the signal amplitude increases significantly Time Frequency Analysis Analysis and with the line of percussion shown in the score of the song.

2.2. Frequency analysis The selection of the buttons to press is determined by the tes- Button Difficulty Allocation Level situra, as mentioned in previous sections. Therefore, in order to decide the buttons to push in the selected time instants, it is necessary to perform a frequency analysis of the song. OUTPUT In order to take into consideration the nonlinear perception notes.mid of frequency by the human auditory system, the Mel scale is Fig. 2. Block Diagram. used [5]. The Mel scale is mostly linear for low frequencies (< 500Hz) and logarithmic for high frequencies. Uniformly The block diagram of the proposed system is shown in spaced frequency bands are defined in the Mel frequency axis Fig. 2. According to this diagram, we consider three dif- to obtain the subjective information about the perception of ferent stages to solve the problem of generating a sequence low and high frequency sound. of buttons synchronized with the audio file: time analysis, to Now, an appropriate frequency resolution for the calcula- perform the rhythm analysis of the song (section 2.1), fre- tion of the power spectrum, before the application of the Mel quency analysis, to obtain the tessitura (section 2.2) and but- filters, must be selected. In our application, the signal is di- ton allocation, to create the button sequence from the rhythm, vided into windows of 46ms without overlap, and the power tessitura and difficulty level (section 2.3). In the following spectrum is calculated in each of them. subsections, these three stages are considered. In the next step, a bank of Mel filters is applied to the power spectrum [6]. We have considered a bank of 10 Mel 2.1. Time analysis filters ranging from 100Hz to 1KHz. The outputs of the five The objective of the temporal analysis of the audio signal is to even filters have been chosen, thus there is no frequency over- obtain its rhythmic pattern. The rhythmic pattern will deter- lap between the filters’ outputs. Each of these outputs will be mine the particular time instants at which the different buttons related to a specific button in the game. The frequency range have to be pressed and how long they have to be pressed. The analyzed has been chosen taking into account that, in pop and rhythm of a musical passage of a song is straightly related to rock music, most of the musical and rhythmic information the points where a significant energy peak appears [4]. are contained in the low frequencies and these are the genres Therefore, in the proposed system, rhythmic patterns are usually played in these applications. Also, note that, this fre- obtained finding significant energy peaks in the audio signal quency range is not directly related to the guitar frequencies. in the time domain. Each peak detected corresponds to a mu- Fig. 4 a) shows the raw values obtained directly at the sical beat. The detection process is as follows: selected outputs of the bank of Mel filters applied to the initial • Music signal energy is calculated in windows of 5ms of fragment of the song ’Love me do’ by The Beatles. Note duration, without overlap. In our evaluations, we have that the amplitude of the filters decreases while the number of found that this is an appropriate temporal resolution. the filter increases because the energy is concentrated on the lower frequencies. • Peaks over a certain threshold are detected. These peaks are considered to correspond to music beats. The 2.3. Button allocation threshold is set to the half of the maximum energy The selected outputs of the Mel filters will be employed to peak. identify the buttons to press. Each button will be related to

1187

Authorized licensed use limited to: Universidad de Malaga. Downloaded on September 25, 2009 at 08:37 from IEEE Xplore. Restrictions apply. Fig. 3. Top) Score of the initial passage of the Beatles’ song ’Love me do’. Middle) Waveform of the initial passage shown in the score. Button) Measured energy and peaks selected.

a) modifying the number of buttons or selecting polyphonic or 1000 Filter 1 monophonic output. The game levels defined are as follows: 800 Filter 2 Filter 3 a) 600 Filter 4 8 Filter 5 Filter 1 400 6 Filter 2 Filter 3 200

Mel filter outputs Filter 4 4 Filter 5 0 0 1 2 3 4 5 6 7 time(s) 2 b) Normalized Mel filter outputs 0 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Filter 1 time(s) 8 Filter 2 b) Filter 3 6 Filter 4 Filter 5 Button 1 4 Button 2 Button 3 2 Button 4 0 Button 5 0 1 2 3 4 5 6 7 Normalized Mel filter outputs time(s)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Fig. 4. Output of the five selected Mel filters for the initial time(s) fragment of the song ’Love me do’. a) Unprocessed. b) Nor- malized. Fig. 5. a) Normalized outputs of the five selected Mel filters for the initial passage of the song ’Love me do’. b) Measured signal energy with selected peaks and selected buttons. one filter output: the close-left button is related to the first Mel • Easy: 3 buttons, monophonic. filter (the lower frequency range) and the close-right to the last • Mel filter. However, there exists an important difference in the Easy-plus: 5 buttons, monophonic. amplitude of the different outputs of the filters, actually, filter • Medium: 4 buttons, polyphonic. 1 always has the highest amplitude. In order to cope with • Amazing: 5 buttons, polyphonic. this characteristic, the outputs of the filters are normalized by their respective standard deviations and the result of this The procedure implemented for the assignment of buttons procedure can be observed in Fig. 4 b). This normalization is as follows: simplifies the task of button assignment. At each detected • The number of outputs of the Mel filters selected equals beat, the normalized filter outputs are observed and the largest the number of buttons used in each level of the game. one defines a button to press (Fig. 5). However, not only • For each time instant in which a beat has been detected, the largest value is considered, since four difficulty levels are the button assigned to the Mel filter with the largest out- defined. More levels of difficulty can be designed by simply put is selected.

1188

Authorized licensed use limited to: Universidad de Malaga. Downloaded on September 25, 2009 at 08:37 from IEEE Xplore. Restrictions apply. • If a polyphonic level is chosen, the following, in mag- 0.45

nitude, Mel filter output, is used to select an additional 0.4 Frets on Fire button. Proposed System 0.35

a) 0.3 1

0.5 0.25

0 0.2 Amplitude Buttons Distribution −0.5 0.15

−1 0 1 2 3 4 5 6 7 0.1 time(s)

b) 0.05

Button 1 0 Button 1 Button 2 Button 3 Button 4 Button 5 Button 2

Button 3 Fig. 7. Comparison between the distributions of buttons to be

Button 4 pressed for the Amazing level of the son ’Defy the machine’: Button 5 distribution chosen manually in Frets on Fire and distribution

0 1 2 3 4 5 6 7 automatically generated by the proposed system. time(s) Fig. 6. a) Waveform of the initial passage of the Beatles’ song ’Love Me Do’ b) Button allocation for the Easy-plus level. description of the song, frequency analysis, to obtain infor- Vertical lines mark the peaks that define the rhythmic pattern mation about the tessitura of the different passages, and the detected. assignation of the buttons to press at each beat. The detec- tion of rhythm is made on the basis of the overall energy of In Fig. 6, the waveform of the initial fragment of the song the audio signal and a bank of Mel filters is used to obtain a ’Love Me Do’, together with the corresponding buttons se- representation of the evolution of the energy in the different lected for the Easy-plus level is presented. frequency bands through the song. Button assignment is car- Finally, the information about when and which button has ried out depending on the difficulty level selected, using the to be pressed is saved in a midi file with the format used by rhythmic pattern extracted, to determine the right instant to Frets on Fire. press a button and its duration, and the output of Mel filters to select the buttons to press. 3. RESULTS The rates of selection of buttons to press obtained by the Up to now, and to the authors’ knowledge, there is not a sim- system proposed and the selection of buttons included in the ilar system available for comparison purposes, so the evalua- distribution package of the Frets on Fire have been compared tion has been done as follows. The distributions of the buttons and similar results have been obtained. Moreover, subjective selected to be pressed of the songs included in Frets on Fire evaluations by persons used and not-used to this kind of ap- have been compared with the ones obtained by the designed plications have been favorable. system. Fig. 7 represents the distribution of buttons to be 5. REFERENCES pressed for the Amazing level of the song ’Defy the machine’ [1] GuitarHero, ”http://www.guitarhero.com”. of Frets on Fire and the distribution generated by the proposed system. It can be observed that both distributions are similar. [2] Frets on Fire, ”http://fretsonfire.sourceforge.net”. The quality of the results generated with the system de- [3] B. Yuan and E. Folmer,”Blind Hero: Enabling Guitar scribed has been checked in the research laboratory by per- Hero for the Visually Impaired”, in 10th ACM Conf. on sonnel with experience in this type of games, and good com- Computers and Accessibility, 2008, pp. 169-176. ments have been given. The results have also been subjec- [4] S. Dixon, ”Automatic extraction of tempo and beat from tively evaluated by persons not used to this kind of applica- expressive performances”, J. New Music Research,vol. tions and their perception of the rhythm and tessitura gener- 30, pp. 39-58, 2001. ated by the proposed system was similar to the one obtained [5]S.S.StevensandJ.VolkmanandE.B.Newman,”A with songs prepared for Frets on Fire and Guitar Hero sam- scale for the measurement of the psychological magni- ples. tude of pitch”, J. Acoustic Society of America, vol. 8, pp. 4. CONCLUSIONS 185-190, 1937. [6] E. Pampalk, Computational Models of Music Similarity In this paper, a new system for the automatic edition of songs and their Application in Music Information Retrieval, for Frets on Fire has been presented. The system performs Technischen Universitt Wien, 2006. three main different tasks: time analysis, to obtain a rhythmic

1189

Authorized licensed use limited to: Universidad de Malaga. Downloaded on September 25, 2009 at 08:37 from IEEE Xplore. Restrictions apply.