PIANO ARRANGEMENT SYSTEM BASED ON COMPOSERS’ ARRANGEMENT PROCESSES Sho Onuma Masatoshi Hamanaka University of Tsukuba University of Tsukuba [email protected] [email protected] ABSTRACT this analysis, we can determine what the problems are in an arrangement and how to solve them. Our goal is to construct an automatic musical arrangement In this protocol analysis, we recorded the five composers’ system by applying analysis results of composers’ thought process for arranging an orchestral score for piano arrangement processes. We arranged an orchestral score by using an eye mark camera. After the music was arranged, for a solo pianist to evaluate our system. Usually, an we queried the composers on how or why they arranged orchestral score has several instrumental parts, which is their music they way they did. The data from eye mark difficult for a soloist to play. Therefore, an orchestral score camera enabled us to perform detailed protocol analysis needs to be arranged with several instrumental parts into using not only their thinking-aloud processes but also visual both hands parts, which enables a solo pianist to play with points. Using results of the think-aloud protocol analysis, we both hands. However, there is a problem with an constructed a piano arrangement system. There were exactly arrangement of more than five notes for each hand, since the same points that a musician arranged and the system one hand has only five fingers. To solve this problem, we warned, and musical novices were able to choose from introduce a protocol analysis for analyzing the process of several arrangement processes. arranging music. We applied this protocol analysis to a We described our protocol analysis in Section 2, a musical arrangement, and analyzed the arrangement system implementation in Section 3, an example of process. By using the analysis results, we constructed a analysis using the system in Section 4, and we making piano arrangement system for warning the composer of concluding remarks in Section 5. problems in the score that need to be arranged. 2. ANALYZING MUSICAL ARRANGEMENT 1. INTRODUCTION PROCESSES We propose an automated method for arranging an In arranging an orchestral score for piano, we have to pay orchestral for piano, which enables a solo pianist to attention to several problems concerning the use of a comfortably play an orchestral score. musician’s fingers. For example, it is difficult to play two Musical arrangements have several variations. For example, notes that are more than one octave distant with one hand a folk song arranged in a jazz style or rock song arranged with because the fingers cannot stretch that far. Also, the a heavy use of strings. There are also arrangements for number of notes that can be played at the same time is soloists. For example, there are musical arrangements limited due to the number of fingers. In addition, it is originally for multiple-instruments for solo pianists. difficult to cross the hands while playing. Furthermore, it Musical arrangements are divided into two types, those is important to maintain an original piece of music and with many notes and those with fewer notes. Although considered melody repetition because there are repetitions previous research has mostly focused on the former [1] [2], of similar melodies in an orchestral score. For example, we investigated musical arrangements with fewer notes. there are two similar orchestral melodies, “p” and “q”, and We used an original orchestral score, and arranged it for a only “q” has a flute part. When “p” and “q” are arranged solo pianist. This type of arrangement is called a “piano- without the flute part, the arranged melodies “p” and “q” arrange score”, which is widely available. However, a are exactly the same. The arrangement cannot be piano-arrange score is usually written for professional or reproduced from an original piece because there is semi-professional soloists. Therefore, a musical novice essentially no difference between “p” and “q” in the may have difficulty in playing it. arrangement. Repetition and maintenance are complicated We used a think-aloud protocol analysis to examine five problems because of recognized the entire score and in a composers’ arrangement processes [3]. Originally, a think- musical arrangement. However, composers are able to aloud protocol analysis a usability testing technique used arrange music that has these problems. Therefore, we to analyze composers’ thinking-aloud process. Based on analyzed a composer’s arrangement process. 191 2.1. RECORDING COMPOSERS’ ARRANGEMENTS In another example listed in Table 1, the Time was 4:48, the “Events” were that the composer’s thought-aloud. He We recorded five composers’ arrangement process. In said, “It is difficult to play cornet because there are many Figure 1, a piano arrangement of an orchestral score. To notes”. The composer’s thinking-aloud indicates that he record a video of a composer’s visual points, we used an was having difficulty because of the “many notes”, and he eye mark camera. We also recorded each composer’s voice solved this problem by not including the cornet part. when he/she is thinking-aloud by using microphones. Thus, using the results of the think-aloud protocol After the music arranging process, we queried the analysis, we labeled the problems and problem-solving composers on the details of their arrangement processes operations which appear plural times and examined the using the video from the eye mark camera. For example, relations between them. We found that certain problems and we asked composers questions such as. Why did you focus problem-solving operations occurred when composers are in on this section for a long time? What is troubling you now trouble in arrangement process. These problems indicate that about your arrangement? Please tell me if you have any problems occurred during arrangement, and solving the good ideas for the arrangement. problems indicated that the problems were also solved. Orchestra Piano In the third example in Table 1, the time 7:43, and the composer is in trouble because of over-lapping notes between several parts. In Table 1, the problems are described as P5 Arrange (Problem 5) under “Problems”. Then, the composer solved the problem by “Removing a note”. In Table 1, the solution is described as j (problem-solving operation 4) under “Operation”. Eye mark camera 4 Consequently, we found nine problems and six operations in the arrangement process. 2.3. MODELING PROBLEMS AND PLOBLEM- SOLVING OPERATIONS Figure 1: Recording of Musical Arrangement Process In Figure 2, “P” indicates a problem, and “j” indicates a problem-solving operation. We used them, to construct a 2.2. PROTOCOL ANALYSIS state-transition model. The model showed the arrangement process started in P and ended in S (Solution 9). P involves We analyzed the arrangement process by using a think- 0 9 0 inputting composed orchestral melodies, and S involves aloud protocol analysis. Table 1 list the details of protocol 9 outputting piano melodies. P is actually P via j or analysis of a composer arrangement processes. “Time” 0 1-8 1-6 transition to another problem except for S . Several problems denotes the time it took to arrange the music. “Events” 9 are formed simultaneously, but a musical novice solved each denotes important actions during the process, for example, problem one by one, enabling him/her to transition to S . playing a midi keyboard or thinking-aloud. “Arrangement 9 For example, when a pianist’s fingers cannot reach (P ) work” denotes details of arrangement. 7 when melodies are composed, Figure 2 produces three In the first example in Table 1, the Time was 1:48, the problem-solving operations, j . When no problems occur, “Events” were that the composer played the first violin and 4-6 the state-transitions to S and the model output the melody. second violin parts in a midi keyboard. When the composer 9 Based on this model, we constructed a piano arrangement looked for the base line, he said, “Difficult to play”. It would system. In the next section, we describe these various be difficult to play if both viola and cello parts are included. problems, and how they are transitioned. The composer solved this problem by including only the Composed cello part and left out the composer viola part. orchestra melodies Time Events Arrangement Problem Operation PROBLEMS P0 min:sec OPERATION 1:48 play：[viola + cello] P1: Dynamics Playing viola and cello parts j1: Rating melodies [3-4 bars] P : Same melodies 2 P1 j : Unison P 2 Looking for base line (Difficult) 0 P3: Repeated melodies If playing both viola and cello parts j3: Unrepeated (consider left hand) P2 P3 Difficult to play P7 Use cello, remove viola j4 4:48 Think-aloud: P “It is difficult to play PROBLEMS 4 OPERATION cornet, because of P6 P : Close notes P5 P8 many notes.” 4 j4: Remove notes P : Over-lapped notes “Not using cornet.” j4 5 j5: Move notes P6 P7 7:43 Play P6: Many notes up/down octave [1st violin + 2nd violin] Playing 1st violin and 2nd violin P : Unreachable notes 7 j6: Change hand [2 bar] P8: Difficult melodies S9 Last 2 bars, there are over-lapped notes P5 Remove note j 4 Piano melodies Table 1: Example of Think-Aloud Protocol Analysis Figure 2: State-Transition Model 192 2.4. PROBLEMS notes because the pianist’s fingers cannot reach them. P7 is described by the following formulas. The system displays P are transitioned by j . P is where there are noticeable 1-3 1-3 1 the notes in red (Figure 4 P ). dynamics. Dynamics are important in deciding a piece’s 7 ͉ ≤ ͉ < ͉ + ͆and ͊ͨ͗͝ℎ + 13 ≤ ͊ͨ͗͝ℎ ambience.