Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 http://asmp.eurasipjournals.com/content/2014/1/7

RESEARCH Open Access An optical music recognition system for traditional Chinese Kunqu Opera scores written in Gong-Che Notation Gen-Fang Chen1* and Jia-Shing Sheu2

Abstract This paper presents an optical music recognition (OMR) system to process the handwritten musical scores of Kunqu Opera written in Gong-Che Notation (GCN). First, it introduces the background of Kunqu Opera and GCN. Kunqu Opera is one of the oldest forms of musical activity, spanning the sixteenth to eighteenth centuries, and GCN has been the most popular notation for recording musical works in since the seventh century. Many Kunqu Operas that use GCN are available as original manuscripts or photocopies, and transforming these versions into a machine-readable format is a pressing need. The OMR system comprises six stages: image pre-processing, segmentation, feature extraction, symbol recognition, musical semantics, and musical instrument digital interface (MIDI) representation. This paper focuses on the symbol recognition stage and obtains the musical information with Bayesian, genetic algorithm, and K-nearest neighbor classifiers. The experimental results indicate that symbol recognition for Kunqu Opera's handwritten musical scores is effective. This work will help to preserve and popularize Chinese cultural heritage and to store Kunqu Opera scores in a machine-readable format, thereby ensuring the possibility of spreading and performing original Kunqu Opera musical scores. Keywords: Optical music recognition; Kunqu Opera; Gong-Che Notation; MIDI; Classifiers

1. Introduction there are three famous script sets that were popular dur- Kunqu Opera is one of the oldest forms of opera per- ing the Qing Dynasty (1636 to 1911 AD): Jiugong formed in China, dating back to the end of the Yuan Dacheng Nanbei Ci Gongpu (A comprehensive anthol- Dynasty (1271 to 1368 AD) approximately 700 years ogy of texts and notation of the Southern and Northern ago. It dominated Chinese theatre from the sixteenth to opera tunes in nine modes, 1746 AD, with 4,466 musical the eighteenth centuries (see Figure 1) and is distinct in works) [4], Nashuying Qupu (Nashu Studio Theatrical the virtuosity of its rhythmic patterns. It has been a Music, 1792 AD, with more than 360 scripts) [5], dominant influence on more recent forms of local opera and Nanci Dinglv (The Law of the Southern Word, in China and is considered the mother of all Chinese op- 1720 AD, with 1,342 musical works) [6]. eras, such as the opera. In 2001, United Nations The risk of losing this rich cultural heritage must be Educational, Cultural and Scientific Organization urgently addressed, as Kunqu Opera is facing competi- (UNESCO) proclaimed Kunqu Opera a Masterpiece of tion from mass culture, as well as a lack of audience the Oral and Intangible Heritage of Humanity [1]. interest since the eighteenth century because of the high Its composers wrote the musical scores of Kunqu level of technical knowledge required. Of the 400 arias Opera in Chinese using seven or ten characters, that is, regularly sung in opera performances in the mid- in Gong-Che Notation (GCN) [2]. Several Kunqu Opera twentieth century, only a few dozen are performed now, scores exist in the ancient literature [3]. In particular, such as Mudan Ting (). Many Kunqu Opera works are available only as original manuscripts * Correspondence: [email protected] or photocopies, and their digitization and transformation 1Department of Computer Science and Technology, Normal into a machine-readable format is a pressing need. University, Hangzhou, Zhejiang 310036, China Full list of author information is available at the end of the article

© 2014 Chen and Sheu; licensee Springer. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 2 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

[20], neural networks [21], fuzzy theory [22], genetic algo- rithms [23], high-level domain knowledge [24], graph grammar [25], and probability theory [26]. Most of these methods are intended for WMN scores and can convert paper-based scores to digital form (such as a MIDI sequence). Replacing the scores' manual input mode in this way will improve the rate of musical score digitization. In this study, an OMR system for Kunqu Opera mu- sical scores is presented. The remainder of this paper is organized as follows: Section 2 describes Kunqu Opera musical scores, Section 3 discusses the structure of the OMR system and some preliminary operations, Section 4 provides the experimental results for selected scores using ‘ ’ Figure 1 Photo of the Kunqu Hall of Longevity . Playwright: the OMR system, and Section 5 offers ideas for future Hong (1645 to 1704); actor and actress from Kunqu Opera Troupe: Zheng-Ren Cai and Jing-Xian Zhang; performance research. venue: Hangzhou Theatre, China; photographer: Gen-Fang Chen; date taken: October 11, 2011. 2. Gong-Che Notation in Kunqu Opera musical scores GCN is a type of musical notation that was developed Optical music recognition (OMR) refers to the auto- and cultivated in East Asia, flourishing mainly in ancient matic processing and analysis of images of musical nota- China. GCN was invented in the Tang Dynasty (618 to tion. This process typically employs a scanner or other 907 AD) [27] and became a popular form of music nota- digital device to transform paper-based musical scores tion in China during the Song (960 to 1279 AD), Ming into a digital image, which is then processed, recognized, (1368 to 1644 AD), and Qing (1636 to 1911 AD) Dynas- and automatically translated into a standard format for ties. Spanning 1,000 years of use, it is a widely accepted music files, such as the musical instrument digital inter- musical notation in East Asia and resembles WMN in face (MIDI) format. OMR represents a comprehensive Europe. Many traditional Chinese musical manuscripts amalgamation of fields and methods, including music- have been written in GCN [28], including the Kunqu ology, artificial intelligence, image engineering, pattern Opera scripts. GCN represents musical notes using recognition, and MIDI technology. OMR technology Chinese characters, with the basic GCN note pitch sym- provides a way to convert paper-based scores and has bols represented by ten Chinese characters, as shown in numerous applications, including computer-assisted Figure 2. The meaning of these characters is presented music teaching, digital music libraries, musical statistics, in Table 1. digital music image automatic classification, and syn- A Kunqu Opera GCN score contains contents such as chronous music and audio communication. the title, key signature, (qupai is the general term The first published OMR work was conducted by for the tune names used in traditional Chinese lyrics and Pruslin [7] at the Massachusetts Institute of Technology. music writing, and every qupai singing tone has its own Pruslin's system recognized a subset of Western Music musical form, tone, and tonality), lyrics, and notes. An Notation (WMN), primarily musical notes. Before the example of a GCN score is provided in Figure 3. Words 1990s, many studies were performed [8,9], and research constitute a primary component of a GCN score. GCN had begun to focus on handwritten formats [9-11] and is scripted in the format of traditionally written Chinese: ancient (or folk) music, including medieval music [12], from top to bottom and from right to left. Rhythm white mensural notation [13,14], early music prints [15], marks are indicated to the right of the note characters. Orthodox Hellenic Byzantine Music notation [16], and The title of a Kunqu Opera script is located in the first Greek traditional music [17]. During the same period, column, and the qupai is located in the second column, several commercial OMR software packages, such as with the key signature in the top row. The lyrics with Capella-scan, Optical Music easy Reader, Photo Score, notes appear below the qupai, and the notes with the Sharp Eye, Smart Score, and Vivaldi Scan, were devel- rhythm marks are to the upper right of the lyrics. A oped. These reached the market for WMN, and their frame with four borders usually encircles all the col- recognition rate exceeded 90% for commonly printed umns. Figure 4 depicts this layout for a GCN score. musical scores [18]. Commonly, a Chinese lyric word includes several Some techniques and methods used in OMR technology GCN notes, as shown in Figure 5. The number of research include projection [19], mathematical morphology pitches in a Chinese lyric word ranges from 1 to 8, and Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 3 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

Figure 2 Ten GCN symbols of a pitch. the rhythm mark number of a pitch is from 0 to 2. Thus, comprising six independent stages: (1) image pre- words, pitches, and rhythms form a complex three- processing, (2) document image segmentation, (3) feature dimensional (3D) spatial relationship in a GCN score, extraction, (4) musical symbol recognition, (5) musical se- with the first dimension containing lyrics, the second di- mantics, and (6) MIDI representation. mension containing pitches, and the third dimension Because of the maturity of image digitizing technology containing rhythm marks. The font size of a Chinese [29], paper-based Kunqu Opera musical scores can easily lyric word is larger than the font size of a pitch, whose be converted into digital images. Therefore, this paper font size is in turn larger than that of a rhythm mark. does not discuss the image acquisition stage. The system GCN does not mark a note's duration but instead processes a gray-level bitmap from a scanner or reads gives the rhythm marks at regular intervals, with each directly from image files, with the input bitmap generally rhythm mark being indicated in the corresponding being 300 dpi with 256 gray levels. note's upper right corner. The basic GCN note rhythm mark comprises eight symbols, as shown in Table 2, and 3.1 Image pre-processing two main types of rhythms are distinguishable: ‘ban’, the Pre-processing might involve any of the standard image- stronger beat, and ‘yan’, the weaker beat. processing operations, including noise removal, blurring, The difficulties encountered in the processing of GCN de-skewing, contrast adjustment, sharpening, binariza- musical scores are due to the following: (1) the complex tion, or morphology. Many operations may be necessary 3D relationship between symbols in a score's image, (2) to prepare a raw input image for recognition, such as symbols being written in different sizes, shapes, and inten- the selection of an interesting area, the elimination of sities, (3) the variation in relative size between different nonmusical elements, image binarization, and the cor- components of a musical symbol, (4) different symbols rection of image rotation. appearing connected to each other and the same musical Many documents, particularly historical documents symbol appearing in separated components, and (5) diffi- such as those in Kunqu Opera GCN scores, rely on culties in converting the musical information in GCN careful pre-processing to ensure good overall system scores to other styles of musical notation, such as WMN. performance and, in some cases, to significantly im- prove the recognition performance. In this work, we 3. Description of the OMR system for Kunqu briefly touch upon the basic pre-processing operations Opera musical scores of Kunqu Opera GCN scores, such as binarization, The OMR system for Kunqu Opera (KOMR), shown noise removal, skew correction, and the selection of an in Figure 6, is an off-line optical recognition system area of interest. Image binarization uses the algorithm of Otsu [30], noise removal is conducted using basic Table 1 Description of a GCN score's general symbols morphological operations [31], and image rotation has (pitch) been corrected using the least squares criterion [32]. Symbols Description TheareaofinterestinaKunqu score is surrounded Phoneticize in Movable doh Solfege byaframe(widelines)(seeFigure3).Theframeisa simplified Chinese connected component containing the longest line in hé 5 sol thescore,soweusetheHoughtransform[33]tolo- sì 6lacate the longest line then delete the connected compo- yī 7tinent with the longest line from the score, leaving the area of interest. Because these are common image- shàng 1do processing operations, we do not give a detailed de- ě Ch 2rescription here. gōng 3mi fán 4fa3.2 Document segmentation and analysis of GCN scores liù 5 sol Document segmentation is a key phase in the KOMR wǔ 6lasystem. With the symbols in a GCN score document having been classified, this stage first segments the docu- yǐ 7ti ment into two sections, one including symbols for the Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 4 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

Figure 3 Musical score sheet of Nashu Studio Theatrical Music (p. 1983).

notes and the other for the non-notes. Elements of non- X-Y projection, RLSA, scale-space analysis, and the Hough note symbols, such as the title, key signature, qupai, transform [40]. lyrics, noise, and border lines of a textural framework, A preliminary result of GCN score segmentation has been are then identified and removed. presented in [41]. A self-adaptive RLSA was used to seg- Because music is a time-based art, the arrangement of ment the image according to an X-axis function (denoted by the notes is one of its most important factors. Therefore, PF(x)) indicating the number of foreground pixels in each obtaining the arrangement of the notes in a GCN score column of the image. This X-axis function uses X-projection is requisite to document the segmentation. Concordant to compute the number of flex points that satisfy the follow- with the writing style of the GCN score, the arrange- ing conditions: PF(x − 1) < PF(x)andPF(x)>PF(x +1)orPF ment of the notes can be organized based on high-level (x − 1) > PF(x)andPF(x)

Table 2 Description of GCN score's general symbols (rhythm) Symbols Phoneticize in Description simplified Chinese ban A note with ‘ban’ starts a stronger beat yan A note with ‘yan’ starts a weaker beat

yaoyan Anotewith‘yaoyan’ starts a weaker beat

yaoban Denotes a note's duration from a weaker beat to the next stronger beat ban A note with ‘ban’ starts a stronger beat

zhengban Denotes the stronger beat in the middle of a bar for an 8/8 beat yaoban Denotes a note's duration from a weaker beat to the next stronger beat in some scores Figure 4 Sample annotation for a sub-section of Figure 3. yaoyan A note with ‘yaoyan’ starts a weaker , Xianlv,aChinesekeysignatureinamusicalwork,andnine beat in some scores key signatures are commonly used in Kunqu Opera; , Bashing ganzhou,isaqupai, a fixed type of rhythm or melody in Chinese poetry and opera. 3.3 Symbol feature extraction Selecting suitable features for pattern classes is a critical process in the OMR system. Feature representation is also Next, the algorithm matches each connected component to a crucial step, because perfect feature data can effectively its corresponding sub-image. According to the experimental enhance the symbol recognition rate. The goal of feature results in [41], the rate of correct segmentation in lines is extraction is to characterize a symbol to be recognized 98.9%, and the loss rate of notes is 2.7%; however, the total by measurements whose values are highly similar for error rate of all notes and lyrics is almost 22%. symbols in the same category, but different for symbols in different categories. The feature data must also be in- variant during relevant transformations, such as transla- tion, rotation, and scaling. Popular feature extraction methods for OCR and Chin- ese character recognition include peripheral feature [42], cellular feature [43], and so on [44]. Because symbols are writtenindifferentsizesinaGCNscore,toconstructa simple and intuitive approach, four types of structural fea- tures have been used in this exploratory work based on the reference [42,43]; these are suited to feature extraction of symbols in a GCN score and to compare with the recogni- tion rate of KOMR. An n × m matrix featuring symbol data is used to obtain symbols of the same size as in the feature matrix. These are obtained using a grid to segment the symbols [45]. In Figure 7, a sample symbol with size H × W isshowninsub-graph(a),andann × m grid is shownÂÃ in ¼ ; ¼ ; ¼ H Â sub-graph (b). In this example,ÂÃ h0 0 hn H hi n ; ¼ ; ¼ ; ¼ W Â i w0 0 wm W wj m j , and the four features of each symbol are given by the following equations:

2 3 …; …; … 4 5 T 1 ¼ …; ti;j; … ; ti;j …; …; …

Xwi Xhj ¼ fxðÞ; y ; 1≤i≤m; 1≤j≤n ð1Þ Figure 5 Sample annotation for a sub-section of Figure 4. ¼ x wi−1 y¼hj−1 Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 6 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

Figure 6 The KOMR system architecture. 2 3 …; …; … 3.4 Musical symbol recognition 4 5 T 2 ¼ …; ti;j; … ; ti;j Computer recognition of handwritten characters has …; …; … been intensely researched for many years. Optical char- Xwi ÀÁ acter recognition (OCR) is an active field, particularly ¼ ; ; ≤ ≤ ; ≤ ≤ ð Þ fxhj 1 i m 1 j n 2 for handwritten documents in such languages as Roman x¼wi−1 2 3 [45], Arabic [46], Chinese [47], and Indian [48]. …; …; … Several Chinese character recognition methods have 4 5 T 3 ¼ …; ti;j; … ; ti;j been proposed, with the best known being the trans- …; …; … formation invariant matching algorithm [49], adaptive

Xhj confidence transform based classifier combination [50], ¼ fwðÞi; y ; 1≤i≤m; 1≤j≤n ð3Þ probabilistic neural networks [51], radical decompos- ¼ y hj−1 ition [52], statistical character structure modeling [53], 8 Markov random fields [54], and affine sparse matrix > 0; for 1 < i < m and 1 < j < n > Xx > ÀÁ ÀÁfactorization [55]. > ; ; ¼ þ ; > x if fkhj 0 and fx 1 hj The basic symbols for musical pitch (see Figure 2) in a > k¼0 > > ¼ ; ¼ Kunqu Opera GCN score are Chinese characters, but > 1 for j 1 > > Xwm ÀÁ ÀÁ > other musical pitch symbols and all rhythm symbols are >W−x; if fk; h ¼ 0 and fx−1; h 2 3 > j j specialized symbols for GCN. Thus, the methods of …; …; … <> k¼x 4 5 ¼ ; ¼ Kunqu Opera GCN score recognition refer to the tech- T ¼ …; t ; ; … ; t ; ¼ 1 for j m 4 i j i j > …; …; … > Xx niques of both OMR and OCR. In this paper, the follow- > ; ðÞ¼; ðÞ; þ >x if fwi k 0 and fwi x 1 > ing three approaches to musical symbol recognition are > k¼0 > > ¼ 1; for i ¼ 1 compared. > > Xhn > >H−x; if fwðÞ¼i; k 0 and fwðÞi; x−1 > 3.4.1 K-nearest neighbor :> k¼x ¼ 1; for i ¼ n The K-nearest neighbor (KNN) classifier is one of the ð4Þ simplest machine learning algorithms. It classifies fore- grounds based on the closest training examples in the where f(x, y) is the value of a pixel at coordinates (x, y) feature space and is a type of instance-based, or lazy, in an image. If f(x, y) = 1, then the pixel is a foreground learning in which the function is only approximated lo- pixel; otherwise, 0 indicates a background pixel. The T1 cally, and all computation is deferred until classification feature is the number of pixels in each cell of the grid as [56]. The neighbors are selected from a set of fore- a cellular feature, T2 feature appears to be the number of grounds for which the correct classification is known, a pixels falling on the upper edge of each grid cell, T3 is set that can be considered the training set for the algo- the number of pixels falling on the right edge of each rithm, though no explicit training step is required. grid cell, and T4 is the number to be some measure of In the experiment described in the manuscript, we how many background pixels there are from the edge to choose half of the test musical symbols as training sam- the first foreground pixel, if f(1,1) = 1, then t1,1 =0;if f ples; then, for the convenience of computing, we set the (n,1) = 1, then tn,1 =0;iff(1,m) = 1, then t1,m =0;iff(n,m) value of K to 1. The feature data of each class are ob- = 1, then tn,m = 0; and if there is no foreground pixel in tained from the samples by calculating the average fea- the image, then ti,j = 0, for 1 ≤ i ≤ m and 1 ≤ j ≤ n. ture data of all samples. Although the distance function Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 7 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

Figure 7 Matrix featuring data of a musical symbol using an n × m grid partition (a to d).

can also be learned during the training, the correspond- simple, probabilistic classifiers that apply Bayesian deci- ing distance functions for the four features above are de- sion theory with conditional independence assumptions, termined by the following equations: providing a simple approach to discriminative classifica- tion learning [57]. Xm Xn In this work, the conditional probabilities P(T|S ) ,1≤ m  n− ρ; ρ ¼ 1ifsi;j ¼ ti;j; other ρ ¼ 0 k r i¼1 j¼1 r ≤ 4 with each of the four features for the Bayesian deci- fSðÞ; T ¼ 1 m  n sion theory classifier are calculated as follows: ð5Þ

Xm Xn Xm Xn k ρ; ρ ¼ 1ifs ; ¼ t ; ; other ρ ¼ 0 si;j−ti;j i j i j ¼ ¼ i¼1 j¼1 ð j Þ ¼ i 1 j 1 fSðÞ; T ¼ ð6Þ P T Sk 1  2 m  n m n ð9Þ m n X X Xm Xn − si;j ti;j k − si;j ti;j i¼1 j¼1 fSðÞ; T ¼ ð7Þ i¼1 j¼1 3  PðTjSkÞ2 ¼ ð10Þ m n m  n Xm Xn  − ρ; α < ti;j < β; ρ ¼ ; ρ ¼ Xm Xn m n if 1 other 0 k s ; − i¼1 j¼1 i j si;j ti;j fSðÞ; T ¼ 4 m  n i¼1 j¼1 PðTjSkÞ3 ð11Þ ð8Þ m  n where f(S,T)1 ‐ 4 is the distance function of the corre- Xm Xn sponding feature 1-4, si,j is the feature matrix of the t ; m  n− ρ; ifα < i j < β; ρ ¼ 1; other ρ ¼ 0 prototype class (S), and t is the feature matrix of the k i,j i¼1 j¼1 si;j ≤ ≤ ≤ ≤ PðTjS Þ4 ¼ test sample (T). In Equation 5,ifsi,j = ti,j,1 i m,1 j k m  n ρ ρ n, then = 1; otherwise, =0.f(S,T) counts the number ð12Þ of unequal elements in the feature matrixes (S and T). In Equations 6 and 7, f(S,T) calculates the sum of the dif- where P(T|Sk)1 ‐ 4 is the conditional probability of the ference between all corresponding elements in the fea- corresponding feature 1-4, S ={S1, …, Sk, …, Sc} is the set k ture matrixes (S and T). Equation 8 counts the number of prototype classes, si;j is the feature matrix of the kth of non-approximate elements in S and T, using the pa- class Sk in the set of prototype classes, and ti,j is the fea- α β α k rameters and as experience values. In this work, = ture matrix of the test sample (T). In Equation 9,ifsi;j = 0.9 and β = 1.1. ti,j, then ρ = 1; otherwise, ρ = 0. In Equation 12, α and β are again experience values, and in this work, we set α = 3.4.2 Bayesian decision theory 0.9 and β = 1.1. Bayesian decision theory is used in many statistics-based In the experiment, the prior probability P(Sk), 1 ≤ methods. Classifiers based on Bayesian decision theory are k ≤ c of different symbols Sk is not equal, and all Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 8 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

prior probabilities come from the prior statistics of the 3.5 Semantic analysis and MIDI representation training dataset. For example, the beat symbol ‘ ’ has After all the stages of the OMR system are complete (see the prior probability 0.354511. If P(Sk|T)=maxP(Sj|T), Figure 6), the recognized symbols can be employed to T is classified to Sk. The Bayesian rule was used. write the score in different data formats, such as MIDI, Xc Nyquist, MusicXML, WEDELMUSIC, MPEG-SMR, nota- ð j Þ ðÞ= ð j Þ ðÞ; ≤ ≤ P T Sk rP Sk PTSi PSi 1 r 4 and has four tion interchange file format (NIFF), and standard music ¼ i 1 description language (SMDL). Although different repre- expressions to estimate P(S |T) which correspond to k sentation formats are now available, no standard exists each of the four features. for computer symbolic music representation. However, MIDI is the most popular digital music format in mod- 3.4.3 Genetic algorithm ern China, analogous to the status of the GCN format In the field of artificial intelligence, a genetic algorithm in ancient China. (GA) is a search heuristic that mimics the process of This work selected MIDI for music representation, be- natural evolution. This heuristic is routinely used to cause the musical melody in a Kunqu Opera GCN score generate useful solutions to optimization and search provides monophonic information. Thus, the symbols problems. GAs generate solutions to optimization prob- recognized from the GCN score can be represented by lems using techniques inspired by natural evolution, such an array melody[L][2], with L representing the number as inheritance, mutation, selection, and crossover. In GAs, of notes in the GCN score, the first dimension in the the search space parameters are encoded as strings, array representing the pitch of all notes, and the second and a collection of strings constitutes a population. The dimension representing the duration of all notes. processes of selection, crossover, and mutation con- Finally, the note information in melody[L][2] can be tinue for a fixed number of generations or until some transformed into a MIDI file using an associated coding condition is satisfied. GAs have been applied in such language, such as Visual C++, and the MIDI file of the fields as image processing, neural networks, and ma- Kunqu Opera GCN score can then be disseminated glo- chine learning [58]. bally via the Internet. In this work, the key GA parameter values are as follows: 4. Experimental results A Chinese Kunqu Opera GCN score may contain infor- mation as Chinese characters and musical notes, such as  Selection-reproduction rate: p = 0.5 s lyrics, title, tune, pitch, rhythm, voices, preface of music,  Crossover rate: p = 0.6 c and so on. In this experiment, a dataset was randomly  Mutation rate: p = 0.05 m selected from two Kunqu score books [4,5]. Because  Population class: C =12 OCR for Chinese characters is actively researched [47]  Number of individuals in the population: N = 200 p and OMR for musical information is a developing field,  Maximum iteration number: G = 300 we consider only the musical note information in this experiment. This includes pitches and rhythm marks An individual's fitness value is determined from the (beats) but neglects other information such as lyrics following fitness function: (Chinese characters). First, all scores in the set were segmented according to [41], and then the musical note information, including XR ÀÁÀÁXR FIðÞ¼u fehu;v; su;v ¼ f ðÞρ ¼ ¼ v 1 v 1 Table 3 Information on the 39 selected images XR ρ ¼ ; if h ; ¼ s ; ; then ρ Name of works m  n i j i j v¼1 Nashu Studio Jiugong Dacheng ¼ 1; else ρ ¼ 0 ð13Þ Theatrical Music Nanbei Ci Gongpu Range of pages pp. 1975-1996 pp. 4326-4344 Range of training pages pp. 1975-1984 pp. 4326-4335 where Iu is the kth individual, F(Iu) is the fitness value of Iu, R is the number of a gene bits in Iu, hi,j is the jth gene Range of testing pages pp. 1985-1996 pp. 4336-4344 of Iu, si,j is the feature matrix of the set of prototype clas- Serial number of testing pages No. 1-10 No. 11 ~ 19 ses corresponding to hi,j, the function e() computes the Number of all pages 20 19 value of the comparative result between h and s , and i,j i,j Number of pitches in all pages 3,100 2,157 m and n are the width and length, respectively, of a sym- Number of beats in all pages 1,413 661 bol's grid. Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 9 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

Figure 8 Recognition rate of 39 images using four features. X-axis: serial number of pages; Y-axis: (a to c) recognition rate of pitch using three classifiers and (d, e) recognition rate of beat using two classifiers. (f) Average recognition rate of pitch and beat.

pitches and beats (rhythm marks), was decoded. Finally, 16 (n×m above). Each symbol was then resized to 16 × the musical information was translated to a MIDI file. 16 pixels and converted to a 256-dimensional vector, a fea- ture datum of the symbol. 4.1 Training and test datasets To create training datasets, we randomly selected im- To estimate the efficiency of KOMR, experiments were ages from the available datasets. Specifically, the images conducted using images randomly selected from Nashu on pp. 1975-1984 and pp. 4326-4335 (see Table 3) con- Studio Theatrical Music [5] and Jiugong Dacheng Nanbei stituted the training sets, and other images were in- Ci Gongpu [4]. Thirty-nine images were used. The pri- cluded in the test sets. mary information from these images is shown in Table 3 and includes the number of pages, pitches, and beats. 4.2 Results The resolution was calculated as (680-720) × (880-900). To evaluate the accuracy of our results, the various clas- Figure 3 displays a common musical score sheet from sifiers were compared based on their test GCN scores. page 1983 of Nashu Studio Theatrical Music. The three classifiers' pitch recognition rates for the test To improve the recognition rate of all classifiers, a sym- images are shown in Figure 8, with (a) the KNN classi- bol's grid was used to render a partition of resolution 16 × fier (in this work, K = 1), (b) the Bayesian classifier, and

Table 4 Experimental statistics of the recognition rate for all pitches and all beats using features 1 to 4 Feature Recognition rate of pitch (%) Recognition rate of beat (%) BC GA KNN BC KNN Feature 1 52.75 43.09 64.80 55.59 70.78 Feature 2 52.59 41.05 59.35 57.10 73.46 Feature 3 51.97 43.60 62.72 58.32 68.17 Feature 4 58.24 42.72 49.85 61.68 56.63 Average rate 53.89 42.62 59.18 58.17 67.26 Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 10 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

(c) the GA classifier. The beat recognition rates for (d) beats in the first book included seven different symbols: the KNN classifier and (e) the Bayesian classifier are also , whereas there were only shown. The recognition rate obtained by KNN is higher four different symbols in the second book: . than that of the other two classifiers. Thus, each beat feature in the first book is very different From the results obtained for the musical notes, shown from that in the second book, and the second has less dis- in Table 4, it can be seen that the KNN classifier outper- crimination. Meanwhile, Figure 8 and Table 4 suggest that formed the other two classifiers. Notably, the performance the four features have little effect on the recognition rate. of the KNN classifier was clearly superior to that of the Following the recognition stage, the musical informa- GA classifier. The recognition rate of beats is greater than tion of a GCN score was saved to an array melody[L][2], that of pitches, and the results in Figure 8 indicate that which was transformed into MIDI format and stored as the recognition rate of the beats in Nashu Studio Theatri- a MIDI file (Additional file 1). Figure 9 shows the sheet cal Music is greater than that in Jiugong Dacheng Nanbei score of the MIDI file in WMN based on the recognition Ci Gongpu. This is because the musical symbols for the of musical information from Figure 3.

Figure 9 Sheet score of the MIDI file in WMN for Figure 3. Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 11 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

5. Conclusions 5. T Ye, Nashu Studio Theatrical Music. new, edn (Shanghai Guji Publishing As one of UNESCO's Oral and Intangible Heritages of House, Shanghai, 1995), pp. 1975–1996 6. S Lv, X Yang, H Liu, S Tang, The Law of the Southern Word. new edn. Humanity, Kunqu Opera had a deep effect on ancient (Shanghai Guji Publishing House, Shanghai, 1995), pp. 1–10 Chinese entertainment and has been studied in many 7. D Pruslin, Automatic recognition of sheet music, Dissertation (Massachusetts ways. GCN is the most widely used recording method for Institute of Technology, Cambridge, 1966) 8. I Fujiinaga, Adaptive optical music recognition, Dissertation (Faculty of Music, ancient Chinese musical information, employed by a sig- McGill University, 1996) nificant number of musical works, including all Kunqu 9. K Ng, Optical music analysis for printed music score and handwritten music Opera scores. manuscript, in Visual Perception of Music Notation: On-Line and Off-Line Recognition, ed. by SE George (Idea Group Publishing, Pennsylvania, 2003), This paper presented the six-stage KOMR system, with pp. 108–127 the key phases being image segmentation and symbol rec- 10. J Wolman, J Choi, S Asghzrzadeh, J Kahana, Recognition of handwritten ognition. This paper focused on the recognition phase, music notation, in Proceedings of the International Computer Music Conference (San Jose, 9–13 Nov 1992), pp. 125–127 studying the effectiveness of three classifiers to obtain mu- 11. J Caldas Pinto, P Vieira, M Ramalho, MO Mengucci, P Pina, F Muge, Ancient sical information from the GCN score. The results indi- music recovery for digital libraries, in Proceedings of the 4th European cate that a KNN classifier is most suitable for this task. Conference on Research and Advanced Technology for Digital Libraries. Lecture – However, because the average recognition rate of Notes in Computer Science (Springer, Heidelberg, 2000), pp. 245 249 12. W McGee, P Merkley, The optical scanning of medieval music. Comput. KOMR is less than 90%, it is necessary to find more Hum. 25(1), 47–53 (1991) powerful methods that can be applied to all phases of 13. LJ Tardón, S Sammartino, I Barbancho, Optical music recognition for scores KOMR. These problems merit further research. written in white mensural notation. EURASIP J. Image Video Process. (2009). doi: 10.1155/2009/843401 This work is preliminary and tentative and will help 14. NP Carter, Segmentation and preliminary recognition of madrigals notated to preserve and popularize Chinese cultural heritage. in white mensural notation. Mach. Vis. Appl. 5(3), 223–230 (1992) In particular, this study will assist the conversion of 15. L Pugin, JA Burgoyne, I Fujinaga, Goal-directed evaluation for the improvement of optical music recognition on early music prints, in Proceedings of the 7th handwritten musical Kunqu Opera scores to a machine- ACM/IEEE Joint Conference on Digital Libraries (JCDL'07, Vancouver, 1 April 2007), readable format, thus ensuring the possibility of dissemin- pp. 303–304 ating and performing the original Kunqu Opera musical 16. VG Gezerlis, S Theodoridis, Optical character recognition of the Orthodox Hellenic Byzantine Music notation. Pattern Recogn. 35(4), 895–914 (2002) scores. 17. A Pikrakis, Recognition of isolated musical patterns in the context of Greek traditional music, in Proceedings of the Third IEEE International Conference on Electronics, Circuits, and Systems, vol 2nd edn. (IEEE Xplore Digital Library, Additional file Rodos, 1996), pp. 1223–1226 18. K Ng, J Barthelemy, B Ong, I Bruno, P Nesi, CIMS: coding images of music Additional file 1: Shuanghe.zip. The file is a MIDI format file which sheets. Leeds University PhysicsWeb (2005). http://music.leeds.ac.uk/people/ includes musical information from Figure 3 in the main body text. kia-ng. Accessed 20 June 2005 19. I Fujinaga, Optical music recognition using projections, Dissertation (Montreal University, Canada, 1988) Competing interests 20. G Chen, The study of optical music recognition and prototype system, The authors declare that they have no competing interests. Dissertation (Zhejiang University, China, 2003) 21. IG Ratiu, Self organizing neural network and direction of arrival estimation Acknowledgements technique MUSIC 2D applied in target radiolocation recognition, in This work is supported by the key project of National Social Science Proceedings of the 4th WSEAS International Conference on Computer Foundation of China Grant No. 12AZD120, the promotion project of National Engineering and Applications (Stevens Point, 2011), pp. 231–236 Culture Science and Technology in China Grant No. [2013]718-6, HangZhou 22. HS Park, JO Yoo, SB Cho, A context-aware music recommendation system Special Funds to Support Cultural Creativity project Grant No. 93, and the using fuzzy Bayesian networks with utility theory. Lecture Notes in Computer planning project of educational technology research in Zhejiang Province Science. vol. 4223 (Springer, Heidelberg, 2006), pp. 970–979 under Grant JB001. 23. I Yoda, K Yamamoto, Automatic construction of recognition procedures for musical notes by genetic algorithm, in International Association for Pattern Author details Recognition Workshop on Document Analysis Systems (World Scientific, 1Department of Computer Science and Technology, Hangzhou Normal Hackensack, 1995), pp. 153–159 University, Hangzhou, Zhejiang 310036, China. 2Department of Computer 24. MV Stuckelberg, C Pellegrini, M Hilario, An architecture for musical score Science, National University of Education, Taipei 10671, Taiwan. recognition using high-level domain knowledge, in Proceedings of the Fourth International Conference on Document Analysis and Recognition, 2nd Received: 18 January 2013 Accepted: 14 February 2014 edn. (Ulm, 1997), pp. 813–818 Published: 4 March 2014 25. S Baumann, G Res, A simplified attributed graph grammar for high-level music recognition, in Proceedings of the Third International Conference on Document References Analysis and Recognition, 2nd edn. (Montreal, 1995), pp. 1080–1083 1. UNESCO, Masterpieces 2001 and 2003: Kunqu Opera. UNESCO Physics 26. G Taubman, Music hand: a handwritten music recognition system, Dissertation Web (2008). http://www.unesco.org/culture/ich/en/RL/00004. Accessed (Brown University, 2005) 12 Oct 2012 27. YL Yang, Chinese Ancient Music Historical Manuscript (People's Music 2. Z Chen, Introduction to Gong-Che Notation, 1st edn. (Huayue's Music Publishing House, Beijing, 2004), pp. 258–259 Publishing House, Beijing, 2006), pp. 1–2 28. Z Tong, J Gu, Y Zhou, X Sun, Chinese Traditional Musicology (Fujian 3. Y-H Wang, Musical Notation of Chinese Traditional Music, 1st edn. (Fujian Education Publishing House, Fuzhou, 2003), pp. 73–75 Education Publishing House, Fuzhou, 2006), pp. 245–246 29. A Rebelo, G Capela, JS Cardoso, Optical recognition of music symbols. 4. X Zhou, J Zhou, X Xu, W Wang, T Zhu, Y Xu, A comprehensive anthology of IJDAR 13,19–31 (2010) texts and notation of the Southern and Northern opera tunes in nine modes. 30. N Otsu, A threshold selection method from gray-level histogram. IEEE. Trans. new, edn (Shanghai Guji Publishing House, Shanghai, 1995), pp. 4326–4344 Syst. Man Cybern. 9,62–66 (1979) Chen and Sheu EURASIP Journal on Audio, Speech, and Music Processing 2014, 2014:7 Page 12 of 12 http://asmp.eurasipjournals.com/content/2014/1/7

31. L Najman, H Talbot, Mathematical Morphology (Wiley, Hoboken, 2010), pp. 56. T Cover, P Hart, Nearest neighbor pattern classification. IEEE Trans. Inf. 12–13 Theory 13(1), 21–27 (1967) 32. YH Liang, ZY Wang, A skew detection method for 2D bar code images 57. R Duda, P Hart, Pattern Classification and Scene Analysis (Wiley, New York, 1973) based on the least square method, in 2006 International Conference on 58. W Gong, Z Cai, L Jia, H Li, A generalized hybrid generation scheme of Machine Learning and Cybernetics, Dalian, 2006, pp. 3974–3977. doi:10.1109/ differential evolution for global numerical optimization, international journal ICMLC.2006.258793 of computational intelligence and applications. Int. J. Comp. Intel. Appl 33. RC Gonzalez, RE Woods, SL Eddins, Digital image processing using MATLAB 35(10), 560–567 (2011). doi: 10.1142/S1469026811002982 (Prentice Hall, 2004), pp. 393–403 34. G Nagy, S Seth, Hierarchical representation of optically scanned documents, doi:10.1186/1687-4722-2014-7 in Seventh International Conference on Pattern Recognition (Montreal, 1984), Cite this article as: Chen and Sheu: An optical music recognition system pp. 347–349 for traditional Chinese Kunqu Opera scores written in Gong-Che 35. FM Wahl, RG Wong, KY Casey, Block segmentation and test extraction Notation. EURASIP Journal on Audio, Speech, and Music Processing in mixed text/image documents. Comput. Graph. Image Process. 2014 2014:7. 20,375–390 (1982) 36. M Feldbach, KD Tonnies, Line detection and segmentation in historical document analysis and recognition. IJDAR 14, 743–747 (2011) 37. R Manmatha, JL Rothfeder, A scale space approach for automatically segmenting words from historical handwritten documents. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1212–1225 (2005) 38. K Kise, A Sato, M Iwata, Segmentation of page images using the area Voronoi diagram. Comput. Vis. Image Underst. 70(3), 370–382 (1998) 39. RD Duda, PE Hart, Use of the Hough transform to detect lines and curves in pictures. Commun. ACM 15(1), 11–15 (1972) 40. N Nikolaou, M Makridis, B Gatos, N Stamatopoulos, N Papamarkos, Segmentation of historical machine-printed documents using adaptive run length smoothing and skeleton segmentation paths. Image Vis. Comput. 28, 590–604 (2010) 41. G-F Chen, W Zhang, H Cui, Extracting notes from Chinese Gong-Che notation musical score image using a self-adaptive smoothing and connected component labeling algorithm. Int. J. Adv. Comput. Technol. 4(1), 86–95 (2012) 42. TH Hildebrandt, WT Liu, Optical recognition of handwritten Chinese characters: advances since 1980. Pattern Recogn. 26, 205–225 (1993) 43. RI Oka, Handwritten Chinese-Japanese character recognition by using cellular feature. Proc. Int. Conf. Pattern Recogn. 2, 783–785 (1982) 44. OD Trier, AK Jain, T Taxt, Feature extraction methods for character recognition-a survey. Pattern Recogn. 29(4), 641–662 (1996) 45. AW Senior, AJ Robinson, An off-line cursive handwriting recognition system. IEEE Trans. Pattern Anal. Mach. Intell. 20, 309–321 (1998) 46. A Amin, Off-line Arabic character recognition: the state of the art. Pattern Recogn. 31(5), 517–530 (1998) 47. CL Liu, F Yin, DH Wang, QF Wang, Online and offline handwritten Chinese character recognition: benchmarking on new databases. Pattern Recogn. 46, 155–162 (2013) 48. S Basu, N Das, R Sarkar, M Kundu, M Nasipuri, DK Basu, A hierarchical approach to recognition of handwritten Bangla characters. Pattern Recogn. 42, 1467–1484 (2009) 49. CW Liao, JS Huang, A transformation invariant matching algorithm for handwritten Chinese character recognition. Pattern Recogn. 23(1), 1167–1188 (1990) 50. X Lin, X Ding, M Chen, R Zhang, W Youshou, Adaptive confidence transform based classier combination for Chinese character recognition. Pattern Recogn. Lett. 19, 975–988 (1998) 51. RD Romero, DS Touretzky, RH Thibadeau, Optical Chinese character recognition using probabilistic neural networks. Pattern Recogn. 30(8), 1279–1292 (1997) 52. D Shi, RI Damper, SR Gunn, Offline handwritten Chinese character recognition by radical decomposition. ACM Transac. Asian Lang. Info. Submit your manuscript to a Process. 2(1), 27–48 (2003) 53. IJ Kim, JH Kim, Statistical character structure modeling and its application to journal and benefi t from: handwritten Chinese character recognition. IEEE Trans. Pattern Anal. Mach. 7 Convenient online submission Intell. 25(11), 1422–1436 (2003) 7 54. J Zeng, ZQ Liu, Markov random field-based statistical character structure Rigorous peer review modeling for handwritten Chinese character recognition. IEEE Trans. Pattern 7 Immediate publication on acceptance Anal. Mach. Intell. 30(5), 767–780 (2008) 7 Open access: articles freely available online 55. J Tan, X Xie, WS Zheng, JH Lai, Radical extraction using affine sparse matrix 7 High visibility within the fi eld factorization for printed Chinese characters recognition. Int. J. Pattern 7 Recognit. Artif. Intell. 26(3) (2012). doi:10.1142/S021800141250005X Retaining the copyright to your article

Submit your next manuscript at 7 springeropen.com