(IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021 Monophonic via Mobile App

1 Edgar García Leyva Elena Fabiola Ruiz Ledesma2 Rosaura Palma Orozco3 4 Instituto Politécnico Nacional. SEPI- Instituto Politécnico Nacional. Lorena Chavarría Báez ESCOM ESCOM, UPIICSA Instituto Politécnico Nacional. Mexico City, Mexico Mexico City, Mexico ESCOM, Mexico City, Mexico

Abstract—In the guild of , it is common to work with and mathematics, it was decided to develop a monophonic guitar because the emulation of a great variety of synthesizer, using a mobile application, which allows the sounds that are produced by different musical instruments, to use a synthesizer through a mobile device such as a starting from just playing the guitar, which means, a piece of smart cell phone or a tablet, which can be transported easily, music is played with a guitar, but other musical instruments are without having to make an additional expense. actually heard such as, a saxophone, a violin, a piano or percussions, depending on the instrument that has been selected. Music can be classified into two main categories: The problem that arises in this article is that synthesizers are monophonic and polyphonic. Monophonic music is made up of expensive and due to their size, the transportation of the a single melodic line, which means, only one musical note equipment is often impractical. As mentioned, the development sounds at a certain time, while polyphonic music is made up of of a mobile application that has the function of a monophonic more than one melodic line, which means, two or more musical synthesizer is proposed as a solution. In this way, the cost is notes sound at the same time [2]. The present study focuses on greatly reduced, and additionally, the user is able to install the monophonic music. application on a mobile device with Android operating system and connect it to an electric or electro-acoustic guitar through an The guitar is a musical instrument that allows its player to audio interface; obtaining as a result, a functional technological express musical notes in different ways. The guitar, like other instrument by offering guitarists an alternative with respect to instruments such as the piano or the violin, generates analog conventional synthesizers. The construction of this application sounds, which need to be converted to digital sounds in order used the Fast Fourier Transform Radix-2 as a signal recognition to be read by the computer. From this digitization, it is possible algorithm, which allowed obtaining the fundamental frequencies to apply signal recognition techniques for different purposes generated by the guitar, which were transformed into MIDI and in this case, to detect the fundamental frequencies notation and later used in sound emulation. generated by the guitar. A transformation was applied to these frequencies using the notation mentioned in the Musical Keywords—Monophonic synthesizer; guitar; sound emulation; Instrument Digital Interface (MIDI), to obtain a discrete mobile application representation of the musical notes, which are defined in an I. INTRODUCTION interval that goes from 0 to 127, where each number corresponds to a musical note. The advent of tablets and smart cell phones has opened a range of possibilities in all areas of knowledge. In the field of The overall objective of this study was to develop a music, technological development has occupied a privileged portable tool that allows the guitarist to emulate monophonic place by having mobile applications that help in tuning musical sounds of other musical instruments or other types of sounds instruments, in measuring time to practice, in recording and using the guitar, all this through computational and editing music, among other aspects. mathematical techniques. For this purpose, the following specific objectives were proposed: The rise of technology has allowed the creation of tools in order to support musicians in a vast diversity of ways. The tool • Recognize frequencies generated by making use that is interesting to highlight in this article is related to the of hardware and software resources of mobile devices. emulation of monophonic sounds of musical instruments or • Transform frequencies generated by guitars to MIDI other types of sounds, which is obtained through the use of notation. synthesizers. The Royal Academy of the Spanish Language defines synthesizer as: "Electronic musical instrument capable • Emulate monophonic sounds of musical instruments or of producing sounds of any frequency and intensity and other sounds based on the obtained MIDI notes. combining them with harmonics, thus providing sounds of any known instrument, or sound effects that do not correspond to This article is divided into 4 sections. The second section any conventional instrument" [1]. Synthesizers are very useful shows some synthesizers available on the market, as well as devices because when connected to the guitar, are able to some application programming interfaces that have been provide a great variety of sounds, increasing the possibilities of developed in order to recognize signals and to support the musical interpretation of the guitarist. execution of MIDI sounds. The third section deals with the theoretical references about the recognition of monophonic Two of the problems that arise are its portability and its sounds and the transformation of frequency to MIDI notation. high cost; Due to the mentioned, based on computer science The fourth section shows the methodology used for the

409 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021 development of the monophonic synthesizer following the B. Fast Fourier Transform (FFT) stages of the incremental Software Engineering model. Fast Fourier Transform is an efficient mathematical Subsequently, the results obtained from the tests carried out are implementation of the Discrete Fourier Transform (DFT), shown and finalized with the conclusions. which is a particular case of the Fourier Transform for sequences of finite length in which the spectrum is evaluated II. RELATED WORK only in a few specific frequencies, and therefore, a discrete At present, there are independent synthesizers for guitar, spectrum is obtained [10, 11]. which means, mounted on electronic circuits, some of them are: MEL9 [3], SY-300 [4], GR-55 [5], among others; Over time, several FFT algorithms have been developed However, when it comes to mobile applications that perform such as: prime factor, split radix, vector radix, split vector the function of a guitar synthesizer on the Android operating radix, Winograd Fourier transform, etc. [12]. system, there are no formal alternatives to it. The FFT algorithm used to develop the mobile application On the other hand, there are some application programming of the monophonic synthesizer is the Radix-2. The Radix-2 interfaces (APIs) aimed at the Android operating system, algorithm is considered the most used for the FFT calculation, which can be useful for the construction of a synthesizer, for it works when the number of data samples is a power of 2, in example, those that serve to perform frequency recognition and case the number of samples does not satisfy this criterion, the those that serve to execute MIDI sounds, some of these missing spaces are filled with value 0, this does not alter the interfaces are: TarsosDSP [6] and MIDI Driver [7] calculated frequency spectrum. The input and output of an FFT respectively. are expressed in complex numbers, in this case, for its implementation, two arrangements are accepted to store the In the present mobile application, it was chosen the real and imaginary components in them, when using the frequency recognition through the Fast Fourier Transform recording tools of mobile devices, all the bytes of audio Radix-2 algorithm, according to [8], making use of the native information are obtained, which are used within the real Java development kit, while for the execution of MIDI sounds, components, while the imaginary components are always filled the model that was done in [7] was retaken. with zeros, the output of the algorithm is contained in two other arrays, one corresponding to each type of component, where III. THEORETICAL ASPECTS the frequency spectra are stored, since only the actual samples In this section, reference is made to the techniques used to are used for input, only the first half of the components need to recognize monophonic sounds and, on the other hand, the be analyzed. Each component of the frequency spectrum is digital interface of musical instruments (MIDI) is presented, related to the previous component, since it is the sampling which shows a notation that serves to discretize frequencies. frequency divided by the number of samples of the FFT [8]. A. Monophonic Sound Recognition In order to recognize monophonic sounds, the power or Due to in this article the monophonic sounds generated by amplitude spectrum (Xp) is analyzed, selecting the frequency f the guitar are taken as a basis, some algorithms that can be with the greatest amplitude, which is calculated through the used for the recognition of monophonic sounds are specifically sum of the squares of its real (Xreal) and imaginary (Ximag) mentioned. components, as shown in (1), 2 2 There are mainly two approaches that are used to perform Xp = Xreal(f) + Ximag(f) (1) monophonic sound recognition. One of them consists in The FFT algorithm used to develop the mobile application analyzing the signal samples in the time domain, and the other is Radix-2, which means that it must work on a group of in analyzing them in the frequency domain. A widely used samples whose number is a power of two [8]. method in the time domain is autocorrelation, which compares a signal with delayed versions of itself at successive intervals C. MIDI to find the highest amplitudes within the signal and measure Musical Instrument Digital Interface (MIDI) is a music the distances between them. Through these distances the period notation system that allows computers to communicate with of the wave can be inferred, and with it, the present frequency musical synthesizers. MIDI files contain instructions to create of the monophonic sound can be detected. On the other hand, the pitch, volume and duration of the notes, this based on a there is the analysis of signals in the frequency domain, where sequence of events called: note_on and note_off [13]. algorithmic implementations of the Discrete Fourier Transform are used, with which a set of frequency intervals and their Musical notes are not encoded by their names, instead amplitudes are obtained. A simple way to detect the frequency numbers from 0 to 127 are assigned as shown in Table I. For of the monophonic sound present quickly is to select the example, the number 57 corresponds to a musical note A with frequency with the greatest amplitude [9]. a frequency of 220 Hertz (Hz). In this article, the signals in the frequency domain are Equation (2) shows the transformation of frequency in Hz analyzed, so some algorithms that can be used to recognize the to MIDI note, monophonic sounds generated by the guitar in that domain are MIDINote = round(69 + 12 × log (f / 440)) (2) specifically mentioned. 2

410 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021

TABLE I. MIDI NOTES ASSOCIATED WITH ITS NAME AND FREQUENCY IN HZ [15]

Name MIDI note Frequency (Hz) Name MIDI note Frequency (Hz)

D 38 73.42 F♯/G♭ 66 369.99

D♯/E♭ 39 77.78 G 67 392.00

E 40 82.41 G♯/A♭ 68 415.30

F 41 87.31 A 69 440.00

F♯/G♭ 42 92.50 A♯/B♭ 70 466.16

G 43 98.00 B 71 493.88

G♯/A♭ 44 103.83 C 72 523.25

A 45 110.00 C♯/D♭ 73 554.37

A♯/B♭ 46 116.54 D 74 587.33

B 47 123.47 D♯/E♭ 75 622.25

C 48 130.81 E 76 659.26

C♯/D♭ 49 138.59 F 77 698.46

D 50 146.83 F♯/G♭ 78 739.99

D♯/E♭ 51 155.56 G 79 783.99

E 52 164.81 G♯/A♭ 80 830.61

F 53 174.61 A 81 880.00

F♯/G♭ 54 185.00 A♯/B♭ 82 932.33

G 55 196.00 B 83 987.77

G♯/A♭ 56 207.65 C 84 1046.50

A 57 220.00 C♯/D♭ 85 1108.73

A♯/B♭ 58 233.08 D 86 1174.66

B 59 246.94 D♯/E♭ 87 1244.51

C 60 261.63 E 88 1318.51

C♯/D♭ 61 277.18 F 89 1396.91

D 62 293.66 F♯/G♭ 90 1479.98

D♯/E♭ 63 311.13 G 91 1567.98

E 64 329.63 G♯/A♭ 92 1661.22

F 65 349.23 A 93 1760.00 Association (MMA) and the Japan MIDI Standards Committee Where round is a function of rounding to one digit, factor (JMSC) and first published in 1991 [16]. 12 is the resulting linear pitch space per octave, factor 69 is note A (440 Hz), which is taken as reference, log2 is used Within the general MIDI specification 128 sounds of according to the logarithmic pitch perception in humans and musical instruments or other types of sounds are included, and the variable f is the input frequency that will be converted to a these are divided in sections such as: Piano, Chromatic MIDI note [14]. Percussion, Organ, Guitar, Bass, Strings, Ensemble, Brass, Reed, Pipe, Synth Lead, Synth Pad, Synth Effects, Ethnic, General MIDI is a standardized specification for electronic Percussive and Sound Effects which are used in this mobile musical instruments that respond to MIDI messages. General application and are shown in Table II. MIDI was developed by the American MIDI Manufacturers

411 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021

TABLE II. SOUNDS OF THE GENERAL MIDI SPECIFICATION [16]

00 - Acoustic Grand Piano 32 - Acoustic Bass 64 - Soprano Sax 96 - FX 1 (rain) 01 - Bright Acoustic Piano 33 - Electric Bass (finger) 65 - Alto Sax 97 - FX 2 (soundtrack) 02 - Electric Grand Piano 34 - Electric Bass (pick) 66 - Tenor Sax 98 - FX 3 (crystal) 03 - Honky-tonk Piano 35 - Fretless Bass 67 - Baritone Sax 99 - FX 4 (atmosphere) 04 - Electric Piano 1 36 - Slap Bass 1 68 – Oboe 100 - FX 5 (brightness) 05 - Electric Piano 2 37 - Slap Bass 2 69 - English Horn 101 - FX 6 (goblins) 06 - Harpsichord 38 - Synth Bass 1 70 – Bassoon 102 - FX 7 (echoes) 07 - Clavi 39 - Synth Bass 2 71 – Clarinet 103 - FX 8 (sci-fi) 08 - Celesta 40 - Violin 72 – Piccolo 104 - Sitar 09 - Glockenspiel 41 - Viola 73 – 105 - Banjo 10 - Music Box 42 - Cello 74 – Recorder 106 - Shamisen 11 - Vibraphone 43 - Contrabass 75 - Pan Flute 107 - Koto 12 - Marimba 44 - Tremolo Strings 76 - Blown Bottle 108 - Kalimba 13 - Xylophone 45 - Pizzicato Strings 77 – Shakuhachi 109 - Bag pipe 14 - Tubular Bells 46 - Orchestral Harp 78 – Whistle 110 - Fiddle 15 - Dulcimer 47 - Timpani 79 – Ocarina 111 - Shanai 16 - Drawbar Organ 48 - String Ensemble 1 80 - Lead 1 (square) 112 - Tinkle Bell 17 - Percussive Organ 49 - String Ensemble 2 81 - Lead 2 (sawtooth) 113 - Agogô 18 - Rock Organ 50 - Synth Strings 1 82 - Lead 3 (calliope) 114 - Steel Drums 19 - Church Organ 51 - Synth Strings 2 83 - Lead 4 (chiff) 115 - Woodblock 20 - Reed Organ 52 - Choir Aahs 84 - Lead 5 (charang) 116 - Taiko Drum 21 - Accordion 53 - Voice Oohs 85 - Lead 6 (voice) 117 - Melodic Tom 22 - Harmonica 54 - Synth Voice 86 - Lead 7 (fifths) 118 - Synth Drum 23 - Tango Accordion 55 - Orchestra Hit 87 - Lead 8 (bass + lead) 119 - Reverse Cymbal 24 - Acoustic Guitar (nylon) 56 - Trumpet 88 - Pad 1 (new age) 120 - Guitar Fret Noise 25 - Acoustic Guitar (steel) 57 - Trombone 89 - Pad 2 (warm) 121 - Breath Noise 26 - (jazz) 58 – Tuba 90 - Pad 3 (polysynth) 122 - Seashore 27 - Electric Guitar (clean) 59 - Muted Trumpet 91 - Pad 4 (choir) 123 - Bird Tweet 28 - Electric Guitar (muted) 60 - French Horn 92 - Pad 5 (bowed) 124 - Telephone Ring 29 - Overdriven Guitar 61 - Brass Section 93 - Pad 6 (metallic) 125 - Helicopter 30 - Distortion Guitar 62 - Synth Brass 1 94 - Pad 7 (halo) 126 - Applause 31 - Guitar harmonics 63 - Synth Brass 2 95 - Pad 8 (sweep) 127 - Gunshot IV. METHODOLOGY The software requirements are first divided into several modules or features that can be built and delivered The mobile application was developed using the phases of incrementally. This is graphically represented in Fig. 2. the incremental Software Engineering model, which applies linear sequences in a staggered manner as the calendar of Returning to what Pressman and Mall [17, 18] point out, 3 activities progresses. Each linear sequence produces modules were created as part of the development of the mobile deliverable software increments [17]. The diagram in Fig. 1 application. describes the stages used for this development. In the first module, the recognition of the frequencies According to what Mall [18] points out, first of all a simple generated by the guitar was carried out, for which the Fast system is built and delivered which implements only a few Fourier Transform Radix-2 algorithm was applied, making use basic characteristics. During a few successive iterations, of the hardware and software resources of mobile devices. improved versions are deployed and delivered, until the desired When playing a musical note with the guitar, the fundamental system is finally realized. frequency corresponding to said musical note was obtained in real time, showing it on the mobile application interface.

412 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021

Fig. 1. Diagram of the Incremental Model.

example, a saxophone, a trumpet, a violin, among others; whereas when there is an absence of sound, the note_off event was activated to keep the mobile application silent. For the construction of all these modules, the Java programming language was used together with the Android Development Kit (SDK). A. Logical Block Diagram of the Structure of the Mobile Application

Fig. 3 shows the block diagram of the mobile application. Fig. 2. Incremental Software Development. As can be seen, an audio input is required, which goes through a sampling process to obtain the discretized signal, making use In the second module, the transformation of frequencies to of the audio recording tools offered by the Android operating MIDI notes was carried out, for which equation 1 shown in the system. Subsequently, the filter allows to eliminate the peaks MIDI section was applied, obtaining as a result, discrete values of the signal, which means, it eliminates the unwanted noise, of frequencies within a range from 0 to 127, showing the and then to apply the Fast Fourier Transform and thus obtain aforementioned transformation together with at the the fundamental frequencies. From these fundamental fundamental frequency in the mobile application interface. frequencies, the transformation to MIDI notation is carried out. From this moment, the sound chosen by the user can be The third module allowed to emulate the monophonic emulated through the execution of the note_on and note_off sounds of musical instruments or other types of sounds, based events. on the MIDI notes obtained. The note_on and note_off events of the digital interface of musical instruments were used to B. Operation of the Mobile Application execute and stop the MIDI notes. Consequently, the moment Fig. 4 represents the operation of the mobile application. In the user plays a musical note with the guitar, the note_on event order to use it, it is necessary to connect it to an electric or is activated, with which the sound chosen by the user within electroacoustic guitar through an audio interface, the last one is the 128 included in the mobile application must be heard; for connected to an audio output device such as an amplifier.

Fig. 3. Block Diagram of the Mobile Application.

413 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021

Fig. 4. Mobile Application Operation.

V. RESULTS The mobile application has been developed using the Java programming language in order to run on mobile devices with the Android operating system. The tests were carried out on a device from the Motorola brand, a Moto G7 Plus model with 64 GB of storage and 4 GB of RAM, with the Android 10 (Android Q) operating system installed. The work was carried out with a sample of three guitarists to carry out the tests of the operation of the mobile application. Two of the guitarists used an electric guitar and the remaining guitarist used an electro-acoustic guitar. The musical instruments were connected to the mobile application through an iRig 2 audio interface. The guitarists selected in the list of the main interface of the mobile application, several of the 128 sounds that were available to be emulated and they played the guitar obtaining the sound expected. Additionally, they were Fig. 5. Example of the Execution of a Musical Note a 440 Hz with the able to observe the frequency of the note played, as well as its Guitar using the Alto Saxophone Sound through the Mobile Application. transformation to a MIDI note in real time. The mobile application was subjectively evaluated with the feedback obtained from the guitarists, who pointed out the great usefulness of this mobile application because it broadens their possibilities of interpretation with their musical instrument, without requiring more than their mobile device and an audio interface. On the other hand, they mentioned that they noticed a slight latency between the moment they played the guitar and the emulation of the chosen sound, which could be reduced using a low-level programming language. Two examples of the mobile application in operation are shown in Fig. 5 and Fig. 6.

Fig. 6. Example of Playing a Musical Note a 440 Hz with the Guitar using the Acoustic Grand Piano Sound through the Mobile Application.

414 | Page www.ijacsa.thesai.org (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 12, No. 5, 2021

VI. DISCUSSION REFERENCES [1] REAL ACADEMIA ESPAÑOLA, “Diccionario de la lengua española”, The mobile application was developed with the Java 23.ª ed. [Online]. Available: https://dle.rae.es. [Accessed 6 April 2021]. programming language. Due to Java uses a virtual machine [2] R. Bennett, “Léxico de música”, Madrid: Ediciones Akal S.A., 2003. (Java Virtual Machine, JVM), which processes the instructions [3] Electro-Harmonix, “MEL9”, 2016. [Online]. Available: before being executed [19], a considerable latency was https://www.ehx.com/products/mel9. [Accessed 25 March 2021]. obtained in the results. To those who want to return to this [4] BOSS, “SY-1000”, 2019. [Online]. Available: article for their research, they are advised to use a lower-level https://www.boss.info/mx/products/sy-1000/. [Accessed 25 March programming language such as C or C++, with which the 2021]. latency would be reduced considerably and obtain better [5] ROLAND, “GR-55”, 2011. [Online]. Available: results, because the instructions are executed directly. Guitar https://www.roland.com/mx/products/gr-55/. [Accessed 25 March effects pedals commonly use recommended programming 2021]. languages [20]. [6] University College Ghent, “TarsosDSP”, 2019. [Online]. Available: https://github.com/JorenSix/TarsosDSP. [Accessed 8 March 2021]. VII. CONCLUSION [7] B. Farmer, “Midi Driver”, 2021. [Online]. Available: https://github.com/billthefarmer/mididriver/. [Accessed 8 March 2021]. The development of mobile applications has acquired great [8] R. Neuenfeld, M. Fonseca y E. Costa, “Design of optimized radix-2 and relevance due to the variety of areas where they can be used. radix-4 butterflies from FFT with decimation in time”, 2016 IEEE 7th As mentioned, there are applications that allow communication Latin American Symposium on Circuits & Systems (LASCAS), pp. 171- between people, planning travel routes, requesting food 174, 2016. delivery and entertainment, to name a few examples. [9] J. Strawn, C. Abbott, J. Gordon and P. Greenspun, “The Computer Additionally, the applications have allowed people to count Music Tutorial”, London: The MIT Press, 1996. with tools that provide the opportunity to explore, grow and [10] R. W. Heath, “Introduction to Wireless Digital Communication”, United develop in other areas such as music, which contributes to their States of America: PRENTICE HALL, 2017. [11] V. Montero, “Software para identificación de música”, Sevilla: comprehensive training. The mobile application presented in Universidad de Sevilla, 2020. this work benefits this aspect because the user has a synthesizer [12] D. Takahashi, “Fast Fourier Transform Algorithms for Parallel in the palm of his hand that is capable of emulating Computers”, Japan: Springer, 2019. monophonic sounds of different musical instruments or other [13] J. Jamrich, “New Perspectives on Computer Concepts”, United States of types of sounds with a guitar. This synthesizer, unlike the ones America: Cengage Learning, 2018. available on the market, is affordable, easily transportable and [14] P. Blanchard and D. Volchenkov, “Random Walks and Diffusions on usable anytime, anywhere. Applications as the one described Graphs and Databases An Introduction”, Germany: Springer, 2011. above allow the practice of music to be accessible to huge [15] R. Izhaki, “Mixing audio concepts, practices and tools”, Great Britain: number of people. Routledge, 2017. [16] M. Association, “General MIDI”, [Online]. Available: It has been considered as future work, to make this mobile https://www.midi.org/specifications-old/item/general-midi. [Accessed 6 application a polyphonic synthesizer, where to obtain the April 2021]. polyphony generated by the guitar, the frequency spectrum [17] R. S. Pressman and B. R. Maxim, “Software Engineering: A obtained by the FFT Radix-2 algorithm will be taken up, and Practitioner's Approach”, New York: Mc Graw Hill Education, 2019. search and decision mechanisms will be used. It is also planned [18] R. Mall, “Fundamentals of Software Engineering”, Sonepat: PHI to adapt the mobile application to a lower-level programming Learning Private Limited, 2018. language like C++ for lower latency. [19] Oracle, "Java Virtual Machine Technology", [Online]. Available: https://docs.oracle.com/javase/8/docs/technotes/guides/vm/index.html. ACKNOWLEDGMENT [Accessed 6 April 2021]. [20] B. Holmes, “Guitar Effects-Pedal Emulation and Identification”, Belfast: This work was supported by Instituto Politécnico Nacional, Queen’s University Belfast, 2019. COFAA, EDD and SIP. (Project SIP20200832).

415 | Page www.ijacsa.thesai.org