gusdevpack: A Python-Based Framework for Audio and Multimedia Newbies Renato Fabbri Marília F. Chiozo NICS, IA, Unicamp IC, Unicamp Rua da Reitoria, 165 Av. Albert Einstein, 1251 Cidade Universitária Zeferino Vaz Cidade Universitária Zeferino Vaz CEP 13091-970 Campinas/SP, CEP 13084-971 Campinas/SP, Brazil, Brazil, [email protected] [email protected]

Abstract such real-life applications is called figus. Doc- In this work we present a cross-platform open- umentation is provided as part of fdp for wrap- Python-based framework for audio and multimedia. per classes and standalone methods of the li- This framework, named gusdevpack, is designed brary, describing their technical aspects such as for reducing application development time and ac- the meanings and domains of method arguments quainting new users with existing resources for mul- and data members. The tutorials, accessible timedia development. It is structured around a from the main fdp frontend along with the ex- wrapper , example applications, tutorials and ample applications and the documentation, are documentation. meant to walk the user through the process of Keywords writing complete applications and also serves as a guide to the documentation for users who are Python, audio/multimedia development not familiar with the wrapper library. SoniPy[4] is an independently developed 1 Introduction project that is similar to ours when one com- To regular users of multimedia creation soft- pares current end products, yet it holds a fun- ware, making the transition to a fully scripted damental dierence in purpose. While fdp is environment such as Python can be a gruelling aimed primarily at the novice , with task. The reasons behind such transition usually secondary goals such as assisting the process of ranges from curiosity to lack of elegant methods learning how to program and forming a com- to achieve an intended goal. Giving up the tran- munity of users with a common speci interest sition halfway through due to time constraints (audio programming), SoniPy targets develop- or loss of interest is not unheard of. ers in a more general way, placing more empha- Although there are eorts focused on teach- sis on the tools themselves than on the users' ing the necessary programming concepts to non- backgrounds. (such as [1] and [2]), existing li- braries of interest in multimedia development 2 Problem Description have, at times, a steep learning curve, and re- In developing audio applications, we encounter quire the user to skim through several pages three main sub-problems that demand special of documentation before actual usage details of attention: 1) choosing a suitable approach to classes and methods become clear. This process sound manipulation for the task at hand; 2) de- is tiresome and inecient. ciding how the user is to control the chosen ap- Our work intends to address this issue. On proach; and 3) deciding on a sound rendering the most basic level, a wrapper library aims method. to provide simpler interfaces to existing mul- Within fdp a developer should be able to ma- timedia libraries. We call our wrapper library nipulate standard Python data structures and, guslib, which is distributed as part of gus- simultaneously, be familiar with at least some devpack (or simply fdp). This fdp is centered basics about the possibilities of the framework on tutorial windows and example applications being employed. As soon as these goals are and code. Among the included example applica- achieved, the user should be able to conceive tions, one can nd, along with programs written and manipulate appropriate data structures for for the sole purpose of showcasing the wrapper the intended functionality. library, use cases drawn from real-life applica- As control interfaces, fdp oers, besides stan- tions, such as the guslib port of [3]. The set of dard command-line control, gui interfaces (pro- vided by the cross-platform wxPython pack- with, a smooth learning curve, easily-accessible age), optional input and visual feedback tools and an existing user base in the eld of through Pygame itself or WxMpl1. Mirra-based multimedia development. As is the case with visual feedback is planned for the near future, other interpreted languages, Python bindings providing 2D graphics via OpenGL. for existing c/c++ libraries can be obtained by Finally, fdp presents osc-controlled ChucK tools such as swig, which automate most of the and CSound as sound rendering backends for process of writing bindings. The presence of a the multimedia developer to work with. osc command-line interpreter enables new users to interfacing is achieved through the SimpleOSC experiment with small segments of code with- package, and CSound has its own Python bind- out going through the entire process of prepar- ings in the form of the Csnd package. The use of ing a source le and compiling it, which is often osc interfaces allows for future expansion such required from users of languages such as Java that fdp support SuperCollider and PureData as and c/c++. Additionally, as a dynamically backends. typed language with an automatic garbage col- lection scheme, it enjoys the status of a language 3 Language Choice that does not force programmers to be particu- The philosophy of code reuse, which has been di- larly attentive to variable denitions and mem- recting programming language eorts since even ory allocation issues, both of which can be some- before the advent of object-oriented program- what overwhelming to novice programmers. Be- ming, is satisfactorily embodied by the Python cause we focus especially on developers with lit- programming language. Through its use, devel- tle or no programming experience, choosing a opers are able to wrap code fragments as compo- language with a steep learning curve would de- nents that can then be reused by other Python feat the very purpose of our work. developers, who employ the language as the log- In the eld of multimedia development, ical glue that holds components together and Python benets from being a market trend. A gives them a larger purpose. However, Python larger user base means higher development ac- certainly is not the only language to possess such tivity, which translates into faster evolution, traits. This raises the question of why we chose more innovations and quicker response to bugs. it over other available languages. This factor, added to endorsement by com- The language is by no means perfect. Several mon distributions such as Fedora, makes of its downfalls are a result of its interpreted Python a language that is easily available, and nature and shared with other interpreted lan- is so at its prime. guages in some level. Speed is one such down- 4 Components and Tools fall. By foregoing compilation (in the sense it goes for compiled languages), interpreted lan- At present moment, fdp deals with the following guages place that overhead on load time, exe- components: cution, or both. Yet, from a novice's point of • NumPy, adds fast and sophisticated array view, undesirable though it might be, a sacrice facilities to the Python programming lan- of program speed is often preferrable to a sac- guage. [5] rice of programming speed. A more specic issue that impacts speed is memory manage- • Audiolab, allows importing and exporting ment. Automatic garbage collection, for exam- between sound les and NumPy arrays. [6] ple, while having the obvious advantadge of cut- • Chuck, a programming language for real- ting down on a large number of common mem- time audio. [7] ory leaks that are disencouraging to novices, the • Csnd, provides a Python interface to the extreme focus on spatial optimization of some Csound . [8] implementations might hurt execution speed in unacceptable ways. There is little that can be • osc Protocol, allows multimedia devices to done about this issue other than relinquishing share music performance data in real time some control to the user. over a network. [9] On the other hand, Python combines a size- • SimpleOSC, provides a simple way to use able number of libraries for developers to work osc Python implementation. [10]

1The WxMpl module provides wxPython gui ele- • Zombi, denes a very basic set of widgets ments with embedded Matplotlib graphics. that sends out osc messages. [10] • wxPython, a cross-platform wxWidgets for over ve years by c/c++ programmers as a wrapper for the Python programming lan- cross-platform api that provides high-level ac- guage. [11] cess to services such as those oered by Mi- crosoft's low-level multimedia api, DirectX. sdl WxMpl, provides interactive plotting capa- • can be used to handle input (mouse, joysticks bilities via the Matplotlib 2D plotting li- and the keyboard) and accelerated graphics. For brary. [12] audio, Pygame oers few and intuitive capabili- • Pygame, a set of modules written on top ties such as playback and audio cd-rom access. of the sdl library and designed for writing games. [13] 4.1 IPython, Easy Install, Editors IPython, an enhanced Python shell. [14] Seeing as this project targets end-users, it is our • concern that these users have a starting point • Easy Install, allows automatic download, when it comes to choosing programming tools building and installation of Python pack- such as interpreters and editors. ages. [15] As of the current fdp version, we strongly recommend the use of IPython as an inter- • Editors: Kate [16] (Linux), Notepad2 [17] (Windows). preter, EasyInstall and the editors Kate (on Linux) or Notepad2 (on Windows). We intend • guslib, a wrapper library targeting audio to include options for automatically installing and multimedia. IPython, and Kate or Notepad2, while EasyIn- stall is already bundled with fdp due to being • figus, a collection of applications built around guslib. our Python module installer of choice for this project. • fdp, is guslib, figus, documentation and tutorials. 4.2 guslib, figus, fdp For cases where the user needs more spe- guslib is a wrapper library built around se- cic mathematical functions, NumPy is avail- lected functionalities of the libraries listed able. Furthermore, through NumPy it is possi- above. It is a structured pool of contributed ble to use SciPy tools, such as signal processing code segments striving towards ease of use of Scikits2. Audiolab, recently added to Scikits, fdp components within audio and multimedia enables users to quickly import and export be- applications. tween NumPy array objects and dierent sound figus is a set of applications using guslib le formats. and fdp components. This set is comprised While Chuck and Csnd target audio ren- of developers' and users' contributions, such as dering, osc, accessible from Python via Sim- open source applications using guslib and ex- pleOSC, is a dedicated protocol for communicat- ternal multimedia modules. ing between compliant applications.3 Addition- fdp is guslib and figus together with tuto- ally, Chuck and Csound can be accessed using rial texts and scripts. These resources are in- the os.system method from Python Standard tended to be presented as wizards, in clean Library. [13] step-by-step expositions, building a knowledge These methods of communicating with sound base around the usage of guslib and external renderers are readily compatible with Python multimedia modules not yet wrapped by gus- itself, therefore gui and graphical utilities can lib. These texts, scripts and wizards are being potentially be interfaced with audio  and video designed to be collaboratively expanded upon  renderers. by guslib developers and users. The Pygame package provides a Python in- Summarizing, these three pieces of software terface to Simple DirectMedia Layer (more com- exist for mainly one purpose: aggregating con- monly known as sdl), a general purpose library tributions of developers and users to compo- for multimedia applications that has been used nents of fdp.

2Scikits are independent projects that are too special- 4.3 Example ized to live in SciPy itself. [18] As an example of some of the components in- 3osc also acts as the core of the dssi plugin api, an evolution of the ladspa api. ladspa and dssi are linux volved, we present the following code, which api meant for audio processing and synthesizing. captures mouse data from a wxPython object and sends it by means of OSC messages to an vice as a sound synthesis controller can a OSC-aware application. more complex application. The documentation # Importing wxPython and simpleOSC to and examples bundled with fdp aim to enable the s c r i p t a user to reuse code similar to this after skim- import wx , osc ming through only a few documentation topics that are directly related to the intended task. class MyFrame(wx . Frame ) : For a proper introduction to the libraries used """Subclass of Frame in which we by fdp, we intend to provide practical exam- read mouse ples and documentation covering from uses with positions (X, Y) and send them barely any integration between components to through simpleOSC uses that fully integrate several components in to localhost , port 9000, address / s i n u s """ order to perform complex tasks.

def __init__( self ): 5 fdp Focus on End-User wx.Frame.__init__(self , None, −1, Although guslib relies mainly on third-party "My Frame", size = (600, 500)) Python packages that generally target devel- panel = wx.Panel(self , −1) opers, fdp is designed with end-users in mind. # Binding the mouse movement event Well-documented example scripts exploring dif- : wx .EVT_MOTION ferent functionalities provided by guslib, links # to the "OnMove" function of our to related projects and further documentation MyFrame c l a s s aims at guiding new users through the transition panel . Bind (wx .EVT_MOTION, s e l f . to Python multimedia development, as long as OnMove) they are willing to read and experiment. Even so, the idea of providing non- wx.StaticText(panel , −1, "Pos : " , programmers (or those new to Python mul- pos = (10, 12)) timedia development) with a well-documented self .posCtrl = wx.TextCtrl(panel , framework can be helpful at usage stages −1, "", pos = (40, 10)) later than introduction. The complexity of def OnMove(self , event): most libraries nowadays requires even seasoned # Getting the position of the programmers to visit the documentation and mouse move event look at code examples often, save for a few pos = event.GetPosition() frequently-used fragments of code. It is our in- # Setting its values on the tention that the fdp framework relieve users of TextCtrl object self.posCtrl the need to mentally index documentation and self .posCtrl.SetValue("%s, %s" % ( pos.x, pos.y)) code examples by adopting an organization in- tuitive enough that even rst-time users should # Sendind the position values be able to hit the desired section of the docu- through OSC mentation with few or no misses at all. # To localhost, port 9000 ( simpleOSC default for sending) 6 Cross-Platform Open-Source # and /sinus address (i.e. what is Software Approach requested of the server) osc.sendMsg("/sinus", [pos.x, pos. The components used in the development of fdp y ] ) are open-source, as will be fdp itself as of its ini- tial public release. Source code documentation serves not only the purpose of aiding the main- i f __name__ == '__main__ ' : tenance cycle, but it also aims to let users learn osc . i n i t ( ) from the very source code of fdp. This is true of app = wx.PySimpleApp() the guslib wrappers as well as of the example frame = MyFrame ( ) applications and the fdp frontend. frame .Show(True) app.MainLoop() The open-source license allows users to con- tribute to the documentation and code, assum- With an OSC-aware sound synthesis appli- ing a cooperative work stance towards develop- cation, this script could be a starting point ers. This is also a design goal of fdp as a whole: from where a user wishing to use the mouse de- inviting users to experiment with new, unfore- seen combinations of guslib components and al- http://lcpd.ime.usp.br/~marioct/ lowing them to play a role that goes one step anais-sbcm2007-sem-capa.pdf. further from simple tests and bug reports. [4] Overcoming Software Inertia in Data 7 Conclusions and Further Work Sonication Research Using the SoniPy Framefork, 2007. Available at: At its current development stage, the fdp eort http://www.avatar.com.au/sonipy/ is focused on providing the end-user with a com- Worrall_SonipyICoMCS07.pdf. prehensive set of tools (in the form of packages), documentation and examples. Once the bare [5] Numpy, 2007. Available at: essentials have been covered, we will move to- http://numpy.scipy.org/. wards broadening the set of choices given to the [6] David Cournapeau. Audiolab, 2007. Avail- user in dierent areas. As mentioned in previous able at: sections, these include visual feedback through http://www.ar.media.kyoto-u.ac.jp/ Mirra, a OpenGL-based 2D graphics package, members/david/softwares/audiola%b/. and the option to use SuperCollider and Pure- [7] Ge Wang and Perry Cook et al. Chuck web- Data as sound rendering backends. Another in- page, 2007. Available at: clusion that has been discussed by the develop- http://chuck.cs.princeton.edu/. ers is ScyPy integration. The SciPy library is [8] Introduction to using csnd: the python built to work with NumPy arrays, and provides wrapper for csound, 2007. Available at: several user-friendly and ecient numerical rou- http://www.avatar.com.au/sonify/ tines such as routines for numerical integration csnd. or optimization. Collecting voluntary user feedback and adapt- [9] The Center For New Music and Au- ing fdp to meet the needs of users will become dio Technology (CNMAT). opensoundcon- of vital importance in the development process. trol.org, 2007. Available at: This is especially true in the case of new users http://opensoundcontrol.org/. who have not yet been exposed to the exist- [10] ixi audio. Ixi backyard, 2007. Available at: ing resources available to assist the transition http://www.ixi-software.net/ to Python multimedia development. The ini- content/backyard.html. tial public release of fdp is scheduled to mid- [11] homepage, 2007. Available at: February, 2008, as an open-source project un- http://www.wxpython.org. der the General Public Licence (gpl), hosted at SourceForge4. [12] Ken McIvor. Wxmpl, 2005. Available at: http://agni.phys.iit.edu/~kmcivor/ 8 Acknowledgements wxmpl/. Our thanks go to the developers of all soft- [13] Pygame homepage, 2007. Available at: ware used in this project, as well as to Professor http://www.pygame.org. Adolfo Maia Júnior. Special thanks go to Rohit [14] Ipython, 2007. Available at: Gupta for all the invaluable tips. http://ipython.scipy.org/moin/. References [15] Easy install, 2007. Available at: [1] Renato Fabbri and Fábio Furlanete. http://peak.telecommunity.com/ Python for audio manipulation, 2007. DevCenter/EasyInstall. Partially presented at the 5th Linux Audio [16] Anders Lund, Christoph Cullmann, and Conference. An ongoing Portuguese version Joseph Wenninger. Kate homepage, 2007. is available at: Available at: http://estudiolivre.org/tiki-index. http://kate-editor.org/. ?page=python-e-som-tutorial%. [17] Florian Balmer. Notepad2, 2007. Available [2] Dive into python, 2007. Available at: at: http://www.diveintopython.org/. http://www.flos-freeware.ch/ notepad2.html. [3] Applications of Group Theory on Granular Synthesis, 2007. Available at: [18] Scikits, 2007. Available at: http://scipy.org/scipy/scikits/wiki. 4http://sourceforge.net/projects/fdpack/