Coding for sight and sound Coding for sight and sound In the last of a three part article exploring artistic practices using Free Software, Martin Howse examines the specifi c concerns of software artists creating real time visuals semi-interpreted scripting languages such as Python, Lisp or Scheme are defi nitely out of the question in this hard core real time arena, but such limitations can often produce artistic workarounds which re-examine computational methodologies. On the other hand, technical issues do perhaps stifl e the fl exibility of some of the apps we’ll examine, in contrast to the supremely JACKed or piped pluggability of Free Software audio. Thankfully, Free Software initiatives such as the Livido project (see Jacking Video), launched during last year’s Piksel conference at Bek in Norway, aim to bring such fl exibility into video work, seriously raising the Vision On game for video software on open platforms. TOASTED early all the artists whose work we’ve Early artists, such as Woody and Steina Vasulka, delved into in previous installments, working with video in the 60s and 70s confi ned Nstraddle both audio and visual realms, their experiments purely to the analogue realm. feeling equally at home with both media and And artist-coders working outside budgets cross-pollinating or feeding through visual as Farmer’s Manual, any audio or visual output which could stretch to buying time on a sweet sources and structures with the physical and is considered merely as the rather irrelevant by blue fridge-sized SGI box, would have to wait audible. If, as Randall Packer, artist and theorist, product of systems and processes. until the birth of the Amiga Video Toaster (now argues, contemporary multimedia work Despite this fl attening of fundamentally open sourced) for affordable real-time video expresses Wagner’s notion of the different materials under the wheels of machinic manipulation. Indeed, the multimedia Gesamtkunstwerk (or total work of art), then process or watchword of raw data, it’s capabilities of the well designed Amiga, opened Free Software coder-artists are defi nitely masters important to assess the specifi c demands and up the way for a whole new generation of artists of this form. As we saw last month, Erich concerns of hacking visuals. Video does admit to begin working creatively with visuals. Around Berger’s Tempest project, making excellent use of a different way of working, and presents both low cost machines such as the Amiga, an of GEM (a graphical extension to Pd, or Pure artistic concerns and technical demands which underground movement, known as the Data, the supremely extendible artistic perhaps make video a tougher medium to work Demoscene, emerged, providing a platform and environment), presents one extremely novel, with successfully. Above all, real time video still conceptually clean integration of both audio presents quite specifi c technological challenges and visual material, using electromagnetic when it comes down to grabbing, processing Just how do radiation from the monitor displaying images to and throwing out live material. The sheer generate live sound. Indeed, in the case of volume of data which uncompressed video artists interact artists such as ap (Martin Howse, Jonathan presents at reasonable resolutions and colour with computers Kemp), or applications such as Pd, little, if any, depth, coupled with demanding, complex and what are the differentiation is made between the two fi elds, mathematics performed across such large data with sound and video material handled solely as sets, make of video coding a daunting task. The most creatively data. And in heavily code-based initiatives such 25 frames per second clock is always ticking in fulfi lling means? the developer’s head, and her code quite simply has to keep up with this frenetic pace. Hardware can only do so much, and artist-coders need to be up-to-speed with specifi cations and instruction sets, pushing machines to the limit with tricksy low-level code. More abstracted, 52 LinuxUser & Developer LinuxUser & Developer 53 Coding for sight and sound Coding for sight and sound but successful nonetheless. Live coders, grouped JOCKEYING FOR POSITION Farmers Manual push Pd extension GEM to the To quickly recap, Pd presents a visual under the TOPLAP (Live Algorithm If the digital audio sequencer is the rather dull limit, fl attening audio and visual data in glitched environment allowing artists to connect math, virtuosic performance Programming and a Temporary Organisation for mainstay of every sound studio, then Non control and generator objects in complex its Promotion) banner, have also explored the Linear Editors (NLE) are the bread and butter of arrangements called patches which represent a use of the early children’s programming video work; a solid dependable tool which is work in progress. Patches can easily be re- Artists are always bound language, Logo, devised by Seymour Papert for used by necessity rather than for true artistic edited, re-run, and controlled by a huge range the production of live visuals. Dave Griffi ths, a exploration. As such, apps such as the ambitious of external and networked sources. A fl attened to some degree by major player on the SuperCollider scene, uses Cinelerra, coded more or less solely by Andraz approach to audio, video and all data is the key The 25 frames per available technologies Scheme, a relative of Logo via a shared Lisp-like Torri, unfortunately fall outside the remit of our to Pd, providing artists with the essential ability second clock is always grammar, to produce somewhat formalist, live investigation. However, as a good many natural to throw stuff together, connect disparate data coded 3D work within his own Fluxus NLE operations, such as transitions, equally well sources to complex processes, control structures ticking in the developer’s community for coders to show off skills and environment. Logo’s turtle graphics and apply to live video processing, there does seem Throw nearly 100 mixes, effects and frame and outlets. Pd itself is more or less exclusively head, and her code quite ideas. The particular aesthetics and concerns of instruction set, designed to command a robot to be a growing trend towards crossover of NLE blends into the toolkit, alongside chaining with audio angled, though of course this doesn’t simply has to keep up this scene shine through in the work of or screen-based turtle to draw often recursive and live video apps, particularly suites which are EffecTV and VeeJay adds up to a powerful stop artists projecting their own patches, as contemporary coders such as Jaromil of FreeJ patterns, crop up again in SuperCollider and geared more around the art of VJing; mixing, package which also boasts a decent level of they re-patch live, or even making use of Pd’s with this frenetic pace fame. Video coding is still very much a skilled, this neat, low tech approach also fi ts easily with matching and creating visual material as an control, with a handy hotkeyed console and decidedly old school graphing functions. virtuosic affair. Tom Schouten’s work on Packet Forth, within adjunct to DJs or live electronic audio. Andraz remote access using either its own VIMS Until recently GEM, kicked off by Mark Danks objects into the mix. Objects are logically Even modern hardware struggles to keep PDP, which he describes as “turtle graphics on Torri is heavily involved in the Piksel initiative, protocol, or OSC (Open Sound Control). and now developed primarily by Gunter Geiger divided into controls, which includes mouse, pace with the complex data manipulations steroids.” and the LiVES (LiVES is a Video Editing System) FreeJ, from master coder Jaromil of dyne:bolic and IOhannes m zmoelnig, was the choice for keyboard and tablet access. The Gem-window dreamt up by today’s artist-coders. Artists are application, coded by Gabriel Finch, aka fame, is another such more accessible, VJ real-time graphics under Pd. GEM throws in a object is also here, and this must be used to always bound to some degree by available HIGH LIVIDO Salsaman, is another case in point. LiVES started oriented app, which does offer a pleasant old full multimedia toolkit, but does impose some provide the rendering context for every patch. technologies, pushing hard against these Technical constraints can work wonders for life as a simple, small GNU/Linux NLE, and it school approach to the problems of working limitations, dividing the audio and video worlds Gemhead specifi es the start of the rendering barriers and in some instances breaking through artistic creativity, and with an often common does still fulfi ll this function admirably, offering a live with visuals. The latest FreeJ 0.7 iteration in a slightly unhealthy manner. Sure, users can chain. Manipulator objects provide for such to fi nd novel ways of working which then feed code base and set of programming concerns decently intuitive GUI with cut and paste presents a range of interfaces to the well written control one medium from the other, but this things as scaling and vector translation and self back into visualisation or creative industries. shared by Free Software artist-coders, it’s less facilities, and support for a good range of input underlying realtime video manipulation engine. doesn’t add up to a pure data approach. That explanatory Geos offer building blocks such as Free Software and a shared codebase supports likely that the bigger artistic picture will become and encoded formats. However, to some extent A GTK2 interface is accessible, but not said, artists such as Erich Berger, make good use spheres or polygons. You’ll also fi nd plenty of this sometimes symbiotic relation, which shares obscured by bits, bytes and low level LiVES has recently morphed into something of a recommended for live use.