Networked Music Workshop ICMC 2005, Barcelona

Presented by: ∏lvaro Barbosa (Pompeu Fabra University,M TG), Scot Gresham-Lancaster (Cogswell College Sunnyvale, CA), Jason Freeman (Georgia Institute of Technology), Ross Bencina (Pompeu Fabra University, M TG)

Participants in this workshop will learn about different types of networked music practice and about tools and techniques which are available to undertake these tasks. The focus will be on the technical, compositional, and aesthetic challenges involved in realizing networked music on local area networks and over the Internet, using both peer-to-peer and client-server networking models. The workshop will discuss systems intended to lead non- musicians towards creative expression, as well as systems for practicing musicians to extend the boundaries of performance.

After a broad overview of historical projects, like from the work of the League of Automatic M usic Composer and the HUB from the 1970's and 1980's and other important approaches to networked music, the workshop will focus on case studies of particular projects and the tools they use. Ross Bencina's network transport infrastructure, OSCgroups, which uses a centralised name lookup server and peer-to-peer data interchange via Open Sound Control, will be discussed in connection with recent work by Scot Gresham- Lancaster and "The Hub." Phil Burk's TransJam, a Java-based server for real-time collaboration over the Internet, will be explored in connection with the Auracle networked sound instrument. Specific issues to be addressed include the logistics of event coordination, the mediation between transparency and complexity in the system, the handling of timing and latency issues, human interface design, and the maintenance and monitoring of client reliability.

ICMC05 Networked Music Workshop part #1.introduction

Álvaro Barbosa [e-mail: [email protected]]

Music Technology Group, Audiovisual Institute Pompeu Fabra University – Barcelona, Spain

Send me an Email for the presentation!!!

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Networked Music Books

[1] At a Distance, MIT Press, Cambridge MA, 2005. Chapter: “Computer Network Music Bands: a History of the League of Automatic Music Composers and the Hub” by Chris Brown and John Bischoff

[2] VIRTUAL MUSIC . Routledge, New York, 2005. “How the web got Wired for Sound” by William Duckworth

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Networked Music Survey Articles

[1] A. Barbosa, Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation. Leonardo Music Journal, Volume 13, Issue 1, pp. 53-59, MIT Press, Cambridge MA, 2003.

[2] G. Weinberg. The Aesthetics, History, and Future Challenges of Interconnected Music Networks. Proceedings of the International Computer Music Conference (ICMC02), pp. 349-356, 2002.

[3] D. Tanzi, Observations about Music and Decentralized Environments. Leonardo Music Journal, Volume 34, Issue 5, pp. 431-436, MIT Press, Cambridge MA, 2001.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Classification Space (Barbosa 2003) based on Rodden’s environmental Facets (Rodden, T. 1992)

Music Composition Support Systems (Project Oriented)

Shared Sonic Environments

Co-Located Musical Networks Remote Music Performance System

Local Remote

LOCATION [ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

The User’s Point of View

From the preliminary discussion during this WS preparation Scot pointed out:

“There are two distinct streams here. One that is regarding network music that is intended for a general and possibly non-musician user, and then techniques for practicing musicians to extend the boundaries of performance. (…) in the context of the workshop they need to be addressed within their separate contexts.”

1. Systems and techniques for practicing musicians I Will Discuss issues related with Network Latency

2. Systems for a general and possibly non-musician user I Will Discuss The concept of Shared Sonic Space and the Public Sound Objects Project (PSOs)

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Network Latency and Network Acoustic

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 For the Human ear to perceive two simultaneous sounds, they should not be displaced in time over 20ms (Hirsh, 1959), which means that for mutual awareness to be supported in a bilateral performance this threshold would be around 40ms

9 The perception of two different sounds performed simultaneously is strongly dependent on sound characteristics (timbre, pitch or loudness), music style and other feedback types, such as visual or physical stimuli. Nevertheless, a 40 ms threshold is reasonably high enough to characterize a worst-case scenario

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 This value is coherent with the results from research carried out in 2002 at Stanford University by Natham Shuett (Schuett, 2002), whish experimentally defined an Ensemble Performance Threshold (EPT) for impulsive rhythmic music lying between 20-30 ms, as well as the outcome from research carried out by Nelson Lago in 2004 (Lago and Kon, 2004) at São Paulo University, whish achieved similar results.

9 Considering a Music Style based on Rhythmic Synchronization, at a Global level Latency will always be an impediment for real time musical communication over computer networks.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 If we consider, as an example, a peer-to-peer connection between Santiago do Chile and Moscow, we have an approximate distance of 14.141 Km. Even with data transfer at the speed of light (299.792,458 Km/s) and unlimited bandwidth, bi-directional latency would be approximately 94,3 ms, which is much higher then the minimum tolerable threshold.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 Furthermore, latency has a highly variable and unpredictable nature, creating time base errors, de- sequencing and even partial loss of content, resulting in a severe condition for performance control.

9 And emergent Mobile Technology has slower transfer rates

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 A number of experiments have been carried out with the purpose of determining the maximum amount of communication latency which can be tolerated between musicians in order to keep up with a synchronous performance. Example: Discovery Channel Interview The Real On-Line Concert Stanford-McGill's

9 in June 2004 at the Sound and Image Department from the Portuguese Catholic University conducted a new experiment aiming, amongst other goals, to study the relationship between Tempo and Latency.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 In the experiment, simulated network latency conditions were applied to the performance of four different musicians playing jazz standard tunes with four different instruments (Bass, Percussion, Piano and Guitar)

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Self-Test for latency tolerance in individual performance

250 Musician #1 - Bass Musician #2 - Percussion Musician #3 - Piano Musician #4 - Guitar

200

150 Latency (ms)

100

50

80 100 120 140 160 180 200 220

Tempo (BPMs)

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 A second part of the experiment addressed the usage of a Delayed Individual Feed-Back Topology, Which allows latency tolerance up to 50ms with instrumental practice.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 Individual Delayed FeedBack is being applied in Alexander Carrot’s SoundJack Application and in the commercial System eJamming

http://soundjack.jxta.org/ http://www.ejamming.com/

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

9 net-delay can also be faced as a natural element when creating music over the internet 9 Think of net-delay is the particular acoustics of Internet.

Atau Tanaka’s Interview from Crossfade http://crossfade.walkerart.org/

Chirs Chafe’s SoundWIRE Group at CCRMA Sound Waves on the Internet from Real-time Echoes

http://ccrma.stanford.edu/groups/soundwire/

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Shared Sonic Spaces

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Community oriented Shared Virtual Environments (SVEs) in the context of Sonic Arts.

➼An aspect of sonic collaboration over the internet, that goes beyond the idea of enhancing the established acoustic communication paradigms.

➼Some Examples: The WebDrum (Burk, P. 2000), MP3Q (Tanaka, A. 2000), Auracle (Neuhaus, M. 2004).

➼Issues can be raise regarding the time scale of shared events on the internet (permanent events???)

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

PUBLIC SOUND OBJECTS (PSOs) An ongoing Public Event in the internet ➼Most public artistic proposals designed for community performance have been developed towards an event that takes place at a specific date during a certain period of time, in the presence of a physical and/or virtual audience in a theatre-like experience. ➼Internet provides permanent connectivity, thus a public Internet event can go on permanently. ➼In this system the audience and performers will be able to dynamically join and leave the performance at any time they want. (Video - PSO.mov)

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

PSOs CLIENTS

System Architecture WEB BROWSER Streaming Audio Client PSOs SOUND INTERNET OBJECTS Controller Interface SERVER DATA-BASE ➼In the Preliminary Prototype the interaction feedback will be STREAMING AUDIO SERVER auditory only and it will convey the WEB BROWSER

Streaming Audio Client performance of all the connected HTTP-SERVER users at a given moment with the Controller Interface respective latency due to network INTERACTION speed limitations. SERVER (...) LOCAL VISUAL ➼ WEB BROWSER REPRESENTATION In the Public installation Site the ENGINE Streaming Audio Client soundscape will have added Controller Interface spacialization, and visual representation of the users performance. Performance Commands (Discrete Connection triggered by client events)

Global Audio Performance Public Installation Site (Continuous Streaming Connection)

[ABARBOSA Sept.05] ICMC05Audio BroadcastNetworked Music Workshop part #1.introduction

The Interface Layer ➜The user to contributes to the musical performance by transforming the characteristics of a visual Sound Object representation, sending normalized parameters over the network to the synthesis engine’s transformers. ➜The GUI should be a behaviour driven metaphorical interface, avoiding a flat mapping of parameters (inspired by Kiyoshi Furukawa’s Small Fish). ➜The Sound Object visual representation has an automatic periodical behaviour that is The ball is moving continuously and the conducted by the user. user can manipulate its size (1), its speed and direction (2), each wall’s acoustic texture - Pitch (3), Ball Tail -Delay (4) and Ball Speed Reduction According to Network latency towards the server (5).

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

An ongoing Public Event in the internet 9The first system prototype is now working for nine simultaneous Sound Objects, and it has a good feel of playability even at highly remote locations, provided by the network adaptive tempo (speed). 9 http://www.iua.upf.es/~abarbosa/ and http://soundserver.porto.ucp.pt/pso 9From the trial installation in October 2004, over a 100 inquiries where obtained from system users. 9Future versions will include the possibility to upload user files and an optional client synthesis engine to provide local individual feed-back.

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

[ABARBOSA Sept.05] ICMC05 Networked Music Workshop part #1.introduction

Thank You !!!!!

[ABARBOSA Sept.05] TwoTwo PlacesPlaces atat OnceOnce

UsingUsing LANLAN andand WANWAN NetworksVideoNetworksVideo ChatChat TechnologyTechnology forfor LiveLive PerformancePerformance ofof MultidisciplinaryMultidisciplinary ArtsArts EventsEvents

ICMCICMC workshopworkshop SundaySunday Sept.Sept. 4th4th 20052005 HistoricalHistorical PerspectivesPerspectives -- AntiphonyAntiphony „ GabrelliGabrelli atat St.St. MarksMarks inin VeniceVenice „ Auditory dislocation „ Metaphysical representation of spirit and body BroadcastBroadcast TVTV

„ EdwardEdward R.R. MurrowMurrow “You“You AreAre There”There” OneOne wayway experienceexperience ofof simultaneitysimultaneity

DeeplyDeeply feltfelt connectionconnection withwith thethe MoonMoon LandingLanding inin JulyJuly 19691969 EATEAT (Experiments(Experiments inin ArtArt andand Technology)Technology)

„ ActiveActive electronicselectronics inin experimentalexperimental sonicsonic environmentsenvironments usingusing speakerspeaker displacementdisplacement andand thethe interactioninteraction ofof electronicselectronics inin livelive performanceperformance TheThe HUBHUB

„ InteractiveInteractive ComputerComputer NetworkNetwork MusicMusic GroupGroup „ Clocktower „ HubRenga „ Points of Presence

Indigenous to the Web GeorgGeorg HaijduHaijdu

„ Quintet.net „ New performance practices

„ Local versus global

„ *Each concert produces five distinct versions of the piece with local flavor added to each performance (particularly if performer uses an acoustic instrument).

„ Real versus virtual

„ * Anonymity, identifiability and masquerading; audience response

„ Private versus public

„ *Audiences of varying size „ *Listeners influence progress of the performance by casting votes ChrisChris BrownBrown

„ "Eternal Network Music", a concert of computer network music on the internet linking 14 live performers at 6 different locations (Mills College, California Institute of the Arts, Princeton University, Renssallaer Polytechnic Institute, Stetson University, and the Zentrum für Kunst und Medientechnologie (ZKM) in Karlsruhe Germany). Part of the "net_condition - Art in the Online Universe"exhibition produced by ZKM, Karlsruhe. HaywardHayward toto BaselBasel

„ Between CSUHayward iCIM and HyperWerk in Basel, Switzerland „ Bill Thibault BeOS software “Grabbo” sends MID stream to CSUH over the internet audio back „ Using RAT and VIC Ab_Time series

ScotScot GreshamGresham--LancasterLancaster

PaulinePauline OliverosOliveros atat RPIRPI QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. inin Troy,Troy, NYNY

QuickTime™ and a Montevideo Center in TIFF (Uncompressed) decompressor are needed to see this picture. Marseilles, France PhilosophicalPhilosophical ConsiderationsConsiderations

„ SolutionSolution lookinglooking forfor aa „ DiscorporationDiscorporation problemproblem transcendstranscends thethe „ Is this truly a musical boundariesboundaries ofof thethe problem that needs to concertconcert hallhall andand be solved? opensopens upup thethe „ Our experience with possibilitiespossibilities forfor newnew the Hub has lead to formsforms ofof sharingsharing more exciting music on networks in the expressionexpression same room CurrentCurrent TechnologyTechnology OptionsOptions

„ RobustRobust AudioAudio ToolTool (RAT)(RAT) && VideoVideo ConferencingConferencing ToolTool (VIC(VIC )) openopen sourcesource „ HardwareHardware solutionssolutions (expensive)(expensive) „ VbrickVbrick ** CommonsCommons MultiMulti--PointPoint „ ** H323H323 gatewaysgateways (Tandberg(Tandberg etc.)etc.) „ UbiquitousUbiquitous ChatChat ClientsClients „ YahooYahoo MessengerMessenger ** NetMeetingNetMeeting „ iChatiChat AVAV ** AIMAIM -- MPEG4MPEG4 -- AACAAC SomeSome CurrentCurrent WorkWork

„ ExpositionExposition -- Calpurnia’sCalpurnia’s DreamDream ObscuredObscured byby MovementMovement @@ thethe Internet2Internet2 conferenceconference USCUSC andand CSUHCSUH „ PeeringsPeerings -- StudentsStudents fromfrom MillsMills atat CSUHCSUH andand StudentsStudents atat RPIRPI „ «« Ab_timeAb_time »» colaborationcolaboration withwith GRIMGRIM atat thethe MontevideoMontevideo CenterCenter inin MarseillesMarseilles (2(2 dancersdancers)) andand iCIMiCIM CSUHaywardCSUHayward QuicktimeQuicktime BroadcasterBroadcaster

„ Apple’sApple’s livelive encodingencoding softwaresoftware forfor streamingstreaming audioaudio andand videovideo overover thethe Internet.Internet. QuickTimeQuickTime BroadcasterBroadcaster capturescaptures andand encodesencodes audioaudio and/orand/or videovideo intointo QuickTimeQuickTime compatiblecompatible formatsformats includingincluding MPEGMPEG--44 DarwinDarwin StreamingStreaming ServerServer

„ the open source version of Apple’s QuickTime Streaming Server technology that allows you to send streaming

QuickTime™ and a media to clients across TIFF (Uncompressed) decompressor are needed to see this picture. the Internet using the industry standard RTP and RTSP protocols

„ Core of REAL’s Helix Producer UnicastUnicast vsvs MulticastMulticast

„ ••UnicastUnicast packetpacket hashas sourcesource IPIP addressaddress andand destinationdestination IPIP addressaddress „ RouterRouter receivesreceives unicastunicast IPIP packetpacket „ RouterRouter forwardsforwards packetpacket toto aa singlesingle destinationdestination basedbased onon destinationdestination IPIP addressaddress „ Can be delivered on the same interface

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture. UnicastUnicast vsvs MulticastMulticast

„ MulticastMulticast packetpacket hashas sourcesource IPIP addressaddress andand groupgroup (multicast)(multicast) addressaddress „ RouterRouter receivesreceives multicastmulticast IPIP packetpacket „ RouterRouter forwardsforwards packetpacket outout allall interfacesinterfaces exceptexcept thethe sourcesource (incoming)(incoming) interfaceinterface „ Only group members listen for traffic

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture. SuperColliderSuperCollider 33

„ OSCOSC communicatescommunicates locallylocally oror globallyglobally „ ShipShip clientsclients toto remoteremote sitesite andand controlcontrol oror tradetrade controlcontrol betweenbetween thethe twotwo spacesspaces „ IntegrateIntegrate thethe acousticacoustic soundsound fromfrom thethe remoteremote spacespace withwith remotelyremotely controlledcontrolled locallocal supercollidersupercollider patchpatch „ TreatTreat acousticacoustic materialmaterial fromfrom bothboth spacesspaces withwith eacheach otherother OSCOSC groupsgroups

Using OCS we define a protocol that allows for interconnected behavior

/Tim /Scot /pitch 60.345 would mean Tim is receiving pitch data from Scot.

/Hub /Chris /pitch 45.222 would be the entire group receiving pitch data from Chris.

Client/Server in OSCgroups to implement this between each of us. MixingMixing bothboth performanceperformance modesmodes „ AsAs ofof yet,yet, II knowknow ofof nono performancesperformances thatthat tightlytightly integrateintegrate livelive internetinternet videovideo withwith computercomputer networknetwork techniquestechniques „ TheThe toolstools areare finallyfinally herehere toto encourageencourage thisthis typetype ofof expressionexpression „ TheThe problemproblem continuescontinues toto bebe havinghaving technicallytechnically competentcompetent IT,IT, video,video, andand computercomputer musiciansmusicians atat allall connectedconnected sitessites ConclusionsConclusions

„ IncreasedIncreased bandwidthbandwidth andand compressioncompression makemake independentindependent twotwo wayway broadcastingbroadcasting integratedintegrated withwith interactiveinteractive electronicselectronics aa possibilitypossibility

„ ThisThis wholewhole approachapproach toto performanceperformance opensopens waysways toto growgrow nationalnational andand internationalinternational communitiescommunities ofof likelike mindedminded performanceperformance

„ ScotScot GreshamGresham--LancasterLancaster slancasterslancaster@@cogswellcogswell.edu.edu Building Shared Sonic Environments with Java

Jason Freeman Music Department Georgia Institute of Technology

http://www.jasonfreeman.net

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech All music is networked music!

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech “client-server” model

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech “peer-to-peer” model

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech So then what makes music Networked Music?

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Networked Music: A Definition

Networked Music describes performance situations where the traditional aural and visual connections between participants are augmented, mediated, or replaced by electronically- controlled connections.

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Shared Sonic Environments

(Barbosa 2003) • Accessible via the Internet • Intended for use by general public • Ongoing • Improvisatory

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Shared Sonic Environments

• Java-based tools (Phil Burk) –Jsyn (Java Synthesis API) –TransJam • Case Study: –Auracle (Neuhaus et al.)

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Why should I use Java to build a Shared Sonic Environment?

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Java

• High-level object-oriented language • Great, free IDEs (Eclipse, Netbeans) • Write once, run anywhere • Web deployment strategies –Applets – Java web start applications – Server-side Java apps, EJBs, servlets, JSPs, etc.

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Java and Audio

• Early API — really pathetic – 8 bit “business” audio • JavaSound — much better – But support varies by platform and JVM • JMF (Sun) and QTJ (Apple) — usable – Focused on playback, capture, and mixing • What about high-level audio synthesis?

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Jsyn: Java Synthesis API

• Developed by Phil Burk / SoftSynth • Unit-generator based synthesis framework • Uses JNI to run DSP natively • Supports Java 1.1+ on all major platforms (Mac 9/X, Win, Linux) • Relatively simple deployment scenario (similar to web browser plugin) • Jsyn is usually run on client-side applets

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Let’s Make a Sine Wave! Demo Applet

Source Code

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Jsyn: Additional Features

• Tons of unit generators: oscillators, filters, math, sample readers / writers, FM, mixers, etc. • SynthNote framework for handling note events, envelopes, etc. • Graphics components (e.g. waveform display) • Summary || Complete JavaDocs

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Wire

• A graphical patch editor for Jsyn • Exports to Java source code (paid version only)

Quick Wire Demo

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Jsyn: The Dark Side

• No support for frequency-domain (FFT) • Can’t write your own unit generators – Must be done in pure Java (slow) • Non-linear API for linear language – Wire a partial solution • GUI components are clunky – But you can always build your own

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Getting JSyn http://www.softsynth.com • Free for most non-commercial uses • $90 license covers most other scenarios ($45 for students) and includes full version of Wire

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Now What About Collaborating?

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Real-time Networking in Java

• Low-level: java.net classes – Open sockets, read and write the bits • Mid-level: JavaOSC API – Still need to write a server to manage clients • High-level: TransJam server and API – End-to-end solution

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech TransJam

•Two parts: – Server-side Java application – Client-side Java API • 100% Pure Java • Beneath the surface, an XML protocol

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech TransJam Features

• Client Management: – Usernames and logins – Rooms of users • Message Passing: – Chat-style text messages – Shared data objects • Bells and Whistles: – IP-to-location service (world map) – Pre-built GUI components to handle common tasks

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Let’s Make a Chat Applet! Demo Applet Source Code JavaDocs XML Protocol

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech TransJam Limitations

• No management of timing and latency • Tricky to transmit non-text data – Inefficient transmission protocol • GUI components are ugly – Overriding existing L&F is tedious • Need permission to run persistent server- side application

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Getting TransJam http://www.transjam.com • Need to contact SoftSynth for evaluation copy • Licensing fees on case-by-case basis

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Auracle (2004)

• A voice-controlled, networked sound instrument

• A shared sonic environment implemented with Java, Jsyn, and TransJam

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech The Auracle Team

• Max Neuhaus • Phil Burk • Kristjan Varnik • C. Ramakrishnan • Jason Freeman • David Birchfield • Akademie Schloss Solitude

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech System Architecture

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Timing and Latency

• “conversational” approach – Focused on gestures – Event distribution • Latency is actually artificially increased in many circumstances!

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Reliability

• Server-side event logging – HTTP / PHP / MYSQL • Unified client-side logging mechanism

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Yet Still…

Fewer than 5% of the people who visit auracle.org successfully launch Auracle, log in, and make a sound. Why? • Don’t want to install Jsyn • Don’t have a mic • Don’t understand how to use it

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech Have Questions? Please save them for our discussion at the end!

ICMC ‘05 Barcelona Building Shared Sonic Environments with http://www.jasonfreeman.net Networked Mus Wkshp Java Georgia Tech

ICMC 2005 Network Music Workshop

Open Sound Control, Resilience and Internet Collaboration

Ross Bencina [email protected] Music Technology Group Universitat Pompeu Fabra, Barcelona Topics

 ReacTable* Networked Component Architecture

 OSC Review

 TUIO protocol

 OSCgroups

 The Distributed reacTable* reacTable* architecture

Open Sound Control (OSC)

 Binary format for packing “messages”

 Messages are identified by textual paths e.g. / synth1/lfo/frequency/set

 Messages can have multiple parameters, with a number of types, overloading is possible

 Usually uses UDP for transport but other options are possible (TCP, serial, etc). OSC on the Internet: Challenges

 Packet loss and reordering

 Router/firewall network address translation NAT

 Peer discovery (naming services) OSC on the Internet: Challenges

 Packet loss and reordering  Router/firewall network address translation NAT

 Peer discovery (naming services) TUIO (Tangible User Interface Objects Protocol)

Kaltenbrunner, M. Bovermann, T. Bencina, R. Costanza, E. 2005. 'TUIO: A Protocol for Table-Top Tangible User Interfaces' Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation; Vannes, France

 A set of semantics and OSC messages for transferring object-existence and state (location, orientation etc.)

 Resilient to UDP packet loss and reordering TUIO Mechanics

 Messages:

 /tuio/*/fseq – frame sequence number, always sent  /tuio/*/alive – list of living objects, always sent  /tuio/*/set – update an object's state

 Packet loss resiliency features:

 Deltas in the alive list indicate additions and removals (no add/remove messages)  Set messages are redundantly re-sent using spare bandwidth OSC on the Internet: Challenges

 Packet loss and reordering  Router/firewall network address translation NAT  Peer discovery (naming services) NAT Traversal (hole punching)

Bryan Ford, Pyda Srisuresh and Dan Kegel, “Peer-to-Peer Communication Across Network Address Translators,” http://www.brynosaurus.com/pub/net/p2pnat/, p5. NAT Traversal (hole punching)

Bryan Ford, Pyda Srisuresh and Dan Kegel, “Peer-to-Peer Communication Across Network Address Translators,” http://www.brynosaurus.com/pub/net/p2pnat/, p5. OSC on the Internet: Challenges

 Packet loss and reordering

 Router/firewall network address translation NAT  Peer discovery (naming services) OSCgroups

 Handles NAT hole punching

 Manages group membership

 Client uses peer-to-peer multicast to other group participants OSCgroups OSCgroups Group Membership Model

 Users + Groups (separate password for each)

 Similar to IRC: if you don't stay connected someone else can use your name after a timeout period, otherwise they need your password. Same for groups. OSCgroups Messages

 From clients to server

 /groupserver/user_alive userName password privateIpAddress privatePort groupName0 groupPassword0 groupName1 groupPassword1 ...

 /groupserver/get_group_users_info groupName groupPassword

 From server to clients

 /groupclient/user_info userName privateIpAddress privatePort publicIpAddress publicPort secondsSinceLastAlive group0 group1 ...

 /groupclient/user_alive_status userName userPassword status

 /groupclient/user_group_status userName userPassword groupName groupPassword status

 From client to client

 /groupclient/ping userName Distributed reacTable*

 TUIO messages are forwarded to remote clients via OSCgroups with the addition of a source message indicating which host the messages are from.