IEEE-Madison ECN Meeting
Virtual Music Jamming with Low Latency Networked Systems
April 15th, 2021 Virtual Meeting Tom Kaminski [email protected] Anton Kapela, [email protected] John Lombardo [email protected]
IEEE-Madison Section Entrepreneurs and Consultants Network
IEEE-Madison ECN April 15, 2021 1 Overview Part I – Basics Part II – Demos The Problem Classes Jambox Pi image Peer-to-Peer, noVNC Browser I/F Client-Server, URLrelay for IP Hierarchical Jack/Qjackctl/Patchage/ALSA Compressed or Not? Sonobus Peer-to-Peer Hardware Jamulus Client/Server Laptop/Desktop Audio Input/Output SoC Part III – Technical Issues RaspBerry Pi Pi 3B, 3B+ or 4B Audio Card USB Interface Meeting End: Live Jazz Jam Software for Music Jams Jack, Sonobus, Jamulus
IEEE-Madison April 15, 2021 2 The Problem
Covid-19 gathering rules make it difficult to get close together indoors Music like Jazz, Bluegrass, etc. require close coordination for success Musicians separated by more than 30 feet (30 millisecond delay in air) experience difficulty performing Most “virtual meeting” technologies introduce much more than 30 milliseconds delay (Cell phone: ~200ms.) We have gotten used to Virtual Meetings, why not “Virtual Jamming? From Quora.com: “Average delay on a phone call is also known as ‘Latency’. Ideally, latency should not be more than 200 milliseconds. If it exceeds 200 ms, quality of the call will degrade significantly.” Worse for Jams... Other Issues: Many musicians are not technically savvy We need to provide a worry-free “Appliance” that can be simply controlled. Home Firewall/Router configuration can be a problem
IEEE-Madison April 15, 2021 3 The Basic Needs
A device to digitize microphone and/or instrument signals A method of transmitting the digitized signals over networks A method of “mixing” signals from two or more musicians A method of receiving the signals from multiple musicians A method to convert the signals back to sound
BOX 1 BOX 2
Network
IEEE-Madison April 15, 2021 4 Peer-to-Peer
Each musician sends and receives data from all of the other musicians Each musician’s Box adjusts the “Mix” Computation is of order n for n separate “Boxes” Network traffic is of order n2
BOX 1 BOX 2
BOX 3 Example Software: Jacktrip, Quacktrip, Sonobus
IEEE-Madison April 15, 2021 5 Client-Server
Each musician sends and receives data to/from a Central Server and sends Mix Levels Central Server adjusts the “Mix” for each musician and sends the results to each individual musician Computation is of order n2 for the Server Network traffic is of order n
BOX 1 BOX 2
BOX 3 Server BOX 3
Example Software: Jamulus, Jammr.net, Jamkazam, Netty McNetface
IEEE-Madison April 15, 2021 6 Hierarchical
Musicians are divided into “M” groups A Group Mixer adjusts the “Mix” for musicians in a Group and sends the results to to the next level (Main mix?) Main Mix sends “M” adjusted Mixes back down to the individual Group Mixers Group Mixers send the combined Mix back to the musicians
String Mix Drum Mix Main Mix
Wind Mix
Example Software: Music 101, JackTrip Virtual Studio IEEE-Madison April 15, 2021 7 Hardware
A Desktop, Laptop, or SoC to run the software
A Microphone (one or more) and Headphone
An appropriate ADC/DAC for the audio signals Desktop with an Audio Add-in card Laptop with built-in audio Low-latency USB Audio Interface SoC with USB or Audio Card
A WIRED internet connection
Some method of interacting with the software Keyboard/mouse/Monitor (Possibly) Phone, iPad, Tablet with a browser or X11 remote device Example: Laptop with an external USB Interface for Musical Instruments and Ethernet Wired I/F
IEEE-Madison April 15, 2021 8 What is A SoC? “A System on a Chip (SoC) is a method of placing all necessary electronics for running a computer on a single chip. Instead of having an individual chip for the CPU, GPU, USB controller, RAM, Northbridge, Southbridge, etc., everything is compressed down into one tidy package” --- Raspberrypi.org
IEEE-Madison April 15, 2021 9 RaspBerry Pi Family
CM-3
3 Model B+
4 Model B+
IEEE-Madison April 15, 2021 10 Hardware: Pi Audio Cards AudioInjector.net Pi Hats https://blokas.io
Pisound - Raspberry Pi Sound Card & All 24bit MIDI Interface ADC up to 192kHz Sampling
Pi Zero
Ultra
IEEE-Madison April 15, 2021 11 Hardware: USB Audio Box
Musician-Quality USB Low Latency Audio Interfaces
● Adjustable Gain ● 1/4”, XLR Inputs ● Phantom 48V Power ● Headphone Level Adjust Behringer M-Audio M-Track-Duo Uphoria UMC-202HD
Focusrite Scarlett 2i2
Prices from $49 to $200
IEEE-Madison April 15, 2021 12 Hardware: USB Plug Ins Low Price USB Audio Interfaces
Stereo Out, Mono In – $7
Stereo In/Out, 24bit to 96ksps, Built-in Mono Mic $15
IEEE-Madison April 15, 2021 13 Software: Jack
JACK – “Jack Audio Connection Kit”
JACK is a low-latency audio server, written for any operating system that is reasonably POSIX compliant. It currently exists for Linux, OS X, Solaris, FreeBSD and Windows. It can connect several client applications to an audio device, and allow them to share audio with each other. Clients can run as separate processes like normal applications, or within the JACK server as "plugins".
JACK was designed from the ground up for professional audio work, and its design focuses on two key areas: synchronous execution of all clients, and low latency operation.
See: https://jackaudio.org/
IEEE-Madison April 15, 2021 14 Software: SonoBus High Quality Network Audio Streaming
SonoBus is an easy to use application for streaming high-quality, low-latency peer-to-peer audio between devices over the internet or a local network.
Multi-user, multi-platform, open- source, completely free.
Simply choose a unique group name (with optional password), and instantly connect multiple people together to make music, remote sessions, podcasts, etc. Easily record the audio from everyone, as well as playback any audio content to the whole group.
See: https://sonobus.net/#
IEEE-Madison April 15, 2021 15 Software: Jamulus
● Server or Client ● Private Server (Hidden IP) with Firewall forwarding Public Server – No Firewall Mapping needed! ● Uses Jack Audio Connected Devices ● Jamulus is free and open source software (FOSS) licensed under the GPL Jamulus is software for playing music, rehearsing, or just jamming with anyone online with low latency. You can use your Windows, macOS or Linux machine to connect to Jamulus servers worldwide. Jamulus is free and you can just use your normal broadband connection. Simply connect to a public See: https://jamulus.io server or host your own private one.
IEEE-Madison April 15, 2021 16 Part II: Demonstrations
Setup: Jambox Pi image is used See: https://github.com/kdoren/jambox-pi-gen URLrelay for finding Pi’s IP address VNC running on the Pi for Virtual Desktop noVNC Browser I/F for access The Jambox Raspbian Desktop Tour Audio Connection Setup Jack/Qjackctl/Patchage/ALSA Sonobus Peer-to-Peer Demo Pi3B to Pi3B Jamulus Client/Server Pi3B to Pi4B Live Jam Pi3B to Server at 5Nines
IEEE-Madison April 15, 2021 17 Part III: Technical Issues
Latency What Causes it? Oversampling ADC/DAC USB, SPI, popular CODECS I/O of modern computers Software abstractions
ERROR Correction CODECS and Forward Error Correction in reliable Multi-path networks Can we do better?
Network Coding Multipath TCP at MIT within Internet Engineering Task Force Application level TCP?
IEEE-Madison April 15, 2021 18 Part III: Latency
What About Latency? Creative Combat with latencies for ~half of my career
Backstory: 2010, friend and I built a microwave network between NYC and Chicago
Goal: round trip to be under 13.3 milliseconds (Spread Networks, now Zayo, hit 13.0 in mid 2011), but practically target 12.5
Day-1 launch: 11.5 msec RTT, few months later, ~9 msec RTT
Full backstory: https://visionscarto.net/enquete-reseaux-hft
IEEE-Madison April 15, 2021 19 Race to c
IEEE-Madison April 15, 2021 20 Race to c What did we learn?
• Sigma/Delta & Successive Approximation isn't usually the major latency source – can be effectively reduced to "0 cycle" latencies at reasonable resolution
• Abstractions are nice, but cost something – usually time
IEEE-Madison April 15, 2021 21 Race to c - modems
What else did we learn?
Filters aren't everything, they're the only thing
...and that abstractions are nice, but cost something – usually time
But really, we mean: I/O abstractions are expensive
• Like, really, really expensive
IEEE-Madison April 15, 2021 22 "Systems, what?"
IEEE-Madison April 15, 2021 23 Systems, oh.
Summary:
• Safety (atomicity) creates unavoidable (serialization) delays • Mutexes, MSI, DMA, etc.
• Going faster isn't always helpful
IEEE-Madison April 15, 2021 24 Latency: It's a sick joke!
IEEE-Madison April 15, 2021 25 Part III: USB 2.0 & 3.x
IEEE-Madison April 15, 2021 26 Choose your own...latency?
April 15, 202118
IEEE-Madison http://blog.ultimateoutsider.com/2018/04/comparing-usb-audio-interface-latency.html April 15, 2021 27 More Systems Stuff
More depth than we have time for today
"Effectively Measure and Reduce Kernel Latencies for Real-time Constraints" https://www.youtube.com/watch?v=epcPeMlBJW0
IEEE-Madison April 15, 2021 28 Systems & modem latency... yikes!
IEEE-Madison April 15, 2021 29 Workarounds for Unreliable Networks
Wait, but why?
• We have three options to address packet loss: • Recover (FEC at Tx/Rx, delay penalty) • Retransmit (ARQ, sequencing, delay penalty) • Make it up (ML/AI, interpolation, synthesis)
IEEE-Madison April 15, 2021 30 Part III: A Way Out Linear Network Coding IETF MPTCP - RFC 8684
"Linear network coding is a technique in which the "MultiPath TCP (MPTCP) is an effort towards enabling the intermediate nodes combine the packets they receive simultaneous use of several IP-addresses/interfaces by a instead of just relaying them. The packets are combined modification of TCP that presents a regular TCP interface using linear coefficients from a finite field." to applications, while in fact spreading data across several subflows."
IEEE-Madison April 15, 2021 31