Pro Audio is Easy, Consumer Audio is Hard Audio Confernce 2010

Lennart Poettering [email protected]

May 2010

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Who Am I?

Software Engineer at , Inc. Developer of PulseAudio, and a few other projects http://0pointer.de/lennart/ [email protected] IRC: mezcalero

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Didn’t graduate in signal processing

Not musician

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Not musician Didn’t graduate in signal processing

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Correct misconceptions Improve cooperation, share code and interfaces It might just be interesting

Why?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Improve cooperation, share code and interfaces It might just be interesting

Why? Correct misconceptions

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard It might just be interesting

Why? Correct misconceptions Improve cooperation, share code and interfaces

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Why? Correct misconceptions Improve cooperation, share code and interfaces It might just be interesting

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Nobody makes money off Linux Audio Only 3 people payed to hack on general purpose infrastructure by the big distributors

General Background

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Only 3 people payed to hack on general purpose infrastructure by the big distributors

General Background Nobody makes money off Linux Audio

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard General Background Nobody makes money off Linux Audio Only 3 people payed to hack on general purpose infrastructure by the big distributors

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard However, we have allies, particularly in RT, Mobile, also V4L

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: laymen users

Pro Audio: knowledgable user

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: knowledgable user Consumer Audio: laymen users

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: crappy hardware (Both the PC, and the Audio HW, Feature Monsters)

Pro Audio: good hardware

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: good hardware Consumer Audio: crappy hardware (Both the PC, and the Audio HW, Feature Monsters)

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: Audio is one thing among many

Pro Audio: Audio is primary purpose of the machine

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: Audio is primary purpose of the machine Consumer Audio: Audio is one thing among many

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: user != administrator

Pro Audio: User owns the machine, full configurability, user == administrator

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: User owns the machine, full configurability, user == administrator Consumer Audio: user != administrator

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: low latencies, functionality Consumer Audio: low power usage, dynamically changing environments, robustness

Different Optimization Targets:

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: low power usage, dynamically changing environments, robustness

Different Optimization Targets: Pro Audio: low latencies, functionality

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Different Optimization Targets: Pro Audio: low latencies, functionality Consumer Audio: low power usage, dynamically changing environments, robustness

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Latency vs. Power Usage

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard In Consumer Audio: choose latency as high as possible, and as low as necessary, allow configuration changes

In Pro Audio: minimize latency at price of power usage, and dynamic configuration changes

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard In Pro Audio: minimize latency at price of power usage, and dynamic configuration changes In Consumer Audio: choose latency as high as possible, and as low as necessary, allow configuration changes

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Unfortunately, most hardware/drivers cannot do that Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s

Result for consumer: we need to change latencies

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s

Result for consumer: we need to change latencies Unfortunately, most hardware/drivers cannot do that

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Result for consumer: we need to change latencies Unfortunately, most hardware/drivers cannot do that Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard That’s difficult, because system timers deviate from sound card timers, and sound card timers are crap

Hence: schedule audio with system timers

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Hence: schedule audio with system timers That’s difficult, because system timers deviate from sound card timers, and sound card timers are crap

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Linear Regression + Spline interpolation Exponentional backoff Everything twice

Time interpolation

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Exponentional backoff Everything twice

Time interpolation Linear Regression + Spline interpolation

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Everything twice

Time interpolation Linear Regression + Spline interpolation Exponentional backoff

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Time interpolation Linear Regression + Spline interpolation Exponentional backoff Everything twice

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Requires history, for remixing

To make long latencies useful: support seeking

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard To make long latencies useful: support seeking Requires history, for remixing

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Dropouts, require dynamic adjustments of sleep times

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Timing and transfer block granularity

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Outlook, moving timing into the kernel?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Sound card IRQs are accurate!

Non-interpolated timing has advantages?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Non-interpolated timing has advantages? Sound card IRQs are accurate!

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Algorithm to use? DLL?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard We cannot trust clients, one misbehaving client may not be allowed to stall the entire pipeline

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Why? Never drop out

RT CPU time calculations change, need to preemptively increase sleep times when streams are moved, removed and added

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard RT CPU time calculations change, need to preemptively increase sleep times when streams are moved, removed and added Why? Never drop out

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Unreliable latencies, dynamically changing

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Compromises in RT code, IO loops are not constant time

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Need to support S16LE and FLOAT32, and a lot more

Multiple encoded formats, conversion/resampling required, but ideally zero-copy

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Multiple encoded formats, conversion/resampling required, but ideally zero-copy Need to support S16LE and FLOAT32, and a lot more

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Zero copy (because sizeof(metadata) ¡ sizeof(payload))

Large latency means exchanging of large blobs

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Large latency means exchanging of large blobs Zero copy (because sizeof(metadata) ¡ sizeof(payload))

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Means ref counting

Cached audio samples

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cached audio samples Means ref counting

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Need to merge and extend in software Difficult: Routing in the Sound HW seldom known (USB!)

Cannot expose low-level hw controls

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Difficult: Routing in the Sound HW seldom known (USB!)

Cannot expose low-level hw controls Need to merge and extend in software

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cannot expose low-level hw controls Need to merge and extend in software Difficult: Routing in the Sound HW seldom known (USB!)

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard If the mixer controls even exist

Common: unreliable dB data

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Common: unreliable dB data If the mixer controls even exist

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Means no low-level control on RT =¿ rtkit

Machine not owned by the user and user not knowledgable

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Machine not owned by the user and user not knowledgable Means no low-level control on RT =¿ rtkit

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard No memory locking

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Compromises, compromises, compromises

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation PA, JACK

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Use more standard desktop infrastructure. Tear down the seperating walls.

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on APIs?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on Timing?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on Protocols? RTP?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Meet RT folks, V4L folks, MM folks

Come to LPC!

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Come to LPC! Meet RT folks, V4L folks, MM folks

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Any questions?

That’s all, folks.

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard That’s all, folks. Any questions?

Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard