Pro Audio is Easy, Consumer Audio is Hard Linux Audio Confernce 2010
Lennart Poettering [email protected]
May 2010
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Who Am I?
Software Engineer at Red Hat, Inc. Developer of PulseAudio, Avahi and a few other Free Software projects http://0pointer.de/lennart/ [email protected] IRC: mezcalero
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Didn’t graduate in signal processing
Not musician
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Not musician Didn’t graduate in signal processing
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Correct misconceptions Improve cooperation, share code and interfaces It might just be interesting
Why?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Improve cooperation, share code and interfaces It might just be interesting
Why? Correct misconceptions
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard It might just be interesting
Why? Correct misconceptions Improve cooperation, share code and interfaces
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Why? Correct misconceptions Improve cooperation, share code and interfaces It might just be interesting
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Nobody makes money off Linux Audio Only 3 people payed to hack on general purpose infrastructure by the big distributors
General Background
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Only 3 people payed to hack on general purpose infrastructure by the big distributors
General Background Nobody makes money off Linux Audio
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard General Background Nobody makes money off Linux Audio Only 3 people payed to hack on general purpose infrastructure by the big distributors
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard However, we have allies, particularly in RT, Mobile, also V4L
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: laymen users
Pro Audio: knowledgable user
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: knowledgable user Consumer Audio: laymen users
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: crappy hardware (Both the PC, and the Audio HW, Feature Monsters)
Pro Audio: good hardware
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: good hardware Consumer Audio: crappy hardware (Both the PC, and the Audio HW, Feature Monsters)
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: Audio is one thing among many
Pro Audio: Audio is primary purpose of the machine
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: Audio is primary purpose of the machine Consumer Audio: Audio is one thing among many
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: user != administrator
Pro Audio: User owns the machine, full configurability, user == administrator
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: User owns the machine, full configurability, user == administrator Consumer Audio: user != administrator
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Pro Audio: low latencies, functionality Consumer Audio: low power usage, dynamically changing environments, robustness
Different Optimization Targets:
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Consumer Audio: low power usage, dynamically changing environments, robustness
Different Optimization Targets: Pro Audio: low latencies, functionality
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Different Optimization Targets: Pro Audio: low latencies, functionality Consumer Audio: low power usage, dynamically changing environments, robustness
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Latency vs. Power Usage
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard In Consumer Audio: choose latency as high as possible, and as low as necessary, allow configuration changes
In Pro Audio: minimize latency at price of power usage, and dynamic configuration changes
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard In Pro Audio: minimize latency at price of power usage, and dynamic configuration changes In Consumer Audio: choose latency as high as possible, and as low as necessary, allow configuration changes
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Unfortunately, most hardware/drivers cannot do that Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s
Result for consumer: we need to change latencies
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s
Result for consumer: we need to change latencies Unfortunately, most hardware/drivers cannot do that
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Result for consumer: we need to change latencies Unfortunately, most hardware/drivers cannot do that Also: hardware/drivers limited in there range: ideally we have one wakeup every 1s
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard That’s difficult, because system timers deviate from sound card timers, and sound card timers are crap
Hence: schedule audio with system timers
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Hence: schedule audio with system timers That’s difficult, because system timers deviate from sound card timers, and sound card timers are crap
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Linear Regression + Spline interpolation Exponentional backoff Everything twice
Time interpolation
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Exponentional backoff Everything twice
Time interpolation Linear Regression + Spline interpolation
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Everything twice
Time interpolation Linear Regression + Spline interpolation Exponentional backoff
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Time interpolation Linear Regression + Spline interpolation Exponentional backoff Everything twice
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Requires history, for remixing
To make long latencies useful: support seeking
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard To make long latencies useful: support seeking Requires history, for remixing
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Dropouts, require dynamic adjustments of sleep times
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Timing and transfer block granularity
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Outlook, moving timing into the kernel?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Sound card IRQs are accurate!
Non-interpolated timing has advantages?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Non-interpolated timing has advantages? Sound card IRQs are accurate!
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Algorithm to use? DLL?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard We cannot trust clients, one misbehaving client may not be allowed to stall the entire pipeline
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Why? Never drop out
RT CPU time calculations change, need to preemptively increase sleep times when streams are moved, removed and added
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard RT CPU time calculations change, need to preemptively increase sleep times when streams are moved, removed and added Why? Never drop out
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Unreliable latencies, dynamically changing
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Compromises in RT code, IO loops are not constant time
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Need to support S16LE and FLOAT32, and a lot more
Multiple encoded formats, conversion/resampling required, but ideally zero-copy
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Multiple encoded formats, conversion/resampling required, but ideally zero-copy Need to support S16LE and FLOAT32, and a lot more
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Zero copy (because sizeof(metadata) ¡ sizeof(payload))
Large latency means exchanging of large blobs
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Large latency means exchanging of large blobs Zero copy (because sizeof(metadata) ¡ sizeof(payload))
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Means ref counting
Cached audio samples
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cached audio samples Means ref counting
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Need to merge and extend in software Difficult: Routing in the Sound HW seldom known (USB!)
Cannot expose low-level hw controls
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Difficult: Routing in the Sound HW seldom known (USB!)
Cannot expose low-level hw controls Need to merge and extend in software
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cannot expose low-level hw controls Need to merge and extend in software Difficult: Routing in the Sound HW seldom known (USB!)
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard If the mixer controls even exist
Common: unreliable dB data
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Common: unreliable dB data If the mixer controls even exist
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Means no low-level control on RT =¿ rtkit
Machine not owned by the user and user not knowledgable
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Machine not owned by the user and user not knowledgable Means no low-level control on RT =¿ rtkit
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard No memory locking
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Compromises, compromises, compromises
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation PA, JACK
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Use more standard desktop infrastructure. Tear down the seperating walls.
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on APIs?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on Timing?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Cooperation on Protocols? RTP?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Meet RT folks, V4L folks, MM folks
Come to LPC!
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Come to LPC! Meet RT folks, V4L folks, MM folks
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard Any questions?
That’s all, folks.
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard That’s all, folks. Any questions?
Lennart Poettering Pro Audio is Easy, Consumer Audio is Hard