Journey from Closed to Open:
Lessons Learned from Open Sourcing Sound Open Firmware Mission
1.Inspire others to open source firmware.
2.Show how firmware can be open sourced.
3.Discuss common challenges open sourcing firmware and how to overcome them.
4.Introduce Sound Open Firmware (SOF) architecture.
2 About Me
Employed by Intel as a software architect.
Linux user and engineer since 1994.
Developed AsoC and PMIC abstraction layers.
Linux audio engineer since 2003.
Working on audio DSP’s since 2008.
Working on SOF since 2015.
3 Sound Open Firmware is an open source audio digital signal processing frmware and driver infrastructure.
4 The Dark Ages
*Darkness due to lack of source code 5 Closed Source Firmware
Historically the standard practice amongst firmware development teams with a few exceptions.
Often seen as a black box by integrators and driver & user space developers.
Makes it more difficult to debug problems in other areas of the stack.
Impossible for upper stack developers to debug hardware problems.
Usually developed alongside a single OS specific driver.
Documentation never truly matches or keeps up with firmware source code.
? Data In Data Out
6 Changing Course
Factors influencing open sourcing. 7 Market Drivers
• Demand for speech and voice authentication and recognition applications and technologies • Impact of AI on accuracy of speech and voice recognitions • Growth in voice control- based devices in consumer and enterprise markets
Audio, speech and voice has become ubiquitous
8 Open Source Hardware
Minnowboard Project
Open Source Hardware.
Baytrail CPU, dual core @1.33GHz, 2GB DDR
Tensilica Xtensa audio DSP @ 400MHz
Open schematics, PCB layout, BOM
Open source Software
Full open source Linux software stack
Open source Firmware
Open source coreboot BIOS
NO open source audio DSP firmware
9 Minnowboard DSP Architecture
Instruction Instruction Cache RAM Xtensa HiFi 2EP core.
to/from HiFi EP Core OCP (Upstream) IOSF2OCP 96kB Instruction RAM bridge
168kB Data RAM JTAGJTAG JTAG EXT 64 bit AXI Trace HiFi EP PIF PIF2AXI 32-bit APB Slave Master and Slave Logic Core I/F SSP0 Interrupts Interrupt (from shim) A M/N 2 * DMACs A CLK Uu LPE D Shim d 3 * I2S ports I Control & 32-bit APB Slave Config Oi signals SSP1 4kB RF o PCI device from host OS Mailbox M/N CLK Data Data RAM 32 bit S Cache OCP Slave MF LPE OCP Master DMA_00 Writes Xa 32-bit APB Slave OCP SSP2 Master Reads b M/N Source: CLK 32 bit r OCP Slave Tangier LPE OCP i Master DMA_01 Writes Sonics OCP c Master VLV Reads
10 Challenges Ahead
Be prepared ! 11 Political Challenges
Challenges Solutions
• Company policy for audio DSP firmware • Be prepared to fight the same battle more was historically closed source. than once.
• Most colleagues were initially either • Build ground swell of opinion and facts strongly in favour or strongly against. behind open source. • • Fears around disclosing IP. Helps if you have proof of concept code - “skunkworks”! • How do we add value? • Tell the world about your project!
“You may have to fight a battle more than once to win it.” Margaret Thatcher 12 Technical Challenges - Compiler
Xtensa ISA differs between cores adding complexity for compilers. SOF Source SOF Source Code Code Cadence provide an optimising compiler for Xtensa Firmware source Firmware source Commercial license for some targets $$$ Cadence GCC Compiler Compiler Free for some targets like Minnowboard :)
ELF ELF ELF Need open source compiler for community. ImageELF Image Image Image
GCC supports xtensa base ISA
GCC does not support xtensa SIMD/VLIW.
Add GCC support for Minnowboard xtensa ISA.
13 Technical Challenges – Image Builder
SOF Source Compilers generate ELF images. Code Firmware source ELF images don’t run on bare metal.
Cadence GCC ELF image needs converted to binary image Compiler Compiler format for loading into DSP memories.
ELF ImageELF Need to have tools that can convert ELF to Image multiple different firmware image formats. Rimage tool Create tool to convert
Unsigned ELF images into multiple ImageImage different image formats.
Should be able to run code now ! 14 Technical Challenges – DSP Emulator
Code not easy to debug
No debugger (yet) Xtensa Qemu
No printf(); SOF Firmware
FW DMA driver JTAG can’t be used – Intel only.
FW SSP driver Emulation can be used to debug bring up. FW Audio mixer
Qemu already supports base Xtensa ISA VM #1
Qemu support added xtensa GDB Minnowboard DSP On host
Extra registers and instruction not in base ISA
15 Technical Challenges – Heterogeneous Emulator
Firmware must be debugged alongside driver.
X86 Xtensa Qemu Qemu/KVM Qemu used to virtualise drivers and firmware together. Guest OS * SOF Firmware Qemu IO Bridge Audio DSP driver FW DMA driver Host side almost real time. IO, DMA IRQs Codec driver FW SSP driver DSP side emulated.
Audio Userspace FW Audio mixer VM #0 VM #1
xtensa x86 GDB GDB On host On host
16 Technical Challenges – Code Signing
Newer DSP hardware has security that validates firmware SOF Source binaries. Code
Firmware source Code signing support was added to open source rimage tool. Cadence GCC
PCKS #1.5 Compiler Compiler
Openssl ELF ImageELF Image Created “public” private key to be used for developer hardware Rimage e.g. UP^2 tool
Implications with GPLv3 “tivoisation” clause. Unsigned ImageImage
Signing Tool
Signed ImageImage
17 Build a Community A healthy open source project needs a healthy community!
Who are the community?
Commercial users who deploy in products.
Non commercial hobbyists.
Academics researching audio processing algorithms.
Use external code hosting platforms like Github, mailing lists, IRC, wiki’s etc and do development in public.
Release patches and code “early and often”.
Don’t do infrequent code drops.
No “abandonware”.
Accept patches from others.
Accept bug reports from others. 18 Hello World
19 Firmware Architecture
DSP or SOF Audio Components Discrete X86 or ARM mixer volume ASRC AEC/NS Chip Host buffer speaker mux EQ queue protection splitter SRC tone
Linux Generic Micro Kernel ASoC Audio (XTOS, Zephyr, FreeRTOS options) Audio Tasks msg work SOF boot queues queues memory Driver Control IPC HW/IPC Data IRQs schd mutex exceptions Configurable driver pipelines RTOS/HV atomic Audio timer threads services semaphores Driver Platform Drivers
DMA HDA I2C
DMIC GNA SPI
SSP USB UART GPIO
2 of 2 Firmware diagram – complete diagram 20 Driver Architecture
ASoC Machine Driver
codec board HW config integration integration
Generic PCM Driver DSP or topology PCMs Kcontrols IPC via Mailbox Discrete Buffers via DMA DAPM Chip
Generic IPC Driver
mixer stream PM
pipeline
DSP Platform Driver doorbell mailbox IRQ
code loader IO PM
2 of 2 Driver diagram – complete diagram 21 History
Year ending 2015 2016 2017 2018
Inception
Platforms: Bay Trail Features: Qemu Xtensa
Firmware open sourced Platforms: Cherry Trail and Braswell Features: Qemu host SOF 1.0 public release Driver open sourced Platforms: Apollo Lake, Haswell, Broadwell Features: BSD/GPL driver, topologies Became Linux Foundation project Driver upstream Platforms: Cannon Lake, Ice Lake, Sue Creek Features: Docker build, SOF on host 22 Development Kit
Qemu DSP & OR Cadence Host Emulator Emulator Android m4 HAL topology debug XML(*.xml OR file ) (*m4) SOF ITT Topology m4 rmbox / SOF Source file trace (*.conf) Firmware Code Tool alsaplg Firmware source Zephyr OS debugfs
Cadence GCC Audio DSP Compiler Compiler Topology Metadata ASoC Linux SOF Kernel SOF ELF /lib/firmware/intel ImageELF Driver / Driver Image Firmware
Signed Image rimage Tool ALSA SoC IPC/Mailbox Topology Legend Unsigned Signing ImageImage Tool SOF Source Tool 3rd Party Tool SDK Binary Image
*Note: Docker contains only the open source components Optional Config Data
SDK diagram – complete SDK 23 Core Pillars
Open source Community driven
Permissive1 Modular Portable Tool-rich • BSD/MITFlexibility licensed in auditing• Configurable andtopology customizing • Platform, firmwarefor desired OS, use• BSD/GPL case licensed and firmware and topologies allows for flexible DSP architecture agnostic proprietary tools available customizations • BSD/GPL dual licensed • Portable to other host • Includes toolchain, driver • Extendable using existing platform and DSP debugger, emulator, APIs to add custom or 3rd architectures scripts, firmware creation • Firmware code changes party binary modules tools can be private or up- • Extendable to different streamed • Allows for custom ABIs real-time DSP OSes • Configurable builds using between modules supported GNU GCC or • Modifiable for any custom proprietary toolchains • Develop and test out new integrations modules before deploying • Virtualize both DSP and onto target HW host OS via QEMU • Virtualize trace data in real time
1: SDK tools are a mixture of BSD, GPL and proprietary licenses. 24 Thanks!
Q & A
Drop by the booth.
www.sofproject.com https://github.com/thesofproject
25 Thank you 26