Journey from Closed to Open: Lessons Learned from Open Sourcing Sound Open Firmware Mission 1.Inspire others to open source firmware. 2.Show how firmware can be open sourced. 3.Discuss common challenges open sourcing firmware and how to overcome them. 4.Introduce Sound Open Firmware (SOF) architecture. 2 About Me Employed by Intel as a software architect. Linux user and engineer since 1994. Developed AsoC and PMIC abstraction layers. Linux audio engineer since 2003. Working on audio DSP’s since 2008. Working on SOF since 2015. 3 Sound Open Firmware is an open source audio digital signal processing frmware and driver infrastructure. 4 The Dark Ages *Darkness due to lack of source code 5 Closed Source Firmware Historically the standard practice amongst firmware development teams with a few exceptions. Often seen as a black box by integrators and driver & user space developers. Makes it more difficult to debug problems in other areas of the stack. Impossible for upper stack developers to debug hardware problems. Usually developed alongside a single OS specific driver. Documentation never truly matches or keeps up with firmware source code. ? Data In Data Out 6 Changing Course Factors influencing open sourcing. 7 Market Drivers • Demand for speech and voice authentication and recognition applications and technologies • Impact of AI on accuracy of speech and voice recognitions • Growth in voice control- based devices in consumer and enterprise markets Audio, speech and voice has become ubiquitous 8 Open Source Hardware Minnowboard Project Open Source Hardware. Baytrail CPU, dual core @1.33GHz, 2GB DDR Tensilica Xtensa audio DSP @ 400MHz Open schematics, PCB layout, BOM Open source Software Full open source Linux software stack Open source Firmware Open source coreboot BIOS NO open source audio DSP firmware 9 Minnowboard DSP Architecture Instruction Instruction Cache RAM Xtensa HiFi 2EP core. to/from HiFi EP Core OCP (Upstream) IOSF2OCP 96kB Instruction RAM bridge 168kB Data RAM JTAGJTAG JTAG EXT 64 bit AXI Trace HiFi EP PIF PIF2AXI 32-bit APB Slave Master and Slave Logic Core I/F SSP0 Interrupts Interrupt (from shim) A M/N 2 * DMACs A CLK Uu LPE D Shim d 3 * I2S ports I Control & 32-bit APB Slave Config Oi signals SSP1 4kB RF o PCI device from host OS Mailbox M/N CLK Data Data RAM 32 bit S Cache OCP Slave MF LPE OCP Master DMA_00 Writes Xa 32-bit APB Slave OCP SSP2 Master Reads b M/N Source: CLK 32 bit r OCP Slave Tangier LPE OCP i Master DMA_01 Writes Sonics OCP c Master VLV Reads 10 Challenges Ahead Be prepared ! 11 Political Challenges Challenges Solutions • Company policy for audio DSP firmware • Be prepared to fight the same battle more was historically closed source. than once. • Most colleagues were initially either • Build ground swell of opinion and facts strongly in favour or strongly against. behind open source. • • Fears around disclosing IP. Helps if you have proof of concept code - “skunkworks”! • How do we add value? • Tell the world about your project! “You may have to fight a battle more than once to win it.” Margaret Thatcher 12 Technical Challenges - Compiler Xtensa ISA differs between cores adding complexity for compilers. SOF Source SOF Source Code Code Cadence provide an optimising compiler for Xtensa Firmware source Firmware source Commercial license for some targets $$$ Cadence GCC Compiler Compiler Free for some targets like Minnowboard :) ELF ELF ELF Need open source compiler for community. ImageELF Image Image Image GCC supports xtensa base ISA GCC does not support xtensa SIMD/VLIW. Add GCC support for Minnowboard xtensa ISA. 13 Technical Challenges – Image Builder SOF Source Compilers generate ELF images. Code Firmware source ELF images don’t run on bare metal. Cadence GCC ELF image needs converted to binary image Compiler Compiler format for loading into DSP memories. ELF ImageELF Need to have tools that can convert ELF to Image multiple different firmware image formats. Rimage tool Create tool to convert Unsigned ELF images into multiple ImageImage different image formats. Should be able to run code now ! 14 Technical Challenges – DSP Emulator Code not easy to debug No debugger (yet) Xtensa Qemu No printf(); SOF Firmware FW DMA driver JTAG can’t be used – Intel only. FW SSP driver Emulation can be used to debug bring up. FW Audio mixer Qemu already supports base Xtensa ISA VM #1 Qemu support added xtensa GDB Minnowboard DSP On host Extra registers and instruction not in base ISA 15 Technical Challenges – Heterogeneous Emulator Firmware must be debugged alongside driver. X86 Xtensa Qemu Qemu/KVM Qemu used to virtualise drivers and firmware together. Guest OS * SOF Firmware Qemu IO Bridge Audio DSP driver FW DMA driver Host side almost real time. IO, DMA IRQs Codec driver FW SSP driver DSP side emulated. Audio Userspace FW Audio mixer VM #0 VM #1 xtensa x86 GDB GDB On host On host 16 Technical Challenges – Code Signing Newer DSP hardware has security that validates firmware SOF Source binaries. Code Firmware source Code signing support was added to open source rimage tool. Cadence GCC PCKS #1.5 Compiler Compiler Openssl ELF ImageELF Image Created “public” private key to be used for developer hardware Rimage e.g. UP^2 tool Implications with GPLv3 “tivoisation” clause. Unsigned ImageImage Signing Tool Signed ImageImage 17 Build a Community A healthy open source project needs a healthy community! Who are the community? Commercial users who deploy in products. Non commercial hobbyists. Academics researching audio processing algorithms. Use external code hosting platforms like Github, mailing lists, IRC, wiki’s etc and do development in public. Release patches and code “early and often”. Don’t do infrequent code drops. No “abandonware”. Accept patches from others. Accept bug reports from others. 18 Hello World 19 Firmware Architecture DSP or SOF Audio Components Discrete X86 or ARM mixer volume ASRC AEC/NS Chip Host buffer speaker mux EQ queue protection splitter SRC tone Linux Generic Micro Kernel ASoC Audio (XTOS, Zephyr, FreeRTOS options) Audio Tasks msg work SOF boot queues queues memory Driver Control IPC HW/IPC Data IRQs schd mutex exceptions Configurable driver pipelines RTOS/HV atomic Audio timer threads services semaphores Driver Platform Drivers DMA HDA I2C DMIC GNA SPI SSP USB UART GPIO 2 of 2 Firmware diagram – complete diagram 20 Driver Architecture ASoC Machine Driver codec board HW config integration integration Generic PCM Driver DSP or topology PCMs Kcontrols IPC via Mailbox Discrete Buffers via DMA DAPM Chip Generic IPC Driver mixer stream PM pipeline DSP Platform Driver doorbell mailbox IRQ code loader IO PM 2 of 2 Driver diagram – complete diagram 21 History Year ending 2015 2016 2017 2018 Inception Platforms: Bay Trail Features: Qemu Xtensa Firmware open sourced Platforms: Cherry Trail and Braswell Features: Qemu host SOF 1.0 public release Driver open sourced Platforms: Apollo Lake, Haswell, Broadwell Features: BSD/GPL driver, topologies Became Linux Foundation project Driver upstream Platforms: Cannon Lake, Ice Lake, Sue Creek Features: Docker build, SOF on host 22 Development Kit Qemu DSP & OR Cadence Host Emulator Emulator Android m4 HAL topology debug XML(*.xml OR file ) (*m4) SOF ITT Topology m4 rmbox / SOF Source file trace (*.conf) Firmware Code Tool alsaplg Firmware source Zephyr OS debugfs Cadence GCC Audio DSP Compiler Compiler Topology Metadata ASoC Linux SOF Kernel SOF ELF /lib/firmware/intel ImageELF Driver / Driver Image Firmware Signed Image rimage Tool ALSA SoC IPC/Mailbox Topology Legend Unsigned Signing ImageImage Tool SOF Source Tool 3rd Party Tool SDK Binary Image *Note: Docker contains only the open source components Optional Config Data SDK diagram – complete SDK 23 Core Pillars Open source Community driven Permissive1 Modular Portable Tool-rich • BSD/MITFlexibility licensed in auditing• Configurable andtopology customizing • Platform, firmwarefor desired OS, use• BSD/GPL case licensed and firmware and topologies allows for flexible DSP architecture agnostic proprietary tools available customizations • BSD/GPL dual licensed • Portable to other host • Includes toolchain, driver • Extendable using existing platform and DSP debugger, emulator, APIs to add custom or 3rd architectures scripts, firmware creation • Firmware code changes party binary modules tools can be private or up- • Extendable to different streamed • Allows for custom ABIs real-time DSP OSes • Configurable builds using between modules supported GNU GCC or • Modifiable for any custom proprietary toolchains • Develop and test out new integrations modules before deploying • Virtualize both DSP and onto target HW host OS via QEMU • Virtualize trace data in real time 1: SDK tools are a mixture of BSD, GPL and proprietary licenses. 24 Thanks! Q & A Drop by the booth. www.sofproject.com https://github.com/thesofproject 25 Thank you 26.