Journey from Closed to Open:

Lessons Learned from Open Sourcing Sound Open Mission

1.Inspire others to open source firmware.

2.Show how firmware can be open sourced.

3.Discuss common challenges open sourcing firmware and how to overcome them.

4.Introduce Sound Open Firmware (SOF) architecture.

2 About Me

 Employed by Intel as a software architect.

 Linux user and engineer since 1994.

 Developed AsoC and PMIC abstraction layers.

 Linux audio engineer since 2003.

 Working on audio DSP’s since 2008.

 Working on SOF since 2015.

3 Sound Open Firmware is an open source audio digital signal processing frmware and driver infrastructure.

4 The Dark Ages

*Darkness due to lack of source code 5 Closed Source Firmware

 Historically the standard practice amongst firmware development teams with a few exceptions.

 Often seen as a black box by integrators and driver & user space developers.

 Makes it more difficult to debug problems in other areas of the stack.

 Impossible for upper stack developers to debug hardware problems.

 Usually developed alongside a single OS specific driver.

 Documentation never truly matches or keeps up with firmware source code.

? Data In Data Out

6 Changing Course

Factors influencing open sourcing. 7 Market Drivers

• Demand for speech and voice authentication and recognition applications and technologies • Impact of AI on accuracy of speech and voice recognitions • Growth in voice control- based devices in consumer and enterprise markets

Audio, speech and voice has become ubiquitous

8 Open Source Hardware

 Minnowboard Project

 Open Source Hardware.

 Baytrail CPU, dual core @1.33GHz, 2GB DDR

 Tensilica Xtensa audio DSP @ 400MHz

 Open schematics, PCB layout, BOM

 Open source Software

 Full open source Linux software stack

 Open source Firmware

 Open source BIOS

 NO open source audio DSP firmware

9 Minnowboard DSP Architecture

Instruction Instruction  Cache RAM Xtensa HiFi 2EP core.

to/from HiFi EP Core OCP (Upstream) IOSF2OCP  96kB Instruction RAM bridge

 168kB Data RAM JTAGJTAG JTAG EXT 64 bit AXI Trace HiFi EP PIF PIF2AXI 32-bit APB Slave Master and Slave Logic Core I/F SSP0 Interrupts  Interrupt (from shim) A M/N 2 * DMACs A CLK Uu LPE D  Shim d 3 * I2S ports I Control & 32-bit APB Slave Config Oi signals SSP1  4kB RF o PCI device from host OS Mailbox M/N CLK Data Data RAM 32 bit S Cache OCP Slave MF LPE OCP Master DMA_00 Writes Xa 32-bit APB Slave OCP SSP2 Master Reads b M/N Source: CLK 32 bit r OCP Slave Tangier LPE OCP i Master DMA_01 Writes Sonics OCP c Master VLV Reads

10 Challenges Ahead

Be prepared ! 11 Political Challenges

Challenges Solutions

• Company policy for audio DSP firmware • Be prepared to fight the same battle more was historically closed source. than once.

• Most colleagues were initially either • Build ground swell of opinion and facts strongly in favour or strongly against. behind open source. • • Fears around disclosing IP. Helps if you have proof of concept code - “skunkworks”! • How do we add value? • Tell the world about your project!

“You may have to fight a battle more than once to win it.” Margaret Thatcher 12 Technical Challenges - Compiler

 Xtensa ISA differs between cores adding complexity for compilers. SOF Source SOF Source Code Code  Cadence provide an optimising compiler for Xtensa Firmware source Firmware source  Commercial license for some targets $$$ Cadence GCC Compiler Compiler  Free for some targets like Minnowboard :)

ELF ELF  ELF Need open source compiler for community. ImageELF Image Image Image

 GCC supports xtensa base ISA

 GCC does not support xtensa SIMD/VLIW.

Add GCC support for Minnowboard xtensa ISA.

13 Technical Challenges – Image Builder

SOF Source  Compilers generate ELF images. Code  Firmware source ELF images don’t run on bare metal.

Cadence GCC  ELF image needs converted to binary image Compiler Compiler format for loading into DSP memories.

ELF  ImageELF Need to have tools that can convert ELF to Image multiple different firmware image formats. Rimage tool Create tool to convert

Unsigned ELF images into multiple ImageImage different image formats.

Should be able to run code now ! 14 Technical Challenges – DSP Emulator

 Code not easy to debug

 No debugger (yet) Xtensa Qemu

 No printf(); SOF Firmware

FW DMA driver  JTAG can’t be used – Intel only.

FW SSP driver  Emulation can be used to debug bring up. FW Audio mixer

 Qemu already supports base Xtensa ISA VM #1

 Qemu support added xtensa  GDB Minnowboard DSP On host

 Extra registers and instruction not in base ISA

15 Technical Challenges – Heterogeneous Emulator

 Firmware must be debugged alongside driver.

X86 Xtensa Qemu  Qemu/KVM Qemu used to virtualise drivers and firmware together. Guest OS * SOF Firmware Qemu IO Bridge  Audio DSP driver FW DMA driver Host side almost real time. IO, DMA IRQs Codec driver FW SSP driver  DSP side emulated.

Audio Userspace FW Audio mixer VM #0 VM #1

xtensa x86 GDB GDB On host On host

16 Technical Challenges – Code Signing

 Newer DSP hardware has security that validates firmware SOF Source binaries. Code

Firmware source  Code signing support was added to open source rimage tool. Cadence GCC

 PCKS #1.5 Compiler Compiler

 Openssl ELF ImageELF Image  Created “public” private key to be used for developer hardware Rimage e.g. UP^2 tool

 Implications with GPLv3 “tivoisation” clause. Unsigned ImageImage

Signing Tool

Signed ImageImage

17 Build a Community  A healthy open source project needs a healthy community!

 Who are the community?

 Commercial users who deploy in products.

 Non commercial hobbyists.

 Academics researching audio processing algorithms.

 Use external code hosting platforms like Github, mailing lists, IRC, wiki’s etc and do development in public.

 Release patches and code “early and often”.

 Don’t do infrequent code drops.

 No “abandonware”.

 Accept patches from others.

 Accept bug reports from others. 18 Hello World

19 Firmware Architecture

DSP or SOF Audio Components Discrete X86 or ARM mixer volume ASRC AEC/NS Chip Host buffer speaker mux EQ queue protection splitter SRC tone

Linux Generic Micro Kernel ASoC Audio (XTOS, Zephyr, FreeRTOS options) Audio Tasks msg work SOF boot queues queues memory Driver Control IPC HW/IPC Data IRQs schd mutex exceptions Configurable driver pipelines RTOS/HV atomic Audio timer threads services semaphores Driver Platform Drivers

DMA HDA I2C

DMIC GNA SPI

SSP USB UART GPIO

2 of 2 Firmware diagram – complete diagram 20 Driver Architecture

ASoC Machine Driver

codec board HW config integration integration

Generic PCM Driver DSP or topology PCMs Kcontrols IPC via Mailbox Discrete Buffers via DMA DAPM Chip

Generic IPC Driver

mixer stream PM

pipeline

DSP Platform Driver doorbell mailbox IRQ

code loader IO PM

2 of 2 Driver diagram – complete diagram 21 History

Year ending 2015 2016 2017 2018

Inception

Platforms: Bay Trail Features: Qemu Xtensa

Firmware open sourced Platforms: Cherry Trail and Braswell Features: Qemu host SOF 1.0 public release Driver open sourced Platforms: Apollo Lake, Haswell, Broadwell Features: BSD/GPL driver, topologies Became Linux Foundation project Driver upstream Platforms: Cannon Lake, Ice Lake, Sue Creek Features: Docker build, SOF on host 22 Development Kit

Qemu DSP & OR Cadence Host Emulator Emulator Android m4 HAL topology debug XML(*.xml OR file ) (*m4) SOF ITT Topology m4 rmbox / SOF Source file trace (*.conf) Firmware Code Tool alsaplg Firmware source Zephyr OS debugfs

Cadence GCC Audio DSP Compiler Compiler Topology Metadata ASoC Linux SOF Kernel SOF ELF /lib/firmware/intel ImageELF Driver / Driver Image Firmware

Signed Image rimage Tool ALSA SoC IPC/Mailbox Topology Legend Unsigned Signing ImageImage Tool SOF Source Tool 3rd Party Tool SDK Binary Image

*Note: Docker contains only the open source components Optional Config Data

SDK diagram – complete SDK 23 Core Pillars

Open source  Community driven

Permissive1 Modular Portable Tool-rich • BSD/MITFlexibility licensed in auditing• Configurable andtopology customizing • Platform, firmwarefor desired OS, use• BSD/GPL case licensed and firmware and topologies allows for flexible DSP architecture agnostic proprietary tools available customizations • BSD/GPL dual licensed • Portable to other host • Includes toolchain, driver • Extendable using existing platform and DSP debugger, emulator, APIs to add custom or 3rd architectures scripts, firmware creation • Firmware code changes party binary modules tools can be private or up- • Extendable to different streamed • Allows for custom ABIs real-time DSP OSes • Configurable builds using between modules supported GNU GCC or • Modifiable for any custom proprietary toolchains • Develop and test out new integrations modules before deploying • Virtualize both DSP and onto target HW host OS via QEMU • Virtualize trace data in real time

1: SDK tools are a mixture of BSD, GPL and proprietary licenses. 24 Thanks!

Q & A

Drop by the booth.

www.sofproject.com https://github.com/thesofproject

25 Thank you 26