NetBSD on Marvell Armada XP System on a Chip

Zbigniew Bodek [email protected]

EuroBSDCon 2013, Malta

1 NetBSD on Marvell Armada XP System on a Chip Presentation outline

‣ Justification for choice ‣ Hardware overview ‣ Porting to a modern ARM ‣ Prerequisites ‣ Essentials ‣ Device drivers ‣ Validation ‣ Integration to mainline NetBSD ‣ Comparison with the mainline FreeBSD

2 NetBSD on Marvell Armada XP System on a Chip Justification for choice

‣ Why should we be interested in ARM?

3 NetBSD on Marvell Armada XP System on a Chip Justification for choice

‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets

4 NetBSD on Marvell Armada XP System on a Chip Justification for choice

‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets

‣ Why we invested our work in ?

5 NetBSD on Marvell Armada XP System on a Chip Justification for choice

‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets

‣ Why we invested our work in ? ‣ Customer requirement ‣ Known of it’s portability ‣ Friendly license ‣ Active community ‣ Good alternative to ”other similar”

6 NetBSD on Marvell Armada XP System on a Chip Justification for choice

‣ Advantages ‣ Building for ARMv7 supported ‣ Memory management supported ‣ Whole set of drivers for older Marvell chips ‣ Disadvantages ‣ No settled support for ARMv7 platform (at the time) ‣ No SMP for ARM

7 NetBSD on Marvell Armada XP System on a Chip Hardware overview

‣ Marvell Armada XP ‣ Quad core PJ4Bv7 ‣ 2MB shared L2 cache ‣ Hardware cache coherency, I/O coherency ‣ Integrated storage (USB 2.0, SATA II, NAND, NOR...) ‣ High speed networking (NETA) ‣ PCI-Express ‣ Cryptographic engines ‣ DMA engines ‣ Advanced power management

8 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Prerequisites ‣ Arrange booting environment ‣ Define code base ‣ Compiler availability ‣ Support for similar platform

6.0-RC1 / BEAGLE 6.0.1 / ARMADAXP

9 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Prerequisites ‣ Understand work to be done ‣ Is the CPU core supported? ‣ Is the SoC family supported? ‣ Is the particular board supported? ‣ What peripheral devices are supported? ‣ Armada XP ➡ New CPU core - PJ4B ➡ New SoC family - Armada XP ➡ New evaluation boards ➡ Some peripherals matching existing drivers

10 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Prerequisites ‣ Arrange configuration files ‣ Processor specific under sys/arch/arm/marvell/files.marvell ‣ Defines core device specification for whole chip family ‣ Evaluation board specific under sys/arch/evbarm/conf/ ‣ Board specific files and option definitions → files.armadaxp ‣ Extra specification for Makefile → mk.armadaxp ‣ Set of board specific configuration options → std.armadaxp ‣ Kernel configuration file → ARMADAXP

11 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ CPU core support ‣ What does it mean?

12 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ CPU core support ‣ What does it mean?

NetBSD/arm utilizes a set of ”cpu functions” defined in:

sys/arch/arm/include/cpufunc.h (see struct cpu_functions)

‣ Function wrappers for basic CPU-related operations such as: ‣ cache manipulation routines ‣ core detection and configuration (including MMU) ‣ read/write buffers, branch predictor management ‣ other

13 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ CPU core support ‣ Do not increase redundancy - sufficient CPU functions can be reused ‣ Remember of the core bugs and/or erratum Example:

ENTRY(pj4b_setttb) !/* Cache synchronization is not required as this core has PIPT caches */ !dsb #if defined(L2CACHE_ENABLE) && defined(AURORA_L2_PT_WALK) !orr!r2, r0, #TTB_FLAGS_UP #else !bic!r2, r0, #0x18 #endif !mcr!p15, 0, r0, c2, c0, 0!/* load new TTB */ !mov!r0, #0 !isb !mcr!p15, 0, r0, c8, c7, 0!/* invalidate I+D TLBs */ !RET

14 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ CPU core support 1. Create custom CPU functions sys/arch/arm/arm/cpufunc_asm_pj4b.S 2.Add new CPU to configuration files sys/arch/arm/conf/files.arm sys/arch/evbarm/conf/ARMADAXP 3.Add CPU class and CPU ID to auto-detection sys/arch/arm/arm32/cpu.c 4.Fill-up cpu_functions structure with the desired CPU ops 4.1.Implement _setup routine

15 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Start code Assumptions: • Basic SoC initialization is done by firmware • First part of the kernel to be executed • Start code file and object is defined in mk.armadaxp • Written in Assembly language: • To allow relocation • No environment for C code Goals to achieve: • Remap kernel to the true VA • Redirect to locore.S for stack initialization

16 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Start code VA PA

MMU ON 0xC0000000

MMU OFF 0x00000000

17 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Bootstrap virtual memory! • Create initial page tables • pmap_bootstrap() • Map devices for bus_space_map() • Allocate and create mappings for exception vectors page, message buffer, stack, etc. • Inform UVM about the available pages

18 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Initialize all crucial chip features • CPU (functions) • Decoding windows • Clocks (time base) • (preconfigure) • L2 cache • I/O cache coherency

19 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Other important configurations • Abort handlers • Stack pointer • System console • KGDB

20 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ L2 cache management

CPU

CACHE

L1 CACHE

L2 CACHE

21 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ L2 cache management 1. Set L2 policy (LRU, WB) 2. Enable cache and TLB maintenance broadcast 3. Invalidate cache 4. Set PoC and PoU 5. Finally... enable L2 cache

‣ CPU functions for cache maintenance will do the rest

22 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric • i.a. forwards transactions initiated by the I/O devices and tagged with a shared attribute onto the processors snoop engine. On-chip caches can then snoop these transactions ↓ COHERENT DMA

23 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric

CPU

L2 CACHE L1 CACHE

DRAM COHERENCY FABRIC controller

BUS DEV DEV

24 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ I/O cache coherency ‣ Adjustments (hacks) to the bus_dma and pmap ‣ Set the pages attributes to shared ‣ Override default non-coherent settings in ‣ _bus_dmamap_load_mbuf() ‣ _bus_dmamap_load_buffer() ‣ Add HW synchronization barrier to ‣ _bus_dmamap_sync()

NO NEED FOR SW CACHE MAINTENANCE ON I/O!

25 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Atomic operations ‣ Issues with atomic byte swapping required for locking Deprecated swp and swpb instructions resulted in undefined instruction exceptions in kernel. • Exchanged to ldrex strex pair • Added barriers after acquiring the lock & before releasing

‣ More issues with the building distribution. Undefined instructions (swp & swpb) in shared libraries. • Need for passing ARM_ARCH_7 definition to distribution build • Inserting barrier instruction opcode for non-kernel builds

DMB ! .word 0xf57ff05f

26 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Interrupts Multiprocessor Controller

CPU HW prioritization 4

MPIC

1 2 4 2 DEV0 DEV1 DEV2 DEV3

27 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Interrupts Multiprocessor Interrupt Controller • Great for implementing NetBSD SPL mechanism • Increasing/decreasing prio done in HW • Controller takes care of proper interrupts notification • Much faster and simpler than the solution for older Marvell chips • All pic_ops needed to be implemented - establishing - blocking - unblocking

28 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Armada XP support attached to the existing mvsoc

‣ mvsoc bus uses direct configuration method ‣ Resource description is not taken from the array dynamically created by config(8) basing on the configuration nodes from kernel configuration file ‣ All resources are defined in advance ‣ Bus needs to probe drivers to know their existence

29 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Generic bus for all Marvell platforms ‣ Unified autoconfiguration process ‣ Unified chip-detection ‣ Common resource management ‣ Was already in the tree

‣ Difficult to maintain for a large number of platforms

30 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Device drivers ‣ Reused in-tree drivers for: • Timers (with significant changes) • Serial IF (added support for ns16750) • PCIe (mvpex with adjustments) • USB (with changes) • SATA • RTC • I2C (with changes) ‣ Developed • Network Accelerator • SPI (for Flash)

31 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM

‣ Validation ‣ Testing with ATF (Automated Testing Framework) ‣ Simple way for overall system reliability check ‣ atf-run | atf-report and wait (then read log) ‣ Helped in finding undefined instructions in libraries ‣ 2904 passed test cases ‣ 71 failed test cases (mostly known issues for that baseline) ‣ Testing suites: ‣ LMbench ‣ iperf ‣ Long runs of distribution buils

32 NetBSD on Marvell Armada XP System on a Chip Integration to mainline NetBSD

‣ Changes in the current development branch ‣ Fixed bugs with atomic operations ‣ Unified and simplified initarm() ‣ NetBSD/arm community remarks ‣ Upstreamed features: ‣ Armada XP essentials ‣ ns16750 support ‣ SPI driver ‣ Bug fixes to: spiflash.c, mvpex.c

33 NetBSD on Marvell Armada XP System on a Chip Integration to mainline NetBSD

‣ Not in the mainline: ‣ Networking driver ‣ I/O cache coherency support

34 NetBSD on Marvell Armada XP System on a Chip Comparison with the mainline FreeBSD

● SMP support (quad core operations) ● UP only ● FDT support ● Hard-coded device descriptions ● Superpages support ● Existing device drivers: ● Existing device drivers: ● MPIC ● MPIC ● Timers ● Timers ● UART ● UART ● PCIe ● PCIe ● SATA ● SATA ● USB 2.0 ● USB 2.0 ● I2C ● I2C ● SDIO ● SPI

35 NetBSD on Marvell Armada XP System on a Chip Acknowledgements

‣ Mentors of the project ‣ Rafał Jaworowski (Semihalf, The FreeBSD Project) ‣ Bartłomiej Sięka (Semihalf)

‣ People involved ‣ Grzegorz Bernacki ‣ Michał Dubiel ‣ Grzegorz Jaszczyk ‣ Pablo Ribalta ‣ Marcin Wojtas

36 NetBSD on Marvell Armada XP System on a Chip Acknowledgements Special thanks to

‣ Radosław Kujawa (NetBSD project)

‣ Shadi Ammouri (Marvell)

37 NetBSD on Marvell Armada XP System on a Chip

Any questions?

38