NetBSD on Marvell Armada XP System on a Chip Zbigniew Bodek [email protected] EuroBSDCon 2013, Malta 1 NetBSD on Marvell Armada XP System on a Chip Presentation outline ‣ Justification for choice ‣ Hardware overview ‣ Porting to a modern ARM ‣ Prerequisites ‣ Essentials ‣ Device drivers ‣ Validation ‣ Integration to mainline NetBSD ‣ Comparison with the mainline FreeBSD 2 NetBSD on Marvell Armada XP System on a Chip Justification for choice ‣ Why should we be interested in ARM? 3 NetBSD on Marvell Armada XP System on a Chip Justification for choice ‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets 4 NetBSD on Marvell Armada XP System on a Chip Justification for choice ‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets ‣ Why we invested our work in ? 5 NetBSD on Marvell Armada XP System on a Chip Justification for choice ‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets ‣ Why we invested our work in ? ‣ Customer requirement ‣ Known of it’s portability ‣ Friendly license ‣ Active community ‣ Good alternative to ”other similar” 6 NetBSD on Marvell Armada XP System on a Chip Justification for choice ‣ Advantages ‣ Building for ARMv7 supported ‣ Memory management supported ‣ Whole set of drivers for older Marvell chips ‣ Disadvantages ‣ No settled support for ARMv7 platform (at the time) ‣ No SMP for ARM 7 NetBSD on Marvell Armada XP System on a Chip Hardware overview ‣ Marvell Armada XP ‣ Quad core PJ4Bv7 ‣ 2MB shared L2 cache ‣ Hardware cache coherency, I/O coherency ‣ Integrated storage (USB 2.0, SATA II, NAND, NOR...) ‣ High speed networking (NETA) ‣ PCI-Express ‣ Cryptographic engines ‣ DMA engines ‣ Advanced power management 8 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Prerequisites ‣ Arrange booting environment ‣ Define code base ‣ Compiler availability ‣ Support for similar platform 6.0-RC1 / BEAGLE 6.0.1 / ARMADAXP 9 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Prerequisites ‣ Understand work to be done ‣ Is the CPU core supported? ‣ Is the SoC family supported? ‣ Is the particular board supported? ‣ What peripheral devices are supported? ‣ Armada XP ➡ New CPU core - PJ4B ➡ New SoC family - Armada XP ➡ New evaluation boards ➡ Some peripherals matching existing drivers 10 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Prerequisites ‣ Arrange configuration files ‣ Processor specific under sys/arch/arm/marvell/files.marvell ‣ Defines core device specification for whole chip family ‣ Evaluation board specific under sys/arch/evbarm/conf/ ‣ Board specific files and option definitions → files.armadaxp ‣ Extra specification for Makefile → mk.armadaxp ‣ Set of board specific configuration options → std.armadaxp ‣ Kernel configuration file → ARMADAXP 11 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ CPU core support ‣ What does it mean? 12 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ CPU core support ‣ What does it mean? NetBSD/arm utilizes a set of ”cpu functions” defined in: sys/arch/arm/include/cpufunc.h (see struct cpu_functions) ‣ Function wrappers for basic CPU-related operations such as: ‣ cache manipulation routines ‣ core detection and configuration (including MMU) ‣ read/write buffers, branch predictor management ‣ other 13 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ CPU core support ‣ Do not increase redundancy - sufficient CPU functions can be reused ‣ Remember of the core bugs and/or erratum Example: ENTRY(pj4b_setttb) !/* Cache synchronization is not required as this core has PIPT caches */ !dsb #if defined(L2CACHE_ENABLE) && defined(AURORA_L2_PT_WALK) !orr!r2, r0, #TTB_FLAGS_UP #else !bic!r2, r0, #0x18 #endif !mcr!p15, 0, r0, c2, c0, 0!/* load new TTB */ !mov!r0, #0 !isb !mcr!p15, 0, r0, c8, c7, 0!/* invalidate I+D TLBs */ !RET 14 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ CPU core support 1. Create custom CPU functions sys/arch/arm/arm/cpufunc_asm_pj4b.S 2.Add new CPU to configuration files sys/arch/arm/conf/files.arm sys/arch/evbarm/conf/ARMADAXP 3.Add CPU class and CPU ID to auto-detection sys/arch/arm/arm32/cpu.c 4.Fill-up cpu_functions structure with the desired CPU ops 4.1.Implement <cpu>_setup routine 15 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Start code Assumptions: • Basic SoC initialization is done by firmware • First part of the kernel to be executed • Start code file and object is defined in mk.armadaxp • Written in Assembly language: • To allow relocation • No environment for C code Goals to achieve: • Remap kernel to the true VA • Redirect to locore.S for stack initialization 16 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Start code VA PA MMU ON 0xC0000000 MMU OFF 0x00000000 17 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Bootstrap virtual memory! • Create initial page tables • pmap_bootstrap() • Map devices for bus_space_map() • Allocate and create mappings for exception vectors page, message buffer, stack, etc. • Inform UVM about the available pages 18 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Initialize all crucial chip features • CPU (functions) • Decoding windows • Clocks (time base) • Interrupts (preconfigure) • L2 cache • I/O cache coherency 19 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Other important configurations • Abort handlers • Stack pointer • System console • KGDB 20 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ L2 cache management CPU CACHE L1 CACHE L2 CACHE 21 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ L2 cache management 1. Set L2 policy (LRU, WB) 2. Enable cache and TLB maintenance broadcast 3. Invalidate cache 4. Set PoC and PoU 5. Finally... enable L2 cache ‣ CPU functions for cache maintenance will do the rest 22 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric • i.a. forwards transactions initiated by the I/O devices and tagged with a shared attribute onto the processors snoop engine. On-chip caches can then snoop these transactions ↓ COHERENT DMA 23 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric CPU L2 CACHE L1 CACHE DRAM COHERENCY FABRIC controller BUS DEV DEV 24 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ I/O cache coherency ‣ Adjustments (hacks) to the bus_dma and pmap ‣ Set the pages attributes to shared ‣ Override default non-coherent settings in ‣ _bus_dmamap_load_mbuf() ‣ _bus_dmamap_load_buffer() ‣ Add HW synchronization barrier to ‣ _bus_dmamap_sync() NO NEED FOR SW CACHE MAINTENANCE ON I/O! 25 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Atomic operations ‣ Issues with atomic byte swapping required for locking Deprecated swp and swpb instructions resulted in undefined instruction exceptions in kernel. • Exchanged to ldrex strex pair • Added barriers after acquiring the lock & before releasing ‣ More issues with the building distribution. Undefined instructions (swp & swpb) in shared libraries. • Need for passing ARM_ARCH_7 definition to distribution build • Inserting barrier instruction opcode for non-kernel builds DMB ! .word 0xf57ff05f 26 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Interrupts Multiprocessor Interrupt Controller CPU HW prioritization 4 MPIC 1 2 4 2 DEV0 DEV1 DEV2 DEV3 27 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Interrupts Multiprocessor Interrupt Controller • Great for implementing NetBSD SPL mechanism • Increasing/decreasing prio done in HW • Controller takes care of proper interrupts notification • Much faster and simpler than the solution for older Marvell chips • All pic_ops needed to be implemented - establishing - blocking - unblocking 28 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Armada XP support attached to the existing mvsoc ‣ mvsoc bus uses direct configuration method ‣ Resource description is not taken from the array dynamically created by config(8) basing on the configuration nodes from kernel configuration file ‣ All resources are defined in advance ‣ Bus needs to probe drivers to know their existence 29 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM ‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Generic bus for all Marvell platforms ‣ Unified autoconfiguration process ‣ Unified