NetBSD on Marvell Armada XP System on a Chip
Zbigniew Bodek [email protected]
EuroBSDCon 2013, Malta
1 NetBSD on Marvell Armada XP System on a Chip Presentation outline
‣ Justification for choice ‣ Hardware overview ‣ Porting to a modern ARM ‣ Prerequisites ‣ Essentials ‣ Device drivers ‣ Validation ‣ Integration to mainline NetBSD ‣ Comparison with the mainline FreeBSD
2 NetBSD on Marvell Armada XP System on a Chip Justification for choice
‣ Why should we be interested in ARM?
3 NetBSD on Marvell Armada XP System on a Chip Justification for choice
‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets
4 NetBSD on Marvell Armada XP System on a Chip Justification for choice
‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets
‣ Why we invested our work in ?
5 NetBSD on Marvell Armada XP System on a Chip Justification for choice
‣ Why should we be interested in ARM? ‣ Industry trend to save energy ‣ High performance solutions = new markets
‣ Why we invested our work in ? ‣ Customer requirement ‣ Known of it’s portability ‣ Friendly license ‣ Active community ‣ Good alternative to ”other similar”
6 NetBSD on Marvell Armada XP System on a Chip Justification for choice
‣ Advantages ‣ Building for ARMv7 supported ‣ Memory management supported ‣ Whole set of drivers for older Marvell chips ‣ Disadvantages ‣ No settled support for ARMv7 platform (at the time) ‣ No SMP for ARM
7 NetBSD on Marvell Armada XP System on a Chip Hardware overview
‣ Marvell Armada XP ‣ Quad core PJ4Bv7 ‣ 2MB shared L2 cache ‣ Hardware cache coherency, I/O coherency ‣ Integrated storage (USB 2.0, SATA II, NAND, NOR...) ‣ High speed networking (NETA) ‣ PCI-Express ‣ Cryptographic engines ‣ DMA engines ‣ Advanced power management
8 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Prerequisites ‣ Arrange booting environment ‣ Define code base ‣ Compiler availability ‣ Support for similar platform
6.0-RC1 / BEAGLE 6.0.1 / ARMADAXP
9 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Prerequisites ‣ Understand work to be done ‣ Is the CPU core supported? ‣ Is the SoC family supported? ‣ Is the particular board supported? ‣ What peripheral devices are supported? ‣ Armada XP ➡ New CPU core - PJ4B ➡ New SoC family - Armada XP ➡ New evaluation boards ➡ Some peripherals matching existing drivers
10 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Prerequisites ‣ Arrange configuration files ‣ Processor specific under sys/arch/arm/marvell/files.marvell ‣ Defines core device specification for whole chip family ‣ Evaluation board specific under sys/arch/evbarm/conf/ ‣ Board specific files and option definitions → files.armadaxp ‣ Extra specification for Makefile → mk.armadaxp ‣ Set of board specific configuration options → std.armadaxp ‣ Kernel configuration file → ARMADAXP
11 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ CPU core support ‣ What does it mean?
12 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ CPU core support ‣ What does it mean?
NetBSD/arm utilizes a set of ”cpu functions” defined in:
sys/arch/arm/include/cpufunc.h (see struct cpu_functions)
‣ Function wrappers for basic CPU-related operations such as: ‣ cache manipulation routines ‣ core detection and configuration (including MMU) ‣ read/write buffers, branch predictor management ‣ other
13 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ CPU core support ‣ Do not increase redundancy - sufficient CPU functions can be reused ‣ Remember of the core bugs and/or erratum Example:
ENTRY(pj4b_setttb) !/* Cache synchronization is not required as this core has PIPT caches */ !dsb #if defined(L2CACHE_ENABLE) && defined(AURORA_L2_PT_WALK) !orr!r2, r0, #TTB_FLAGS_UP #else !bic!r2, r0, #0x18 #endif !mcr!p15, 0, r0, c2, c0, 0!/* load new TTB */ !mov!r0, #0 !isb !mcr!p15, 0, r0, c8, c7, 0!/* invalidate I+D TLBs */ !RET
14 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ CPU core support 1. Create custom CPU functions sys/arch/arm/arm/cpufunc_asm_pj4b.S 2.Add new CPU to configuration files sys/arch/arm/conf/files.arm sys/arch/evbarm/conf/ARMADAXP 3.Add CPU class and CPU ID to auto-detection sys/arch/arm/arm32/cpu.c 4.Fill-up cpu_functions structure with the desired CPU ops 4.1.Implement
15 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Start code Assumptions: • Basic SoC initialization is done by firmware • First part of the kernel to be executed • Start code file and object is defined in mk.armadaxp • Written in Assembly language: • To allow relocation • No environment for C code Goals to achieve: • Remap kernel to the true VA • Redirect to locore.S for stack initialization
16 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Start code VA PA
MMU ON 0xC0000000
MMU OFF 0x00000000
17 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Bootstrap virtual memory! • Create initial page tables • pmap_bootstrap() • Map devices for bus_space_map() • Allocate and create mappings for exception vectors page, message buffer, stack, etc. • Inform UVM about the available pages
18 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Initialize all crucial chip features • CPU (functions) • Decoding windows • Clocks (time base) • Interrupts (preconfigure) • L2 cache • I/O cache coherency
19 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Machine initialization initarm() from sys/arch/evbarm/armadaxp/armadaxp_machdep.c • First C code executed • Includes all actions that need to be performed before jumping to main() Goals to achieve: • Other important configurations • Abort handlers • Stack pointer • System console • KGDB
20 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ L2 cache management
CPU
CACHE
L1 CACHE
L2 CACHE
21 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ L2 cache management 1. Set L2 policy (LRU, WB) 2. Enable cache and TLB maintenance broadcast 3. Invalidate cache 4. Set PoC and PoU 5. Finally... enable L2 cache
‣ CPU functions for cache maintenance will do the rest
22 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric • i.a. forwards transactions initiated by the I/O devices and tagged with a shared attribute onto the processors snoop engine. On-chip caches can then snoop these transactions ↓ COHERENT DMA
23 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ I/O cache coherency ‣ The Coherency Fabric
CPU
L2 CACHE L1 CACHE
DRAM COHERENCY FABRIC controller
BUS DEV DEV
24 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ I/O cache coherency ‣ Adjustments (hacks) to the bus_dma and pmap ‣ Set the pages attributes to shared ‣ Override default non-coherent settings in ‣ _bus_dmamap_load_mbuf() ‣ _bus_dmamap_load_buffer() ‣ Add HW synchronization barrier to ‣ _bus_dmamap_sync()
NO NEED FOR SW CACHE MAINTENANCE ON I/O!
25 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Atomic operations ‣ Issues with atomic byte swapping required for locking Deprecated swp and swpb instructions resulted in undefined instruction exceptions in kernel. • Exchanged to ldrex strex pair • Added barriers after acquiring the lock & before releasing
‣ More issues with the building distribution. Undefined instructions (swp & swpb) in shared libraries. • Need for passing ARM_ARCH_7 definition to distribution build • Inserting barrier instruction opcode for non-kernel builds
DMB ! .word 0xf57ff05f
26 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Interrupts Multiprocessor Interrupt Controller
CPU HW prioritization 4
MPIC
1 2 4 2 DEV0 DEV1 DEV2 DEV3
27 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Interrupts Multiprocessor Interrupt Controller • Great for implementing NetBSD SPL mechanism • Increasing/decreasing prio done in HW • Controller takes care of proper interrupts notification • Much faster and simpler than the solution for older Marvell chips • All pic_ops needed to be implemented - establishing - blocking - unblocking
28 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Armada XP support attached to the existing mvsoc
‣ mvsoc bus uses direct configuration method ‣ Resource description is not taken from the array dynamically created by config(8) basing on the configuration nodes from kernel configuration file ‣ All resources are defined in advance ‣ Bus needs to probe drivers to know their existence
29 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Essentials ‣ Marvell SoC abstract bus (mvsoc) ‣ Generic bus for all Marvell platforms ‣ Unified autoconfiguration process ‣ Unified chip-detection ‣ Common resource management ‣ Was already in the tree
‣ Difficult to maintain for a large number of platforms
30 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Device drivers ‣ Reused in-tree drivers for: • Timers (with significant changes) • Serial IF (added support for ns16750) • PCIe (mvpex with adjustments) • USB (with changes) • SATA • RTC • I2C (with changes) ‣ Developed • Network Accelerator • SPI (for Flash)
31 NetBSD on Marvell Armada XP System on a Chip Porting to a modern ARM
‣ Validation ‣ Testing with ATF (Automated Testing Framework) ‣ Simple way for overall system reliability check ‣ atf-run | atf-report and wait (then read log) ‣ Helped in finding undefined instructions in libraries ‣ 2904 passed test cases ‣ 71 failed test cases (mostly known issues for that baseline) ‣ Testing suites: ‣ LMbench ‣ iperf ‣ Long runs of distribution buils
32 NetBSD on Marvell Armada XP System on a Chip Integration to mainline NetBSD
‣ Changes in the current development branch ‣ Fixed bugs with atomic operations ‣ Unified and simplified initarm() ‣ NetBSD/arm community remarks ‣ Upstreamed features: ‣ Armada XP essentials ‣ ns16750 support ‣ SPI driver ‣ Bug fixes to: spiflash.c, mvpex.c
33 NetBSD on Marvell Armada XP System on a Chip Integration to mainline NetBSD
‣ Not in the mainline: ‣ Networking driver ‣ I/O cache coherency support
34 NetBSD on Marvell Armada XP System on a Chip Comparison with the mainline FreeBSD
● SMP support (quad core operations) ● UP only ● FDT support ● Hard-coded device descriptions ● Superpages support ● Existing device drivers: ● Existing device drivers: ● MPIC ● MPIC ● Timers ● Timers ● UART ● UART ● PCIe ● PCIe ● SATA ● SATA ● USB 2.0 ● USB 2.0 ● I2C ● I2C ● SDIO ● SPI
35 NetBSD on Marvell Armada XP System on a Chip Acknowledgements
‣ Mentors of the project ‣ Rafał Jaworowski (Semihalf, The FreeBSD Project) ‣ Bartłomiej Sięka (Semihalf)
‣ People involved ‣ Grzegorz Bernacki ‣ Michał Dubiel ‣ Grzegorz Jaszczyk ‣ Pablo Ribalta ‣ Marcin Wojtas
36 NetBSD on Marvell Armada XP System on a Chip Acknowledgements Special thanks to
‣ Radosław Kujawa (NetBSD project)
‣ Shadi Ammouri (Marvell)
37 NetBSD on Marvell Armada XP System on a Chip
Any questions?
38