Nomadik Application Processor Andrea Gallo Giancarlo Asnaghi ST is #1 world-wide leader in Digital TV and Consumer Audio
MP3 Portable Digital Satellite Radio Set Top Box Player Digital Car Radio DVD Player
MMDSP+ inside more than 200 million produced chips
January 14, 2009 ST leader in mobile phone chips
January 14, 2009 Nomadik
Nomadik is based on this heritage providing: – Unrivalled multimedia performances – Very low power consumption – Scalable performances
January 14, 2009 BestBest ApplicationApplication ProcessorProcessor 20042004
9 Lowest power consumption 9 Scalable performance 9 Video/Audio quality 9 Cost-effective
Nominees: Intel XScale PXA260, NeoMagic MiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611
January 14, 2009 Nomadik
Nomadik is a family of Application Processors – Distributed processing architecture ARM9 + multiple Smart Accelerators – Support of a wide range of OS and applications – Seamless integration in the OS through standard API drivers and MM framework
January 14, 2009 roadmap
...
January 14, 2009 Some Nomadik products on the market...
January 14, 2009 STn8815 block diagram
January 14, 2009 Nomadik : a true real time multiprocessor platform
ARM926 SDRAM SRAM General (L1 + L2) Purpose •Unlimited Space (Level 2 •Limited Bandwidth Cache System for Video) DMA Master OS Memory Controller Memory Peripherals
multi-layer AHB bus
RTOS RTOS Multi-thread (Scheduler FSM)
NAND Flash MMDSP+ Video •Unlimited Space MMDSP+ Audio 66 MHz, 16-bit •“No” Bandwidth 133 MHz, 24-bit •Mass storage HW Acceleration
Audio Peripherals Video Peripherals January 14, 2009 Context & task relation: Logical view
ARM boundary (WinCE & Linux OS type)
GUI, User Applications Virtual world Application engines Multiple & servers Address Space
User side System calls IO process System side (kernel)
Single IO Address Space Drivers & Kernel components
HW/IO process Flat IO Address Space
Physical world
SAA boundary SVA boundary
January 14, 2009 Picture Post Processing
January 14, 2009 Examples of Video Resize filters targeting Mobile TV
Resized: zoom out (preview)
Regular: display 1:1 scale
Wide: zoom to full height
Panoramic crop and zoom to full width
Stretch zoom to full width and height
January 14, 2009 Rotate
January 14, 2009 Hardware/Software partitioning
MPEG4 software stack
Sequence Handled by Software on the host CPU. Group of picture High flexibility, low computation OS friendly ARM SW Picture
Group of Macro-Block Firmware
Handled by Hardware. Macro-Block No flexibility, high computation HW/FW Block
January 14, 2009 MM integration in an open OS
Application (Player,…)
MM Framework Audio Audio Filter Filter/Codec Sink (plug-ins based) Source Codec Video Video Filter/Codec Sink Virtual Media pipeline
Integration/Adaptation layer User side
Kernel side SAA HW Driver (control path) SVA HW Driver (control path) PCM if
SAA SVA Display FrameBuffer Audio Video Post- Codec Codec Processing
January 14, 2009 Concurrent execution
t1 t2 t3 t4 t5
ARM Read & Read & Read & Read & Read & Parse Parse Parse Parse Parse
Audio Audio Audio Audio Decode Decode Decode Decode SAA Audio Audio Audio Playback Playback Playback
Video Video Video Video Decode Decode Decode Decode SVA Video Video Video Display Display Display
January 14, 2009 Power Management support
RUN mode: Full clock speed ~200mA Full64 clock – 393MHz speed Doze mode: • ARM in WFI mode • PLL1 & 2 OFF, clocks =Æ 3232 kHz kHz • All peripheral clocks are gated Standby mode: ARM in WFI mode (Wait For Interrupt) • PLL1 & 2 are running Æ 64-393MHz Wake up: • All peripheral clocks are gated Sleep mode: 3ms • ARM in WFI mode ~mA • PLL1 & 2 OFF, clocks are stopped except the -non switchable part @ 32 kHz • All peripheral clocks are gated
PowerPower Save Save mode mode: Deep-Sleep mode: • ARM in WFI mode - Main part of the logic is no more powered ~µA • PLL1 OFF, PLL2 Running Æ 19MHz - The non switchable part is clocked @32 kHz • CPU & Bus clocked at XTAL frequency • All peripheral clocks unused are gated
January 14, 2009 SW architecture Linux Screenshots and pictures
WVGA 800x480 @ 24bpp Philips TPO display
20% CPU
3% CPU
January 14, 2009 Average CPU load
VGA 30fps 2Mbps Xine FFMPEG 10-20% GSTLaunch MPEG4 27% Lauterbach GSTLaunch H264 37% ETM trace MPEG4 VGA 30fps 2Mbps MP3 Playbin +20pp
OpenGL-ES panel carousel demo ARM load 3%
Lauterbach ETM trace H264 VGA Khronos compliance ongoing 30fps 2Mbps AAC
January 14, 2009 System overview for STn8815
cmp K+ architecture
Application Framew ork
+ Application Framework + Azingo + Google Android Full open source + Access ALP + Trolltech Qtopia Linux 2.6.20 kernel + OpenMoko + Poky
Middleware Communication Camera OpenGL-ES 1.1 GStreamer Pow er Management OpenedHand + Connection Manager + Camera ISP + Clutter + GStreamer + PM framework + Bluetooth + ISP Driver + EGL + OpenMAX-IL + Governor PokyLinux for + Wi-Fi + OpenGL Driver + FFMPEG + Battery Manager + Ethernet + OpenGL-ES 1.1 + SAA Block Library + CPUFreq + Telephony system test + GLES Library + Linux DPM Connectiv ity MEMS + APM Graphic User Interface + SD/MMC + Gesture Library Security + LiMo PM
+ USB Hi-Speed + MEMS Driver + Cryptoengine driver + Matchbox + IPSec + GTK+ + OpenSSL In line with GMAE, + Security Limo and Maemo
Kernel and driver level
Multimedia Interfaces File systems Kernel and Drivers
+ ALSA + JFFS2 + DMA Multimedia HCL Multimedia Drivers Power + V4L2 + YAFFS + I2C + SAA HCL + SAA Driver + X.11 + CRAMFS + Linux kernel 2.6.20 + SGA HCL + SGA Lib and Driver + xvideo + RAMFS + MSP Management + SVA HCL + SVA Driver + EXA + MTD + SPI + DirectFB + UART
January 14, 2009 A/V features at a glance
Video – MPEG4 / H.264 decoder up to VGA 30fps 2Mbps – MPEG4 encoder up to VGA 30fps – H.263 codec – JPEG codec Audio – MP3, AAC / eAAC+, WMA – G.711, G.723, G.726, G.729 – AMRnb / AMRwb – Full duplex noise and acoustic echo cancelling – Effects, e.g. equaliser, reverberation, spatial sound
January 14, 2009 Multimedia interfaces
cmp Multimedia Interfaces
Standard Linux ALSA X.11 DirectFB V4L2 interfaces
xvideo EXA
«flow» «use» «use»«use»
«use» «use»
Kernel and Drivers: Multimedia Drivers: Multimedia Drivers: Nomadik drivers :MSP :SVA Driver :SGA Lib and Driver
January 14, 2009 Graphic and video subsystem 2D Graphics – X.11 EXA for copy, solid fill, compositing – Good acceleration up to 3x on big pixmaps – Penalties for smaller primitives Æ ARM + L2C is much faster 3D Graphics – ARM for EGL and parsing – SVA+FPU for TnL – SGA for rendering – Good fit for smart GUI, not for 3D games Video rendering – Xvideo extension with hw acceleration in SVA – Resize, rotate, colour conversion and blitting
January 14, 2009 Multimedia with FFMPEG cmp Karakum
PMP application Linphone Web Browser - Xine Minimo Matchbox
«use» «use» «use» «use» «flow» «flow» GTK+
«flow» «flow» «flow» «use» Xvideo «flow» «flow» Xserver «use»
«flow» FFMPEG audio «use» codecs FFMPEG video EXA codecs «flow» «flow» «use» BlockLib V4L2 API SGA user lib «flow» «flow» «flow» «use» ALSA SAA Driver SVA driver SGA Driver
«flow» «flow» «flow» «flow»
AudioCodec frame buffer
January 14, 2009 GStreamer and OpenMAX layers cmp Static View
APPLICATION
«Gstreamer Based» Player App
«instantiate» FRAMEWORK «instantiate» «instantiate» «instantiate»
PCM_Data_Buffer GSTREAMER:: GSTREAMER_OMX: GSTREAMER_OMX: «flow» GSTREAMER:: GstAlsaSink :GstOmxMp3Dec :GstOmxMpeg4Dec «flow» GstXVImageSink
OMX::OMX_Core
INTEGRATION
OMX:: OMX:: XV i deo OMX_AudioDecComponent OMX_VideoDecComponent
ST_FFmpeg::ffmpeg libavcodec
SAA User Lib::Blocklib
KERNEL
Linux DDrivers::ALSA driver Linux DDrivers::SAA driver Linux DDrivers::SVA driver
NOMADIK HW
Nomadik HW::Audio Nomadik HW:: Nomadik HW:: Nomadik HW:: Codec (ex. 5095) SAA SVA Frame Buffer
January 14, 2009 Google Android on Nomadik
APPLICATIONS
Home Contacts Phone Browser . . .
APPLICATION FRAMEWORK Activity Window Content Notification View System Manager Manager Providers Manager
Package Telephony Resource Location XMPP Service Manager Manager Manager Manager
LIBRARIES ANDROID RUNTIME Surface Media SQLite Manager Framework Core libraries
OpenGL|ES FreeType WebKit Dalvik Virtual Machine
SGL SSL libc
LINUX KERNEL Camera Bluetooth Modem Flash Memory Binder (IPC) Display Driver Driver Driver Driver Driver Driver Keypad Crypto Engine Power USB Driver WiFi Driver A/V Driver Driver Driver Management
January 14, 2009 Multiple clipping video rendering
Minimise bandwidth usage when not needed!!!
R1
R2 R3
January 14, 2009 Thread Model
Play bin thread
gst player
Input file gst media parser
gst XVImageSink gst-omx mpeg4 gst-omx mp3 dec gst ALSA sink dec plugin plugin
XVLib omx mpeg4 dec omx mp3 dec ALSA lib component component
omx mpeg4 msg thread OMX mpeg4 dec thread OMX mp3 dec thread omx mp3 msg thread
omx mpeg4 dec omx mpeg4 dec omx mp3 dec buf omx mp3 dec msg msg handler buf manager manager handler
ffmpeg mpeg4 dec ffmpeg mp3 dec
SAA block lib
Linux SVA driver Linux SAA driver
January 14, 2009 Power Management
CPUfreq to scale voltage and frequency CPUIdle to switch amongst running, idle, sleep and deep sleep Custom API to control clock gating to all internal IP’s Feasibility on-going with DPM PowerTop
«flow»/sysfs /sysfs /sysfs
Kernel Space
CpuIdle cpufreq Constraint Manager «use»
Drivers and Bus Drivers Driver Interface
Governor Dynamic Tick (Wakeup event)
January 14, 2009 DVFS – Dynamic Voltage Frequency Scaling
stm Operating Points PH1
100.8/100.8 MHz 201.6/100.8 MHz 1.0V 1.2V
264.0/132.0 MHz 1.2V 326.4/163.2 MHz Boot 1.2V
Initial
393.6/131.2 MHz 1.26V 489.6/163.2 MHz 1.45V (lab only)
January 14, 2009 PM Dump Tool
PM monitor to optimise operating modes and ULP, estimate consumption – (Under development)
UART SDMMC MCDE TVOUT AUDIO SAA SVA BT WIFI LAN CPU 0 CPU 1
I (mA) 100 95 20 50 10 100 95 20 50 10 100 95 20
35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms
V 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9
Clock 393 393 64 166 19 393 393 64 166 19 393 393 64
January 14, 2009 Thank you!