Nomadik Application Processor Andrea Gallo Giancarlo Asnaghi ST Is #1 World-Wide Leader in Digital TV and Consumer Audio
Total Page:16
File Type:pdf, Size:1020Kb
Nomadik Application Processor Andrea Gallo Giancarlo Asnaghi ST is #1 world-wide leader in Digital TV and Consumer Audio MP3 Portable Digital Satellite Radio Set Top Box Player Digital Car Radio DVD Player MMDSP+ inside more than 200 million produced chips January 14, 2009 ST leader in mobile phone chips January 14, 2009 Nomadik Nomadik is based on this heritage providing: – Unrivalled multimedia performances – Very low power consumption – Scalable performances January 14, 2009 BestBest ApplicationApplication ProcessorProcessor 20042004 9 Lowest power consumption 9 Scalable performance 9 Video/Audio quality 9 Cost-effective Nominees: Intel XScale PXA260, NeoMagic MiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611 January 14, 2009 Nomadik Nomadik is a family of Application Processors – Distributed processing architecture ARM9 + multiple Smart Accelerators – Support of a wide range of OS and applications – Seamless integration in the OS through standard API drivers and MM framework January 14, 2009 roadmap ... January 14, 2009 Some Nomadik products on the market... January 14, 2009 STn8815 block diagram January 14, 2009 Nomadik : a true real time multiprocessor platform ARM926 SDRAM SRAM General (L1 + L2) Purpose •Unlimited Space (Level 2 •Limited Bandwidth Cache System for Video) DMA Master OS Memory Controller Peripherals multi-layer AHB bus RTOS RTOS Multi-thread (Scheduler FSM) NAND Flash MMDSP+ Video •Unlimited Space MMDSP+ Audio 66 MHz, 16-bit •“No” Bandwidth 133 MHz, 24-bit •Mass storage HW Acceleration Audio Peripherals Video Peripherals January 14, 2009 Context & task relation: Logical view ARM boundary (WinCE & Linux OS type) GUI, User Applications Virtual world Application engines Multiple & servers Address Space User side System calls IO process System side (kernel) Single IO Address Space Drivers & Kernel components HW/IO process Flat IO Address Space Physical world SAA boundary SVA boundary January 14, 2009 Picture Post Processing January 14, 2009 Examples of Video Resize filters targeting Mobile TV Resized: zoom out (preview) Regular: display 1:1 scale Wide: zoom to full height Panoramic crop and zoom to full width Stretch zoom to full width and height January 14, 2009 Rotate January 14, 2009 Hardware/Software partitioning MPEG4 software stack Sequence Handled by Software on the host CPU. Group of picture High flexibility, low computation OS friendly ARM SW Picture Group of Macro-Block Firmware Handled by Hardware. Macro-Block No flexibility, high computation HW/FW Block January 14, 2009 MM integration in an open OS Application (Player,…) MM Framework Audio Audio Filter Filter/Codec Sink (plug-ins based) Source Codec Video Video Filter/Codec Sink Virtual Media pipeline Integration/Adaptation layer User side Kernel side SAA HW Driver (control path) SVA HW Driver (control path) PCM if SAA SVA Display FrameBuffer Audio Video Post- Codec Codec Processing January 14, 2009 Concurrent execution t1 t2 t3 t4 t5 ARM Read & Read & Read & Read & Read & Parse Parse Parse Parse Parse Audio Audio Audio Audio Decode Decode Decode Decode SAA Audio Audio Audio Playback Playback Playback Video Video Video Video Decode Decode Decode Decode SVA Video Video Video Display Display Display January 14, 2009 Power Management support RUN mode: Full clock speed ~200mA Full64 clock – 393MHz speed Doze mode: • ARM in WFI mode • PLL1 & 2 OFF, clocks =Æ 3232 kHz kHz • All peripheral clocks are gated Standby mode: ARM in WFI mode (Wait For Interrupt) • PLL1 & 2 are running Æ 64-393MHz Wake up: • All peripheral clocks are gated Sleep mode: 3ms • ARM in WFI mode ~mA • PLL1 & 2 OFF, clocks are stopped except the -non switchable part @ 32 kHz • All peripheral clocks are gated PowerPower Save Save mode mode: Deep-Sleep mode: • ARM in WFI mode - Main part of the logic is no more powered ~µA • PLL1 OFF, PLL2 Running Æ 19MHz - The non switchable part is clocked @32 kHz • CPU & Bus clocked at XTAL frequency • All peripheral clocks unused are gated January 14, 2009 SW architecture Linux Screenshots and pictures WVGA 800x480 @ 24bpp Philips TPO display 20% CPU 3% CPU January 14, 2009 Average CPU load VGA 30fps 2Mbps Xine FFMPEG 10-20% GSTLaunch MPEG4 27% Lauterbach GSTLaunch H264 37% ETM trace MPEG4 VGA 30fps 2Mbps MP3 Playbin +20pp OpenGL-ES panel carousel demo ARM load 3% Lauterbach ETM trace H264 VGA Khronos compliance ongoing 30fps 2Mbps AAC January 14, 2009 System overview for STn8815 cmp K+ architecture Application Framew ork + Application Framework + Azingo + Google Android Full open source + Access ALP + Trolltech Qtopia Linux 2.6.20 kernel + OpenMoko + Poky Middleware Communication Camera OpenGL-ES 1.1 GStreamer Pow er Management OpenedHand + Connection Manager + Camera ISP + Clutter + GStreamer + PM framework + Bluetooth + ISP Driver + EGL + OpenMAX-IL + Governor PokyLinux for + Wi-Fi + OpenGL Driver + FFMPEG + Battery Manager + Ethernet + OpenGL-ES 1.1 + SAA Block Library + CPUFreq + Telephony system test + GLES Library + Linux DPM Connectiv ity MEMS + APM Graphic User Interface + SD/MMC + Gesture Library Security + LiMo PM + USB Hi-Speed + MEMS Driver + Cryptoengine driver + Matchbox + IPSec + GTK+ + OpenSSL In line with GMAE, + Security Limo and Maemo Kernel and driver level Multimedia Interfaces File systems Kernel and Drivers + ALSA + JFFS2 + DMA Multimedia HCL Multimedia Drivers Power + V4L2 + YAFFS + I2C + SAA HCL + SAA Driver + X.11 + CRAMFS + Linux kernel 2.6.20 + SGA HCL + SGA Lib and Driver + xvideo + RAMFS + MSP Management + SVA HCL + SVA Driver + EXA + MTD + SPI + DirectFB + UART January 14, 2009 A/V features at a glance Video – MPEG4 / H.264 decoder up to VGA 30fps 2Mbps – MPEG4 encoder up to VGA 30fps – H.263 codec – JPEG codec Audio – MP3, AAC / eAAC+, WMA – G.711, G.723, G.726, G.729 – AMRnb / AMRwb – Full duplex noise and acoustic echo cancelling – Effects, e.g. equaliser, reverberation, spatial sound January 14, 2009 Multimedia interfaces cmp Multimedia Interfaces Standard Linux ALSA X.11 DirectFB V4L2 interfaces xvideo EXA «flow» «use» «use»«use» «use» «use» Kernel and Drivers: Multimedia Drivers: Multimedia Drivers: Nomadik drivers :MSP :SVA Driver :SGA Lib and Driver January 14, 2009 Graphic and video subsystem 2D Graphics – X.11 EXA for copy, solid fill, compositing – Good acceleration up to 3x on big pixmaps – Penalties for smaller primitives Æ ARM + L2C is much faster 3D Graphics – ARM for EGL and parsing – SVA+FPU for TnL – SGA for rendering – Good fit for smart GUI, not for 3D games Video rendering – Xvideo extension with hw acceleration in SVA – Resize, rotate, colour conversion and blitting January 14, 2009 Multimedia with FFMPEG cmp Karakum PMP application Linphone Web Browser - Xine Minimo Matchbox «use» «use» «use» «use» «flow» «flow» GTK+ «flow» «flow» «flow» «use» Xvideo «flow» «flow» Xserver «use» «flow» FFMPEG audio «use» codecs FFMPEG video EXA codecs «flow» «flow» «use» BlockLib V4L2 API SGA user lib «flow» «flow» «flow» «use» ALSA SAA Driver SVA driver SGA Driver «flow» «flow» «flow» «flow» AudioCodec frame buffer January 14, 2009 GStreamer and OpenMAX layers cmp Static View APPLICATION «Gstreamer Based» Player App «instantiate» FRAMEWORK «instantiate» «instantiate» «instantiate» PCM_Data_Buffer GSTREAMER:: GSTREAMER_OMX: GSTREAMER_OMX: «flow» GSTREAMER:: GstAlsaSink :GstOmxMp3Dec :GstOmxMpeg4Dec «flow» GstXVImageSink OMX::OMX_Core INTEGRATION OMX:: OMX:: XV i deo OMX_AudioDecComponent OMX_VideoDecComponent ST_FFmpeg::ffmpeg libavcodec SAA User Lib::Blocklib KERNEL Linux DDrivers::ALSA driver Linux DDrivers::SAA driver Linux DDrivers::SVA driver NOMADIK HW Nomadik HW::Audio Nomadik HW:: Nomadik HW:: Nomadik HW:: Codec (ex. 5095) SAA SVA Frame Buffer January 14, 2009 Google Android on Nomadik APPLICATIONS Home Contacts Phone Browser . APPLICATION FRAMEWORK Activity Window Content Notification View System Manager Manager Providers Manager Package Telephony Resource Location XMPP Service Manager Manager Manager Manager LIBRARIES ANDROID RUNTIME Surface Media SQLite Manager Framework Core libraries OpenGL|ES FreeType WebKit Dalvik Virtual Machine SGL SSL libc LINUX KERNEL Camera Bluetooth Modem Flash Memory Binder (IPC) Display Driver Driver Driver Driver Driver Driver Keypad Crypto Engine Power USB Driver WiFi Driver A/V Driver Driver Driver Management January 14, 2009 Multiple clipping video rendering Minimise bandwidth usage when not needed!!! R1 R2 R3 January 14, 2009 Thread Model Play bin thread gst player Input file gst media parser gst XVImageSink gst-omx mpeg4 gst-omx mp3 dec gst ALSA sink dec plugin plugin XVLib omx mpeg4 dec omx mp3 dec ALSA lib component component omx mpeg4 msg thread OMX mpeg4 dec thread OMX mp3 dec thread omx mp3 msg thread omx mpeg4 dec omx mpeg4 dec omx mp3 dec buf omx mp3 dec msg msg handler buf manager manager handler ffmpeg mpeg4 dec ffmpeg mp3 dec SAA block lib Linux SVA driver Linux SAA driver January 14, 2009 Power Management CPUfreq to scale voltage and frequency CPUIdle to switch amongst running, idle, sleep and deep sleep Custom API to control clock gating to all internal IP’s Feasibility on-going with DPM PowerTop «flow»/sysfs /sysfs /sysfs Kernel Space CpuIdle cpufreq Constraint Manager «use» Drivers and Bus Drivers Driver Interface Governor Dynamic Tick (Wakeup event) January 14, 2009 DVFS – Dynamic Voltage Frequency Scaling stm Operating Points PH1 100.8/100.8 MHz 201.6/100.8 MHz 1.0V 1.2V 264.0/132.0 MHz 1.2V 326.4/163.2 MHz Boot 1.2V Initial 393.6/131.2 MHz 1.26V 489.6/163.2 MHz 1.45V (lab only) January 14, 2009 PM Dump Tool PM monitor to optimise operating modes and ULP, estimate consumption – (Under development) UART SDMMC MCDE TVOUT AUDIO SAA SVA BT WIFI LAN CPU 0 CPU 1 35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms I (mA) 100 95 20 50 10 100 95 20 50 10 100 95 20 V 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9 1 0.8 1.2 1.2 0.9 Clock 393 393 64 166 19 393 393 64 166 19 393 393 64 January 14, 2009 Thank you!.