Middleware for Embedded Systems
Total Page:16
File Type:pdf, Size:1020Kb
Embedded Middleware National Chiao Tung University Chun-Jen Tsai 3/21/2011 Embedded Middleware “Middleware”is a very popular terminology in embedded system software design Just like “cloud computing,”it is merely a new name for old technologies We can further divide embedded middleware into two layers System middleware –an API library that provides system abstraction to application software to increase application portability Hardware abstraction middleware –an API specification that acts as a hardware abstraction layer for chip-level integration 2/48 Why System Middleware The purpose of system middleware is to provide a homogeneous environment for applications Operating system does the same thing, but due to the variety of embedded hardware and competitions among OS companies, we need something beyond OS application Defined by an international standard organization Middleware Middleware Middleware Designed by a OS 1 OS 2 OS 1 system manufacturer Hardware 1 Hardware 1 Hardware 2 3/48 System Middleware Components A system middleware must provide the following abstractions for (multimedia) applications Application model GUI subsystem Graphics, video, audio rendering User event handling Timing control Menu File system e.g. DVB-MHP application 4/48 HW SW DVB Multimedia Home Platform functional blocks: I/O Devices Video Codec (H.264, MPEG-2) Application Operating Systems Native Audio Codec (HE-AAC) - MHP Architecture Revisited Graphic library (e.g. Microwin) RISC Processor (MPEG (< 300MHz) RTSP/RTP) Transport Demux - 2 TS, Data MHP Middleware accelerator ( Application Graphics Java JMF 1.1 Model Sun - TV ) Java Standard Classes (CDC/PBP) ( Event, Services accelerator Application Manager Abstraction audio DVB (Navigator) Xlets ) accelerator Video Abstraction HAVI ( UI Platform Xlets ) Processor based on GEM - dependent Java Abstraction modules DAVIC ( FS SoC ) 5 /48 Java-based Middleware DVB-MHP 1.0 (aka DVB-J) system abstraction is based on Java RE, just like many other middleware In particular, CDC/PBP is adopted for MHP Google Android is also based on Java Model Google defined their own “profile”and VM (not JVM byte- code compatible) Rationale: Java is “virtually a computer plus an OS!” Applets, Xlets, Applications MIDlets, etc. Class loader, GC, OS API classes, etc. CPU VM Exec. Engine Real Computer Model Java Model 6/48 Native vs. Java-based Middleware We can use native code to design system middleware. For example, BREW from Qualcomm is a native middleware for mobile handsets: A C/C++ development environment for ARM-based platforms Application: BREW “Applet”model; cooperative multitasking (IThread API) Graphics API: OpenGL/ES Multimedia API: IMedia Supposedly, BREW is faster than Java, but with a properly designed system, this may not be true 7/48 MHP Application Model Abstraction Application implements the javax.tv.xlet.Xlet interface Similar to java.applet.Applet, but simpler Methods are called in the following sequence: initXlet() startXlet() : resource allocation begins here! destroyXlet() Xlet life cycle (initiated by the MHP application manager, which sets up an XletContext for the Xlet) initXlet() startXlet() Start Loaded Paused Active destroyXlet() pauseXlet() destroyXlet() destroyXlet() Destoryed 8/48 Middleware Architecture for DVB-J MHP middleware layers†: Application Manager and Xlet API DVB UI HAVi Service Selection Conditional Return Inter-Xlet JMF Access Channel AWT Comm. Tuning UI Events DSM-CC SI Transport Section Filtering Stream MPEG Java †S. Morris and A. Smith-Chaigneau, Interactive TV Standards, Elsevier, 2005 9/48 DVB-J APIs Java Language API Application Model (Life Cycle) API Based on Java TV xlets Graphical API Based on AWT and HAVi UI level 2 Data Access API Multimedia API Based on JMF Service and User Settings API 10/48 MHP Application UI Construction An MHP application can allocate screen resource, called HScene, using HAVi API An HScene represents a region on the TV display HScreen HScene UI components are mainly drawn using java.awt.Graphics primitives HAVi widget classes 11/48 MHP Display Model Abstraction The MHP display model is defined in Sun Java-TV middleware: Background Background HBackgroundDevice Data Decoder Video Video Decoder JMF Decoder Format scaling HVideoDevice Data Conversion Manipulation + HScreen broadcast TV metadata user preference Behavior Control display info. Format signaling Application to display Behavior Control Graphics Graphics HGraphicsDevice Implementation? Data Composition Graphics primitives created using 12/48 java.awt and HAVi level 2 UI MHP Display Model Implementation If MHP adopts Sun’s PBP reference implementation: add() widgets org.havi.ui.HScene Should be java.awt.Container Java PBP Class Libraries merged for Portable Graphics Library (MicroWindows) efficiency! java.awt.Component Native Graphics Library (X11) OS (Linux) fbdev driver HScene VGA/DVI HScreen HScreen HW video frame buffer Controller has has has HVideoDevice HGraphicsDevice HBackgroundDevice HScreenDevice 13/48 Building a GUI Application During initXlet() Configure the required HScene and retrieve the handle to the HScene object During startXlet() Create widget components under the HScene Display the HScene using setVisible(true) Request the focus During destroyXlet Hide the HScene Release the allocated resources (components, images, etc.) 14/48 Controlling Media from Xlets Xlets can control audio/video media using Java Media Framework Based on JMF 1.0 Use org.davic.net.dvb.DVBLocator to select what to show MHP provides extra JMF controls for Video scaling Subtitle control and service components monitoring Audio playback fine control (for local playback) Notification of changes in incoming TV signals 15/48 JMF Class Architecture JMF is used as media playback Manager interface in MHP has a Clock TimeBase Only JMF 1.1 API Duration extends are used: JMF extends Player Controller MediaHandler DataSource extends extends has a Control Player DataSource Class AWT Networking JNI Lang Library JVM 16/48 Video-Graphics Integration†for MHP getBestConfiguration() 1 n HScreen HScreenConfiguration Java TV 1 getDefaultVideoDevice() extends getSource() n 1 1 1 n locator ServiceContext HVideoDevice HVideoConfiguration 1 1 1 select() 1 Implements 1 getVideoDevice() HVideoComponent Constructed HAVi L2 UI Constructed with new 1 1 with new DataSource ServiceContext MediaSelector (MediaLocator) (locator) 1 getSource() 1 getVideoComponent() 1 1 1 MediaLocator MediaPlayer Player Implements Java Media Framework †iTV Handbook, Prentice Hall 2003 17/48 DVB File System Model† to cable Remote network file Head station system Up-link Tuner, demodulation, & video encoder Channel channel decode MPEG coding & TV 1 audio encoder program modulation MUX data encoder MPEG Transport User input, DEMUX program selection 38 Mbps conditional access MPEG encoder video TV 2 transport MPEG Program Local TV 3 MUX encoder audio file DEMUX system TV 4 encoder data †L. Atzori et al., “Multimedia Information Broadcasting Using Digital TV Channels,”IEEE T-Broadcasting, Sep. 199718/48 DVB File System Abstraction DVB applications (Xlets, etc.) and data can be transmitted to STBs using DSM-CC Object Carousel† Local virtual file system †D.-H. Park et al., “Real-Time Carousel Caching and Monitoring in Data Broadcasting,”IEEE T-Consumer 19/48 Electronics, Feb. 2006 Extraction of Xlets from DSM-CC One possible implementation of Xlets extractor: DSM-CC Parser Xlet Object DII Cache DII Cache Cache Cache Real-Time Thread Pool Xlet Running Application Space Manager Version Manager Monitor Download Threads Threads DSM-CC Cache Space Incoming data stream Section Filter User Input 20/48 Synchronization Synchronization of application with media are facilitated by the following API functions: DSM-CC stream events org.dvb.dsmcc.DSMCCStreamEvent.subscribe() org.davic.media.StreamEventControl.subscribeStreamEvent() Media Time (NPT) org.davic.media.MediaTimeEventControl Private section events org.davic.mpeg.sections 21/48 Java for 2.5/3G Mobile Services 3GPP adopts the concept of middleware for CPE For application middleware, Java RE has been selected (CLDC/MIDP 2.0) For multimedia device integration middleware, OpenMAX has been proposed (by ARM, Motorola, Nokia, TI, etc) Google also selects Java R.E. for G-phone, but use their own virtual machine and profile libraries This avoids Sun’s J2ME licensing fee 22/48 Google Android Architecture Applications Home Contacts Phone Browser ... Application Framework Activity Window Content View Notification Manager Manager Providers System Manager Package Telephony Resource Location XMPP Manager Manager Manager Manager Service System Libraries Android Runtime Surface Manager Media Framework SQLite Core Libraries OpenGL/ES FreeType WebKit Dalvik Virtual Machine SGL SSL C Library Linux Kernel 23/48 Linux’s Role in Android Java model does not have an I/O subsystem, Linux kernel is only (hopefully) used to provide Hardware abstraction layer (for different drivers) Multi-threading management Shared library interface The following “infamous”Linux components are not included Native windowing system (e.g. X11) glibc 24/48 Android Applications An Android application package (APK) is a collection of components which usually run in one thread Services or ContentProviders sometimes need another thread (process) Activity Activity Activity Activity ContentProvider ContentProvider Process Service Service Process Process APK Package 1 APK Package