Memory Management in Tizen SW Platform Team, SW R&D Center Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel • Memory Size Optimization 2 tizen.org Tizen Kernel Overview 3 tizen.org Tizen Kernel Overview Core Multimedia Telephony Sensor FW System FW . FW FW OpenGL ES X11 EFL GStreamer . 1.0/2.0 OAL (OEM Adaptation Layer) Telephony GStreamer Sensor System OpenGL ES/EGL Kernel . Plug-ins Plug-ins Plug-ins Plug-ins Graphics Driver Kernel Memory Graphics Multimedia Power Management Storage Peripheral Management Block Layer DMA DRM CPUfre Devf Ther Input, Sensor, CMA IOMMU V4L2 … BUF Link1 Link2 q req mal … MMC/Flash More at http://source.tizen.org/documentation/porting-guide DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- 4 the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf tizen.org Tizen Kernel Overview Core Service Layer Multimedia Telephony • Memory ManagementSensor FW in TizenSystem FW . FW FW – Coupled with Graphics & Multimedia devices. OpenGL ES X11 EFL GStreamer . • Graphics & Multimedia devices = DMA1.0/2.0 devices with HUGE buffers OAL (OEM Adaptation Layer) Telephony GStreamer Sensor System OpenGL ES/EGL Kernel . Plug-ins Plug-ins Plug-ins Plug-ins Graphics Driver Kernel Memory Graphics Multimedia Power Management Storage Peripheral Management Block Layer CM IOMM DMA CPUfre Devf Ther Input, Sensor, DRM … V4L2 q req mal … A U BUF Link1 Link2 MMC/Flash More at http://source.tizen.org/documentation/porting-guide DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410- 5 the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf tizen.org Tizen Kernel Overview Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 UMM DRM Framework GPU DDK VB2 DMABUF GEM (G3D) Kernel / Device Drivers (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor Core Core External OEM Kernel 6 Framework Library Adaptation * Images from MS Office clipart and Samsung tizen.org Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization 7 tizen.org Tizen Kernel MM, Graphics (DRM) Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 UMM DRM Framework GPU DDK VB2 DMABUF GEM (G3D) Kernel / Device Drivers (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor Core Core External OEM Kernel 8 Framework Library Adaptation * Images from MS Office clipart and Samsung tizen.org Graphics: DRM/GEM (Graphics Execution Manager) Linux DRM Common Framework drm_mode_config drm_fb_helper GEM Connector Crtc Plane drm_framebuffer Encoder Connector Crtc Plane drm_framebuffer fb_info GEM Encoder Allocator Common Specific 9 tizen.org Graphics: DRM/GEM (Graphics Execution Manager) Linux Graphics ExecutionDRM CommonManager Framework drm_mode_config drm_fb_helper GEM • FrameworkConnector developed by Intel Crtc Plane drm_framebuffer • To manageEncoder graphics memory • FrameworkConnector for buffer management. Crtc Plane drm_framebuffer fb_info GEM • AllocationEncoder and sharing. Allocator Common Specific 10 tizen.org Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 1. DRM_IOCTL_MODE_CREATE_DUMB • Create GEM object(global) & user GEM handle(per process) • dumb_create() of struct drm_driver • No physical memory allocated. 2. DRM_IOCTL_MODE_MAP_DUMB • Create fake mmap offset of a gem object and relay the object to user • A hash key of the gem object. • dumb_map_offset() of struct drm_driver 3. MMAP • Request mmap based on the hash key as the offset • Create user address space • Setup cache attribute. • Not mapped to physical memory, yet 11 tizen.org Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Generic) 4. On-demand Paging – Implement & Register a fault handler that • With a page fault, allocate a page and map the page. – vma->vm_ops->fault = xxx_drm_gem_fault 5. Use! 6. DRM_IOCTL_MODE_DESTROY_DUMB – Remove GEM handle & object – Free memory – Implement dumb_destroy() of struct drm_driver 12 tizen.org Graphics: DRM/GEM Allocation • GEM Allocation steps @ Tizen (Exynos Only) 1. DRM_IOCTL_EXYNOS_GEM_CREATE • Only use user-desired size and buffer types. • Create gem object(global) & user gem handle(per process) • physical memory allocated. 2. DRM_IOCTL_EXYNOS_GEM_MMAP • Create user address space • Map the user address space to physical memory • LIBDRM of Exynos uses these APIs, not the generic. 13 tizen.org Graphics: DRM/GEM Sharing • GEM Sharing @ Tizen • DRM_IOCTL_GEM_FLINK – “I will share this GEM to others.” – Create GEM object name for the given GEM handle • Global key vaue for sharing • DRM_IOCTL_GEM_OPEN – “I want to use the shared GEM.” – Create GEM handle based on the given GEM object name • DRM_IOCTL_GEM_CLOSE • You don’t need to implement. It’s already there with DRM. Process 1 Process 2 GEM GEM GEM Create FLINK OPEN gem handle gem object name gem handle 14 tizen.org Tizen Kernel MM, Multimedia (V4L2/VB2) Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 UMM DRM Framework GPU DDK VB2 DMABUF GEM (G3D) Kernel / Device Drivers (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor Core Core External OEM Kernel 15 Framework Library Adaptation * Images from MS Office clipart and Samsung tizen.org Multimedia: V4L2/VB2 • Tizen recommends to use V4L2 at Tizen kernel for Multimedia devices • Video input (codec & camera) & Radio • However, as long as the kernel has: – Gstreamer/OpenMAX plugins – A method to share with other F/W via DMABUF of UMM, • Tizen multimedia works. • If V4L2/VB2 is used, things get easier. 16 tizen.org Tizen Kernel MM, OpenGL/G3D-GPU Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 UMM DRM Framework GPU DDK VB2 DMABUF GEM (G3D) Kernel / Device Drivers (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor Core Core External OEM Kernel 17 Framework Library Adaptation * Images from MS Office clipart and Samsung tizen.org OpenGL / G3D-GPU • Most ARM SoC GPUs (MALI, SGX, …) use their own memory manager • E.g., Exynos4412/4210 Tizen Reference Kernel has Mali w/ UMP. – Mali DDK modified to be compatible with UMM-DMABUF. • If GPU drivers use DRM, it would be great. • (and make them GPL) 18 tizen.org Contents • Tizen Kernel Overview • Memory Management in Tizen Kernel – To Run Tizen OS. • Basic Memory Management Components • Buffer Sharing (UMM/DMABUF) • Buffer Allocation (UMM/DMAAPI) • Buffer Synchronization • Memory Size Optimization 19 tizen.org NEED FOR SOMETHING, A Scenario Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 DRM Framework GPU DDK VB2 GEM (G3D) CameraKernel fetches / Device a video Drivers stream (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor CoreThis scenarioCore External is simplified forOEM presentation. Kernel 20 Framework Not anLibrary actual exampleAdaptation of Tizen * Images from MS Office clipart and Samsung tizen.org NEED FOR SOMETHING, A Scenario Draws its own images Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem V4L2 DRM Framework GPU DDK VB2 GEM (G3D) CameraKernel fetches / Device a video Drivers stream (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor CoreThis scenarioCore External is simplified forOEM presentation. Kernel 21 Framework Not anLibrary actual exampleAdaptation of Tizen * Images from MS Office clipart and Samsung tizen.org NEED FOR SOMETHING, A Scenario Draws its own images Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem GPU Merges the two / Puts the merged image V4L2 DRM Framework GPU DDK VB2 GEM (G3D) CameraKernel fetches / Device a video Drivers stream (BSP) Virtual Post Codec Camera Radio Display HDMI G2D Display Processor CoreThis scenarioCore External is simplified forOEM presentation. Kernel 22 Framework Not anLibrary actual exampleAdaptation of Tizen * Images from MS Office clipart and Samsung tizen.org NEED FOR SOMETHING, A Scenario Draws its own images Multimedia FW X Server EFL Open GL XvImageSink XV Extension X Video Drivers EXA Evas GStreamer 2D GPU PIXMAN Libdrm backend Gstreamer- OpenMAX 2D GPU CameraSrc NEON backend Kernel Userptr Kernel / Core & Subsystem GPU Merges the two / Puts