The OpenVX™ OpenCL™ Interop Extension Editor: Radhakrishna Giduthuri, The Khronos OpenVX Working Group Version 1.0 (provisional), 25th January 2018 Table of Contents 1. Introduction . 3 1.1. Purpose . 3 1.2. Features of the extension specification. 3 1.3. Preserve the asynchronous property of OpenCL . 4 1.4. Coordination Command Queues . 5 1.5. Usage Example of the vx_khr_opencl_interop API . 5 2. Additional Functionality of Existing OpenVX APIs . 8 2.1. Global OpenCL context . 8 2.2. Global Coordination Command Queue . 8 2.3. User kernels and nodes . 8 2.3.1. User kernel using the vx_khr_opencl_interop extension . 8 2.3.2. User node coordination command queue . 8 2.4. OpenCL buffer memory type . 9 3. Constants . 11 4. The OpenCL Interop API functions . 12 4.1. vxCreateContextFromCL . 12 Index . 14 Copyright 2013-2018 The Khronos Group Inc. This specification is protected by copyright laws and contains material proprietary to Khronos. Except as described by these terms, it or any components may not be reproduced, republished, distributed, transmitted, displayed, broadcast or otherwise exploited in any manner without the express prior written permission of Khronos. This specification has been created under the Khronos Intellectual Property Rights Policy, which is Attachment A of the Khronos Group Membership Agreement available at www.khronos.org/files/member_agreement.pdf. Khronos Group grants a conditional copyright license to use and reproduce the unmodified specification for any purpose, without fee or royalty, EXCEPT no licenses to any patent, trademark or other intellectual property rights are granted under these terms. Parties desiring to implement the specification and make use of Khronos trademarks in relation to that implementation, and receive reciprocal patent license protection under the Khronos IP Policy must become Adopters and confirm the implementation as conformant under the process defined by Khronos for this specification; see https://www.khronos.org/adopters. Khronos makes no, and expressly disclaims any, representations or warranties, express or implied, regarding this specification, including, without limitation: merchantability, fitness for a particular purpose, non-infringement of any intellectual property, correctness, accuracy, completeness, timeliness, and reliability. Under no circumstances will Khronos, or any of its Promoters, Contributors or Members, or their respective partners, officers, directors, employees, agents or representatives be liable for any damages, whether direct, indirect, special or consequential damages for lost revenues, lost profits, or otherwise, arising from or in connection with these materials. Khronos and OpenVX are trademarks of The Khronos Group Inc. OpenCL is a trademark of Apple Inc., used under license by Khronos. All other product names, trademarks, and/or company names are used solely for identification and belong to their respective owners. Technical Contributors • Radhakrishna Giduthuri, AMD • Niclas Danielsson, Axis Communications AB • Thierry Lepley, Cadence • Frank Brill, Cadence • John MacCallum, Imagination • Ben Ashbaugh, Intel • Adam Herr, Intel 1. Introduction This document details an extension to OpenVX 1.2, and references some APIs and symbols that may be found in that API, at https://www.khronos.org/registry/OpenVX/specs/1.2/html/index.html. Refer to the OpenCL 1.2 specification at https://www.khronos.org/registry/OpenCL/sdk/1.2/docs/man/xhtml/. The name of this extension is vx_khr_opencl_interop. 1.1. Purpose This OpenVX extension provides a mechanism for interoperation between an OpenVX implementation and an OpenCL application/user-kernel. Efficient communication is key to successful interoperation. The data exchange mechanism needs features such that: - OpenCL data objects can be imported into the OpenVX environment - OpenVX data objects can be accessed as OpenCL data objects - fully asynchronous host-device operations are enabled during data exchange 1.2. Features of the extension specification This interop extension supports six essential features. - share common cl_context object between OpenVX and the OpenCL application - share a set of common in-order cl_command_queue objects for coordination between OpenVX and the OpenCL application/user-kernel - mechanism for an OpenCL application to export cl_mem buffers to OpenVX - mechanism for an OpenCL application to reclaim exported cl_mem buffers back from OpenVX - mechanism for an OpenCL application/user-kernel to temporarily map OpenVX data objects into cl_mem buffers - mechanism to copy between cl_mem buffers and OpenVX data objects. 1.3. Preserve the asynchronous property of OpenCL OpenCL has an asynchronous host API where command queues are executed asynchronously unless the application issues a blocking command. A cl_mem buffer is a handle to an opaque OpenCL data object. OpenCL kernels that consume such data objects can be enqueued into a command queue before the command that produces the content of this data object starts executing. This asynchronous property of the OpenCL API is fundamental, since it allows hiding various overheads/latencies such as creating OpenCL objects or launching kernels on an accelerator. The vx_khr_opencl_interop API intends to preserve this asynchronous property by avoiding unnecessary host-accelerator synchronizations when the ownership of data objects switch between OpenCL and OpenVX, or when data are copied between OpenCL and OpenVX. There are two actors in the vx_khr_opencl_interop API: - the application/user-kernel that uses OpenCL in addition to OpenVX - the OpenVX implementation When a cl_mem buffer is imported into OpenVX or when an OpenVX object is mapped as an OpenCL object, the ownership of the related data object temporarily changes from one actor to the other. Since the actual production of the data object content is considered as asynchronous, the actor that gets the ownership of the data object can’t assume the data content is available. It then has two alternatives: - launch asynchronous tasks (such as OpenCL kernels), making sure that these tasks will not start executing before the tasks of the other actor that produces the data object content are complete. - wait until the data is ready before executing a synchronous task that consumes the content of the data object To ensure the proper coordination between the two actors that exchange data objects, the vx_khr_opencl_interop API relies on an OpenCL command queue, referred to in this document as a coordination command queue. A coordination command queue is the contract between the two actors exchanging the ownership of the data object. It allows both producer and consumer actors to make sure that any data production task of the former cl_mem owner actor is executed before any data consumption tasks of the new owner actor. The coordination command queue enables this coordination to happen asynchronously while still supporting synchronization if this is the choice of one of the actors. 1.4. Coordination Command Queues There are two types of coordination command queues: - A unique global coordination command queue for coordination between the application OpenCL workload and the OpenVX implementation, outside of OpenVX graphs. The application can provide the global coordination command queue when creating the OpenVX context using the vxCreateContextFromCL API; otherwise it will be created by the OpenVX implementation. - User node coordination command queues for coordination between the user nodes and the OpenVX implementation. These coordination command queues will be created and managed by the OpenVX implementation. An OpenVX implementation may use one or more command queues across all the user nodes. The VX_NODE_CL_COMMAND_QUEUE node attribute can be queried from within vx_kernel_initialize_f, vx_kernel_f, and vx_kernel_deinitialize_f callbacks, to get access to a user node’s coordination command queue. When ownership of an OpenCL object changes from actor A (producer of data) to actor B (consumer of data), the Actor A must make sure that its tasks producing the exchanged cl_mem buffers are enqueued into the coordination command queue before the ownership change. Note: In OpenCL, command queues are associated with a target device. In the context of the vx_khr_opencl_interop API, the device to which the coordination command queue is associated is not important, since it does not prevent actors from using other command queues targeting other devices. The application can manage any dependencies between command queues using OpenCL markers (or barriers) and events. 1.5. Usage Example of the vx_khr_opencl_interop API In this example, OpenVX is used to process an image provided by a camera in the host memory. Then OpenCL is used to post-process the two outputs of the OpenVX graph on different OpenCL devices. The example goes through the following steps: - create a vx_context using a cl_context supplied by the application - create multiple cl_mem buffers for the input capture device to cycle through - create graph output images using vxCreateImageFromHandle() with the memory type as VX_MEMORY_TYPE_OPENCL_BUFFER - import a pre-verified OpenVX graph - execute the OpenVX graph - reclaim the OpenCL buffer and enqueue post-processing OpenCL kernels #include <VX/vx_khr_opencl_interop.h> ... // Create an OpenCL context with two OpenCL devices. cl_context_properties ctxprop[] = { CL_CONTEXT_PLATFORM, (cl_context_properties)platform_id, 0, 0 }; cl_device_id device_list[2] = { device_id1, device_id2 }; cl_context