diff --git a/sycl/doc/PluginInterface.md b/sycl/doc/PluginInterface.md index 4c0f18982ea5b..4d13f882650e6 100644 --- a/sycl/doc/PluginInterface.md +++ b/sycl/doc/PluginInterface.md @@ -1,14 +1,13 @@ # The DPC++ Runtime Plugin Interface. - ## Overview -The DPC++ Runtime Plugin Interface (PI) is the interface layer between -device-agnostic part of the DPC++ runtime and the device-specific runtime layers +The DPC++ Runtime Plugin Interface (PI) is an interface layer between the +device-agnostic part of DPC++ runtime and the device-specific runtime layers which control execution on devices. It employs the “plugin” mechanism to bind -to the device specific runtime layers similarly to what is used by libomptarget +to the device specific runtime layers similar to what is used by libomptarget or OpenCL. -The picture below illustrates the placement of the PI within the overall DPC++ +The picture below illustrates the placement of PI within the overall DPC++ runtime stack. Dotted lines show components or paths which are not yet available in the runtime, but are likely to be developed. ![PI in DPC++ runtime architecture](images/RuntimeArchitecture.svg) @@ -22,13 +21,13 @@ points implementing the PI interface. The DPC++ runtime collects those function pointers into a PI interface dispatch table - one per plugin - and uses this table to dispatch to the device(s) covered by the corresponding plugin. -PI is based on a subset of OpenCL 1.2 runtime specification, it follows its -platform, execution and memory models in all aspects except those explicitly -mentioned in this document. A part of PI API types and functions have exact +PI is based on a subset of OpenCL 1.2 runtime specification, it follows OpenCL's +platform, execution and memory models in all aspects except for those explicitly +mentioned in this document. Some of PI API types and functions have exact matches in OpenCL. Whenever there is such a match, the semantics also fully -matches unless the differences are explicitly specified in this document. While +match unless the differences are explicitly specified in this document. While PI has roots in OpenCL, it does have many differences, and the gap is likely -to grow, for example in the areas of memory model and management, program +to grow, for example in areas of memory model and management, program management. ## Discovery and linkage of PI implementations @@ -36,17 +35,71 @@ management. ![PI implementation discovery](images/PluginDiscovery.svg) Device discovery phase enumerates all available devices and their features by -querying underlying plugins found in the system. This process is only performed -once before any actual offload is attempted. +querying underlying plugins found in the system. This process is performed when +all attached platforms or devices are queried in an application; for example, +during device selection. ### Plugin discovery -Plugins are physically dynamic libraries stored somewhere in the system where -the DPC++ runtime runs. TBD - design and describe the process in details. +Plugins are physically dynamic libraries or shared objects. +The process to discover plugins follows the following guidelines. + +The DPC++ Runtime reads the names of the plugins from a configuration file +at a predetermined location (TBD - Add this location). These plugins are +searched at locations in env LD_LIBRARY_PATH on Linux and env PATH on Windows. +(TBD - Extend to search the plugins at a path relative to the SYCL Runtime +installation directory by using DT_RPATH on Linux. Similar functionality can be +achieved on Windows using SetDllDirectory. This will help avoiding extra setting +of LD_LIBRARY_PATH.) +To avoid any issues with read-only access, an environment variable SYCL_PI_CONFIG +can be set to point to the configuration file which lists the Plugin names. The +enviroment variable if set overrides the predetermined location's config file. +These Plugins are then be searched in LD_LIBRARY_PATH locations. +It is the developer's responsibility to include the plugin names from the +predetermined location's config file to enable discovery of all plugins. +(TBD - Extend to support search in DT_RPATH as above.) +In the current implementation the plugin names are hardcoded in the library. +Configuration file or env SYCL_PI_CONFIG is currently not being considered. + +A trace mechanism is provided using env SYCL_PI_TRACE to log the discovery/ +binding/ device enumeration process. Different levels of tracing can be achieved +with different values of SYCL_PI_TRACE. +SYCL_PI_TRACE=0x01 provides basic trace of plugins discovered and bound. It also +lists the device selector's selected device information. +SYCL_PI_TRACE=0x02 provides trace of all PI calls made from the DPC++ runtime +with arguments and returned values. +SYCL_PI_TRACE=-1 lists all PI Traces above and more debug messages. #### Plugin binary interface -TBD - list and describe all the symbols plugin must export in order to be picked -up by the DPC++ runtime for offload. +Plugins should implement all the Interface APIs required for the PI Version +it supports. There is [pi.def](../include/CL/sycl/detail/pi.def)/ +[pi.h](../include/CL/sycl/detail/pi.h) file listing all PI API names that can be +called by the specific version of Plugin Interface. +It exports a function - "piPluginInit" that returns the plugin details and function pointer +table containing the list of pointers to implemented Interface Functions defined in pi.h. +In the future, this document will list the minimum set of Interface APIs +to be supported by Plugins. This will also require adding functionality to SYCL +Runtime to work with such limited functionality plugins. + +(TBD - list and describe the symbols that a plugin must implement in order to +be picked up by the DPC++ runtime for offload.) + +#### Binding a Plugin +The DPC++ Runtime loads all discovered Plugins and tries to bind them by calling +piPluginInit API for each loaded Plugin. The Plugins return the information of +supported PI version and the list of implemented PI API Function pointers. +(TBD - Use the PI API Version information and check for compatibility. +Extend to support version compatibility checks without loading the library. +Eg:Changing the plugin name to reflect the supported Plugin Interface version.) +The information of compatible plugins (with the Function Pointer Table) is +stored in the associated platforms during platform object construction. +The PI API calls are later forwarded using this information. +A plugin is said to "bind" after this process completes with no errors. +During device selection, the user can prefer selection of a device from a +specific Plugin or Backend using the env SYCL_BE. The correspondence between +a plugin and a SYCL_BE value is currently hardcoded in the runtime. +( TBD: Make this a part of configuration file). +Eg: SYCL_BE=PI_OPENCL corresponds to OpenCL Plugin. #### OpenCL plugin @@ -56,12 +109,18 @@ OpenCL implementations. They can be installed either in the standard Khronos ICD-compatible way (e.g. listed in files under /etc/OpenCL/vendors on Linux) or not, and the OpenCL plugin can hook up with both. -TBD describe the nested OpenCL implementation discovery process performed by +TBD - implement and describe the nested OpenCL implementation discovery process performed by the OpenCL plugin ### Device enumeration by plugins +Devices from all bound plugins are queried and listed as and when required, eg: during device selection in device_selector. +The trace shows the PI API calls made when using SYCL_PI_TRACE=-1. +(TBD - Add the trace to list all available devices when plugins are successfully bound.) + +### Plugin Unloading +The plugins not chosen to be connected to should be unloaded. -TBD +TBD - Unloading a bound plugin. ## PI API Specification @@ -71,11 +130,11 @@ able to operate on the corresponding device. The core API further breaks down into - **OpenCL-based** APIs which have OpenCL origin and semantics - **Extension** APIs which don't have counterparts in the OpenCL -- **Interoperability API** which allows interoperability with underlying APIs +- **Interoperability API** which allows interoperability with underlying runtimes such as OpenCL. See [pi.h](../include/CL/sycl/detail/pi.h) header for the full list and -descriptions of PI APIs. [TBD: link to pi.h doxygen here] +descriptions of PI APIs. ### The Core OpenCL-based PI APIs @@ -114,10 +173,10 @@ in a data section. ### The Interoperability PI APIs These are APIs needed to implement DPC++ runtime interoperability with -underlying "native" device runtimes such as OpenCL. Currently there are only -OpenCL interoperability APIs, which is to be implemented by the OpenCL PI -plugin only. These APIs match semantics of the corresponding OpenCL APIs -exactly. +underlying "native" device runtimes such as OpenCL. +There are some OpenCL interoperability APIs, which are to be implemented +by the OpenCL PI plugin only. These APIs match semantics of the corresponding +OpenCL APIs exactly. For example: ``` @@ -129,6 +188,21 @@ pi_result piclProgramCreateWithSource( pi_program * ret_program); ``` +Some interoperability extension APIs have been added to get native runtime +handles from the backend-agnostic PI Objects or to create PI Objects using the +native handles. Eg: + +``` +pi_result piextDeviceGetNativeHandle( + pi_device device, + pi_native_handle *nativeHandle); + +pi_result piextDeviceCreateWithNativeHandle( + pi_native_handle nativeHandle, + pi_device *device); + +``` + ### PI Extension mechanism TBD This section describes a mechanism for DPC++ or other runtimes to detect