[SYCL] Introduce the Level Zero plugin (#1718)

Plugin itself consists of the header and the source file plus cmake
file to build the plugin. Also the following changes were made to
suport the Level Zero plugin in SYCL RT:
* New level0 value was added to the backend enum
* New PI_LEVEL0 value support was added to the SYCL_BE config.
* Docs were updated. Mentioned Level Zero backend and provided the
link to the Level Zero runtime for Intel GPU.
* Changes in sycl cmake file to build Level Zero plugin by default and to
install it with sycl toolchain.

LIT testing with PI_LEVEL0 backend will be enabled in the following
commits. This commits introduces the plugin and makes it buildable.

Signed-off-by: Artur Gainullin [email protected]

Author: againull

buildbot/configure.py

@@ -82,6 +82,14 @@ def do_configure(args):
             "-DOpenCL_INCLUDE_DIR={}".format(ocl_header_dir),
             "-DOpenCL_LIBRARY={}".format(icd_loader_lib)])
 
+    if args.l0_headers and args.l0_loader:
+      cmake_cmd.extend([
+            "-DL0_INCLUDE_DIR={}".format(args.l0_headers),
+            "-DL0_LIBRARY={}".format(args.l0_loader)])
+    elif args.l0_headers or args.l0_loader:
+      sys.exit("Please specify both Level Zero headers and loader or don't specify "
+               "none of them to let download from github.com")
+
     # Add additional CMake options if provided
     if args.cmake_opt:
       cmake_cmd += args.cmake_opt
@@ -115,6 +123,8 @@ def main():
     # User options
     parser.add_argument("-s", "--src-dir", metavar="SRC_DIR", help="source directory (autodetected by default)")
     parser.add_argument("-o", "--obj-dir", metavar="OBJ_DIR", help="build directory. (<src>/build by default)")
+    parser.add_argument("--l0-headers", metavar="L0_HEADER_DIR", help="directory with Level Zero headers")
+    parser.add_argument("--l0-loader", metavar="L0_LOADER", help="path to the Level Zero loader")
     parser.add_argument("-t", "--build-type",
                         metavar="BUILD_TYPE", default="Release", help="build type: Debug, Release")
     parser.add_argument("--cuda", action='store_true', help="switch from OpenCL to CUDA")

buildbot/dependency.conf

@@ -3,8 +3,10 @@
 ocl_cpu_rt_ver=2020.10.4.0.15
 # https://github.com/intel/llvm/releases/download/2020-03/win-oclcpuexp-2020.10.4.0.15_rel.zip
 ocl_cpu_rt_ver_win=2020.10.4.0.15
+# Same GPU driver supports Level Zero and OpenCL:
 # https://github.com/intel/compute-runtime/releases/tag/20.19.16754
 ocl_gpu_rt_ver=20.19.16754
+# Same GPU driver supports Level Zero and OpenCL:
 # https://downloadcenter.intel.com/download/29557/Intel-Graphics-Windows-10-DCH-Drivers
 ocl_gpu_rt_ver_win=ci-neo-015900
 intel_sycl_ver=build

sycl/CMakeLists.txt

@@ -305,6 +305,7 @@ set( SYCL_TOOLCHAIN_DEPLOY_COMPONENTS
      sycl-headers-extras
      sycl
      pi_opencl
+     pi_level0
      libsycldevice
 )
 if(OpenCL_INSTALL_KHRONOS_ICD_LOADER AND TARGET ocl-icd)

sycl/doc/EnvironmentVariables.md

@@ -12,7 +12,7 @@ subject to change. Do not rely on these variables in production code.
 | Environment variable | Values | Description |
 | -------------------- | ------ | ----------- |
 | SYCL_PI_TRACE | Described [below](#sycl_pi_trace-options)  | Enable specified level of tracing for PI. |
-| SYCL_BE | PI_OPENCL, PI_CUDA | Force SYCL RT to consider only devices of the specified backend during the device selection. |
+| SYCL_BE | PI_OPENCL, PI_LEVEL0, PI_CUDA | Force SYCL RT to consider only devices of the specified backend during the device selection. |
 | SYCL_DEVICE_TYPE | One of: CPU, GPU, ACC, HOST | Force SYCL to use the specified device type. If unset, default selection rules are applied. If set to any unlisted value, this control has no effect. If the requested device type is not found, a `cl::sycl::runtime_error` exception is thrown. If a non-default device selector is used, a device must satisfy both the selector and this control to be chosen. This control only has effect on devices created with a selector. |
 | SYCL_PROGRAM_COMPILE_OPTIONS | String of valid OpenCL compile options | Override compile options for all programs. |
 | SYCL_PROGRAM_LINK_OPTIONS | String of valid OpenCL link options | Override link options for all programs. |

sycl/doc/GetStartedGuide.md

@@ -161,8 +161,12 @@ which contains all the symbols required.
 To run DPC++ applications on OpenCL devices, OpenCL implementation(s) must be
 present in the system.
 
-The OpenCL RT for `GPU`, OpenCL RT for `CPU` and TBB runtime which are needed
-to run DPC++ application on Intel `GPU` or Intel `CPU` devices can be
+To run DPC++ applications on Level Zero devices, Level Zero implementation(s)
+must be present in the system. You can find the link to the Level Zero spec in
+the following section [Find More](#find-more).
+
+The Level Zero RT for `GPU`, OpenCL RT for `GPU`, OpenCL RT for `CPU` and TBB runtime
+which are needed to run DPC++ application on Intel `GPU` or Intel `CPU` devices can be
 downloaded using links in
 [the dependency configuration file](../../buildbot/dependency.conf)
 and installed following the instructions below. The same versions are used in
@@ -551,5 +555,7 @@ class CUDASelector : public cl::sycl::device_selector {
 [https://spec.oneapi.com/versions/latest/elements/dpcpp/source/index.html](https://spec.oneapi.com/versions/latest/elements/dpcpp/source/index.html)
 * SYCL\* 1.2.1 specification:
 [www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf](https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf)
+* oneAPI Level Zero specification:
+[https://spec.oneapi.com/versions/latest/oneL0/index.html](https://spec.oneapi.com/versions/latest/oneL0/index.html)
 
 \*Other names and brands may be claimed as the property of others.

sycl/include/CL/sycl/backend_types.hpp

@@ -13,7 +13,7 @@
 __SYCL_INLINE_NAMESPACE(cl) {
 namespace sycl {
 
-enum class backend { host, opencl, cuda };
+enum class backend : char { host, opencl, level0, cuda };
 
 template <backend name, typename SYCLObjectT> struct interop;
 

sycl/include/CL/sycl/detail/pi.hpp

@@ -57,9 +57,11 @@ bool trace(TraceLevel level);
 
 #ifdef SYCL_RT_OS_WINDOWS
 #define OPENCL_PLUGIN_NAME "pi_opencl.dll"
+#define LEVEL0_PLUGIN_NAME "pi_level0.dll"
 #define CUDA_PLUGIN_NAME "pi_cuda.dll"
 #else
 #define OPENCL_PLUGIN_NAME "libpi_opencl.so"
+#define LEVEL0_PLUGIN_NAME "libpi_level0.so"
 #define CUDA_PLUGIN_NAME "libpi_cuda.so"
 #endif
 

sycl/plugins/CMakeLists.txt

@@ -5,3 +5,4 @@ if(SYCL_BUILD_PI_CUDA)
 endif()
 
 add_subdirectory(opencl)
+add_subdirectory(level_zero)

sycl/plugins/level_zero/CMakeLists.txt

@@ -0,0 +1,94 @@
+# PI Level0 plugin library
+
+if (NOT DEFINED L0_LIBRARY OR NOT DEFINED L0_INCLUDE_DIR)
+  message(STATUS "Download Level Zero loader and headers from github.com")
+  if(MSVC)
+    set(L0_LIBRARY
+        "${LLVM_LIBRARY_OUTPUT_INTDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}ze_loader${CMAKE_STATIC_LIBRARY_SUFFIX}")
+  else()
+    set(L0_LIBRARY
+        "${LLVM_LIBRARY_OUTPUT_INTDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}ze_loader${CMAKE_SHARED_LIBRARY_SUFFIX}")
+  endif()
+  if (CMAKE_C_COMPILER)
+      list(APPEND AUX_CMAKE_FLAGS -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER})
+  endif()
+  if (CMAKE_CXX_COMPILER)
+      list(APPEND AUX_CMAKE_FLAGS -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER})
+  endif()
+  file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/l0_loader_build)
+  ExternalProject_Add(l0-loader
+    GIT_REPOSITORY    https://github.com/oneapi-src/level-zero.git
+    GIT_TAG           origin/master
+    SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/Level0/l0_loader"
+    BINARY_DIR        "${CMAKE_CURRENT_BINARY_DIR}/l0_loader_build"
+    INSTALL_DIR       "${CMAKE_CURRENT_BINARY_DIR}/l0_loader_install"
+    CMAKE_ARGS -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
+               -DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}
+               -DOpenCL_INCLUDE_DIR=${OpenCL_INCLUDE_DIRS}
+               -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
+               -DCMAKE_INSTALL_LIBDIR:PATH=lib${LLVM_LIBDIR_SUFFIX}
+               ${AUX_CMAKE_FLAGS}
+    STEP_TARGETS      configure,build,install
+    DEPENDS           ocl-headers
+    BUILD_BYPRODUCTS ${L0_LIBRARY}
+  )
+  ExternalProject_Add_Step(l0-loader llvminstall
+    COMMAND ${CMAKE_COMMAND} -E copy_directory <INSTALL_DIR>/ ${LLVM_BINARY_DIR}
+    COMMENT "Installing l0-loader into the LLVM binary directory"
+    DEPENDEES install
+  )
+
+  install(DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/l0_loader_install/"
+    DESTINATION "."
+    COMPONENT l0-loader
+  )
+
+  list(APPEND SYCL_TOOLCHAIN_DEPLOY_COMPONENTS l0-loader)
+else()
+  include_directories("${L0_INCLUDE_DIR}")
+endif()
+
+include_directories("${sycl_inc_dir}")
+include_directories(${OPENCL_INCLUDE})
+
+add_library(pi_level0 SHARED
+  "${sycl_inc_dir}/CL/sycl/detail/pi.h"
+  "${CMAKE_CURRENT_SOURCE_DIR}/pi_level0.cpp"
+  "${CMAKE_CURRENT_SOURCE_DIR}/pi_level0.hpp"
+)
+
+if (MSVC)
+  # by defining __SYCL_BUILD_SYCL_DLL, we can use __declspec(dllexport)
+  # which are individually tagged for all pi* symbols in pi.h
+  target_compile_definitions(pi_level0 PRIVATE __SYCL_BUILD_SYCL_DLL)
+else()
+  # we set the visibility of all symbols 'hidden' by default.
+  # In pi.h file, we set exported symbols with visibility==default individually
+  target_compile_options(pi_level0 PUBLIC -fvisibility=hidden)
+
+  # This script file is used to allow exporting pi* symbols only.
+  # All other symbols are regarded as local (hidden)
+  set(linker_script "${CMAKE_CURRENT_SOURCE_DIR}/../ld-version-script.txt")
+
+  # Filter symbols based on the scope defined in the script file,
+  # and export pi* function symbols in the library.
+  target_link_libraries( pi_level0
+    PRIVATE "-Wl,--version-script=${linker_script}"
+  )
+endif()
+
+if (TARGET l0-loader)
+  add_dependencies(pi_level0 l0-loader)
+endif()
+  add_dependencies(sycl-toolchain pi_level0)
+
+target_link_libraries(pi_level0 PRIVATE "${L0_LIBRARY}")
+if (UNIX)
+  target_link_libraries(pi_level0 PRIVATE pthread)
+endif()
+
+add_common_options(pi_level0)
+
+install(TARGETS pi_level0
+    LIBRARY DESTINATION "lib" COMPONENT pi_level0
+    RUNTIME DESTINATION "bin" COMPONENT pi_level0)