[SYCL] Embed bfloat16 devicelib into executable if necessary #16729

jinge90 · 2025-01-22T08:58:36Z

Currently, sycl bfloat16 conversion functions are implemented in 2 devicelib spvs(fallback version and native version).
The native version targets for any platform which supports "cl_intel_bfloat16_conversions" extension and fallback version is used for all other platforms.
SYCL runtime will select the bfloat16 spvs during execution time by checking bfloat16 extension.
The design requires us to ship 2 spv files together with sycl runtime which some users may dislike.
The PR uses sycl dynamic library mechanism to re-implement this behavior. These 2 bfloat16 lib files are regarded as dynamic library and embedded to final executable, so we don't need to ship any bfloat16 spv libs.
The PR consists following changes:

Driver: pass the devicelib file location to sycl-post-link tool.
sycl-post-link: analyze user's device image to see whether bfloat16 devicelib functions are used. If yes, add the 2 bfloat16 devicelib files as "required" dynamic library. All required bfloat16 devicelib functions are treated as "imported" symbols in user's device image and all functions in bfloat16 devicelib are "exported" symbols.
Sycl runtime will load and link the required bfloat16 devicelib image and resolve the imported symbols.

Fallback and native version of bfloat16 devicelib files have exactly same exported functions, we add a new metadata("SYCL_DEVICELIB_BF16_TYPE") to indicate the version in them. SYCL runtime will check cl_intel_bfloat16_conversions extension and this metadata to decide which version will be linked.

Signed-off-by: jinge90 <[email protected]>

…b_spv

Signed-off-by: jinge90 <[email protected]>

…b_spv

Signed-off-by: jinge90 <[email protected]>

…b_spv

jinge90 · 2025-02-21T02:43:37Z

Hi, @asudarsa and @AlexeySachkov
Kind ping~.
Thanks very much.

jopperm · 2025-02-24T00:08:19Z

I'm looking at this in the context of SYCL-RTC. Overall this change LGTM and I think we can adapt our equivalent of sycl-post-link to embed the bf16 libraries as device images as well.

However, is it a problem if multiple of these special images are present at runtime? This would be the case if the application compiles multiple kernel bundles at runtime using bf16, or even if the application contains normal kernels using bf16 and also runtime-compiles a single bf16-using bundle.

…b_spv

jinge90 · 2025-02-24T05:47:39Z

I'm looking at this in the context of SYCL-RTC. Overall this change LGTM and I think we can adapt our equivalent of sycl-post-link to embed the bf16 libraries as device images as well.

However, is it a problem if multiple of these special images are present at runtime? This would be the case if the application compiles multiple kernel bundles at runtime using bf16, or even if the application contains normal kernels using bf16 and also runtime-compiles a single bf16-using bundle.

Hi, @jopperm
No matter how many user's device images are present, sycl-post-link tool will analyze all of them and once any user's device image calls bf16 devicelib functions, 2 actions will be taken:

For the user device image using bf16 devicelib functions, the used function name will be added to its imported symbols
bf16 devicelib image will be bundled into final executable and all functions included will be added to exported symbols.

When sycl runtime begins to launch a user kernel, it will resolves all imported symbols from the belonging user's device image, if the the imported symbols include bf16 devicelib functions, sycl runtime will link the user's device image with corresponding devicelib image.

Thanks very much.

jopperm · 2025-02-24T10:22:07Z

Thanks, got it, after post-link, a single sycl_device_binaries contains N user images plus 1 fallback- and 1 native-bfloat16-devicelib-image. But is it problematic if multiple sycl_device_binaries, each containing the special bfloat16-images, are registered with the runtime?

jinge90 · 2025-02-25T02:06:26Z

Thanks, got it, after post-link, a single sycl_device_binaries contains N user images plus 1 fallback- and 1 native-bfloat16-devicelib-image. But is it problematic if multiple sycl_device_binaries, each containing the special bfloat16-images, are registered with the runtime?

Hi, @jopperm
Do you mean there are multiple user device images and they all use bf16 devicelib functions? If so, the scenario is same, suppose we have 3 user kernels and we use device-code-split=per_kernel, then we will have 3 device images and each of them includes one kernel, the 3 kernels all use bf16 devicelib functions.
In sycl-post-link, it will scan all of these 3 device images and find bf16 devicelib functions are needed by them, so all of these 3 user device images will have a imported symbol property and corresponding bf16 devicelib function name will be added to each device image's imported symbol list. sycl-post-link will also embed fallback and native bf16 device image into final program, all of the bf16 functions are added to there exported symbol list, so there are 5 device images included(3 user device image + 2 bf16 devicelib image(native and fallback version)).
In execution time, when one user kernel is going to be executed, sycl runtime needs to build the program from corresponding device image first. Sycl runtime will collect deps for this device image and find it has imported symbols, so bf16 devicelib image will be linked with this device image to resolve the all imported symbols. After it is done, sycl runtime will build final kernel and run it. For each of the user kernel, the process is same, if it is to be launched, sycl runtime need to link corresponding device image with dependent bf16 devicelib image and first.
Thanks very much.

asudarsa · 2025-02-26T02:42:28Z

clang/test/Driver/linker-wrapper-sycl-win.cpp

+//
+// Run clang-linker-wrapper test
+//
+// RUN: clang-linker-wrapper -sycl-post-link-options="SYCL_POST_LINK_OPTIONS" -llvm-spirv-options="LLVM_SPIRV_OPTIONS" "--host-triple=x86_64-pc-windows-msvc" "--linker-path=/usr/bin/ld" "--" HOST_LINKER_FLAGS "-dynamic-linker" HOST_DYN_LIB "-o" "a.out" HOST_LIB_PATH HOST_STAT_LIB %t5.o -sycl-device-libraries=libsycl-crt.new.o -sycl-device-library-location=%S/Inputs/SYCL/lib --dry-run 2>&1 | FileCheck -check-prefix=CHK-CMDS-DEVICE-LIB-DIR %s


Do we need this extra file? Can we not just use a dummy file? Please see

llvm/clang/test/Driver/linker-wrapper-sycl.cpp

Line 61 in b512633

// Generate .o file as SYCL device library file.

If you do not care about what's inside the file, then touch %t1.devicelib.o will suffice.

Hi, @asudarsa
We need to pass "-sycl-device-library-location" to clang-linker-wrapper tool and clang-linker-wrapper will look for specified devicelib files in this specified "-sycl-device-library-location", so we put an empty devicelib file in Inputs/SYCL/lib
Thanks very much.

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp

asudarsa · 2025-02-26T03:00:13Z

clang/test/Driver/linker-wrapper-sycl-win.cpp

+//
+// Run clang-linker-wrapper test
+//
+// RUN: clang-linker-wrapper -sycl-post-link-options="SYCL_POST_LINK_OPTIONS" -llvm-spirv-options="LLVM_SPIRV_OPTIONS" "--host-triple=x86_64-pc-windows-msvc" "--linker-path=/usr/bin/ld" "--" HOST_LINKER_FLAGS "-dynamic-linker" HOST_DYN_LIB "-o" "a.out" HOST_LIB_PATH HOST_STAT_LIB %t5.o -sycl-device-libraries=libsycl-crt.new.o -sycl-device-library-location=%S/Inputs/SYCL/lib --dry-run 2>&1 | FileCheck -check-prefix=CHK-CMDS-DEVICE-LIB-DIR %s


Also, a bit confused. Why name this '-crt' when you are interested only in bfloat16 device lib file?

Hi, @asudarsa
Since the device library list can't be empty, I just randomly put libsycl-crt in the list to avoid compiler warning.
Thanks very much.

llvm/lib/SYCLLowerIR/ComputeModuleRuntimeInfo.cpp

asudarsa

Some minor nits. Overall looks good. Adding @maksimsab to list of reviewers to get feedback on sycl-post-link related changes.

Thanks

…b_spv

Signed-off-by: jinge90 <[email protected]>

jinge90 · 2025-02-26T06:10:43Z

Thanks, got it, after post-link, a single sycl_device_binaries contains N user images plus 1 fallback- and 1 native-bfloat16-devicelib-image. But is it problematic if multiple sycl_device_binaries, each containing the special bfloat16-images, are registered with the runtime?

Hi, @jopperm Do you mean there are multiple user device images and they all use bf16 devicelib functions? If so, the scenario is same, suppose we have 3 user kernels and we use device-code-split=per_kernel, then we will have 3 device images and each of them includes one kernel, the 3 kernels all use bf16 devicelib functions. In sycl-post-link, it will scan all of these 3 device images and find bf16 devicelib functions are needed by them, so all of these 3 user device images will have a imported symbol property and corresponding bf16 devicelib function name will be added to each device image's imported symbol list. sycl-post-link will also embed fallback and native bf16 device image into final program, all of the bf16 functions are added to there exported symbol list, so there are 5 device images included(3 user device image + 2 bf16 devicelib image(native and fallback version)). In execution time, when one user kernel is going to be executed, sycl runtime needs to build the program from corresponding device image first. Sycl runtime will collect deps for this device image and find it has imported symbols, so bf16 devicelib image will be linked with this device image to resolve the all imported symbols. After it is done, sycl runtime will build final kernel and run it. For each of the user kernel, the process is same, if it is to be launched, sycl runtime need to link corresponding device image with dependent bf16 devicelib image and first. Thanks very much.

Hi, @jopperm
Does the explanation above remove your concern?
Thanks very much.

jopperm · 2025-02-26T07:20:09Z

Does the explanation above remove your concern?

Not entirely, your explanation is clear, thank you, but I was asking something else. Let me try to rephrase. Assume you have libary libX containing kernels using bf16. Your application also contains kernels using bf16 and links to libX. At runtime, the library's and the app's sycl_device_binaries (= the result of post-link embedded into the binaries) are registered with the program manager. Both would contain the special bfloat images exporting the symbols of the conversion builtins. Does that interfere with linking either the app's or libX's device images importing these symbols?

jinge90 · 2025-02-26T08:02:55Z

Does the explanation above remove your concern?

Not entirely, your explanation is clear, thank you, but I was asking something else. Let me try to rephrase. Assume you have libary libX containing kernels using bf16. Your application also contains kernels using bf16 and links to libX. At runtime, the library's and the app's sycl_device_binaries (= the result of post-link embedded into the binaries) are registered with the program manager. Both would contain the special bfloat images exporting the symbols of the conversion builtins. Does that interfere with linking either the app's or libX's device images importing these symbols?

Hi, @jopperm
Do you mean sycl dynamic library when referring to "libX"?
Thanks very much.

jopperm · 2025-02-26T08:06:10Z

Do you mean sycl dynamic library when referring to "libX"?

Let's say it's a static lib. The situation I'm interested in is __sycl_register_lib is called more than once.

jinge90 · 2025-02-26T08:11:53Z

__sycl_register_lib

Hi, @jopperm
I expect that the bf16 devicelib functions still work as expected but we may add duplicate device images for bf16 devicelib, this should be optimized. I will double check, thanks for pointing out this!

maksimsab

Please, add tests corresponding to sycl-post-link changes.

mdtoguchi · 2025-02-26T15:40:31Z

clang/test/Driver/sycl-post-link-options-win.cpp

@@ -3,7 +3,7 @@
 // RUN: %clangxx -### --target=x86_64-pc-windows-msvc -fsycl \
 // RUN:          -Xdevice-post-link -O0 %s 2>&1 \
 // RUN:   | FileCheck -check-prefix OPTIONS_POSTLINK_JIT_OLD %s


Add --sysroot=%S/Inputs/SYCL to the command line as there is no guarantee the base lib will be created with the compiler build (for example with the sanitizer based builds).

mdtoguchi · 2025-02-26T15:41:32Z

clang/test/Driver/sycl-post-link-options.cpp

@@ -3,7 +3,7 @@
 // RUN: %clangxx --target=x86_64-unknown-linux-gnu -fsycl -### \
 // RUN:          --no-offload-new-driver -Xdevice-post-link -O0 %s 2>&1 \
 // RUN:   | FileCheck -check-prefix OPTIONS_POSTLINK_JIT_OLD %s


Add --sysroot=%S/Inputs/SYCL here as well

AlexeySachkov

Overall LGTM, but I agree with @maksimsab's point about adding tests. Some unit-tests for SYCL RT should also be added, you can look at https://github.com/intel/llvm/blob/sycl/sycl/unittests/program_manager/DynamicLinking/DynamicLinking.cpp tests for reference

AlexeySachkov · 2025-02-27T10:09:41Z

llvm/include/llvm/SYCLLowerIR/ModuleSplitter.h

+    // DeviceLib module doesn't include any entry point,it can be constructed
+    // using ctor without any entry point related parameter.
+    for (auto Fn : SYCLDeviceLibs) {
+      if (Name == std::string(Fn))


Suggested change

if (Name == std::string(Fn))

if (Name == StringRef(Fn))

There is no need to allocate heap memory and make a string copy for a sake of a simple comparison

AlexeySachkov · 2025-02-27T10:10:33Z

llvm/include/llvm/SYCLLowerIR/ModuleSplitter.h

+    // using ctor without any entry point related parameter.
+    for (auto Fn : SYCLDeviceLibs) {
+      if (Name == std::string(Fn))
+        IsSYCLDeviceLib = true;


We should add break;, there is no need to scan other function names once we've found what we were looking for

AlexeySachkov · 2025-02-27T10:19:06Z

sycl/source/detail/program_manager/program_manager.cpp

@@ -1798,14 +1837,28 @@ ProgramManager::kernelImplicitLocalArgPos(const std::string &KernelName) const {
  return {};
 }

+static bool skipEmptyImage(sycl_device_binary RawImg) {


Suggested change

static bool skipEmptyImage(sycl_device_binary RawImg) {

static bool shouldSkipEmptyImage(sycl_device_binary RawImg) {

jinge90 requested review from a team as code owners January 22, 2025 08:58

jinge90 marked this pull request as draft January 22, 2025 08:58

jinge90 temporarily deployed to WindowsCILock January 22, 2025 08:58 — with GitHub Actions Inactive

[SYCL] Embed bfloat16 devicelib into executable if necessary

5187e85

Signed-off-by: jinge90 <[email protected]>

jinge90 had a problem deploying to WindowsCILock January 22, 2025 09:46 — with GitHub Actions Failure

jinge90 added 2 commits January 23, 2025 10:27

pass sycl_devicelib_loc in clang-linker-wrapper

0f22cde

Signed-off-by: jinge90 <[email protected]>

add bf16 devicelib property to distingush fallback and native version

d71c3d8

Signed-off-by: jinge90 <[email protected]>

jinge90 temporarily deployed to WindowsCILock January 23, 2025 03:10 — with GitHub Actions Inactive

Mark __devicelib_* func from bf16 devicelib as imported

e453e70

Signed-off-by: jinge90 <[email protected]>

jinge90 had a problem deploying to WindowsCILock January 23, 2025 03:57 — with GitHub Actions Failure

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

738a679

…b_spv

jinge90 had a problem deploying to WindowsCILock January 24, 2025 07:05 — with GitHub Actions Error

Import and Export all bf16 devicelib functions

6e1ab96

Signed-off-by: jinge90 <[email protected]>

jinge90 temporarily deployed to WindowsCILock January 24, 2025 07:34 — with GitHub Actions Inactive

Fix clang-format

1fea227

Signed-off-by: jinge90 <[email protected]>

jinge90 had a problem deploying to WindowsCILock January 24, 2025 08:37 — with GitHub Actions Error

jinge90 temporarily deployed to WindowsCILock January 24, 2025 08:55 — with GitHub Actions Inactive

link fallback bf16 devicelib as shared libs

edb3ace

Signed-off-by: jinge90 <[email protected]>

jinge90 temporarily deployed to WindowsCILock January 24, 2025 09:59 — with GitHub Actions Inactive

embed native bf16 devicelib

3bf5b0d

Signed-off-by: jinge90 <[email protected]>

jinge90 had a problem deploying to WindowsCILock January 26, 2025 04:00 — with GitHub Actions Error

jinge90 temporarily deployed to WindowsCILock January 26, 2025 04:20 — with GitHub Actions Inactive

jinge90 added 3 commits January 26, 2025 12:26

select correct version of bf16 devicelib according to platform bf16 ext

6b67d79

Signed-off-by: jinge90 <[email protected]>

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

5e7fbd0

…b_spv

fix clang format

0d92d37

Signed-off-by: jinge90 <[email protected]>

jinge90 temporarily deployed to WindowsCILock January 26, 2025 05:07 — with GitHub Actions Inactive

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

5c0e024

…b_spv

jinge90 temporarily deployed to WindowsCILock February 5, 2025 02:00 — with GitHub Actions Inactive

jinge90 temporarily deployed to WindowsCILock February 5, 2025 02:47 — with GitHub Actions Inactive

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

bbbb34c

…b_spv

jinge90 temporarily deployed to WindowsCILock February 20, 2025 06:21 — with GitHub Actions Inactive

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

4c2d8bc

…b_spv

jinge90 temporarily deployed to WindowsCILock February 24, 2025 05:34 — with GitHub Actions Inactive

jinge90 temporarily deployed to WindowsCILock February 24, 2025 06:26 — with GitHub Actions Inactive

jopperm mentioned this pull request Feb 24, 2025

[SYCL][RTC] Add device library E2E test #17131

Open

asudarsa reviewed Feb 26, 2025

View reviewed changes

clang/tools/clang-linker-wrapper/ClangLinkerWrapper.cpp Outdated Show resolved Hide resolved

asudarsa reviewed Feb 26, 2025

View reviewed changes

llvm/lib/SYCLLowerIR/ComputeModuleRuntimeInfo.cpp Outdated Show resolved Hide resolved

asudarsa approved these changes Feb 26, 2025

View reviewed changes

asudarsa requested a review from maksimsab February 26, 2025 03:10

Merge remote-tracking branch 'upstream/sycl' into embed_bf16_deviceli…

0adfac1

…b_spv

jinge90 temporarily deployed to WindowsCILock February 26, 2025 05:06 — with GitHub Actions Inactive

jinge90 added 2 commits February 26, 2025 13:22

fix typo

024b699

Signed-off-by: jinge90 <[email protected]>

fix incorrect devicelib name

5576b32

Signed-off-by: jinge90 <[email protected]>

jinge90 temporarily deployed to WindowsCILock February 26, 2025 05:56 — with GitHub Actions Inactive

maksimsab requested changes Feb 26, 2025

View reviewed changes

mdtoguchi reviewed Feb 26, 2025

View reviewed changes

AlexeySachkov reviewed Feb 27, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SYCL] Embed bfloat16 devicelib into executable if necessary #16729

[SYCL] Embed bfloat16 devicelib into executable if necessary #16729

jinge90 commented Jan 22, 2025 •

edited

Loading

jinge90 commented Feb 21, 2025

jopperm commented Feb 24, 2025

jinge90 commented Feb 24, 2025

jopperm commented Feb 24, 2025

jinge90 commented Feb 25, 2025

asudarsa Feb 26, 2025

jinge90 Feb 26, 2025

asudarsa Feb 26, 2025

jinge90 Feb 26, 2025

asudarsa left a comment

jinge90 commented Feb 26, 2025

jopperm commented Feb 26, 2025

jinge90 commented Feb 26, 2025

jopperm commented Feb 26, 2025

jinge90 commented Feb 26, 2025

maksimsab left a comment

mdtoguchi Feb 26, 2025

mdtoguchi Feb 26, 2025

AlexeySachkov left a comment

AlexeySachkov Feb 27, 2025

AlexeySachkov Feb 27, 2025

AlexeySachkov Feb 27, 2025

	static bool skipEmptyImage(sycl_device_binary RawImg) {
	static bool shouldSkipEmptyImage(sycl_device_binary RawImg) {

[SYCL] Embed bfloat16 devicelib into executable if necessary #16729

Are you sure you want to change the base?

[SYCL] Embed bfloat16 devicelib into executable if necessary #16729

Conversation

jinge90 commented Jan 22, 2025 • edited Loading

jinge90 commented Feb 21, 2025

jopperm commented Feb 24, 2025

jinge90 commented Feb 24, 2025

jopperm commented Feb 24, 2025

jinge90 commented Feb 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asudarsa left a comment

Choose a reason for hiding this comment

jinge90 commented Feb 26, 2025

jopperm commented Feb 26, 2025

jinge90 commented Feb 26, 2025

jopperm commented Feb 26, 2025

jinge90 commented Feb 26, 2025

maksimsab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexeySachkov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinge90 commented Jan 22, 2025 •

edited

Loading