Build GTK3 support for linux x86-64 to maximize compatibility #1795

jonahgraham · 2025-02-03T20:06:42Z

Recently all SWT builds for Linux x86-64 have been done on a new Debian image. This leads to requiring such a new distro to be able to run SWT successfully.

This change builds GTK3 and common code on an older debian container to increase the range of versions we support. Only GTK4's key library, libswt-pi4-gtk*.so is shipped from a build on the new debaian container.

We continue to build all of the natives on latest debian so we benefit from compiler warnings/errors and other such tooling.

Prerequisite eclipse-platform/eclipse.platform.releng.aggregator#2802

Fixes #1791

The way I have tested this concept locally is to run the build.sh twice on my local machine using the two docker containers mentioned in runOnNativeBuildAgent of the Jenkinsfile*:

Start in root of SWT repo: cd /scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/
Collect the sources to build java -Dws=gtk build-scripts/CollectSources.java -nativeSources 'target/natives-build-temp/gtk'
cd 'target/natives-build-temp/gtk'
Build GTK4 version: docker run --rm -it -v/scratch:/scratch -w$PWD -e SWT_JAVA_HOME=/scratch/java/jdk-17.0.13+11 -e OUTPUT_DIR=/scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64 swtnativebuild ./build.sh -gtk4 install
Build GTK3 version docker run --rm -it -v/scratch:/scratch -w$PWD -e SWT_JAVA_HOME=/scratch/java/jdk-17.0.13+11 -e OUTPUT_DIR=/scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64 swtgtk3nativebuild ./build.sh -gtk3 install
Then try running various snippets with GTK3 and GTK4 in the IDE
Using "Show Command Line" option in launch configuration extract full command line and reproduce the run with x11docker

* Note that I actually built them manually with USER 1000 at the end of the file so I can share directories with the container. There are more details in the step-by-step in eclipse-platform/eclipse.platform.releng.aggregator#2802 (comment)

jonahgraham · 2025-02-03T20:17:38Z

@laeubi and/or @HannesWell How to test this?

Can I push to a branch of SWT so I can run builds on https://ci.eclipse.org/releng/job/eclipse.platform.swt/?

If so, any objections to me merging eclipse-platform/eclipse.platform.releng.aggregator#2802 now so that I can start to test this?

Also, assuming that is the way to test, can you check out if always push new natives is ok?

github-actions · 2025-02-03T20:27:13Z

Test Results

494 files ±0 494 suites ±0 11m 9s ⏱️ + 1m 40s
4 333 tests ±0 4 319 ✅ ±0 14 💤 ±0 0 ❌ ±0
16 574 runs ±0 16 465 ✅ ±0 109 💤 ±0 0 ❌ ±0

Results for commit fd1ea9c. ± Comparison against base commit 9d3673c.

♻️ This comment has been updated with latest results.

HannesWell

Also, assuming that is the way to test, can you check out if always push new natives is ok?

In general the policy is that all SWT native binaries have to be build on project infrastructure and to ensure this we usually don't accept changes in the binaries directly from PRs but just receive the code changes and only rebuild on master what is eventually added to the git/p2-repo.

In order to test the newly built binaries you can just download the built artifacts from Jenkins. E.g. for the second build:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/2/artifact/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64/target/

I've just created a PR to clean up the archived artifacts a bit: #1799

In the past there have been discussions about how to stream-line this process, especially as the required build-resources are currently rare (I hope this changes in the future: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4231).
But I think if this is changed, it should be a separate, dedicated PR.

For all the other changes I need a bit more time to review. I intend to do it tomorrow.

Jenkinsfile

jonahgraham · 2025-02-03T22:38:21Z

Also, assuming that is the way to test, can you check out if always push new natives is ok?

In general the policy is that all SWT native binaries have to be build on project infrastructure [...]

I will remove that commit then and apply your ternary change.

laeubi · 2025-02-04T05:02:20Z

@jonahgraham If you change any of the C files or build scripts a little bit (e.g. add a temporary comment) then the "build natives" workflow should kick in and build things with your changed containers.

What I have done for GTK4 back then was that I first checked the build is running, and then look at the build output (e.g. did the GTK4 binaries produced) and finally that all the so files are attached as desired.

If that works (and the tests not go completely wild) there is a good chance that the change is good to be submited and a committer can then run a build with forced binary generation.

as recommended in eclipse-platform#1795 (comment)

Recently all SWT builds for Linux x86-64 have been done on a new Debian image. This leads to requiring such a new distro to be able to run SWT successfully. This change builds GTK3 and common code on an older debian container to increase the range of versions we support. Only GTK4's key library, libswt-pi4-gtk*.so is shipped from a build on the new debaian container. We continue to build all of the natives on latest debian so we benefit from compiler warnings/errors and other such tooling. Prerequisite eclipse-platform/eclipse.platform.releng.aggregator#2802 Fixes eclipse-platform#1791

as recommended in eclipse-platform#1795 (comment)

HannesWell · 2025-02-04T17:20:31Z

@jonahgraham If you change any of the C files or build scripts a little bit (e.g. add a temporary comment) then the "build natives" workflow should kick in and build things with your changed containers.

Alternatively you can just start a build via Build with parameters and check the forceNativeBuild box:

With #1799 being submitted the build native fragments are now prominently displayed as the only Build artifacts in the corresponding Jenkins job and can be downloaded and further inspected and tests. The current build is waiting for the Y-build tests to complete and will therefore need some more time, but the previous build failed:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/5/

For example for the master you can see it already:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/master/921/

I have to leave now for a few hours but, when back I'll review this in detail and make a few more suggestions for enhancements.

HannesWell

Below I have made a few suggestions for making this change a bit more compact and one required change to fix the pipeline failure.

With that applied the build succeeds:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/

You can obtain the newly built native jars from that runs Build-artifacts section on the overview page or just visit it directly at:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/artifact/eclipse.platform.swt/binaries/

HannesWell · 2025-02-04T23:22:24Z

Jenkinsfile

+	} else if (platform == 'gtk.linux.x86_64') {
+		podTemplate(inheritFrom: 'basic' /* inherit general configuration */, containers: [
+			containerTemplate(name: 'swtbuild', image: 'eclipse/platformreleng-debian-swtgtk3nativebuild:10',
+				resourceRequestCpu:'1000m', resourceRequestMemory:'512Mi',
+				resourceLimitCpu:'2000m', resourceLimitMemory:'4096Mi',
+				alwaysPullImage: true, command: 'cat', ttyEnabled: true)
+		]) {
+			node(POD_LABEL) { stage(nativeBuildStageName) { container('swtbuild') { body() } } }
+		}


Instead of having two blocks one for each config with a lot of shared configuration, we could only have one block and the different images defined before:

def dockerImage = null; switch (platform) { case 'gtk.linux.x86_64': dockerImage = 'eclipse/platformreleng-debian-swtgtk3nativebuild:10'; case 'gtk4.linux.x86_64': dockerImage = 'eclipse/platformreleng-debian-swtnativebuild:12'; } if (dockerImage != null) { podTemplate(inheritFrom: 'basic' /* inherit general configuration */, containers: [ containerTemplate(name: 'swtbuild', image: dockerImage, resourceRequestCpu:'1000m', resourceRequestMemory:'512Mi', resourceLimitCpu:'2000m', resourceLimitMemory:'4096Mi', alwaysPullImage: true, command: 'cat', ttyEnabled: true) ]) { node(POD_LABEL) { stage(nativeBuildStageName) { container('swtbuild') { body() } } } }

The above has a small mistake, I needed to add break to the case statements. See latest commit fd1ea9c The result is the https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/ built both gtk3 and gtk4 with the gtk4 Debian image.

Once build finishes (which may be tomorrow for me now) I will look if that worked.

The good news is that my manual testing showed the problem! This change is still in draft state because I also want to bring in equinox/features/org.eclipse.equinox.executable.feature/library/gtk/check_dependencies.sh that is part of eclipse-equinox/equinox#834 to make sure the compiled libs match expectation at build time.

HannesWell · 2025-02-04T23:23:34Z

Jenkinsfile

@@ -220,22 +230,29 @@ pipeline {
 								runOnNativeBuildAgent("${PLATFORM}") {
 									cleanWs() // Workspace is not cleaned up by default, so we do it explicitly
 									echo "OS: ${os}, ARCH: ${arch}"
-									unstash "swt.binaries.sources.${ws}"
+									unstash "swt.binaries.sources.${ws == 'gtk4' ? 'gtk' : ws }"
 									dir('jdk.resources') {
 										unstash "jdk.resources.${os}.${arch}"
 									}
 									withEnv(['MODEL=' + arch, "OUTPUT_DIR=${WORKSPACE}/libs", "SWT_JAVA_HOME=${WORKSPACE}/jdk.resources"]) {
 										if (isUnix()){
 											sh '''


As the script below that runs on the native machine now uses bash specific commands ([[) and bash seems not to be the default shell on all native machines, we have to add a shebang.

Suggested change

sh '''

sh '''#!/bin/bash -x

👍 that makes sense, the block further down that relies on bash runs on the main build machine. I will include the shebang on it too.

HannesWell · 2025-02-04T23:28:18Z

Jenkinsfile

+									if [[ ${PLATFORM} == gtk4.linux.x86_64 ]]; then
+										cp libswt-pi4-gtk*.so "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.${DEST_PLATFORM}/"
+									else
+										cp *.$binariesExtension "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.${DEST_PLATFORM}/"
+									fi


Couldn't we implement this without the extra DEST_PLATFORM variable and therefore slightly simpler?

Suggested change

if [[ ${PLATFORM} == gtk4.linux.x86_64 ]]; then

cp libswt-pi4-gtk*.so "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.${DEST_PLATFORM}/"

else

cp *.$binariesExtension "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.${DEST_PLATFORM}/"

fi

if [[ ${PLATFORM} == gtk4.linux.x86_64 ]]; then

cp libswt-pi4-gtk*.so "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64/"

else

cp *.$binariesExtension "${WORKSPACE}/eclipse.platform.swt/binaries/org.eclipse.swt.${PLATFORM}/"

fi

HannesWell · 2025-02-04T23:37:23Z

Jenkinsfile

+									elif [[ ${PLATFORM} == gtk4.linux.x86_64 ]]; then
+										binariesExtension='so'
+										DEST_PLATFORM=gtk.linux.x86_64


We could probably save this block when using a slightly adjusted pattern in the next block(given the change below is applied):
elif [[ ${PLATFORM} == gtk*.linux.* ]]; then

But it's probably simpler to understand if we are explicit there. On the other hand we could still be explicit and save this block if we just combine both block and 'or' the arguments.

See the updated commit when I submit, but I reworked this section to make it cleaner and achieved something I wanted to do originally.

This reverts commit 4109c4a.

jonahgraham · 2025-02-05T01:39:15Z

With that applied the build succeeds:

Is the "that" in this comment that you changed the Jenkinsfile via "Replay" feauture? Because AFAICT run 6 failed and run 7 succeeded, but they were both built from same SHA1 but there is a diff in the replay https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/replay/diff

Not a feature I had been familiar with until now. Thanks for the demo.

laeubi · 2025-02-05T05:16:07Z

Alternatively you can just start a build via Build with parameters and check the forceNativeBuild box

Yes but this only works for committers and especially when testing different approaches it is a bit inconvenient, but of course many things work (including replay).

HannesWell reviewed Feb 3, 2025

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

jonahgraham force-pushed the fix-1791 branch from b95417f to 067369f Compare February 3, 2025 22:41

jonahgraham added a commit to jonahgraham/eclipse.platform.swt that referenced this pull request Feb 4, 2025

Temporary edit to force natives to be rebuilt

10038ab

as recommended in eclipse-platform#1795 (comment)

jonahgraham added 2 commits February 4, 2025 17:50

Temporary edit to force natives to be rebuilt

4109c4a

as recommended in eclipse-platform#1795 (comment)

HannesWell force-pushed the fix-1791 branch from 10038ab to 4109c4a Compare February 4, 2025 16:50

HannesWell requested changes Feb 4, 2025

View reviewed changes

jonahgraham added 4 commits February 4, 2025 20:23

Simplify runOnNativeBuildAgent and remove duplicated code

cfa6a6e

Explicitly request bash for scripts using bash

9d1226f

Remove the clunky DEST_PLATFORM and simplify

3b4c79c

Revert "Temporary edit to force natives to be rebuilt"

d4618d4

This reverts commit 4109c4a.

Add missing break on switch

fd1ea9c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build GTK3 support for linux x86-64 to maximize compatibility #1795

Build GTK3 support for linux x86-64 to maximize compatibility #1795

jonahgraham commented Feb 3, 2025

jonahgraham commented Feb 3, 2025

github-actions bot commented Feb 3, 2025 •

edited

Loading

HannesWell left a comment

jonahgraham commented Feb 3, 2025

laeubi commented Feb 4, 2025

HannesWell commented Feb 4, 2025

HannesWell left a comment

HannesWell Feb 4, 2025

jonahgraham Feb 5, 2025

HannesWell Feb 4, 2025

jonahgraham Feb 5, 2025

HannesWell Feb 4, 2025

HannesWell Feb 4, 2025

jonahgraham Feb 5, 2025

jonahgraham commented Feb 5, 2025

laeubi commented Feb 5, 2025

Build GTK3 support for linux x86-64 to maximize compatibility #1795

Are you sure you want to change the base?

Build GTK3 support for linux x86-64 to maximize compatibility #1795

Conversation

jonahgraham commented Feb 3, 2025

jonahgraham commented Feb 3, 2025

github-actions bot commented Feb 3, 2025 • edited Loading

Test Results

HannesWell left a comment

Choose a reason for hiding this comment

jonahgraham commented Feb 3, 2025

laeubi commented Feb 4, 2025

HannesWell commented Feb 4, 2025

HannesWell left a comment

Choose a reason for hiding this comment

HannesWell Feb 4, 2025

Choose a reason for hiding this comment

jonahgraham Feb 5, 2025

Choose a reason for hiding this comment

HannesWell Feb 4, 2025

Choose a reason for hiding this comment

jonahgraham Feb 5, 2025

Choose a reason for hiding this comment

HannesWell Feb 4, 2025

Choose a reason for hiding this comment

HannesWell Feb 4, 2025

Choose a reason for hiding this comment

jonahgraham Feb 5, 2025

Choose a reason for hiding this comment

jonahgraham commented Feb 5, 2025

laeubi commented Feb 5, 2025

github-actions bot commented Feb 3, 2025 •

edited

Loading