GitHub Actions CI Port #16082

illusion0001 · 2024-09-15T12:42:14Z

Cache is supported on the penguin
Cold: https://github.com/illusion0001/rpcs3/actions/runs/10871089443/attempts/1
Warm: https://github.com/illusion0001/rpcs3/actions/runs/10871089443

Plans for later, build in clang and gcc on Windows and OSX

.github/workflows/rpcs3.yml

hcorion · 2024-09-23T19:01:31Z

What's the motivation for porting to GitHub Actions? Doesn't it have the exact same restrictions as Azure?

illusion0001 · 2024-09-28T08:53:08Z

What's the motivation for porting to GitHub Actions? Doesn't it have the exact same restrictions as Azure?

so users don't have to sign up to extra services to run auto builds, and yeah i think

q4a · 2025-01-19T11:29:56Z

What's the motivation for porting to GitHub Actions? Doesn't it have the exact same restrictions as Azure?

I think that GitHub Actions is useful now because of free arm64 runners. I improved illusion0001's version a bit:
https://github.com/q4a/rpcs3/commits/arm-ci/
But don't understand how to pass $COMPILER env to docker?
Check fail on github actions:
https://github.com/q4a/rpcs3/actions/runs/12852962562/job/35835578225
vs sucess on Cirrus CI:
https://cirrus-ci.com/task/6366890357948416

illusion0001 · 2025-01-19T11:34:08Z

$COMPILER is used to distinguish between GCC and clang builds

q4a · 2025-01-19T11:42:09Z

But don't understand how to pass $COMPILER env to docker?

I know that)
Cirrus CI has this lines in log:
https://cirrus-ci.com/task/6366890357948416

+ export CC=clang-18
+ export CXX=clang++-18

Actions x86_64 has this lines in log:
https://github.com/q4a/rpcs3/actions/runs/12852962562/job/35835578464

+ export CC=clang-18
+ export CXX=clang++-18

But Actions arm64 has only this lines in log:
https://github.com/q4a/rpcs3/actions/runs/12852962562/job/35835578225

+ export CC=
+ export CXX=

q4a · 2025-01-19T13:49:20Z

Ok. I fixed arm64 build with simple chmod +x ‎.ci/build-linux-aarch64.sh.
@illusion0001 can you cherry-pick or just copy my changes to this PR?
https://github.com/q4a/rpcs3/commits/arm-ci/
Here is build result:
https://github.com/q4a/rpcs3/actions/runs/12853858021

Darkhost1999 · 2025-01-19T15:06:50Z

Maybe instead of matrix: Linux Build it can be matrix: Linux Aarch 64 and matrix: Linux X64 by creating new name categories.
I was looking into splitting the files and then just using uses ./.workflows/linux_aarch64.yml
That way there's a main file for rpcs3.yml and then maintainable smaller jobs. Especially after adding MacOS then Windows Arm.

illusion0001 · 2025-01-19T16:01:49Z

Ok. I fixed arm64 build with simple chmod +x ‎.ci/build-linux-aarch64.sh. @illusion0001 can you cherry-pick or just copy my changes to this PR? https://github.com/q4a/rpcs3/commits/arm-ci/ Here is build result: https://github.com/q4a/rpcs3/actions/runs/12853858021

Pushed

illusion0001 · 2025-01-20T00:56:29Z

Arm gcc build is clang, needs fixing
https://github.com/illusion0001/rpcs3/actions/runs/12855079318/job/35840192563

q4a · 2025-01-20T20:48:25Z

Arm gcc build is clang, needs fixing https://github.com/illusion0001/rpcs3/actions/runs/12855079318/job/35840192563

I tested few changes and this one works pretty good:
q4a@3e2d9f7
CI result:
https://github.com/q4a/rpcs3/actions/runs/12875043123

illusion0001 · 2025-01-21T02:35:25Z

Arm gcc build is clang, needs fixing https://github.com/illusion0001/rpcs3/actions/runs/12855079318/job/35840192563

I tested few changes and this one works pretty good: q4a@3e2d9f7 CI result: https://github.com/q4a/rpcs3/actions/runs/12875043123

Updated. Also fixed windows artifact uploading build files

Jack54guythecoder · 2025-01-21T20:18:41Z

What's the motivation for porting to GitHub Actions? Doesn't it have the exact same restrictions as Azure?

You can use nightly.link with it.

hcorion · 2025-01-26T01:58:02Z

.github/workflows/rpcs3.yml

+      matrix:
+        compiler: [clang, gcc]
+        cfg:
+          - os: ubuntu-24.04-arm


To save on CI time, we only want to run the clang compiler on arm, since we're already testing both compilers on the x86 build. For arm64 just use the compiler we want to publish binaries with (clang).

Should be fixed

hcorion · 2025-01-26T02:03:39Z

.github/workflows/rpcs3.yml

+      - name: Append Compiler version to appimage
+        run: |
+          file=".ci/deploy-linux.sh"
+          sed -i 's/RPCS3_APPIMAGE="rpcs3-v${COMM_TAG}-${COMM_COUNT}-${COMM_HASH}_linux64.AppImage"/RPCS3_APPIMAGE="rpcs3-v${COMM_TAG}-${COMM_COUNT}-${COMM_HASH}-${COMPILER}_linux64.AppImage"/g' $file


Why are you changing the output filename here? If we want to change the output filename just change the deploy script. I'm not sure why you're changing it in the first place though.

It's to separate the artifacts from being overwritten, i.e clang finishes before gcc, when GCC is uploaded, clang appimage is overwritten. I didn't want to modify script to keep compatibility for other ci

Well, now that I had a second look at this, it's not needed after all because artifact have compiler name in it

hcorion · 2025-01-26T02:15:10Z

.github/workflows/rpcs3.yml

+            ${{ matrix.cfg.build_sh }}
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@main


I'm not familiar with actions/upload-artifact@main, is this going to upload binaries to RPCS3/rpcs3-binaries-win/linux/linux-arm64/etc ? I don't see where we set any variables related to the upload repository, so I don't think this will work how we want it to.

Please also only run this step with the same restrictions we use in Azure:
condition: and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.Repository.Name'], 'RPCS3/rpcs3'), eq(variables['Build.SourceBranch'], 'refs/heads/master'), eq(variables['COMPILER'], 'clang'))

Added

if: | github.event_name != 'pull_request' && github.repository == 'RPCS3/rpcs3' && github.ref == 'refs/heads/master' && matrix.compiler == 'clang'

I'm not familiar with actions/upload-artifact@main, is this going to upload binaries to RPCS3/rpcs3-binaries-win/linux/linux-arm64/etc ? I don't see where we set any variables related to the upload repository, so I don't think this will work how we want it to.

Nope, there will have to be a separate step for it. Using appropriate tokens like azure to upload it

hcorion · 2025-01-26T02:17:46Z

.github/workflows/rpcs3.yml

+      DEPLOY_APPIMAGE: true
+      APPDIR: "/rpcs3/build/appdir"
+      ARTDIR: "/root/artifacts"
+      RELEASE_MESSAGE: "/rpcs3/GitHubReleaseMessage-${{ matrix.compiler }}.txt"


We shouldn't need to have release messages per-compiler, since we only upload the binaries from one compiler to the GitHub Release.

hcorion · 2025-01-26T02:40:08Z

.github/workflows/rpcs3.yml

+        uses: actions/cache@main
+        with:
+          path: ${{ env.CACHE_DIR }}
+          key: ${{ runner.os }}-${{ env.COMPILER }}-${{ env.QT_VER }}-${{ env.VULKAN_SDK_SHA }}-llvm.lock-glslang.lock


I'm wondering if the llvm.lock and glslang.lock keys are working? In Azure providing the filename means it will hash the file for the key. However, in https://github.com/actions/cache#creating-a-cache-key I see that they use this function: hashFiles('**/lockfiles') to hash files.

hcorion · 2025-01-26T02:54:24Z

.github/workflows/rpcs3.yml

+        shell: bash
+        run: .ci/setup-windows.sh
+
+      - name: Export Variables


This should should be in a script file. Also, is it possible to do this step in bash?

It can be done in bash, as for making it a file, I'm afraid that github.env won't be resolved there, hence why it's a step.

hcorion · 2025-01-26T02:59:15Z

.github/workflows/rpcs3.yml

+      - name: Pack up build artifacts
+        shell: bash
+        run: |
+          mkdir -p build


This should mkdir $BUILD_ARTIFACTSTAGINGDIRECTORY

Should be sorted

hcorion · 2025-01-26T03:02:56Z

.github/workflows/rpcs3.yml

+          path: bin2
+          if-no-files-found: error
+
+      - name: Upload artifacts (7z)


Same problems with the linux version of this step

Should be sorted

hcorion · 2025-01-26T03:05:04Z

.github/workflows/rpcs3.yml

+        run: |
+          mkdir -p build
+          .ci/deploy-windows.sh
+          mkdir -p bin2/${{ env.BUILD_RAW }}


Why are we using this bin2 folder instead of just using the existing bin folder for the zip artifact upload.

Bin2 is really only for zipped version artifact to not be double compressed. I'll remove it in favor of 7z

Zip removed

hcorion · 2025-01-26T03:20:06Z

.github/workflows/rpcs3.yml

+          mkdir -p bin2/${{ env.BUILD_RAW }}
+          mv bin/* bin2/${{ env.BUILD_RAW }}
+
+      - name: Upload artifacts (zip)


If I understand correctly, we upload this seperate .zip because the .7z step gets double compressed? Perhaps we only deploy the 7z artifact in the case of a master build for upload.

I'll remove zip

Zip removed

Margen67 reviewed Sep 17, 2024

View reviewed changes

.github/workflows/rpcs3.yml Show resolved Hide resolved

Megamouse added the Build and CI Anything related to the build process and continuous integration label Sep 19, 2024

illusion0001 marked this pull request as draft September 23, 2024 10:17

illusion0001 force-pushed the patch-1 branch 3 times, most recently from 1cdde6d to 1b3a7cd Compare September 28, 2024 08:52

illusion0001 force-pushed the patch-1 branch from 1b3a7cd to a0cb74e Compare January 19, 2025 16:01

illusion0001 marked this pull request as ready for review January 19, 2025 16:03

AniLeo requested a review from hcorion January 19, 2025 17:03

illusion0001 force-pushed the patch-1 branch from a0cb74e to 1c6ff0a Compare January 21, 2025 02:32

illusion0001 force-pushed the patch-1 branch 2 times, most recently from f198291 to 11a5643 Compare January 21, 2025 05:20

illusion0001 and others added 2 commits January 22, 2025 07:28

GitHub Actions CI Port

720113e

Add arm64 build, update Qt version and docker image for github actions

91ef7e0

illusion0001 force-pushed the patch-1 branch from 11a5643 to 91ef7e0 Compare January 22, 2025 07:28

hcorion suggested changes Jan 26, 2025

View reviewed changes

illusion0001 added 2 commits January 26, 2025 01:22

Update rpcs3.yml

9146b3f

Update rpcs3.yml

ff3ab91

illusion0001 added 9 commits January 26, 2025 01:29

Update rpcs3.yml

a914120

Update rpcs3.yml

3000bd5

Update rpcs3.yml

34f0b8e

Update rpcs3.yml

cb2464e

Update rpcs3.yml

e434a5e

Update rpcs3.yml

9cd3b58

Update rpcs3.yml

992b010

Update rpcs3.yml

dddd71f

Update rpcs3.yml

8eb3db7

GitHub Actions CI Port #16082

Are you sure you want to change the base?

GitHub Actions CI Port #16082

Conversation

illusion0001 commented Sep 15, 2024 • edited Loading

hcorion commented Sep 23, 2024

illusion0001 commented Sep 28, 2024 • edited Loading

q4a commented Jan 19, 2025 • edited Loading

illusion0001 commented Jan 19, 2025

q4a commented Jan 19, 2025

q4a commented Jan 19, 2025 • edited Loading

Darkhost1999 commented Jan 19, 2025 • edited Loading

illusion0001 commented Jan 19, 2025

illusion0001 commented Jan 20, 2025 • edited Loading

q4a commented Jan 20, 2025

illusion0001 commented Jan 21, 2025

Jack54guythecoder commented Jan 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illusion0001 commented Sep 15, 2024 •

edited

Loading

illusion0001 commented Sep 28, 2024 •

edited

Loading

q4a commented Jan 19, 2025 •

edited

Loading

q4a commented Jan 19, 2025 •

edited

Loading

Darkhost1999 commented Jan 19, 2025 •

edited

Loading

illusion0001 commented Jan 20, 2025 •

edited

Loading