Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: upload contracts to CF R2 from main #100

Merged
merged 16 commits into from
Feb 4, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/workflows/pre-release.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Workflow to fetch the latest commit hash on the main branch and upload artifacts to CF storage.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`# Build Stellar contracts on every commit to main and upload the wasm code to Cloudflare R2

name: Build and upload from main
on:
workflow_dispatch:

concurrency: ${{ github.workflow }}-${{ github.ref }}

jobs:
stellar-contract-names:
name: Get all Stellar contracts
runs-on: blacksmith-2vcpu-ubuntu-2204
outputs:
releases: ${{ steps.prepare-release.outputs.releases }}
commit_hash: ${{ steps.get-commit-hash.outputs.hash }}

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install dependencies
run: sudo apt-get install -y jq

- name: Get latest commit hash
id: get-commit-hash
run: echo "hash=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"

- name: Prepare JSON output for release
id: prepare-release
run: |
RELEASES_JSON=$(find contracts -maxdepth 1 -mindepth 1 -type d | sed 's|contracts/||' | jq -R . | jq -s --arg commit "${{ steps.get-commit-hash.outputs.hash }}" 'map({
package_name: .,
version: $commit,
package_git_tag: "\(.)_\($commit)"
})')
Comment on lines +30 to +34
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is overkill, the commit is the same now, we just need to list all contract names

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also consider the existing release workflow, and this is the same format where MarcoIeni/release-plz-action@fff938ea2923fdfa35797fff63537b00f30fe6f9 uses.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we aren't releasing to crates.io here. Take a look at how the version and package_git_tag are being used below. The commit hash can be used there directly instead of creating this confusing JSON

echo "releases=$(echo "$RELEASES_JSON" | jq -c)" >> "$GITHUB_OUTPUT"

build:
needs: stellar-contract-names
uses: ./.github/workflows/reusable-build.yaml
with:
commit-hash: ${{ needs.stellar-contract-names.outputs.commit_hash }}

upload:
needs: [stellar-contract-names, build]
strategy:
matrix:
releases: ${{ fromJson(needs.stellar-contract-names.outputs.releases) }}

uses: ./.github/workflows/reusable-upload.yaml
permissions:
id-token: write
contents: read
with:
package-name: ${{ matrix.releases.package_name }}
package-version: ${{ matrix.releases.version }}
package-git-tag: ${{ matrix.releases.package_git_tag }}
artifact-name: ${{ needs.build.outputs.artifact-name }}
artifact-path: ${{ needs.build.outputs.artifact-path }}
cf-bucket-name: ${{ vars.CF_BUCKET_NAME }}
cf-config-bucket-root-key: ${{ vars.CF_BUCKET_ROOT_KEY }}
github-release: false
secrets:
github-token: ${{ secrets.PAT_TOKEN }}
cf-endpoint-url: ${{ secrets.CF_ENDPOINT_URL }}
cf-bucket-access-key-id: ${{ secrets.CF_BUCKET_ACCESS_KEY_ID }}
cf-bucket-secret-access-key: ${{ secrets.CF_BUCKET_SECRET_ACCESS_KEY }}
47 changes: 24 additions & 23 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,14 @@ on:
pull_request:
branches:
- main
- 'releases/**'
- "releases/**"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just releases/** is fine, consistent with other actions

types: [closed]

workflow_dispatch:

concurrency: ${{ github.workflow }}-${{ github.ref }}

jobs:

# Publishes a release in case the release isn't published
publish-release:
name: Publish releases
Expand All @@ -24,21 +23,25 @@ jobs:
((github.event.pull_request.merged == true) &&
contains(github.event.pull_request.labels.*.name, 'release'))

runs-on: blacksmith-8vcpu-ubuntu-2204
runs-on: blacksmith-2vcpu-ubuntu-2204

outputs:
releases: ${{ steps.prepare-matrix.outputs.releases }}
commit_hash: ${{ steps.get-commit-hash.outputs.hash }}

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.PAT_TOKEN }}

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable

- name: Get commit hash
id: get-commit-hash
run: echo "hash=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"

# Creates git tags and publishes the crates of the new releases
- name: Publish release
id: publish-release
Expand All @@ -56,35 +59,33 @@ jobs:
run: |
echo "releases=$(echo '${{ steps.publish-release.outputs.releases }}' | jq -c '.')" >> $GITHUB_OUTPUT

# Creates other artifacts needed (`wasm` files)
build-and-upload:
name: Build artifacts for ${{ matrix.releases.package_name }}-v${{ matrix.releases.version }}
build:
needs: publish-release
uses: ./.github/workflows/reusable-build.yaml
with:
commit-hash: ${{ needs.publish-release.outputs.commit_hash }}
Comment on lines +64 to +66
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correct? This is creating a build on the current commit hash instead of using the release tags created for every contract by release-plz. So, the built wasm files won't correspond to the release tag? The release matrix needs to be applied for both build and upload, unless all release tags are the same commit


# Once a release is done for a package, we iterate on each of these packages and build its corresponding artifacts and upload them
upload:
needs: [publish-release, build]
strategy:
matrix:
releases: ${{ fromJson(needs.publish-release.outputs.releases) }}

uses: ./.github/workflows/reusable-build-upload.yaml

uses: ./.github/workflows/reusable-upload.yaml
permissions:
id-token: write
contents: read

with:
package-name: "${{ matrix.releases.package_name }}"
package-version: "${{ matrix.releases.version }}"
package-git-tag: "${{ matrix.releases.tag }}"

# CF Bucket related variables
cf-bucket-name: "${{ vars.CF_BUCKET_NAME }}"

# The root key to be used for accessing the configs. (ex: `test-root-key` puts releases in `test-root-key/*`)
cf-config-bucket-root-key: "${{ vars.CF_BUCKET_ROOT_KEY }}"

package-name: ${{ matrix.releases.package_name }}
package-version: ${{ matrix.releases.version }}
package-git-tag: ${{ matrix.releases.tag }}
artifact-name: ${{ needs.build.outputs.artifact-name }}
artifact-path: ${{ needs.build.outputs.artifact-path }}
cf-bucket-name: ${{ vars.CF_BUCKET_NAME }}
cf-config-bucket-root-key: ${{ vars.CF_BUCKET_ROOT_KEY }}
github-release: true
secrets:
github-token: "${{ secrets.PAT_TOKEN }}"
cf-endpoint-url: "${{ secrets.CF_ENDPOINT_URL }}"
github-token: ${{ secrets.PAT_TOKEN }}
cf-endpoint-url: ${{ secrets.CF_ENDPOINT_URL }}
cf-bucket-access-key-id: ${{ secrets.CF_BUCKET_ACCESS_KEY_ID }}
cf-bucket-secret-access-key: ${{ secrets.CF_BUCKET_SECRET_ACCESS_KEY }}
67 changes: 67 additions & 0 deletions .github/workflows/reusable-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: "Build Contracts"

on:
workflow_call:
inputs:
commit-hash:
description: "The commit hash to build from"
type: string
required: true
outputs:
artifact-name:
description: "Name of the uploaded artifact containing all builds"
value: ${{ jobs.build.outputs.artifact-name }}
artifact-path:
description: "Path of the uploaded artifact containing all builds"
value: ${{ jobs.build.outputs.artifact-path }}

jobs:
build:
runs-on: blacksmith-8vcpu-ubuntu-2204
outputs:
artifact-name: ${{ steps.set-artifact-name.outputs.name }}
artifact-path: ${{ steps.set-artifact-name.outputs.path }}

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Checkout specific commit
run: git checkout ${{ inputs.commit-hash }}

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: 1.81.0
ahramy marked this conversation as resolved.
Show resolved Hide resolved
targets: wasm32-unknown-unknown

- name: Set artifact name
id: set-artifact-name
run: |
echo "name=wasm-builds-${{ inputs.commit-hash }}" >> $GITHUB_OUTPUT
echo "path=target/wasm32-unknown-unknown/release" >> $GITHUB_OUTPUT

- name: Build all contracts
run: |
# Install Stellar CLI compatible with the soroban-sdk version in Cargo.toml
cargo install --locked stellar-cli --version ^22 --features opt

# Build all contracts
stellar contract build
./optimize.sh

# Process in the release directory
cd target/wasm32-unknown-unknown/release

# Remove unoptimized files and rename optimized ones
# This ensures we only keep the optimized versions
find . -type f -name "*.wasm" ! -name "*.optimized.wasm" -maxdepth 1 -delete
find . -name "*.optimized.wasm" -maxdepth 1 -exec sh -c 'mv "$0" "${0%.optimized.wasm}.wasm"' {} \;
find . -type f ! -name "*.wasm" -delete

- name: Upload artifact
uses: actions/upload-artifact@v4
ahramy marked this conversation as resolved.
Show resolved Hide resolved
with:
name: ${{ steps.set-artifact-name.outputs.name }}
path: target/wasm32-unknown-unknown/release
retention-days: 1
Original file line number Diff line number Diff line change
@@ -1,37 +1,43 @@
name: "Publish specific rust package"
name: "Upload Contract wasm to Cloudflare R2"

on:
workflow_call:
inputs:
# Package related variables
package-name:
description: "The package name to use (ex: gz-srv)"
type: string
required: true
default: ""

package-version:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove newlines? it's more readable that way

description: "The package version to use (ex: 0.1.2)"
type: string
required: true
default: ""

package-git-tag:
description: "the release tag name (ex: gz-srv-v0.1.2)"
type: string
required: true
default: ""

# CF Bucket related variables
artifact-name:
description: "Name of the artifact containing the builds"
type: string
required: true
artifact-path:
description: "Path of the artifact containing the builds"
type: string
required: true
cf-bucket-name:
description: "The CF bucket name to use"
required: true
type: string

cf-config-bucket-root-key:
description: "The root key to be used for accessing the configs. (ex: `test-root-key` puts releases in `test-root-key/*`)"
required: true
type: string
github-release:
description: "Whether to upload as a github release"
type: boolean
default: true

secrets:
github-token:
Expand All @@ -48,36 +54,22 @@ on:
required: true

jobs:
build-and-upload:
runs-on: blacksmith-8vcpu-ubuntu-2204
upload:
name: Upload ${{ inputs.package-git-tag }}
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.github-token }}

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- name: Download artifact
uses: actions/download-artifact@v4
with:
toolchain: 1.81.0
targets: wasm32-unknown-unknown
name: ${{ inputs.artifact-name }}
path: ${{ inputs.artifact-path }}

- name: Build artifacts for ${{ inputs.package-name }}-v${{ inputs.package-version }}
run: |
echo "Building wasm for '${{ inputs.package-name }}-v${{ inputs.package-version }}'";
cargo install --locked stellar-cli --version 22.2.0 --features opt
cargo wasm -p ${{ inputs.package-name }}
stellar contract build
./optimize.sh

# Prepare the variables that will be used across the different next steps
- name: Prepare cross-steps variables
run: |
export PACKAGE_NAME='${{ inputs.package-name }}'
export PACKAGE_VERSION='v${{ inputs.package-version }}'
export PACKAGE_VERSION=${{ inputs.github-release && format('v{0}', inputs.package-version) || inputs.package-version }}

export BASE_ARTIFACTS_DIR="./target/wasm32-unknown-unknown/release"
export BASE_ARTIFACTS_DIR='${{ inputs.artifact-path }}'
export ARTIFACT_NAME="axelar-cgp-stellar-wasm-${PACKAGE_NAME}-${PACKAGE_VERSION}"
export BASE_ARTIFACTS_VERSIONED_DIR="$(dirname ${BASE_ARTIFACTS_DIR})/${ARTIFACT_NAME}" # Regardless of the dir type, relative or absolute

Expand Down Expand Up @@ -115,12 +107,6 @@ jobs:
# This cd to keep the dir structure of the artifacts archive
cd ${{ env.BASE_ARTIFACTS_VERSIONED_DIR }}

# Remove "unoptimized" built wasm files
find "." -type f -name "*.wasm" ! -name "*.optimized.wasm" -maxdepth 1 -delete

# Rename the optimized ones and remove the ".optimized" suffix
find . -name "*.optimized.wasm" -maxdepth 1 -exec sh -c 'mv "$0" "${0%.optimized.wasm}.wasm"' {} \;

# Archive the wasm
find "." -type f -name "*.wasm" -maxdepth 1 -print | zip "${{ env.ZIP_ARCHIVE_FILE }}" -@
find "." -type f -name "*.wasm" -maxdepth 1 -print | tar -czvf "${{ env.TAR_ARCHIVE_FILE }}" -T -
Expand Down Expand Up @@ -176,6 +162,7 @@ jobs:
# https://github.com/orgs/community/discussions/26263#discussioncomment-3251069
- name: Update the GitHub Release
uses: softprops/action-gh-release@c062e08bd532815e2082a85e87e3ef29c3e6d191 # v2.0.8
if: inputs.github-release
with:
tag_name: ${{ inputs.package-git-tag }} # This uses the tag from the push
files: |
Expand Down
Loading