Skip to content

Commit

Permalink
Upgrade to v8 13.2 and use native aarch64 runners
Browse files Browse the repository at this point in the history
  • Loading branch information
bpcreech committed Feb 3, 2025
1 parent cbe0447 commit e6a5544
Show file tree
Hide file tree
Showing 7 changed files with 54 additions and 41 deletions.
66 changes: 34 additions & 32 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,44 +32,40 @@ jobs:
- name: Build
run: python -m build --sdist

- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: sdist
path: dist/*
if-no-files-found: error

# We build for Linux using uraimo/run-on-arch-action@v2, which runs a container under
# the runner in order to reach different platforms (notably Alpine with its musl) and
# architectures (notably aarch64) via emulation. uraimo/run-on-arch-action@v2 doesn't
# support Mac or Windows, so we run a separate job for those.
# architectures. uraimo/run-on-arch-action@v2 doesn't support Mac or Windows, so we
# run a separate job for those.
linux-wheels:
name: Build wheel for ${{ matrix.image }}
runs-on: ubuntu-latest
name: Build wheel for ${{ matrix.config.image }} on ${{ matrix.config.os }}
runs-on: ${{ matrix.config.os }}
strategy:
fail-fast: false
matrix:
image:
config:
# To maximize compatibility of generated wheels, we should theoreticaly build
# on the *oldest* supported distros.
# But V8 ships its own bullseye sysroot and links against that, so we may as
# well run on bullseye (even though buster would provide an older supported
# build distro):
- debian:bullseye
- arm64v8/debian:bullseye
# Alpine 3.19 includes a clang new enough for V8 to build (with only minor
# patches!). Builds on 3.19 seem incompatible with <= 3.18 due to libstdc++
# symbols. (And we can't just run on an old Alpine and update clang from the
# llvm site, because unlike Debian, the llvm project doesn't maintain
# updated packages for old Alpine distros.)
- alpine:3.19
- arm64v8/alpine:3.19
exclude:
# The aarch64 builds run really slowly, so let's skip them except for
# releases.
# For more info on this GitHub Actions hack, see:
# https://stackoverflow.com/questions/65384420/how-do-i-make-a-github-action-matrix-element-conditional#answer-73822998
- image: ${{ !startsWith(github.ref, 'refs/heads/release/') && 'arm64v8/debian:bullseye' }}
- image: ${{ !startsWith(github.ref, 'refs/heads/release/') && 'arm64v8/alpine:3.19' }}
- os: ubuntu-latest # x86_64 base OS (on which we run a container)
distro: bullseye
arch: none
- os: ubuntu-latest
distro: alpine-latest
arch: none
- os: ubuntu-24.04-arm
distro: bullseye
arch: aarch64
- os: ubuntu-24.04-arm
distro: alpine-latest
arch: aarch64

steps:
- name: Configure git
Expand All @@ -88,9 +84,9 @@ jobs:
- uses: uraimo/run-on-arch-action@v2
name: Build wheel
with:
arch: none
arch: ${{ matrix.config.arch }}
distro: none
base_image: ${{ matrix.image }}
base_image: ${{ matrix.config.image }}

setup: |
mkdir -p "${PWD}/wheels"
Expand All @@ -101,7 +97,7 @@ jobs:
shell: /bin/sh

install: |
case "${{ matrix.image }}" in
case "${{ matrix.config.image }}" in
*debian*)
# Let's download some system packages!
# Note that the precise list of packages we need is intertwined not just
Expand Down Expand Up @@ -232,7 +228,7 @@ jobs:
cp dist/*.whl /wheels/
chmod a+rwx /wheels/*.whl
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: wheels
path: ./wheels/*
Expand All @@ -243,7 +239,7 @@ jobs:
with:
arch: none
distro: none
base_image: ${{ matrix.image }}
base_image: ${{ matrix.config.image }}

setup: |
mkdir -p "${PWD}/wheels"
Expand All @@ -254,7 +250,7 @@ jobs:
shell: /bin/sh

install: |
case "${{ matrix.image }}" in
case "${{ matrix.config.image }}" in
*debian*)
PACKAGES=""
PACKAGES="${PACKAGES} python3"
Expand All @@ -280,7 +276,7 @@ jobs:
python3 -m venv /venv
. /venv/bin/activate
case "${{ matrix.image }}" in
case "${{ matrix.config.image }}" in
*debian*)
python3 -m pip install --upgrade hatch hatch-fancy-pypi-readme hatch-mkdocs
hatch run testinstalled:install /wheels/*.whl
Expand Down Expand Up @@ -330,13 +326,19 @@ jobs:
# depot_tools still uses distutils which is gone in 3.12:
python-version: '3.11'

- name: Install Windows SDK
uses: GuillaumeFalourd/[email protected]
with:
sdk-version: 26100
if: startsWith(matrix.config.os, 'windows')

- name: Build wheel
run: |
set -e
python3 -m pip install --upgrade build
python3 -m build --wheel
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: wheels
path: dist/*
Expand All @@ -359,7 +361,7 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/download-artifact@v3
- uses: actions/download-artifact@v4

- name: Compute release version
run: |
Expand All @@ -386,7 +388,7 @@ jobs:
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
steps:
- uses: actions/download-artifact@v3
- uses: actions/download-artifact@v4

- name: Make dist directory
run: mkdir dist && cp wheels/* sdist/* dist/
Expand Down
9 changes: 7 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,8 +154,8 @@ PyMiniRacer!) which wants to integrate V8 must first build it.
### Build PyPI wheels

Because V8 takes so long to build (about 2-3 hours at present on the free GitHub Actions
runners, and >12 hours when emulating `aarch64` on them), we want to build wheels for
PyPI. We don't want folks to have to build V8 when they `pip install mini-racer`!
runners), we want to build wheels for PyPI. We don't want folks to have to build V8 when
they `pip install mini-racer`!

We build wheels for many operating systems and architectures based on popular demand via
GitHib issues. Currently the list is
Expand Down Expand Up @@ -238,6 +238,11 @@ wouldn't simplify the overall workflow management.

### Use `sccache` to patch around build timeouts

**Update: We still use `sscache` but with
[the new Github-hosted `aarch64` runners](https://github.com/actions/runner-images/issues/10820),
this should no longer be strictly required. We no longer run builds on emulation as
discussed below.**

As of this writing, the Linux `aarch64` builds run on emulation becaues GitHub Actions
has no free hosted `aarch64` runners for Linux. This makes them so slow, they struggle
to complete at all. They take about 24 hours to run. The GitHub Actions
Expand Down
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ If you are proposing a feature:
Ready to contribute? Here's how to set up `PyMiniRacer` for local development.

!!! warning

Building this package from source takes several GB of disk space and takes 1-2 hours.

1. Do a quick scan through [the architecture guide](ARCHITECTURE.md) before diving in.
Expand Down Expand Up @@ -220,6 +221,7 @@ To make an ordinary release from `main`:
wheels to PyPI automatically.
!!! warning
As of this writing, the `aarch64` Linux builds are slow because they're running on
emulation. They time out on the first try (and second and third and...) after 6
hours. If you "restart failed jobs", they will quickly catch up to where where they
Expand Down
2 changes: 1 addition & 1 deletion HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@

## 0.1.9 (2017-03-24)

- Fix the compilation for Ubuntu 12.04 and glibc \< 2.17.
- Fix the compilation for Ubuntu 12.04 and glibc < 2.17.

## 0.1.8 (2017-03-02)

Expand Down
2 changes: 1 addition & 1 deletion helpers/v8_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
LOGGER = getLogger(__name__)
LOGGER.setLevel(DEBUG)
ROOT_DIR = dirname(abspath(__file__))
V8_VERSION = "branch-heads/12.6"
V8_VERSION = "branch-heads/13.2"


def local_path(path="."):
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ dependencies = [
]

[[tool.hatch.envs.test.matrix]]
python = ["3.8", "3.9", "3.10", "3.11", "3.12"]
python = ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]

# An environment to test installed wheels
[tool.hatch.envs.testinstalled]
Expand All @@ -120,7 +120,7 @@ dependencies = [
]

[[tool.hatch.envs.testinstalled.matrix]]
python = ["3.8", "3.9", "3.10", "3.11", "3.12"]
python = ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]

[tool.hatch.envs.testinstalled.scripts]
# A weird trick to smuggle the just-built wheel into the env, for matrix testing:
Expand Down
10 changes: 7 additions & 3 deletions src/py_mini_racer/_value_handle.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from datetime import datetime, timezone
from typing import (
TYPE_CHECKING,
Any,
ClassVar,
)

Expand All @@ -23,9 +24,12 @@
PythonJSConvertedTypes,
)

if TYPE_CHECKING:
from collections.abc import Sequence


class _RawValueUnion(ctypes.Union):
_fields_: ClassVar[list[tuple[str, object]]] = [
_fields_: ClassVar[Sequence[tuple[str, Any]]] = [
("value_ptr", ctypes.c_void_p),
("bytes_val", ctypes.POINTER(ctypes.c_char)),
("char_p_val", ctypes.c_char_p),
Expand All @@ -35,7 +39,7 @@ class _RawValueUnion(ctypes.Union):


class _RawValue(ctypes.Structure):
_fields_: ClassVar[list[tuple[str, object]]] = [
_fields_: ClassVar[Sequence[tuple[str, Any]]] = [
("value", _RawValueUnion),
("len", ctypes.c_size_t),
("type", ctypes.c_uint8),
Expand All @@ -52,7 +56,7 @@ class _RawValue(ctypes.Structure):
class _ArrayBufferByte(ctypes.Structure):
# Cannot use c_ubyte directly because it uses <B
# as an internal type but we need B for memoryview.
_fields_: ClassVar[list[tuple[str, object]]] = [
_fields_: ClassVar[list[tuple[str, Any]]] = [
("b", ctypes.c_ubyte),
]
_pack_ = 1
Expand Down

0 comments on commit e6a5544

Please sign in to comment.