doc: How to Build Relocatable Postgres 17 for macOS

[coolaj86]

Table of Contents

• How to Install
• Goal
• How to Build
• References

How to Install

Webi (macOS, Linux, etc)

curl -sS https://webi.sh/postgres | sh

Manually

At the time of this writing, these binaries are not signed.
(though I've renewed my Apple Developer account and I'm working through the signing process now)

This means you must manually remove them from quarantine to run them:

# xattr -r -d com.apple.quarantine <path-to-bins>
xattr -r -d com.apple.quarantine ./postgres-17.0.0-x86_64-darwin

curl -L -O https://github.com/bnnanet/postgresql-releases/releases/download/REL_17_0/postgres-17.0.0-x86_64-darwin.tar.gz
tar xvf ./postgres-17.0.0-x86_64-darwin.tar.gz
xattr -r -d 'com.apple.quarantine' ./postgres-17.0.0-x86_64-darwin/
mv ./postgres-17.0.0-x86_64-darwin ~/.local/opt/postgres-17.0.0
ln -s 'postgres-17.0.0' ~/.local/opt/postgres

export PATH="$HOME/.local/opt/postgres/bin:$PATH"

Goal

Build postgres on one mac that will also run on others -- without having to install brew, etc.

Update: Haha! I've cracked the code! Full script below.

Build Overview

1. Install Xcode, and GNU and BSD build dependencies
2. Download ProstgreSQL source
3. Configure and build (do NOT use DYLD_RUN_PATH due to sip issues)
4. Bundle shared library dependencies (static builds are actively prevented)
5. Update linker rpaths and resign libraries
6. Package tarball for distribution

Note: this attempts to match the install of llvm to clang --version to avoid incompatibilities when compiling.

Install Notes

1. If the code isn't signed with a valid certificate, the installer must remove the quarantine bits, once unpacked:

   xattr -r -d com.apple.quarantine ~/Downloads/postgres-17/

2. llvm is optional, but must be installed (at the same version as the build) for JIT query optimizations to function

Complete Build Script

brew install llvm@16 icu4u

Build

Update: This script is now maintained as https://github.com/bnnanet/pg-essentials/blob/main/pg-build-macos

#!/bin/sh
set -e
set -u

g_vendor="${1:-}"
g_ver="${2:-}"
g_patch="${POSTGRES_PATCH_VERSION:-0}"

g_semver="${g_ver}.${g_patch}"

#g_exts="$(ls ./cortrib/ | grep -v 'README|*.mk|perl|python|ossp|start-scripts|xml2')"
g_exts="amcheck auth_delay auto_explain basebackup_to_shell basic_archive bloom btree_gin btree_gist citext cube dblink dict_int dict_xsyn earthdistance file_fdw fuzzystrmatch hstore isn lo ltree pageinspect passwordcheck pg_buffercache pg_freespacemap pg_prewarm pg_stat_statements pg_surgery pg_trgm pg_visibility pg_walinspect pgcrypto pgrowlocks pgstattuple postgres_fdw seg sslinfo tablefunc tcn test_decoding tsm_system_rows tsm_system_time unaccent"
g_exts_only="intarray intagg oid2name spi vacuumlo"

g_prof=""
g_libc=""
# darwin
g_platform="$(uname -s | tr '[:upper:]' '[:lower:]')" # darwin
# x86_64 or arm64
g_arch="$(uname -m)"
if test "arm64" = "${g_arch}"; then
    g_arch="aarch64"
fi
# aarch64-darwin
g_target="${g_prof}${g_arch}-${g_platform}${g_libc}"

main() { (
    if test -z "${g_vendor}"; then
        echo ""
        echo "USAGE"
        echo "    pg-build-macos <vendor-name> <pg-version>"
        echo ""
        echo "EXAMPLE"
        echo "    pg-build-macos 'custom' 17.0"
        echo ""
        echo "ENVs"
        echo "    Use ENVs to set the (cosmetic) patch version"
        echo "    POSTGRES_PATCH_VERSION=0"
        echo "    (the LLVM version will be set the same as Xcode clang)"
        echo ""
        return 1
    fi

    echo ""
    echo ""
    echo "Installing build dependencies ..."
    if ! test -e ./"postgresql-${g_ver}-${g_target}"; then
        sleep 1
        fn_install_build_deps
    fi

    echo ""
    echo ""
    echo "Downloading PostgreSQL ${g_ver} source ..."
    sleep 1
    fn_download_pg

    echo ""
    echo ""
    echo "Building 'postgres+psql' in ./postgresql-${g_ver}-${g_target}/"
    sleep 1
    fn_build_pg

    echo ""
    echo "Bundling 'postgres+psql' libs in ~/relocatable/postgres-${g_semver}-${g_target}/lib/"
    echo "     and 'psql' libs in ~/relocatable/psql-${g_ver}-${g_target}/lib/"
    sleep 1
    fn_bundle_icu
    fn_bundle_libs ~/relocatable/"postgres-${g_semver}-${g_target}"/lib/
    fn_bundle_libs ~/relocatable/"psql-${g_semver}-${g_target}"/lib/

    echo ""
    echo "Updating 'postgres+psql' linker paths and resigning ..."
    echo "     and 'psql' linker paths and resigning ..."
    sleep 1
    fn_patch_rpaths_extensions ~/relocatable/"postgres-${g_semver}-${g_target}"
    fn_patch_rpaths_libs ~/relocatable/"postgres-${g_semver}-${g_target}"
    fn_patch_rpaths_libs ~/relocatable/"psql-${g_semver}-${g_target}"

    echo ""
    echo "Creating distributable packages for 'postgres+psql' and 'psql'"
    sleep 1
    fn_package

    rm -rf ~/relocatable
    echo "Done"

    b_pad="$(echo "${g_target}" | tr '[:graph:]' ' ')"
    echo ""
    echo "To start from scratch, remove the following:"
    echo "    ./postgresql-${g_ver}.tar.gz ${b_pad}# official source"
    echo "    ./postgresql-${g_ver}-${g_target}/      # intermediate build files"
    echo "    ./postgres-${g_ver}-${g_target}.tar.gz  # distributable server+client"
    echo "    ./psql-${g_ver}-${g_target}.tar.gz      # distributable client"
    echo ""
); }

fn_install_build_deps() { (
    xcode-select --install 2> /dev/null || true
    if ! git --version > /dev/null; then
        echo ""
        echo ""
        echo "ERROR"
        echo "    first install Xcode Tools, then try building again"
        echo ""
        return 1
    fi

    if ! command -v brew > /dev/null; then
        curl https://webi.sh/brew | sh
        #shellcheck disable=SC2030
        export PATH="$HOME/.local/opt/brew/sbin:$PATH"
        export PATH="$HOME/.local/opt/brew/bin:$PATH"
    fi

    b_clang_ver="$(clang --version | cut -d' ' -f4 | head -n 1 | cut -d'.' -f1)"
    # note: this is always true for idempotency - installing again would otherwise cause non-zero exit status
    brew install llvm@"${b_clang_ver}" openssl@3 icu4c libedit lz4 zstd || true
); }

fn_download_pg() { (
    if ! test -f ./"postgresql-${g_ver}".tar.gz; then
        (
            cd /tmp/
            curl -L -O "https://ftp.postgresql.org/pub/source/v${g_ver}/postgresql-${g_ver}.tar.gz"
        )
        mv /tmp/"postgresql-${g_ver}".tar.gz .
    fi

    #rm -rf ./"postgresql-${g_ver}-${g_target}"/

    if ! test -d ./"postgresql-${g_ver}-${g_target}"/; then
        echo "Unpacking ./postgresql-${g_ver}.tar.gz ..."
        tar xzf ./"postgresql-${g_ver}".tar.gz
        mv ./"postgresql-${g_ver}"/ ./"postgresql-${g_ver}-${g_target}"/
    fi
); }

#shellcheck disable=SC2155
fn_build_pg() { (
    if ! command -v brew > /dev/null; then
        #shellcheck disable=SC2030,SC2031
        export PATH="$HOME/.local/opt/brew/sbin:$PATH"
        export PATH="$HOME/.local/opt/brew/bin:$PATH"
    fi

    b_clang_ver="$(clang --version | cut -d' ' -f4 | head -n 1 | cut -d'.' -f1)"

    export CLANG="$(brew --prefix llvm@"${b_clang_ver}")/bin/clang"
    export LLVM_CONFIG="$(brew --prefix llvm@"${b_clang_ver}")/bin/llvm-config"

    export ICU_CFLAGS="-I$(brew --prefix icu4c)/include"
    export ICU_LIBS="-L$(brew --prefix icu4c)/lib -licui18n -licuuc -licudata"

    export LZ4_CFLAGS="-I$(brew --prefix lz4)/include"
    export LZ4_LIBS="-L$(brew --prefix lz4)/lib -llz4"

    export ZSTD_CFLAGS="-I$(brew --prefix zstd)/include"
    export ZSTD_LIBS="-L$(brew --prefix zstd)/lib -lzstd"

    #export CFLAGS="-march=x86-64-v3 -mtune=generic -O2 -pipe -fstack-protector-strong -flto=auto"
    #export CFLAGS="-arch x86_64 -arch arm64 -mtune=generic -O2 -pipe -fstack-protector-strong -flto=auto"
    export CFLAGS="-I$(brew --prefix llvm@"${b_clang_ver}")/include -I$(brew --prefix openssl@3)/include -I$(brew --prefix libedit)/include"
    export CXXFLAGS="${CFLAGS}"
    export CPPFLAGS="${CFLAGS}"

    export LDFLAGS="-L$(brew --prefix llvm@"${b_clang_ver}")/lib -L$(brew --prefix openssl@3)/lib -L$(brew --prefix libedit)/lib"

    # DYLD_RUN_PATH may be removed by sip
    #export DYLD_RUN_PATH="@loader_path/../lib"
    # DYLD_FALLBACK_LIBRARY_PATH may cause unexpected conflicts
    #export DYLD_FALLBACK_LIBRARY_PATH="@loader_path/../lib"

    cd ./"postgresql-${g_ver}-${g_target}"/ || return 1

    # Clean (uncomment when needed)
    #make clean

    # Configure

    # turned on: llvm,lz4,ssl,zstd
    # disabled: -
    # not turned off: icu,readline,zlib,spinlocks,atomics
    # built-in: -
    # custom-location: libedit,tzdata
    # not turned on: bonjour,gssapi,ldap,nls,ossp,pam,perl,python,tcl,xml,xslt
    #
    # Note:
    #     'postgres' and 'pgsql' are special prefix strings for 'opt' builds
    if ! test -f ./config.status; then
        ./configure \
            --prefix="${HOME}/relocatable/pgsql-${g_semver}-${g_target}" \
            --exec-prefix="${HOME}/relocatable/pgsql-${g_semver}-${g_target}" \
            --disable-rpath \
            --with-libedit-preferred \
            --with-llvm \
            --with-lz4 \
            --with-ssl=openssl \
            --with-system-tzdata=/usr/share/zoneinfo \
            --with-zstd \
            --with-extra-version=" ${g_vendor} +icu,llvm-${b_clang_ver},openssl-3,readline,zlib -tzdata"
    fi

    # Build
    make -j"$(nproc --ignore=1)"
    for b_ext in $g_exts; do
        (
            echo ""
            echo ""
            echo "#### Building Extension ${b_ext} ####"
            sleep 0.3
            cd ./contrib/"${b_ext}"
            make
        )
    done
    for b_ext in $g_exts_only; do
        (
            echo ""
            echo ""
            echo "#### Building Extension ${b_ext} ####"
            sleep 0.3
            cd ./contrib/"${b_ext}"
            make
        )
    done

    # Install
    rm -rf ~/relocatable
    mkdir -p ~/relocatable

    # Server + Client
    make install
    for b_ext in $g_exts; do
        (
            echo ""
            echo ""
            echo "#### Installing Extension ${b_ext} ####"
            sleep 0.3
            cd ./contrib/"${b_ext}"
            make install
        )
    done
    for b_ext in $g_exts_only; do
        (
            echo ""
            echo ""
            echo "#### Installing Extension ${b_ext} ####"
            sleep 0.3
            cd ./contrib/"${b_ext}"
            make install
        )
    done
    mv ~/relocatable/"pgsql-${g_semver}-${g_target}" ~/relocatable/"postgres-${g_semver}-${g_target}"
    rm -rf ~/relocatable/"postgres-${g_semver}-${g_target}"/include

    # (Mostly) Client Tools
    b_client_targets="./src/bin/ ./src/include/ ./src/interfaces/"
    # this excludes './doc/' due to failing linter errors
    for b_target in ${b_client_targets}; do
        echo ""
        echo ""
        echo "#### Installing client ${b_target} ####"
        sleep 0.3
        make -C "${b_target}" install
    done
    mv ~/relocatable/"pgsql-${g_semver}-${g_target}" ~/relocatable/"psql-${g_semver}-${g_target}"
    rm -rf ~/relocatable/"psql-${g_semver}-${g_target}"/include
); }

fn_bundle_icu() { (
    if ! command -v brew > /dev/null; then
        #shellcheck disable=SC2030,SC2031
        export PATH="$HOME/.local/opt/brew/sbin:$PATH"
        export PATH="$HOME/.local/opt/brew/bin:$PATH"
    fi

    b_icudir="$(brew --prefix icu4c)"
    cp -RPp \
        "${b_icudir}"/lib/libicuuc*.dylib \
        "${b_icudir}"/lib/libicui18n*.dylib \
        "${b_icudir}"/lib/libicudata*.dylib \
        ~/relocatable/"postgres-${g_semver}-${g_target}"/lib/
); }

fn_bundle_libs() { (
    a_pgqsl_dir="${1}"

    if ! command -v brew > /dev/null; then
        #shellcheck disable=SC2031
        export PATH="$HOME/.local/opt/brew/sbin:$PATH"
        export PATH="$HOME/.local/opt/brew/bin:$PATH"
    fi

    b_lz4dir="$(brew --prefix lz4)"
    cp -RPp "${b_lz4dir}"/lib/*.dylib "${a_pgqsl_dir}"

    # readline for bsd/macos
    b_editdir="$(brew --prefix libedit)"
    cp -RPp "${b_editdir}"/lib/*.dylib "${a_pgqsl_dir}"

    # macos' openssl isn't always compatible with mainline openssl
    b_ssldir="$(brew --prefix openssl@3)"
    cp -RPp "${b_ssldir}"/lib/*.dylib "${a_pgqsl_dir}"

    # note: zlib is part of the macos base

    b_zstddir="$(brew --prefix zstd)"
    cp -RPp "${b_zstddir}"/lib/*.dylib "${a_pgqsl_dir}"
); }

# SIMPLE EXPLANATION
#
#     For each binary or library file, run otool:
#         otool -L ./my-bin-or-lib
#
#     For each linked path embedded in the file, update it to the bundled path:
#         install_name_tool -change <builtin-path> @loader_path/../lib/<lib> ./my-bin-or-lib
#
fn_patch_rpaths() { (
    a_file="${1}"

    case "$a_file" in
        *.dylib)
            # skip the name of the file, and the first line (its self-link)
            b_libs="$(otool -L "${a_file}" | tail -n +3 | grep '/Users/' | cut -f2 | cut -d' ' -f1)"
            ;;
        *)
            # skip the name of the file, but get the linked non-system lib names
            b_libs="$(otool -L "${a_file}" | tail -n +2 | grep '/Users/' | cut -f2 | cut -d' ' -f1)"
            ;;
    esac

    if test -n "${b_libs}"; then
        echo "    updating ${a_file}"
        for b_lib in ${b_libs}; do
            b_name="$(basename "${b_lib}")"

            echo "        ${b_name}"
            b_path="@loader_path/../lib/${b_name}"
            install_name_tool -change "${b_lib}" "${b_path}" "${a_file}"
        done
    fi

    codesign --force --sign - "$a_file"
); }

fn_patch_rpaths_extensions() { (
    a_dir="${1}"
    for b_ext in $g_exts; do
        fn_patch_rpaths "${a_dir}"/lib/"${b_ext}".dylib
    done
); }

fn_patch_rpaths_libs() { (
    a_dir="${1}"

    for b_file in "${a_dir}"/bin/*; do
        fn_patch_rpaths "${b_file}"
    done

    for b_file in "${a_dir}"/lib/*.*.dylib; do
        fn_patch_rpaths "${b_file}"
    done
); }

# In theory we could create a universal binary but... not worth it
# lipo -create -output my_binary my_binary_arm64.o my_binary_x86_64.o

fn_package() { (
    echo ""
    tar czf ./"postgres-${g_semver}-${g_target}".tar.gz \
        -C ~/relocatable/ ./"postgres-${g_semver}-${g_target}"/
    echo "    ./postgres-${g_semver}-${g_target}.tar.gz"

    tar czf ./"psql-${g_semver}-${g_target}".tar.gz \
        -C ~/relocatable/ ./"psql-${g_semver}-${g_target}"/
    echo "    ./psql-${g_semver}-${g_target}.tar.gz"
    echo ""
); }

main

References

See:

• https://stackoverflow.com/a/33992190
• https://matthew-brett.github.io/docosx/mac_runtime_link.html

[coolaj86]

closing as sticky documentation