Skip to content
This repository has been archived by the owner on Nov 25, 2022. It is now read-only.

Commit

Permalink
replace travis with circle-ci, cleanup of build steps (#639)
Browse files Browse the repository at this point in the history
Travis is getting slower and was already hard to debug. This commit: 

- moves from travis to Circle-CI
- has a ci-scripts/README.md which explains what is contained there
- puts all build and CI scripts into "ci_scripts" directory
- allows to run full ci runs locally
- trims down the base Docker image where tests/docs/wheels are run from 4.4GB to 1.7GB
- fixes #628

Using circle-ci also means that we can probably integrate Mac-OS build steps more easily.

I am sure that my initial circle-ci config (helped by kind hints from @dignifiedquire ) can be improved.
  • Loading branch information
hpk42 authored Apr 1, 2019
1 parent 7f3bf1f commit 515e5eb
Show file tree
Hide file tree
Showing 34 changed files with 608 additions and 309 deletions.
1 change: 1 addition & 0 deletions .circleci/config.yml
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# ignore backup files
*~

# ignore tox and docker stuff
.docker-*
wheelhouse
workspace

# ignore build temporary files
*.o
.deps
Expand Down
25 changes: 0 additions & 25 deletions .scripts/buildwheel.sh

This file was deleted.

40 changes: 0 additions & 40 deletions .scripts/deploy.sh

This file was deleted.

70 changes: 0 additions & 70 deletions .travis.yml

This file was deleted.

7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Delta Chat Core Library

[![Build Status](https://travis-ci.org/deltachat/deltachat-core.svg?branch=master)](https://travis-ci.org/deltachat/deltachat-core)
[![CircleCI](https://circleci.com/gh/deltachat/deltachat-core.svg?style=svg)](https://circleci.com/gh/deltachat/deltachat-core)

The _Delta Chat Core Library_ is written in cross-platform **C**,
documented at <https://c.delta.chat>.
Expand All @@ -9,13 +9,18 @@ The ``deltachat`` Python bindings can be found in the
[python subdirectory](https://github.com/deltachat/deltachat-core/tree/master/python)
and are documented at <https://py.delta.chat>.

In the [ci_scripts directory](https://github.com/deltachat/deltachat-core/tree/master/ci_scripts/README.md)
you'll find docker- and library-building scripts.

## binary/distribution packages (work-in-progress)

There are work-in-progress efforts for creating (binary) packages which
do not require that you build the library manually:

- [libdeltachat-core-git archlinux package](https://aur.archlinux.org/packages/libdeltachat-core-git/>)

- [python-wheel packaging](https://m.devpi.net/dc/master)

- [Debian packaging](https://github.com/deltachat/deltachat-core/issues/299)

- [Windows building](https://github.com/deltachat/deltachat-core/issues/306)
Expand Down
52 changes: 52 additions & 0 deletions ci_scripts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@

# Continuous Integration Scripts for Delta Chat

Continuous Integration is run through CircleCI
but is largely independent of it.


## Generating docker containers for performing build step work

All tests, docs and wheel building is run in docker containers:

- **coredeps/Dockerfile** specifies an image that contains all
of Delta Chat's core dependencies as linkable libraries.
It also serves to run python tests and build wheels
(binary packages for Python).

- **doxygen/Dockerfile** specifies an image that contains
the doxygen tool which is used to generate C-docs.

To run tests locally you can pull existing images from "docker.io",
the hub for sharing Docker images::

docker pull deltachat/coredeps
docker pull deltachat/doxygen

or you can build the docker images yourself locally
to avoid the relatively large download::

cd ci_scripts # where all CI things are
docker build -t deltachat/coredeps docker-coredeps
docker build -t deltachat/doxygen docker-doxygen

## ci_run.sh (main entrypoint called by circle-ci)

Once you have the docker images available
you can run python testing, documentation generation
and building binary wheels::

sh DOCS=1 TESTS=1 ci_scripts/ci_run.sh

## ci_upload.sh (uploading artifacts on success)

- python docs to `https://py.delta.chat/_unofficial_unreleased_docs/<BRANCH>`

- doxygen docs to `https://c.delta.chat/_unofficial_unreleased_docs/<BRANCH>`

- python wheels to `https://m.devpi.net/dc/<BRANCH>`
so that you install fully self-contained wheels like this:
`pip install -U -i https://m.devpi.net/dc/<BRANCH> deltachat`



20 changes: 20 additions & 0 deletions ci_scripts/ci_run.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# perform CI jobs on PRs and after merges to master.
# triggered from .circleci/config.yml

set -e -x

export BRANCH=${CIRCLE_BRANCH:-test7}

# run doxygen on c-source (needed by later doc-generation steps).
# XXX modifies the host filesystem docs/xml and docs/html directories
# XXX which you can then only remove with sudo as they belong to root

if [ -n "$DOCS" ] ; then
docker run --rm -it -v $PWD:/mnt -w /mnt/docs deltachat/doxygen doxygen
fi

# run everything else inside docker (TESTS, DOCS, WHEELS)
docker run -e BRANCH -e MESONARGS -e TESTS -e DOCS \
--rm -it -v $(pwd):/mnt -w /mnt \
deltachat/coredeps ci_scripts/run_all.sh

47 changes: 47 additions & 0 deletions ci_scripts/ci_upload.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/bin/bash

if [ -z "$DEVPI_LOGIN" ] ; then
echo "required: password for 'dc' user on https://m.devpi/net/dc index"
exit 1
fi

set -xe

DOXYDOCDIR=${1:?directory where doxygen docs to be found}
PYDOCDIR=${2:?directory with python docs}
WHEELHOUSEDIR=${3:?directory with pre-built wheels}

export BRANCH=${CIRCLE_BRANCH:?specify branch for uploading purposes}


# python docs to py.delta.chat
rsync -avz \
-e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
"$PYDOCDIR/html/" \
[email protected]:build/${BRANCH}

# C docs to c.delta.chat
rsync -avz \
-e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" \
"$DOXYDOCDIR/html/" \
[email protected]:build-c/${BRANCH}

echo -----------------------
echo upload wheels
echo -----------------------

# Bundle external shared libraries into the wheels
pushd $WHEELHOUSEDIR

pip install devpi-client
devpi use https://m.devpi.net
devpi login dc --password $DEVPI_LOGIN

devpi use dc/$BRANCH || {
devpi index -c $BRANCH
devpi use dc/$BRANCH
}
devpi index $BRANCH bases=/root/pypi
devpi upload deltachat*.whl

popd
90 changes: 90 additions & 0 deletions ci_scripts/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@

version: 2
jobs:
build_test_docs_wheel:
machine: True
steps:
- checkout
- run: docker pull deltachat/doxygen
- run: docker pull deltachat/coredeps
- run:
command: ci_scripts/ci_run.sh
environment:
MESONARGS:
DOCS: 1
TESTS: 1

- run:
name: copying C docs, python docs and wheels to workspace
command: |
mkdir -p workspace/python
cp -av docs workspace/c-docs
cp -av python/.docker-tox/wheelhouse workspace/
cp -av python/doc/_build/ workspace/py-docs
- persist_to_workspace:
root: workspace
paths:
- c-docs
- py-docs
- wheelhouse

upload_docs_wheels:
machine: True
steps:
- run:
command: |
echo 'export MESONARGS=""' >> $BASH_ENV
- checkout
- attach_workspace:
at: workspace
- run: ls -laR workspace
- run: ci_scripts/ci_upload.sh workspace/c-docs workspace/py-docs workspace/wheelhouse

build_monolith_tests:
machine: True
steps:
- checkout
- run: docker pull deltachat/coredeps
- run:
command: |
echo 'export MESONARGS="-Dmonolith=true"' >> $BASH_ENV
echo 'export TESTS=1' >> $BASH_ENV
- run: ci_scripts/ci_run.sh

build_static:
machine: True
steps:
- checkout
- run: docker pull deltachat/coredeps
- run:
command: |
echo 'export MESONARGS="--default-library=static"' >> $BASH_ENV
- run: ci_scripts/ci_run.sh

build_forcefallback:
machine: True
steps:
- checkout
- run: docker pull deltachat/coredeps
- run:
command: |
echo 'export MESONARGS="--wrap-mode=forcefallback --default-library=static"' >> $BASH_ENV
- run: ci_scripts/ci_run.sh

workflows:
version: 2
build_all:
jobs:
- build_test_docs_wheel
- build_monolith_tests
- build_static
- build_forcefallback
- upload_docs_wheels:
requires:
- build_test_docs_wheel
- build_monolith_tests
- build_static
- build_forcefallback


Loading

0 comments on commit 515e5eb

Please sign in to comment.