General test harness improvements (#77)

* Use treon to run notebooks in their own directory

treon==0.1.3 runs notebooks in their directory so we can discard the
workarounds that we have

* Fix output testing failing silently

* Use new notebook exclusion in treon==0.1.3

This means that we can throw away the workaround we had in `.travis.yml`.
Not a super big change, but it feels nice.

* Cache pip for Travis builds

* Use 3.8 over 3.8-dev for Travis builds

Matplotlib is choking on something that it didn't choke on before.
Maybe this will fix it.

* Temporarily disable Python 3.8 testing

See #88. Waiting on Python 3.8 support in numba/llvmlite.

* Fix installing requirements.txt failing silently

.travis.yml

@@ -1,25 +1,21 @@
 language: python
-dist: xenial  # for python3.7+ testing
+dist: bionic
+
+cache: pip
 
 python:
 - 3.6
 - 3.7
-- 3.8-dev
 
 install:
 # numpy requires some additional dependencies
 - sudo apt-get update
 - sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran libhdf5-dev
 - pip install -r test/requirements-dev.txt
-- find . -name "requirements.txt" -exec pip install -r "{}" \;
-
-before_script:
-# To ignore a directory, we can just delete the directory
-# so that treon doesn't see it.
-- xargs -t rm -rf < test/ignore
+- find . -name "requirements.txt" | tr '\n' '\0' | xargs -0 -n1 pip install -r
 
 script:
 # Test that notebooks run successfully
-- find "$PWD" -name "*.ipynb" -exec sh test/test_notebook.sh "{}" +
+- eval "treon $(awk '{printf " --exclude " $0}' test/ignore)"
 # Naive test to check that output committed with notebook-
-- find . -name "*.ipynb" -exec sh test/test_output.sh "{}" \;
+- find . -name "*.ipynb" | tr '\n' '\0' | xargs -0 -n1 sh test/test_output.sh

test/README.md

@@ -12,7 +12,7 @@ on Travis.
 
 The tests also check if output is committed with the notebook (which it should be).
 This test is naive - it tests if the number of code cells in a notebook is equal to
-the number of cell outputs in the same notebook.
+the number of cell outputs in the same notebook. (See `.travis.yml`.)
 
 ## Structure
 

test/requirements-dev.txt

@@ -1,2 +1,2 @@
 -r requirements.txt
-treon==0.1.2
+treon==0.1.3

test/test_output.sh

@@ -1,7 +1,4 @@
 #!/bin/sh
 # test/test_output.sh 
-test $(grep -oc '"outputs": ' "$1") -eq $(grep -oc '"cell_type": "code"' "$1")
-if [ $? -gt 0 ]; then
-    echo "FAIL: Was output committed with this notebook? $1"
-fi
-exit 0
+echo $1
+test $(grep -oc '"outputs": ' "$1") -eq $(grep -oc '"cell_type": "code"' "$1") || exit 1