Automated Functional Testing

If you just want to run the functional tests once, it's straightforward in Android Studio: create a Run Configuration that runs all tests in the androidTest/ directory, then plug in a tablet and click Run.

This page is about how we automate the testing process using Jenkins. Our Jenkins server at jenkins.projectbuendia.org automatically builds and tests both the client and the server whenever new commits are pushed to GitHub, helping us catch bugs quickly. Although the functional tests take the most work to maintain, they also produce the most value — they save us a great deal of time that would be spent manually tapping and swiping to test everything when we release a new APK.

Jenkins has a integration-dev job that runs an integration test by installing the latest server build at integration.projectbuendia.org, installing the latest client build on a tablet, and then running all the functional tests on the tablet, with that client running against that server.

For this job to work, there must be a Jenkins node set up to run these tests—that is, a Linux machine that remains up all the time, communicates with Jenkins, and has an attached tablet.

Setting up a new Jenkins node

If you need to add a new node, this is how to do it. This procedure has been tested with Ubuntu 14.04.

Logged in as root

• Make sure basic tools are installed: apt-get install git

• Install JDK 7 with the command: apt-get install openjdk-7-jdk

• Download the Android SDK for Linux ("SDK Tools Only").

• Go to the /opt directory and unpack the tar file you downloaded.

• Try /opt/android-sdk-linux/platform-tools/adb. It should print out a long page of help on its various commands and options. If you are running 64-bit Ubuntu and get a "No such file or directory" error even though the file exists, you probably need to install 32-bit libraries:

  apt-get install libc6-i386 lib32stdc++6 zlib1g:i386
  

• Install the necessary platform packages with this command (and type "y" to accept the license):

  /opt/android-sdk-linux/tools/android update sdk --no-ui --all --filter \
      android-21,build-tools-19.1.0,extra-android-support,extra-android-m2repository,platform-tools
  

• Add a user named "buendia" with the home directory /home/buendia

Logged in as buendia

• Append this line to /home/buendia/.bashrc:

  export ANDROID_HOME=/opt/android-sdk-linux
  

• Run ssh-keygen -t dsa to generate an SSH key and don't enter any passphrase; just hit Enter twice.

• Get your key in .ssh/id_dsa.pub added to the .ssh/authorized_keys files for the "jenkins" user on both jenkins.projectbuendia.org and integration.projectbuendia.org.

• Log in to each of those accounts once, answering "yes" to accept the new host (so that when Jenkins tries to ssh into these machines, it doesn't get stuck at the ssh prompt waiting for a "yes"). After you've done this once, the host keys will get saved and ssh won't ask for confirmation again.

  ssh [email protected]  # answer yes, then log out
  ssh -p 9022 [email protected]  # answer yes, then log out
  

In Jenkins settings

In a browser, go to http://jenkins.projectbuendia.org/ and log in.

• Click Jenkins > Manage Jenkins > Manage Nodes > New node
• Enter a name for your node
• Select Dumb Slave
• Click OK
• For Remote root directory enter "/home/buendia"
• For Labels enter "android-test"
• Select "Only build jobs with label restrictions matching this node"
• For Launch method select Java Web Start
• Click Save

Then go to the settings page for your newly created node and follow the instructions there to download slave.jar and run it on your machine. To keep it running all the time, you could use a small shell script like this (you'll need to fill in the appropriate URL and secret provided by Jenkins):

#!/bin/bash

while true; do
    java -jar slave.jar -jnlpUrl http://jenkins.projectbuendia.org/computer/.../slave-agent.jnlp -secret ...
    sleep 1
done

With your tablet

Connect the tablet to your machine with a USB cable and answer the prompt to allow USB debugging.

Now you're all set. Just make sure slave.jar is always running. When Jenkins has an Android test job to run, it will launch the job on your machine.

Troubleshooting

If you want to try running the functional tests on your machine directly (without involving Jenkins) to confirm that communication with the tablet is working, you can clone the repo and run gradlew like this:

git clone --recursive https://github.com/projectbuendia/client
cd client
./gradlew connectedAndroidTest