Web Image Processing Pipelines (WIPP) is an open-source web-based algorithmic plugin platform for trusted image-based measurements from terabyte-sized images developped at the National Institute of Standards and Technology (NIST), in collaboration with the National Institutes of Health (NIH) - National Center for Advancing Translational Science (NCATS).
The goal of this project is to lower the bar for users and algorithm developers to execute and share image analyses over terabyte-sized image collections in high-throughput and high content microscopy imaging applications.
- Upload collections of microscopy images,
- Register algorithmic plugins,
- Configure image processing workflows,
- Stitch grids of small fields of view,
- Visualize and analyze large fields of view interactively using the pyramid view,
- Extract image features and customize scatterplots,
- Train and use Tensorflow models for image segmentation,
- Prototype algorithms with polyglot Jupyter notebooks.
WIPP is designed as an expandable platform leveraging containerization and orchestration technologies such as Docker and Kubernetes and is meant to be deployed on a Kubernetes cluster (see Installation instructions).
The core of the WIPP framework consists of:
- the WIPP REST API, based on the Java Spring framework,
- the WIPP web UI, an Angular 7 application,
- a MongoDB database,
- [Keycloak], an open source identity and access management server,
- Argo Workflows, a container native workflow engine for Kubernetes,
- WIPP Plugins, algorithmic plugins packaged as Docker images.
Additional tools can be added to the WIPP core for a more complete system:
- JupyterHub, to develop polyglot Jupyter notebooks that can be used in WIPP workflows,
- Plots, for scalable and interactive scatterplots visualization,
- Tensorboard, for Tensorflow jobs monitoring,
- Elastic Stack for application, logs and system monitoring.
For a quick start, we provide instructions for the deployment of a complete installation of WIPP on a single-node Kubernetes cluster.
Installing WIPP on a single-node cluster
Instructions are provided for popular operating systems using simple solutions for setting up a single-node Kubernetes cluster:
- macOS (with Docker Desktop)
- macOS (with Multipass+microk8s)
- Linux (Multipass+microk8s)
- Windows 10 (Multipass+microk8s)
The following components are included in the single-node deployment:
- WIPP REST API and UI,
- MongoDB,
- Keycloak,
- Argo Workflows,
- JupyterLab,
- Plots,
- Tensorboard.
We provide WIPP plugins for CNN-based segmentation using Tensorflow. These plugins require access to a GPU in the Kubernetes cluster.
The requirements for running the using the WIPP UNet CNN Semantic-Segmentation Training plugin and WIPP UNet CNN Semantic-Segmentation Inference plugin are:
- Cuda 10.1 installed,
- nvidia-docker-runtime and set as default docker runtime
- GPU support enabled in the cluster (for example, by installing the NVIDIA Device Plugin for Kubernetes or enabling the
gpuaddon in microk8s)
Alternatively, some GPU cards may be compatible with the preview release of the NVIDIA GPU Operator for Kubernetes, which aims to simplify NVIDIA GPU resources in Kubernetes clusters.
The WIPP framework deployment instructions are provided for deploying a testing instance of the system. Additional configurations may be required for a secure and scalable deployment. We recommend deployments on private networks behind a firewall.
User guide detailing how to perform routine operations in WIPP, such as uploading images, registering plugins and configuring workflows is available in the user guide folder.
To test a fresh installation of WIPP, we provide three sample datasets with instructions on how to run test workflows using these datasets in the data folder.
Links to manifest files to install the currently available WIPP plugins can be found in the plugins folder.
Part of the WIPP ecosystem, the WIPP Registry allows the registration and search of available plugins:
- Demo instance https://wipp-plugins.nist.gov/
- Deployment instructions available here
Existing WIPP plugins can be used as learn-by-example templates for creating new plugins. A basic template for Python-based plugins is provided here: Python sample plugin.
The WIPP plugin manifest schema is available at this link.
If you encounter issues during the installation or usage of WIPP, you can get help using one of the following options:
- open an issue on the repository,
- send an email to the WIPP team.
- NIST article and video:
- A Map App to Track Stem Cells, NIST news, February 13, 2018
- Reference book:
- Web Microanalysis of Big Image Data, published in January 2018 by Springer Verlag.
- Publications:
- Peter Bajcsy, Antoine Vandecreme, Julien Amelot, Joe Chalfoun, Michael Majurski, and Mary Brady, “Enabling Stem Cell Characterization from Large Coverage Microscopy,” IEEE Computer, 2016: vol. 49, no. 7, pp. 70-79, URL
- Antoine Vandecreme, Tim Blattner, Michael Majurski, Peter Bajcsy, Keana Scott, and John Henry J. Scott, “From Image Tiles to Web-Based Interactive Measurements in One Stop,” Microscopy and Microanalyses, Portland, OR, August 2-6, 2015, URL
If you are using WIPP in your research, please use the following references to cite us.
Peter Bajcsy, Joe Chalfoun, and Mylene Simon (2018), "Web Microanalysis of Big Image Data", Springer-Verlag International, URL