The Runner API provides a RESTful interface for submitting and managing Apache Beam streaming data processing jobs, and job contexts. It was originally started at Ananas Analytics.
-
Execute pipelines on multiple execution environments ( Spark and Flink )
-
Synchronous job API for testing only with Standalone Flink.
-
Asynchronous job API
-
Currently works with Standalone Spark as well on cluster, Mesos, YARN client and on EMR
-
Support Google Dataflow
-
Support Flink (in beta)
-
Support Bounded and unbounded sources
-
Synchronous CLI for testing and submitting streaming data processing jobs for developers
Install Java 8 ( or more recent version)
-
Build the fat jar
../build-cli.sh
-
Run
java -jar build/libs/ananas-cli-xxx.jar start --host [host] --port 8888
or
- Using gradle run task
gradle :runner:run -DfilesToStage=mock.jar
- New I/O Connector : Redis, Kafka, Pubsub
- New transformer step javascript
- Find a non assigned feature request or a create it if it does not exist with your request
- Push a Pull Request