Live dashboard with Kafka Streaming and Websockets

This is an example on how to use Kafka Streamin API to aggregate logs and display results online in a dashboard.

Tools:

1. Java 8+
2. Apache Kafka 0.11
3. Apache Kafka Streams API
4. Spring Boot
5. STOMP
6. Chart.js

How to run:

1. Follow step 1 and step 2 of Kafka Quickstart

2. Create two topics: data-in and data-out

   bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic data-in

   bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic data-out

3. Clone and build java project

   git clone https://github.com/sajmmon/kafka-streaming-websockets.git

   cd kafka-streaming-websockets

   mvn clean package

4. Run producer

   java -cp target/shade-kafka-streaming-websockets-0.1.0.jar uk.ac.ebi.produce.KafkaExampleProducer

5. Run streaming client

   java -cp target/shade-kafka-streaming-websockets-0.1.0.jar uk.ac.ebi.streaming.KafkaStreamingMain

6. Run spring boot

   java -cp target/shade-kafka-streaming-websockets-0.1.0.jar uk.ac.ebi.Application

7. See live dashboard in localhost:8080

When you click "Connect" you should see:

Recommended resources:

1. Kafka Streaming (by Gwen Shapira)
2. Using WebSocket with Spring Boot (by Pivotal)
3. Apache Kafka with Spring (by Baeldung)
4. Kafka Streams Examples (by Confluent)
5. Creating custom serializers (by Niels.nu)