Viewing Real-time data streams (MySql, Logstash & WebSockets) - Part 1

Recently I've had to do a proof of concept (POC) on how to provide our analytics team with a way of viewing monetary transactions on a web-site, as real time as possible.

In this article I'll be showing you the basic architecture and the intended data flow for the POC.

I'll share the steps to do recreate this yourself in a later post (part 2).

The Architecture

For my architecture I wanted to make use of only 3 components; the data-source, the data stream and the front end.

Data Source

Given that the data source was already defined as a relational database I wanted to use a lightweight, popular and open-source database  so I chose MySQL (I tried with SQLite but couldn't get it working on Windows).

This also means that I needed to have a data-streaming component that could poll a database over a JDBC connection for changes to this database.

Front End

With the front end I chose html because it is easy to develop, it can be deployed locally, it supports cross-platform development, and I've got the ability to use web sockets and other push notification methods to receive data in a stream format.

Data Stream

For the streaming component I could have used something like Kafka or RabbitMQ, but I wanted something that is lightweight, doesn't have a sharp learning curve and could support all of my communication protocols and Logstash ticks all of these boxes.

Component View

Data flow

1. The web-client registers on the web-socket defined on Logstash.
2. The data flow starts with a database insert, which then gets detected using Logstash.
3. Logstash detects the change by polling the database every N-seconds (10 was real-time enough for my purposes) and checking the changes against a last inserted timestamp column.
4. Logstash then published these changes to a web-client using a web-socket.