Meet us in Singapore: Samuel Chan is speaking at GovTech STACK 2026, Marina Bay Sands on 3 November.
Real-time architectures with Kafka, Spark, and the open source streaming stack.
6 articles in this topic
Featuring the use of Cassandra for OLTP, MySQL for OLAP, Kafka and Spark Streaming to ingest and process transactional data streams in a streaming dashboard that visualizes the total volume of money flow in the stock market
High Level Introduction to Apache Kafka, an Event Streaming Platform
In the first part of the article, we will discuss the overview of an open-source streaming data pipline and its environment setup using Docker Compose.
In the second part of the article, we will walk through the design and implementation of OLTP and OLAP databases using Cassandra and MySQL respectively.
In the third part of this series, we will demonstrate how to ingest and process transactional data using Kafka and Spark Streaming.
In the fourth and final installment of this series, we will perform some analytical queries on the MySQL database and create a dashboard using Streamlit.
The same thinking runs all the way through this 12-chapter guide, written by Samuel Chan and ready to read in an afternoon.
Build and ship a live IDX stock app in one afternoon, with GitHub Copilot as your AI pair programmer in VS Code and the Sectors API feeding it live market data. A casual, hands-on community meetup in Jakarta.
October 31, 2026Block71
This is one slice of the catalogue. Browse every topic on the notes index, or subscribe and we will send the next piece your way.