Let’s say you want to create a realtime visualization of a Kafka data stream. Maybe you want to process it with FlinkSQL before visualizing it.

Your keyboard clatters, “mkdir data_viz_with_kafka_and_flinksql”. You cd in, activate your virtual environment, crack your knuckles, and… the complexities hit. How to create open connections to your data stream? Feed the data into a frontend component? What happens when multithreading rears its ugly head? Your feelings of bravado slowly dissipate… Don’t worry, I’ve got you covered!

In this session, we’ll cover how to take a stream of data in Kafka and visualize it with Streamlit. It’s sourced from the Alpaca API, before being sent to Kafka and processed with FlinkSQL for surfacing the Streamlit component. We’ll go through the ins and outs of creating Kafka producers and consumers in python, processing realtime data via windowing using FlinkSQL in Confluent Cloud, and visualizing that data clearly for an audience using a component built with Streamlit.

By the end of the talk, attendees will be confident in creating live data visualizations using Kafka, FlinkSQL, and Streamlit and be equipped to take their realtime use cases to the next level.