2026-06-06 –, Hardwick Hub
Sensors operating in complex environments produce noisy data. Determining exactly when a system transitions between states — and what values it is recording — is surprisingly hard: vibrations, environmental changes, and gradual shifts all conspire against simple threshold approaches. This talk walks through a real-world Python pipeline that solves this problem, starting with classical signal processing, exposing its failure modes, and then building a principled solution using a Kalman filter for noise reduction coupled with a Hidden Markov Model (HMM) for state inference. Attendees will leave understanding how to frame sensor problems as state estimation tasks and how to apply these techniques in Python using necessary libraries.
Objective
Many operations depend on accurate data from continuous sensor streams. Knowing when a system transitions between states, when a process cycle completes, and how much change occurred per cycle drives scheduling, monitoring, and operational reporting. This talk presents a complete data science pipeline — built entirely in Python — that automates event detection and value estimation from noisy sensor streams. The goal is to give attendees both a worked real-world case study and a transferable toolkit for tackling noisy, event-driven sensor data in any domain.
The Problem
Sensors record measurements continuously, but the raw signal is far from clean. Vibrations, speed changes, and environmental shifts all create noise that masks the true underlying state of the system (for example: wake, light sleep, deep sleep, REM sleep). A naive threshold-based approach — the initial "traditional method" — is brittle: it misfires on transient spikes, misses gradual transitions, and cannot estimate values reliably. This section sets up the problem visually with annotated sensor traces and shows concretely where simple methods break down.
Why Kalman Filter + Hidden Markov Model?
The key insight is that the system operates as a latent state machine: at any moment it is in one of a small number of discrete states (idle, transitioning, active, completing), and what we observe is a noisy function of that state. This framing motivates a two-stage approach: Kalman Filter — smooths the raw signal, handles sensor noise, and provides a principled estimate of the true instantaneous value with an associated uncertainty. Hidden Markov Model — takes the smoothed signal and infers the sequence of hidden states, including the timing of transitions and the most probable value estimate at peak. The talk explains the intuition behind both models without heavy mathematics, and then shows how to implement them in Python with filterpy (Kalman) and hmmlearn (HMM).
I am a data scientist at a international mining group.