PyCon UK 2019

Real-time data acquisition, from Arduino to the web, using PubSub with Redis, Django and other friends
2019-09-16 , Assembly Room

I'll apply PubSub to collect data in real-time from TCP-enabled peripherals, like Arduino or others, and deliver them to a data server for storage, processing, and broadcasting to remote clients (typically smartphones or tablets) for real-time monitoring and inspection


We all know and love our fancy web framework, but when it comes to collecting data in real-time from remote devices, we have to take a step in a new direction.

This talk aims to add an extra dimension to the skills of Python/Django developers, who may be familiar with processes and techniques involved in data publishing, but would like to explore new ways of collecting data in the first place.

The talk covers data collection, processing and publishing.

Collection: TCP-enabled devices such as Arduino make it possible to collect real-time data extremely cheaply and simply. I will demonstrate how easy it is to deliver data from such devices to our data servers, using robust PubSub communication patterns.

Processing: a Python backend provides multiple options for processing data and extracting useful information and insights.

Publishing: to complete the cycle, the information must be delivered in real-time to the user in a comprehensible, usable form; we’ll assume our users are working in the field with smartphones or tablets.

Target audience

I will assume that my audience has a basic understanding of Python and Django, and build on that to introduce tools (Redis, Arduino) and techniques (PubSub, real-time messaging, Django Channels) in a practical context.

We'll dig into the code of several practical use cases, to explore how the solution can be adapted to different scenarios.
The software technologies selected for this purpose are: Python, Redis and Django Channels.

I hope that this talk will give other programmers the confidence to explore this fascinating and useful sphere of programming.


Is your proposal suitable for beginners?: yes

I work as a consultant in different fields, applying open source and web technologies to automation.
In the past, I made extensive use of C/C++ and other languages, but lately abandoned them in favour of Python based frameworks, with particular interest in Django.
I got a master’s degree in Physics in 1987 at University of Modena, Italy