2025-04-25 –, Hassium
In this talk, we show how to leverage publicly available tools to control any game or program using hand or body movements. To achieve this, we introduce PosePIE, an open-source programmable input emulator that generates input events on virtual gamepads, keyboards and mice based on gestures recognized by using AI-driven pose estimation. PosePIE is fully configurable by the user through Python scripts, making it easily adaptable to new applications.
Recent advancements in machine learning and AI hardware acceleration have enabled the use of complex models for solving computer vision problems in real-time applications. Pose estimation is one such problem, involving the detection of keypoints of the human body within an image.
In this talk, we show how PosePIE uses pose estimation to control any game or program using hand or body movements. By using state-of-the-art models, PosePIE does not require expensive specialized sensors but works entirely on the monocular image from an off-the-shelf webcam. By leveraging readily available Graphics Processing Unit (GPU) hardware, it is able to do all processing at a high frame rate to support interactive applications.
As PosePIE is fully configurable by the user through Python scripts, it can be easily adapted to new applications. This lowers the barrier to use pose estimation and gesture recognition in creative ways and for novel applications.
The source code of PosePIE is available on GitHub under the GNU GPLv3+ license.
Novice
Expected audience expertise: Python:Novice
Daniel Stolpmann received the B.Sc. and M.Sc. degrees in computer science and engineering from Hamburg University of Technology (TUHH), Germany, in 2017 and 2019. During his master studies, he started working at the Institute of Communication Networks (ComNets) as a student assistant and became a research fellow after his graduation. At ComNets, he conducted research on machine learning for communication networks, network coding and network emulation. In 2024, he joined Tegtmeier Inkubator as a senior software developer and started working on AI-enabled smart home systems.