, Titanium [2nd Floor]
Weather and environmental data power analytics, ML, and operations—but APIs differ wildly and data prep is slow. Wetterdienst is a Python library that provides a unified, Polars‑first interface to multiple weather services (DWD, ECCC, EA, NOAA/NWS, Geosphere Austria, IMGW, Eaufrance, WSV, and more). It standardizes request patterns, returns tidy (long) data, converts to SI units, handles caching, timezones (UTC by default), and retries—so teams can focus on analysis instead of plumbing. This talk introduces Wetterdienst’s provider architecture, core request patterns, performance practices with Polars, and how to integrate via Python, CLI, or its REST API. We’ll walk through real examples (station discovery, parameter selection, timeseries retrieval), exporting to databases, and patterns for robust pipelines in ETL and ML.
Problem
Accessing weather data means wrestling with inconsistent APIs, formats, and units—slowing down data engineering and making pipelines hard to reproduce.
Solution
Wetterdienst is a Python library providing a unified, Polars-first interface to multiple open weather services (DWD, ECCC, EA, NOAA/NWS, Geosphere Austria, IMGW, Eaufrance, WSV, and more). It standardizes request patterns, returns tidy long-format data in SI units, and handles caching, timezones, and retries—so teams can focus on analysis instead of plumbing.
Core concepts:
- Polars-first — All data operations use Polars (v1.15+); pandas supported for some I/O
- Declarative request pattern — Provider → stations → values; tidy/long output by default
- Sensible defaults — UTC timestamps, SI units, humanized parameter names
- Reliability — Disk-based caching via diskcache, stamina-based retries, timezone handling
- Provider architecture — Consistent interfaces across DWD, ECCC, EA, NOAA/NWS, Geosphere, IMGW, Eaufrance, WSV, and more
- Multiple interfaces — Python API, CLI, and REST
Outline
- Introduction
- Journey — How Wetterdienst came to life
- Wetterdienst — Architecture, concepts, and request patterns
- Value — What wetterdienst offers you, me and everyone else
- Demo — Live: station discovery, timeseries retrieval, station metadata, climate stripes and more via app
Target Audience
Data engineers, scientists, and platform teams who need reliable weather data for analytics, ML, and operations.
Prerequisites
Basic Python and DataFrame experience (Polars or pandas); familiarity with ETL/ML pipelines helpful.
Key Takeaways
- A unified, Polars-first workflow to access and normalize open weather data
- Practical patterns for station discovery, timeseries retrieval, unit conversion, and caching
- How to integrate Wetterdienst via Python, CLI, and REST, and export to common formats and databases
Links
📦 Repo https://github.com/earthobservations/wetterdienst
📖 Docs https://wetterdienst.readthedocs.io/
🌐 App https://wetterdienst.eobs.org/
💡 Examples https://github.com/earthobservations/wetterdienst/tree/main/examples
Benjamin Gutzmann is a 32 year old Python/data engineer and maintainer of Wetterdienst, currently at Otto Group data.works (Data Engineer since 2023; previously Junior Data Engineer), working across Generative AI and data engineering on GCP with Python, SQL, Argo, and Terraform. He has built the Wetterdienst library at earth observations (hobby project, since 2018). Before his start into work life he has studied Hydrology (BSc, MSc) at TU Dresden.