2024-04-23 –, Kuppelsaal
As applications grow, so do the amount of configurable features. Managing consistent defaults, maintaining user and developer documentation, and ensuring uniform parsing among a growing number of client applications can become a challenge. Adding constraints like complex fallback hierarchies and backwards compatibility, increases the probability of runtime errors. We show how Pydantic's strong data validation and integration into Python's type annotations can help building a strict specification for your configuration format, catch misconfiguration early, and mitigate the aforementioned problems with a non-formalized configuration management system.
We describe how we moved our configuration management system from a simple unstructured YAML format loaded into dictionaries into a fully formalized, typed, class-based system using Pydantic's data validation.
While simple enough to begin with, we discuss the problems that emerged from the lack of tight specification of our early configuration system: Missing ahead-of-time validation and resulting runtime errors; out-of-sync code and browsable user documentation; incompatible defaults and subtle differences in various separate parsers scattered throughout many microservices; duplicated and brittle fallback logic. Using a strict specification can mitigate these issues by enabling static validation of configuration files, automatic documentation generation, centralized defaults, and flexible data transformation.
After discussing various available configuration management systems, we explain
the motivation to hand-roll a simple system based on the data validation
library Pydantic. Popularized by it's usage in FastAPI has become the de-facto standard for data validation in Python. It's deep integration into Python's type annotation system makes it a powerful tool for configuration management.
After an introduction into Pydantic capabilities and usage, specifically it's features tailored to configuration management (pydantic.BaseSettings), we share some tips-and-tricks encountered while speccing out our configuration file format. Additionally, we share some inspiration on our internal tooling to load and validate configuration, render up-to-date browsable user documentation, integration with CI systems, and lessons learned for a incremental transition from the lose dict-based system to the strictly typed class-based type strict system powerd by Pydantic.
Novice
Expected audience expertise: Python:Intermediate
Abstract as a tweet (X) or toot (Mastodon):How Pydantic's strong data validation based on type annotations can help build a strict spec for your configuration format, catch misconfiguration early, and mitigate the aforementioned problems with a non-formalized configuration management system.
