The Invisible Work of the Stack: Measuring the Impact of Public Investment in PyPI
Almost every Python data science workflow depends on PyPI, but few people see the maintenance work behind it. PyPI hosts over 780,000 packages, making it the largest package repository to gather the necessary toolkit to develop everything that happens in the Data Science stack: from data engineering and machine learning to visualization and AI tooling.
Despite its importance for the Python ecosystem, PyPI shares a problem common to many OSS projects. Although it forms part of our critical digital infrastructure, its maintenance depends heavily on the often unseen labour of volunteer communities. The “care” work in OSS maintenance is often overlooked and undervalued: these maintainers do not receive reciprocal contributions for their work, making entire ecosystems vulnerable to collapse if communities can no longer sustain their efforts.
This talk examines what happens when public institutions invest in that infrastructure, and how we might evaluate whether such investments actually help. Between 2023 and 2024, the Sovereign Tech Agency invested over €1 million in PyPI and related parts of the Python cryptographic ecosystem Work was done across the cryptographic ecosystem (PyCA Cryptography, pyOpenSSL, M2Crypto, BoringSSL, the ssl module, and PyPI/Warehouse), covering security improvements, API modernization, legacy deprecation, and alignment with sigstore for supply chain verification.
Two years later, the difficult question is impact: how can we measure the effects of public investment in open source without reducing maintenance to simplistic metrics? This question comes with a lot of caveats: we need to find comparable software projects to find our counterfactual, gather data, control for confounders without flattening the socio-technical realities of maintenance. In this talk, we open up these challenges by introducing an impact evaluation framework for public investment in OSS projects. We construct a goals-to-metrics dictionary to evaluate change over time, and apply a causal inference framework combining Propensity Score Matching with Generalized Synthetic Control Methods to estimate the effects of the funding by comparing treated repositories with a donor pool.
Attendees will gain a practical understanding of how to evaluate the impact of open source funding: how to translate funding goals into measurable indicators, gather relevant OSS data, and use causal methods responsibly while preserving the socio-technical context of maintenance work. We also encourage feedback from maintainers, researchers, and funders on what responsible, community-sensitive evaluation should look like.
This talk lies in the intersection of domain application and education & outreach topics. It aims to share the work of the Sovereign Tech Agency and its relevance for the Python ecosystem. It is addressed to two main audiences: OSS maintainers who may benefit from the agency's funding programmes, and researchers, public institutions, and private actors interested in evaluating the impact of OSS investments.
For maintainers, the central thesis is that public investment mechanisms are essential for sustaining critical digital infrastructure, where OSS plays a central role in it. The talk makes the case for why institutions like the Sovereign Tech Agency matter and how maintainers can directly benefit from them. A core goal is to build bridges between OSS communities and the Sovereign Tech Agency as a public agency whose mission aligns with the long-term sustainability of open source ecosystems.
For those working with OSS data and investment decisions (whether in academia, public institutions, or the private sector) the talk presents a novel framework for evaluating the impact of OSS funding, an emerging field with many open questions. The approach proposes a mixed-methods pipeline that moves from the high-level goals of a funding initiative (sustaining a project's health and long-term viability) down to quantitative, causal measurement of outcomes.
This talk also aims to open up a conversation. Impact evaluation in OSS is still an emerging field, with more questions than answers. While we propose a possible answer, we are looking forward to hearing back from the attendants. How else can responsible, community-aware evaluation be conducted? What other novel methods and data sources could be used to solve the aforementioned challenges?
Time outline of the presentation:
- 3 mins - Introduction: who we are, what the talk covers, and why OSS impact evaluation matters
- 5 mins -
pip, PyPI, and the invisible “care” work of open source maintenance - 2 mins - Sovereign Tech Agency
- 3 mins - Sovereign Tech Fund's investment in PyPI
- 9 mins - Evaluation Framework
- 5 mins of methods
- 4 mins of results
- 8 mins - Q&A
Required background knowledge is minimal. Attendees should have a general familiarity with Python, open source software, or software repositories, but no prior knowledge of causal inference methods is required.
I am a Data Scientist and Sociologist working at the Sovereign Tech Agency. My main project focuses on researching frameworks for evaluating the impact of public funding in open-source software. I’m interested in open data, feminism, and the human dynamics behind tech structures. In my free time, I enjoy practicing yoga and reading.