SciPy 2026

Background and Motivation

Electric aviation represents a critical frontier in sustainable transportation, and Vermont-based BETA Technologies is pioneering this transformation. BETA is taking a methodical "crawl, walk, run" approach to FAA certification: first our H500A electric motor, then our ALIA CTOL (Conventional Takeoff and Landing) aircraft, and finally our ALIA VTOL (Vertical Takeoff and Landing) aircraft.
Fast design iteration is fundamental to our development process, making quickly accessible, accurate data integral to the business. Our testing program generates terabytes of data as we validate motor performance and safety. Engineers need access to both near real-time and historical data, supporting everything from millisecond-level debugging to fleet-wide trend analysis. For FAA certification, we must preserve raw data indefinitely, and the pipeline architecture itself must be simple and explainable to regulators.

We needed a solution our cross-functional team could own, understand, and modify. A Python-based pipeline built on open-source tools aligned perfectly with BETA's collaborative culture, allowing one language to unify our entire data stack from edge to cloud.

Methods

We built an end-to-end pipeline entirely in Python, integrating established open-source tools with modern cloud data infrastructure. Our entire AWS infrastructure is provisioned and managed using CDK, ensuring our infrastructure is as maintainable and version controlled as our application code.

Data Ingestion:
Custom Python decoders parse CAN (Controller Area Network) log formats from test stands. Decode specifications vary frequently as we iterate on motor designs, so we built CLI tools that allow test conductors to push new decode files whenever needed (sometimes 25 times per week!), ensuring the pipeline adapts to evolving requirements without data engineering intervention.

Raw Storage:
We leverage Apache Iceberg as our lakehouse format on AWS S3, using PySpark for writes. Iceberg provides schema evolution, time travel, and hidden partitioning, which is essential for managing growing datasets while maintaining data quality. This layer preserves tall-format, full-fidelity data indefinitely. Engineers can access any signal on demand, easily add or remove instrumentation as testing needs evolve, and the transparent storage architecture is easily explainable to the FAA for certification purposes.

Transformation:
Dbt defines SQL transformations that aggregate raw time series data and event metadata into well-defined, consistent data mart schemas and custom views. Its testing framework ensures data quality, the open-source Python library sqlfluff handles SQL linting, and its documentation features maintain living documentation directly in the code.

Analytics Layer:
Transformed data lands in Amazon Redshift, optimized for fast queries and ready for analysis. This data mart serves multiple downstream uses: engineers perform ad-hoc analysis in Python; automated Python scripts generate derived insights that are written back to our data platform and stored alongside observed data; and a dedicated time tracking database stores aggregated test hours by component and operating condition, critical for FAA testing requirements.

We use Grafana, an open-source visualization tool that connects to any data source, seamlessly unifying our many databases into a single visualization layer. We've built a library of reusable Grafana dashboards that make stored data immediately accessible, serving analytics across the organization and transforming data into actionable insights for day-to-day decision making.

Orchestration:
Apache Airflow DAGs coordinate the entire pipeline through a mix of scheduled and event-driven jobs, with custom operators written in Python for our specific workflow needs.

Results

The pipeline processes half a terabyte of test data monthly, supporting 50+ engineers across multiple test stands and development programs. Data access that previously required hours of manual data extraction now completes in seconds, allowing engineers to spend less time hunting for data and more time iterating on designs.

The best outcome: learnings from this project extend well beyond this single pipeline. By building reusable BETA-specific CDK constructs and establishing common architectural patterns, we've created a platform that accelerates development across all our data sources, from manufacturing sensors to flight test telemetry. This unified approach reduces development time and makes our entire data ecosystem more consistent and maintainable.