2026-07-15 –, Thomas Swain Room
Adapt v0.1 is a real-time, reproducible data-analysis framework developed to support adaptive radar scanning within the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) facility. It implements a declarative, store-driven architecture that separates acquisition, processing, and visualization into independent, thread-safe components. A continuous ingestion worker registers incoming radar data, while processing workers poll a central DataStore for newly available items and execute configured analysis chains. Visualization and external systems interact only with completed outputs, preventing interference with internal logic. The framework is built on the Scientific Python ecosystem, including Py-ART, Xarray, Scikit-learn, OpenCV, and SciPy, and is designed for maintainability and extensibility through well-defined input–output protocols.
Adaptive radar scanning enables real-time response to evolving convective storms, overcoming limitations of fixed, omnidirectional volume scans that often miss rapid microphysical transitions. Because radar beam physics constrains full-volume update rates, dynamically focusing on sectors of interest can significantly improve temporal resolution. Achieving this requires low-latency analysis, forecasting, and decision support integrated directly into operational workflows. While legacy systems such as TITAN demonstrated real-time storm tracking decades ago, most modern Python-based radar and tracking tools were designed for offline analysis. Campaign-driven ARM operations require continuous ingestion, event-driven execution, streaming outputs, flexible configuration, and robust integration with operational infrastructure. Adapt addresses these needs through a lightweight, modular design that cleanly separates orchestration, scientific logic, and downstream consumers.
The architecture consists of three loosely coupled layers. The scientific layer contains deterministic modules for detection, analysis, projection, and tracking that operate on structured inputs and produce explicit outputs. The orchestration layer manages item lifecycles, scheduling, and metadata state transitions including creation, queuing, processing, completion, or failure, enabling recoverability and preventing race conditions. The data access layer provides a client abstraction over the repository so downstream systems query structured metadata rather than raw files. Configuration files and CLI arguments define algorithm selection, runtime parameters, radar sources, and product definitions, supporting campaign-specific objectives.
To prevent silent numerical corruption, Adapt enforces algorithm contracts that validate outputs immediately after execution. Segmentation products are checked for dimensional consistency, contiguous labeling, and mask integrity; projection products are verified for spatial alignment, finite motion vectors, and forecast horizon consistency; analytical outputs undergo schema and metadata validation. Violations halt processing for that item and record diagnostic state in the catalog, ensuring fail-fast behavior and reproducible debugging.
The processing pipeline operates as an external script transitioning toward modular CLI tools. A downloader thread monitors configured sources and constructs items containing scan metadata, input paths, and expected outputs. Processor threads consume queued items, resolve dependencies through the catalog, execute scientific modules, validate outputs, write results atomically, and update state. Threads communicate exclusively through queues without shared mutable state, and algorithm modules remain stateless. The orchestrator supervises queue depth and dependency conditions without directly controlling thread execution.
Multidimensional grids are stored in NetCDF, while tabular analysis and tracking outputs use Parquet for efficient columnar access. Partitioned directory structures enable scalable time-range queries. A metadata catalog records radar inventories, processing runs, product definitions, and lineage relationships. A data client supports batch queries and streaming mode, polling for newly completed products so dashboards can visualize segmentation masks, projected motion, and lifecycle metrics without disrupting active processing. Each execution is registered as a uniquely identified run storing configuration, radar selection, and product relationships, enabling deterministic replay of historical datasets using the same logic as real-time operation.
Xarray provides labeled multidimensional data structures that preserve spatial coordinates and metadata, preventing index misalignment common in raw array workflows. Pydantic enforces strict configuration schemas and validates runtime parameters before execution. Dense motion fields are estimated using OpenCV’s Farnebäck optical flow on consecutive reflectivity frames, and cell geometries are derived using SciPy spatial triangulation methods. Py-ART provides Level-II decoding, coordinate transforms, and radar-specific processing foundations.
Adapt remains in an alpha stage. Key development priorities include stronger dataset versioning and provenance tracking within the repository layer, improved support for concurrent reads during active writes, exploration of structured streaming and event-driven orchestration models, and development of interactive dashboards for operational visualization. Future work will also address containerized and distributed deployment for cloud-native scalability and object-storage–first architectures. The modular separation between orchestration, scientific computation, and data APIs allows independent evolution of components and invites community contributions in data management, streaming frameworks, visualization systems, distributed execution, and reproducibility practices.
In summary, Adapt provides a modular, real-time architecture for adaptive radar scanning that enforces deterministic state management, contract-based validation, and repository abstraction. By eliminating thread entanglement and clearly separating system layers, it supports both historical reprocessing and operational guidance for live adaptive radar campaigns.
Bhupendra A. Raut is a Computational Environmental Scientist at Argonne National Laboratory, where his research focuses on the analysis of clouds and precipitation in large-scale remote sensing datasets and numerical model outputs. He has developed convection identification, tracking, and analysis algorithms and applied various clustering and machine learning methods to produce value-added products for multi-platform observational campaigns. Currently, he is co-developing an adaptive sensing framework for the U.S. Department of Energy’s Atmospheric Radiation Measurement (ARM) user facility and leads the Chicago Urban Flux Network.
His interdisciplinary research leverages statistics, computer vision, machine learning, and edge computing. An active member of the scientific community, he contributes to several prominent open-source atmospheric tools, including Py-ART, TINT, and tobac. Dr. Raut holds a Ph.D. and an M.Sc. in Atmospheric and Space Sciences from the University of Pune, following a B.Sc. from J. B. College of Science.