BEGIN:VCALENDAR VERSION:2.0 PRODID:-//pretalx//pretalx.com//scipy-2026//speaker//8DARNL BEGIN:VTIMEZONE TZID:CST BEGIN:STANDARD DTSTART:20001029T030000 RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10;UNTIL=20061029T080000Z TZNAME:CST TZOFFSETFROM:-0500 TZOFFSETTO:-0600 END:STANDARD BEGIN:STANDARD DTSTART:20071104T030000 RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=11 TZNAME:CST TZOFFSETFROM:-0500 TZOFFSETTO:-0600 END:STANDARD BEGIN:DAYLIGHT DTSTART:20000402T030000 RRULE:FREQ=YEARLY;BYDAY=1SU;BYMONTH=4;UNTIL=20060402T090000Z TZNAME:CDT TZOFFSETFROM:-0600 TZOFFSETTO:-0500 END:DAYLIGHT BEGIN:DAYLIGHT DTSTART:20070311T030000 RRULE:FREQ=YEARLY;BYDAY=2SU;BYMONTH=3 TZNAME:CDT TZOFFSETFROM:-0600 TZOFFSETTO:-0500 END:DAYLIGHT END:VTIMEZONE BEGIN:VEVENT UID:pretalx-scipy-2026-XZGSV7@pretalx.com DTSTART;TZID=CST:20260715T152500 DTEND;TZID=CST:20260715T155500 DESCRIPTION:_Adapt v0.1_ is a real-time\, reproducible data-analysis framew ork developed to support adaptive radar scanning within the U.S. Departmen t of Energy Atmospheric Radiation Measurement (ARM) facility. It implement s a declarative\, store-driven architecture that separates acquisition\, p rocessing\, and visualization into independent\, thread-safe components. A continuous ingestion worker registers incoming radar data\, while process ing workers poll a central DataStore for newly available items and execute configured analysis chains. Visualization and external systems interact o nly with completed outputs\, preventing interference with internal logic. The framework is built on the Scientific Python ecosystem\, including Py-A RT\, Xarray\, Scikit-learn\, OpenCV\, and SciPy\, and is designed for main tainability and extensibility through well-defined input–output protocol s.\n\nAdaptive radar scanning enables real-time response to evolving conve ctive storms\, overcoming limitations of fixed\, omnidirectional volume sc ans that often miss rapid microphysical transitions. Because radar beam ph ysics constrains full-volume update rates\, dynamically focusing on sector s of interest can significantly improve temporal resolution. Achieving thi s requires low-latency analysis\, forecasting\, and decision support integ rated directly into operational workflows. While legacy systems such as TI TAN demonstrated real-time storm tracking decades ago\, most modern Python -based radar and tracking tools were designed for offline analysis. Campai gn-driven ARM operations require continuous ingestion\, event-driven execu tion\, streaming outputs\, flexible configuration\, and robust integration with operational infrastructure. Adapt addresses these needs through a li ghtweight\, modular design that cleanly separates orchestration\, scientif ic logic\, and downstream consumers.\n\nThe architecture consists of three loosely coupled layers. The scientific layer contains deterministic modul es for detection\, analysis\, projection\, and tracking that operate on st ructured inputs and produce explicit outputs. The orchestration layer mana ges item lifecycles\, scheduling\, and metadata state transitions includin g creation\, queuing\, processing\, completion\, or failure\, enabling rec overability and preventing race conditions. The data access layer provides a client abstraction over the repository so downstream systems query stru ctured metadata rather than raw files. Configuration files and CLI argumen ts define algorithm selection\, runtime parameters\, radar sources\, and p roduct definitions\, supporting campaign-specific objectives.\n\nTo preven t silent numerical corruption\, Adapt enforces algorithm contracts that va lidate outputs immediately after execution. Segmentation products are chec ked for dimensional consistency\, contiguous labeling\, and mask integrity \; projection products are verified for spatial alignment\, finite motion vectors\, and forecast horizon consistency\; analytical outputs undergo sc hema and metadata validation. Violations halt processing for that item and record diagnostic state in the catalog\, ensuring fail-fast behavior and reproducible debugging.\n\nThe processing pipeline operates as an external script transitioning toward modular CLI tools. A downloader thread monito rs configured sources and constructs items containing scan metadata\, inpu t paths\, and expected outputs. Processor threads consume queued items\, r esolve dependencies through the catalog\, execute scientific modules\, val idate outputs\, write results atomically\, and update state. Threads commu nicate exclusively through queues without shared mutable state\, and algor ithm modules remain stateless. The orchestrator supervises queue depth and dependency conditions without directly controlling thread execution.\n\nM ultidimensional grids are stored in NetCDF\, while tabular analysis and tr acking outputs use Parquet for efficient columnar access. Partitioned dire ctory structures enable scalable time-range queries. A metadata catalog re cords radar inventories\, processing runs\, product definitions\, and line age relationships. A data client supports batch queries and streaming mode \, polling for newly completed products so dashboards can visualize segmen tation masks\, projected motion\, and lifecycle metrics without disrupting active processing. Each execution is registered as a uniquely identified run storing configuration\, radar selection\, and product relationships\, enabling deterministic replay of historical datasets using the same logic as real-time operation.\n\nXarray provides labeled multidimensional data s tructures that preserve spatial coordinates and metadata\, preventing inde x misalignment common in raw array workflows. Pydantic enforces strict con figuration schemas and validates runtime parameters before execution. Dens e motion fields are estimated using OpenCV’s Farnebäck optical flow on consecutive reflectivity frames\, and cell geometries are derived using Sc iPy spatial triangulation methods. Py-ART provides Level-II decoding\, coo rdinate transforms\, and radar-specific processing foundations.\n\nAdapt r emains in an alpha stage. Key development priorities include stronger data set versioning and provenance tracking within the repository layer\, impro ved support for concurrent reads during active writes\, exploration of str uctured streaming and event-driven orchestration models\, and development of interactive dashboards for operational visualization. Future work will also address containerized and distributed deployment for cloud-native sca lability and object-storage–first architectures. The modular separation between orchestration\, scientific computation\, and data APIs allows inde pendent evolution of components and invites community contributions in dat a management\, streaming frameworks\, visualization systems\, distributed execution\, and reproducibility practices.\n\nIn summary\, Adapt provides a modular\, real-time architecture for adaptive radar scanning that enforc es deterministic state management\, contract-based validation\, and reposi tory abstraction. By eliminating thread entanglement and clearly separatin g system layers\, it supports both historical reprocessing and operational guidance for live adaptive radar campaigns. DTSTAMP:20260715T021113Z LOCATION:Thomas Swain Room SUMMARY:Adapt: Prototyping a Real-Time\, Reproducible Data Analysis Framewo rk for Adaptive Radar Scanning - Bhupendra Raut URL:https://pretalx.com/scipy-2026/talk/XZGSV7/ END:VEVENT END:VCALENDAR