2026-07-15 –, Thomas Swain Room
The advent of multimodal large language models (mLLMs) provides new opportunities for automated data exploration tasks on multi-petabyte atmospheric data sets. In this presentation, we present Nepho, a Python package for parallel mLLM prompting on collections of atmospheric quicklook data plots. We then evaluate the accuracy of several mLLMs in answering questions about Atmospheric Radiation Measurement (ARM) atmospheric datasets. We demonstrate that the GPT 4/5 and llama3-vision models were the most accurate models for quicklook plot exploration and recommend prompt engineering and retrieval-augmented generation for such data exploration workflows.
Atmospheric datasets, such as the U.S. Department of Energy Atmospheric Radiation Measurement Facility’s archive span several petabytes and decades. This makes exploring such datasets difficult for users that are interested in specific weather phenomena. However, multimodal LLMs such as GPT 5.0 now support basic analyses of atmospheric data plots. Given that quicklooks are available on ARM’s dqplotbrowser website for most of ARM’s instrument and value added product data, mLLMs present a potential new opportunity for automated data exploration using agents.
In this presentation, we present a feasibility study for using mLLMs for data exploration. In order to perform this study, we developed Nepho, a Python package that supports parallel mLLM inference of prompts on sets of quicklook plots. Nepho supports a wide variety of mLLMs using OpenAI, RESTful API, and ollama endpoints through a backend abstraction. Nepho encodes image timeseries into an embedding along with the prompt and performs inference of specific prompt-data plot pairs automatically for the user, making automated mLLM workflows easier on image collections. Nepho supports parallel inference for faster processing and therefore can scale to multiple processors.
Nepho was used for a feasibility study for using mLLMs to explore atmospheric datasets through quicklook plots. As a part of this effort, atmospheric scientists developed a testing dataset of 132 prompt-data plot-answer triplets from a wide array of atmospheric datasets. An example of such a triplet is shown in Figure 1. In this example, we use an mLLM to explore spikes in eddy correlation flux data from the ARM Southern Great Plains site. We provide the multiple choice question about the plot and then assess accuracy by comparing against human-generated answers about the plot. We evaluated 12 mLLMs in total. GPT-4.1 and GPT-5 provided the best accuracy, both around 68%. The best open source model performance we evaluated was llama3.2-vision:90b with 57.58% accuracy. This shows that, without any effort to provide domain-specific information to the mLLMs, that mLLMs have fair accuracy on answering multiple-choice questions for this testing dataset. Since we did not include any domain-specific information in our prompt, we recommend methods to increase the accuracy for specific datastreams by including domain-specific information through retrieval-augmented generation to improve accuracy.
Nepho has enabled other community efforts exploring the feasibility of mLLM-assisted data exploration. For example, the ARM Facility plans further feasibility studies on weather radar scene classification and exploration of data quality issues in atmospheric plots for the ARM Data Quality Office, incorporating these recommendations. LLM-Assisted Radar Scenes (LARS), a weather radar classification package based on Nepho, is already under development.
Bobby Jackson is an atmospheric scientist at Argonne National Laboratory. His research interests include radar meteorology, using AI and edge computing to improve atmospheric observations, and open source software development for the atmospheric sciences. He is a lead developer on PySP2 and PyDDA, two open source Python packages for aerosol and radar wind retrievals. In addition, he is a contributing developer to numerous packages in the Pangeo and Open Source Radar communities, including PyART.