2026-07-15 –, University Hall
In the past year Xarray has seen increased usage across various sub-fields of biology, revealing interesting challenges. It can be difficult to determine the best way to represent a data structure (e.g. anndata, NGFF-Zarr) as an Xarray object. Furthermore, some use cases such as whole brain imaging require the use of lesser known Xarray features such as custom indexes.
In this talk I will showcase examples of how to encode common biological data structures as Xarray objects. Finally, I will demonstrate how the custom index infrastructure has expanded what types of data can be usefully encoded in Xarray.
Background
Biological datasets come in a wide variety of shapes, sizes, and types. However, there are common challenges faced across biology when dealing with complex structured data, such as keeping track of real-world coordinates. Xarray provides a powerful solution to these issues. Additionally, Xarray provides first class support for HDF and Zarr files, formats already in wide use in biology.
Issues
Increased usage in various projects has revealed issues around converting existing data structures into Xarray. For example some Napari developers use Xarray to keep track of physical units from images, but they struggled with the fact that various libraries had different conventions for encoding metadata into Xarray.
That struggle is exemplary of a larger issue: The best way to convert an existing data structure (on disk or in memory) to Xarray may not be obvious, especially for newer users of Xarray. Or it is possible to be unaware of functionality (e.g. Custom Indexes) necessary to fully represent a data structure.
Success Stories
These conversion difficulties are solvable.
I will present three examples of successful conversion of common biological data structures to Xarray. Through these I will discuss, what worked, what was hard, and recommendations for anyone interested in using Xarray for biology.
- Microscope Images: OME-Zarr (NGFF)
- Omics Data: AnnData
- Multimodal data (Single cell Raman Spectroscopy + Microscope Images + Lipidomics)
Indexes
A key enabling technology to allow some biological data structures to be represented in Xarray is the ability to write custom indexes. Custom indexes are powerful tools that can also encode complex interconnected relationships in metadata data structures and allow sophisticated selection queries. However they are not yet well known in the community.
To showcase their use I will demonstrate the indexes developed for a real world use case of combined speech and intracranial EEG data. These indexes also show the benefits of cross field collaboration as they are useful in non-biological applications as well.
Xarray also has newly built-in Indexes built using the custom index infrastructure. These indexes allow for opening huge data sets, such as whole brain images, which would previously have resulted in out of memory errors. I will show how these indexes enable opening a sectioned brain image in Xarray.
Conclusion
To conclude I will summarize the advice on how to convert a biological data structure into an Xarray object, and how to fully leverage Xarray’s functionality.
This will include how to think through:
- How metadata maps to Xarray
- What kinds of selection queries you need
- The practicalities of data loading
Finally, and most importantly, advice on how to do this as a community, and where to get help.
Context
Blog posts:
https://xarray.dev/blog/xarray-napari-plan
https://xarray.dev/blog/flexible-indexing
https://xarray.dev/blog/xarray-biology
Prior SciPy Talk about Xarray and Biology:
https://www.youtube.com/watch?v=ujOseM1Bk1g
That talk focused on introducing the idea of Xarray - this talk is more concrete with examples and advice on loading data into xarray and what to do with it once there.
Speaker
I am a multimodal-microscopist who has since branched out to support multiple areas of Biology in my role as the Xarray Community Developer where I focus on ensuring Xarray has the tools biologists need and educating biologists about how Xarray might be useful for them.
I am working as an Xarray community developer at Earthmover. In this role I am focused on improving Xarray’s support and documentation for the biology/biomedical community. Prior to this I completed my PhD in which I extensively used Xarray, zarr and the Pydata stack to implement custom microscope control software and analyze multimodal timelapse single cell microscopy data. I loved the open source scientific software so much that now I get to work full time improving it and sharing it with others.