2026-07-17 –, Johnson Great Room
Tiled is a full-fledged data management service designed specifically to help scientists store, find, and access scientific data at scale easily.
The concept of data structures is the cornerstone of Tiled; it allows us to abstract the inherent diversity of various file formats and data storage types to a handful of scientifically meaningful representations: arrays, tables, nested hierarchies, and even awkward, ragged, and sparse arrays. Tiled provides a consistent API to such disparate datasets and naturally integrates with the SciPy ecosystem, including NumPy, pandas, xarray, Dask, and more. The users can slice, convert, and retrieve only the data they need, or even subscribe to live streams from external instruments and send updates to a dashboard. Importantly, Tiled supports operations with rich metadata – including search – making the data registered in Tiled discoverable and interactable with minimal overhead, by the human users and AI agents alike. Tiled runs equally well on a private laptop or in a large facility’s data center. Its built-in authentication and authorization mechanisms make the data access controllable and secure. Finally, Tiled is a fully open-source project developed under a multi-institutional governance model, which reflects our commitment to open science and the FAIR principles in scientific computing.
In this talk we will introduce Tiled’s architecture, demonstrate its most popular use cases using the native Python client, discuss deployment and integration strategies, and show how it can simplify practical scientific data workflows.
Motivation and Background
Scientific datasets, especially from large experimental facilities, are growing in size and complexity. Researchers increasingly face the challenges of securely sharing large volumes of data between institutions while juggling a multitude of storage formats. These obstacles emphasize the need for decoupling the storage infrastructure from the analytical workflows, so that scientists can focus on computation and interpretation rather than data plumbing. Traditional access patterns, where entire files are transferred and parsed locally, strain bandwidth, memory, and compute resources. The recent advent and widespread adoption of agentic workflows further highlight this problem: to operate efficiently, an AI agent often benefits from having a direct access to certain dataset slices enriched with metadata – a requirement, which is difficult to fulfil with the file-centric approach.
Tiled was developed within the synchrotron light source community (e.g., National Synchrotron Light Source II and other facilities using the Bluesky ecosystem) to address these challenges, but it is agnostic to the specifics of the application domain. The project aims to provide a unified, high-performance, feature-rich service that lets users interact with their data without making any considerations about the underlying storage formats and infrastructure. It lets users slice, search, and stream only the pieces of data they need – whether arrays, tables, or hierarchical datasets – and treat them as familiar NumPy or pandas objects.
What is Tiled?
Tiled’s core offering is a data access and management service with:
A web server that exposes structured datasets via HTTP APIs.
A Python client that seamlessly integrates with popular tools in scientific computing, such as NumPy, pandas, xarray, Dask, AwkwardArray, etc.; users of h5py or zarr, for example, would find Tiled’s interface familiar.
Support for multiple underlying data sources: filesystems, databases, remote servers, blob storages, or combinations thereof.
Efficient format transcoding and chunked data access: users can retrieve just the subset of data they need, reducing I/O and network costs.
An easily expandable set of supported storage formats (e.g. zarr, parquet, csv, hdf5, etc.) and extensible data structures beyond simple arrays and tables (e.g. sparse, awkward, and ragged arrays).
Integrated caching, both client-side and server-side, to accelerate repeated access and interactive exploration.
Streaming capabilities via WebSockets, enabling real-time data updates and interactive workflows, which is particularly valuable for live experiments, monitoring dashboards, and adaptive analysis pipelines.
Built-in authentication and authorization (authN/authZ) mechanisms, allowing deployments to enforce fine-grained access control. Tiled supports multiple authentication providers and role-based permissions, making it suitable for multi-user facilities, collaborative research groups, and cloud deployments.
Relevance to SciPy Community and Broader Audience
Even though Tiled has originated from and is used widely in the synchrotron light source community, the problems it solves are universal wherever large or complex datasets are involved, from genomics to environmental science to astronomy. Tiled fills a gap by offering a flexible, secure, and convenient data access abstraction layer that complements computational tools. Its integration with the SciPy ecosystem and standards makes it directly applicable to users in practically any scientific domain.
