PyConDE & PyData Berlin 2024

Zarr is a data format for storing chunked, compressed N-dimensional arrays and is sponsored by NumFOCUS under their umbrella.

It is based on open-source technical specification and has implementations in several languages, with Zarr-Python being the most used.

Outline

First, I’d be talking about:

Understanding Zarr basics (5 mins.)

• What is Zarr, and how it works?
  • The inner workings of Zarr using illustrated graphics
• What is the Zarr Specification?
  • How is Zarr different when compared to other storage formats?

Then, I'll be talking about the new Zarr Specification V3 and its significant features:

What's new in Zarr Spec V3? (15 mins.)

• What is the motivation for the evolution of the specification?
  • High-latency storage → Better support for technologies, particularly systems with relatively high latency per operation, such as cloud object stores
  • Interoperability → Language-agnostic approach towards the new specification by slimming down the specification to achieve interoperability across major programming languages
• Major design updates
  • Greater flexibility in how groups and arrays are created
    • Support for implicit groups that do not have a metadata document but whose existence is implied by descendant nodes
  • Restructuring of the JSON metadata document and storage path in both arrays and groups
    • Why is the Zarr V3 metadata consolidated compared to the Zarr V2 metadata?
  • Explicit support for extensions via defined extension points and mechanisms
    • How do extensions allow the community to add innovative and cutting-edge features to help their specific use cases?
  • Chunk encoding and supported codecs for V3
    • How are chunks encoded into binary representation for storage in the store, using the chain of codecs specified by the codecs metadata field?
• ZEP Process
  • Need and origin of a community feedback process for the evolution of Zarr specification
  • Transformation from steering council governed to community-owned specification
  • Learnings when migrating from Spec V2 → Spec V3

Then, I’d be doing a hands-on session, which would cover the following:

Hands-on (5 mins.)

• Creating Zarr arrays and groups using Zarr-Python V3.0
  • Walk through of the new features (mentioned above)
• Demo of Sharding Codec extension
  • Creating a sharded array and group and showing how a large number of chunks can be grouped together into a single shard
• Looking under the hood
  • Use store functions to explain how your Zarr data is stored

I'd be closing the talk by:

Conclusion (5 mins.)

• Key takeaways
• How can you get involved?
• QnA

This talk aims to address an audience that works with large amounts of data and is looking for a transparent, open-source, reliable, cloud-optimised, and environmentally friendly format. Also, I’d like to invite anyone interested in the lessons I learned by maintaining the project throughout the years.

The tone of the talk is set to be informative, story-telling and fun.

Intermediate knowledge of Python and NumPy arrays is required for the attendees to attend this talk.

After this talk, you’d:

• understand the basics of Zarr and what's new in V3,
• using Zarr V3 for local and cloud storage,
• make an informed decision on what data format to use for your data

and also you'd:

• know why should you have a process for your project,
• have essential takeaways regarding when an OSS project transitions from a young to a mature stage