Have you ever put JSON into a string column and thought there HAS to be a better way?
Well now there is! The Parquet community has ratified the Variant type, a native binary encoding for semi-structured data with this one clever trick called shredding that lets columnar storage do what it does best, even on your messiest nested data. Variant stores field names and offsets separately from values or if they show up consistently enough as a regular column. This makes an otherwise unwieldy JSON blob much more easily optimised. You get the speed of a fixed schema for the parts that are predictable, and the flexibility of schemaless for the parts that aren't.
We know that picking column types can sometimes feel overwhelming, so we collected benchmarks for you and we'll show you why projects like Arrow, Spark, Iceberg, Delta Lake and DuckDB are all rushing to adopt it. Stop parsing those messy JSONs and don't store JSON blobs anymore!