Rediscovering DataFrames: 100× Analytics Without Leaving Pandas
You've hit the pandas ceiling: 10 GB DataFrames choke, you need vector search for embeddings, and reading from S3 takes longer than the query itself. The usual upgrades hurt — Spark rewrite, new ORM, separate vector DB. There's a shorter path: keep the pandas API, swap the engine underneath. Trade up to 100× faster analytics, native vector search, direct querying of S3 / Postgres / 40+ formats as SQL table functions, and time-series functions. Built on chDB — the in-process build of ClickHouse, which I created and lead at ClickHouse Inc. We'll cover the migration cheat-sheet, concrete latency numbers, honest tradeoffs vs DuckDB and native pandas, and close with a live build: a pandas-heavy data pipeline migrated column-by-column. Code published as an open repo.
このトピックに関するあなた自身の経験を教えてください
I created chDB — the in-process build of ClickHouse — and lead it as Technical Director at ClickHouse Inc. (acquired 2024, 2,600+ GitHub stars). The pandas/DataFrame story has been my main focus for 18 months: chDB v4 shipped pandas-compatible chdb.query() and zero-copy DataFrame interop. Earlier versions: EuroPython 2024, PythonAsia 2026, ClickHouse Singapore Meetup, ClickHouse 26.3 webinar. Open source contributor: ClickHouse, Jemalloc, K8s, Memcached, CockroachDB, Superset.
この発表を通じて、参加者とどんな議論が可能ですか？
I want to spark conversations on: (1) when keeping the pandas API matters more than picking the 'best' DataFrame library; (2) honest tradeoffs between in-process engines (chDB vs DuckDB vs Polars vs vanilla pandas); (3) what pandas idioms don't translate to SQL and what to do about it; (4) whether the future of Python data work is one engine that does everything, or thoughtful composition of specialized tools. I'd love to hear how attendees handle the same scaling pain in their work.
Auxten Wang is Technical Director at ClickHouse Inc. and the creator of chDB — the in-process build of ClickHouse, acquired by ClickHouse Inc. in 2024. Before ClickHouse, he was Principal Engineer at Shopee, Cofounder & CTO of CovenantSQL (a decentralized BFT SQL database), and RecSys & Infra Director at 4Paradigm. Based in Singapore. Co-organizes the OpenClaw meetup.