2023-08-16 –, Aula
We love to use Python in our day jobs, but that enterprise database you run your ETL job against may have other ideas. It probably speaks SQL, because SQL is ubiquitous, it’s been around for a while, it’s standardized, and it’s concise.
But is it really standardized? And is it always concise? No!
Do we still need to use it? Probably!
What’s a data-person to do? String-templated SQL?
print(f”That way lies {{ m̴͕̰̻̏́ͅa̸̟̜͉͑d̵̨̫̑n̵̖̲̒͑̾e̸̘̼̭͌s̵͇̖̜̽s̸̢̲̖͗͌̏̊͜ }}”.)
Instead, come and learn about Ibis! It offers a dataframe-like interface to construct concise and composable queries and then executes them against a wide variety of backends (Postgres, DuckDB, Spark, Snowflake, BigQuery, you name it.).
Ibis is a pure Python library that lets you write Python to build up expressions that can be executed on a wide array of backends (SQLite, DuckDB, Postgres, Spark, Clickhouse, Snowflake, BigQuery, and more!). It offers a dataframe-like interface and helps you to write concise and composable interactive analytics code.
Have you ever had to translate a proof-of-concept from Pandas to PySpark to run on the “real data”?
Or download a huge parquet file because the upstream data is the result of 500 lines of dense SQL and you’re afraid to mess with it?
Have a love/hate relationship with SQL because it lets you get your job done, but think, there must be a better way?
Well, if you’re a data-engineer, data-scientist, data-hobbyist, or data-anything, come and join us for a tour of what Ibis can do for you!
Ibis: Because SQL is everywhere but you don't want to use it
Category [Data Science and Visualization] –Data Analysis and Data Engineering
Expected audience expertise: Domain –some
Expected audience expertise: Python –some
Public link to supporting material – Project Homepage / Git –
I'm fascinated by a variety of problems related to computers. I've solved hard problems in a variety of software engineering domains including digital video, Rust, systems programming, computer vision, and analytics. I'm currently helping build next generation Python analytics tooling at Voltron Data.