Unspoken overhead of the Inverted Index
2024-05-07 , Asgabat

Inverted Index - a fundamental data structure behind full-text search. However, everything comes with its trade-offs - this presentation will show you when it makes sense not using it while still delivering the ultimate search experience.


In this presentation I'd like to explore the technical fundamentals of Inverted Index. We'll see where it shines and where its biggest trade-offs are. We’re going to explore usage patterns, where full-text search doesn’t work best, and how one can optimize OpenSearch for the ultimate experience. We’ll dive into structures like keywords, match-only text fields or doc values.

This rather entry-level talk will focus on technical fundamentals and necessary implementation details. The audience is going to build intuition immensely useful for designing their data structures.

Today, dozens of companies struggle escaping from OpenSearch to columnar-based SQL warehouses. Primary concern here being of course the cost. Understanding how one can optimize their setup first will help squeezing the most from their existing resources.

Seasoned software engineer and architect with strong systems design and operations background.
Experienced in highly-scalable, globally distributed environments. Currently Founding Engineer at Quesma.