Retrieval-Augmented Generation (RAG) chatbots are a key use case of GenAI in organizations, allowing users to conveniently access and query internal company data. A first RAG prototype can often be created in a matter of days. But why are the majority of prototypes still in the pilot stage? [1]

In this talk we share our insights from developing a production-grade chatbot at Merck. Our RAG chatbot for R&D experts accesses over 50,000 documents across numerous SharePoint sites and other sources. We identified three technical key success factors:
1. Building a robust data pipeline that syncs documents from source systems and that handles enterprise features such as replicating user permissions.
2. Developing a chatbot workflow from user question to answer with retrieval components such as hybrid search and reranking
3. Establishing a comprehensive evaluation framework with a clear optimization metric.

We think that many of these lessons are broadly applicable to RAG chatbots, making this talk valuable for practitioners aiming to implement GenAI solutions in business contexts.