2026-02-12 –, RoboCon
Learn how I sped up the Robot Framework’s test suite to find ⅔ of the bugs in ¼ of the time!
We want to run our tests as often and early as possible, so we immediately notice when we break something. However, many teams can't test as often as they'd like because their tests take hours or even days to run.
Innovative testing methods can identify most errors with just a fraction of test execution time, thereby significantly accelerating our testing. I’ll show you how to use AI to find most bugs in a fraction of the test runtime. With this we can give feedback on new bugs much more frequently.
Learn how I used mutation testing to introduce hundreds of bugs into the Robot Framework’s own code and how I applied an AI-based testing approach to the Robot Framework’s test suite to find ⅔ of these bugs in ¼ of the time.
Running tests as often and as early as possible is the dream of many agile testers. Ideally, after every commit and on all branches, so that we immediately notice when we break something.
But what if my tests take hours or even days? For many, the dream of an accelerated testing process seems unattainable or at least impractical.
However, research shows a possible solution: One approach to providing quick feedback even with slow tests is to run a small subset that is fast enough. This is worthwhile if this subset finds a majority of the defects in a fraction of the time. For example, 80% of defects in 10% of the time it takes to execute all tests. We need innovative methods to accomplish this, but they also need to be practically feasible.
In this presentation, I introduce an approach that can be implemented with little effort in existing projects to uncover most defects with minimal testing effort and without changing anything about your tests!
The method uses large language models (AI) and clustering to create an effective smoke test suite. This can be used for arbitrary changes, to identify defects across the entire code base with minimal testing effort. Thus, providing quick feedback on new bugs.
I’ll present the fundamentals, explain how it works and show research results about the effectiveness of the technique.
- Even with very large E2E test suites (hours/days to run), it’s possible to give really fast feedback about new bugs, e.g. in every pull request!
- We will cover current research into how to speed up long-running test suites
- Modern test selection techniques are easy to set up, even in legacy software projects, and effective
- I will show a practical application of the technique on the Robot Framework’s own tests
Anyone with a test suite that takes more than a few minutes to run.
Anyone who wants faster feedback from their tests when new bugs are introduced.
Anyone who wants to run ALL tests for every commit but can't, because tests take too long.
in-person
Fabian Streitel studied computer science at TU Munich and has been working for over 10 years to increase the impact of test automation for his customers. Over the years, Fabian has seen the testing processes of hundreds of companies and has helped improve them. He leads the Test Intelligence team and all research in that area at CQSE.