2026-02-12 –, RoboCon
This talk, is about using image recognition libraries to automate tasks or testing, when it is costly or difficult to obtain object identifiers in the applications under test.
We will talk about the libraries SikuliLibrary and ImageHorizonLibrary to answer these questions:
What, Why, Where, When, and How to use them?
Contents:
- About Image Recognition Libraries - SikuliLibrary and ImageHorizonLibrary
- Future of SikuliLibrary
Image recognition libraries are used to automate tasks or testing, when it is costly or difficult to obtain object identifiers in the applications under test. These libraries use Computer Vision (OpenCV) to match reference images with a copy of the computer screen, and also Optical Character Recognition (OCR) for text extraction. With these techniques and operation system actions like mouse movement and keyboard strokes, the system can replicate the actions of the human user.
-
SikuliLibrary is a Robot Framework library that allows to use the SikuliX (http://sikulix.com/) Java API. It uses Robot Framework Remote to interface Python functions with the SikuliX Java libraries, so it needs to have Java Runtime Environment installed in your system.
-- diagram from project: https://github.com/MarketSquare/robotframework-SikuliLibrary/blob/master/docs/img/architecture.png --
The usual workflow for a Test Case or Task is:
- Import SikuliLibrary and start its server
- Define the location for the reference images
- Start the Application Under Test (AUT)
- Interact with the AUT by actions of mouse, keyboard,
matching of reference images on the screen,
and Optical Character Recognition (OCR) for text extraction.
- Complete the workflow by stopping the server.
SikuliLibrary defines 78 keywords, which can be groupped as: Configuration, Actions (Mouse, Keyboard), Assertions and Verifications, and Information.
SikuliLibrary is operating system agnostic, but care must be taken regarding the reference images resolution, which needs to be consistent for reproducibility of tests or tasks executions.
-
ImageHorizonLibrary is a Robot Framework library, based on pyautogui and other Python modules, and optionally opencv-python for adjusting the image recognition precision. This library does not have Optical Character Recognition (OCR) keywords. Similarly to SikuliLibrary, it uses reference images to interact with the AUT on the screen. We can say that the usual workflow is the same as the one with SikuliLibrary, except for the server and OCR parts.
ImageHorizonLibrary defines 34 keywords, organized like in SikuliLibrary.
Like SikuliLibrary, the library ImageHorizonLibrary is operating system agnostic, and care must be taken regarding the reference images resolution, which needs to be consistent for reproducibility of tests or tasks executions. One advantage when comparing with SikuliLibrary, is not needing to have a Java Runtime Environment.
-
The future development of SikuliLibrary, is dependent of the advance of the original Java project SikuliX, because its maintainer has suspended its development for now. However, Raimund Hocke, https://github.com/RaiMan, has prepared an integration of SikuliX fully in Python, named sikulix4python. This would make the use of the library easier and universal.
There is also the possibility to use keywords of ImageHorizonLibrary, because they complement SikuliLibrary, and the development of ImageHorizonLibrary is stale.
Participants will learn about Image Recognition libraries, and how to use them for Black Box testing.
Categorize / Tags:libraries, automation technology, Computer Vision, OCR, Black Box Testing
Is this suitable for ..?: Beginner RF User, Intermediate RF User, Advanced RF User Describe your intended audience:Any Software Tester can take advantage of using Image Recognition libraries for testing or automating in graphical environments. Knowing about Black Box testing may uncover solutions for challenges the Software Tester may encounter in their activities.
In-Person or Online talk/workshop?:I am available for In-Person and Online
Hélio Guilherme is an experienced Software Tester since 2008 when he first had contact with Robot Framework at Nokia Networks in Lisbon, Portugal.
During his work activities he used all the Robot Framework internal libraries, and other libraries like: SikuliLibrary, SSHLibrary, SeleniumLibrary, SwingLibrary, Browser, RequestsLibrary and AppiumLibrary.
He is currently the lead developer and maintainer of RIDE (https://github.com/robotframework/RIDE/) and maintainer of SikuliLibrary (https://marketsquare.github.io/robotframework-SikuliLibrary/). He says he does not know if "he is a Software Tester who likes to do Software Development, or a Software Developer who likes to Software Testing". Professionally, he is DevOps and QA Engineer at LOAD in Aveiro, Portugal (https://load.digital/).