2026-03-06 –, Tutorial on Automation with Image Recognition Libraries - SikuliLibrary (and ImageHorizonLibrary)
This tutorial, is about using image recognition libraries to automate tasks or testing, when it is costly or difficult to obtain object identifiers in the applications under test.
We will use the libraries SikuliLibrary and ImageHorizonLibrary to automate applications we do not know about their internal components. This is what is called Black Box Testing.
We will practice automating a Login in a VMWare/VirtualBox Windows system and then doing some actions.
Contents:
- About Image Recognition Libraries - SikuliLibrary (and ImageHorizonLibrary)
- Knowing the Java based SikuliX IDE and its possibility to run Robot Framework test cases.
- SikuliLibrary:
-- Installation
-- Planning the Test Suites file structure
-- Defining Test Cases and Resources
-- Running Test Suites
- Combining SikuliLibrary keywords with ImageHorizonLibrary
- Practice in automating a Login in a VMWare/VirtualBox Windows system and then doing some actions.
About:
Image recognition libraries are used to automate tasks or testing, when it is costly or difficult to obtain object identifiers in the applications under test. These libraries use Computer Vision (OpenCV) to match reference images with a copy of the computer screen, and also Optical Character Recognition (OCR) for text extraction. With these techniques and operation system actions like mouse movement and keyboard strokes, the system can replicate the actions of the human user.
SikuliLibrary is a Robot Framework library that allows to use the SikuliX Java API. It uses Robot Framework Remote to interface Python functions with the SikuliX Java libraries, so it needs to have Java Runtime Environment installed in your system.
-- diagram from project: https://github.com/MarketSquare/robotframework-SikuliLibrary/blob/master/docs/img/architecture.png --
The usual workflow for a Test Case or Task is:
- Import SikuliLibrary and start its server
- Define the location for the reference images
- Start the Application Under Test (AUT)
- Interact with the AUT by actions of mouse, keyboard,
matching of reference images on the screen,
and Optical Character Recognition (OCR) for text extraction.
- Complete the workflow by stopping the server.
SikuliX IDE:
- Installation SikuliX IDE, which requires Java
- Creating and Running a Test Case with SikuliX IDE
SikuliLibrary:
- Installation
- Planning the Test Suites file structure
- Defining Test Cases and Resources
- Running Test Suites
ImageHorizonLibrary is a Robot Framework library, based on pyautogui and other Python modules, and optionally opencv-python for adjusting the image recognition precision. This library does not have Optical Character Recognition (OCR) keywords. Similarly to SikuliLibrary, it uses reference images to interact with the AUT on the screen. We can say that the usual workflow is the same as the one with SikuliLibrary, except for the server and OCR parts.
Combining SikuliLibrary keywords with ImageHorizonLibrary:
-- Installation of ImageHorizonLibrary
-- Adjusting Test Suites to use SikuliLibrary and ImageHorizonLibrary simultaneously (conflicting keyword names)
-- Running Test Suites
Practice in automating a Login in a VMWare/VirtualBox Windows system and then doing some actions.
Participants will learn about Image Recognition libraries, and how to use them for Black Box testing. Practicing in automating a RDP Windows connection, doing Login and executing some actions.
Categorize / Tags:libraries, automation technology, Computer Vision, OCR, Black Box Testing
Is this suitable for ..?: Beginner RF User, Intermediate RF User, Advanced RF User Describe your intended audience:Any Software Tester can take advantage of using Image Recognition libraries for testing or automating in graphical environments. Knowing about Black Box testing may uncover solutions for challenges the Software Tester may encounter in their activities.
Hélio Guilherme is an experienced Software Tester since 2008 when he first had contact with Robot Framework at Nokia Networks in Lisbon, Portugal.
During his work activities he used all the Robot Framework internal libraries, and other libraries like: SikuliLibrary, SSHLibrary, SeleniumLibrary, SwingLibrary, Browser, RequestsLibrary and AppiumLibrary.
He is currently the lead developer and maintainer of RIDE (https://github.com/robotframework/RIDE/) and maintainer of SikuliLibrary (https://marketsquare.github.io/robotframework-SikuliLibrary/). He says he does not know if "he is a Software Tester who likes to do Software Development, or a Software Developer who likes to Software Testing". Professionally, he is DevOps and QA Engineer at LOAD in Aveiro, Portugal (https://load.digital/).