Image Retrieval consists in searching in a large database for the most similar images to one or more query images. It has many applications in various fields, e.g., to validate whether a person's photo is contained in your database of people's photos; to build a visual recommendation system; or to create a video deduplication system.

Huge progress in Computer Vision in the deep learning era highlighted Content-based Image Retrieval (CBIR) techniques that use the image contents (features, colors, shapes, etc) rather than metadata (keywords, tags). This gets rid of time-consuming, costly and error-prone human annotations to produce the metadata.

A classic CBIR approach consists of three steps:

1. A deep neural network called the feature extractor (typically a CNN, or a ViT) computes a representation of each image of the database in the form of an embedding vector.
2. The same feature extractor is used to compute an embedding of a query image.
3. The search is performed by retrieving the closest representations in this vector space using a distance metric (cosine, L1, or more complex ones).

Thereafter, two main challenges arise:

• Quality of image representations - the embeddings should capture the visual features that are relevant to your searches/tasks. For instance, if you intend to do face recognition, embeddings should encode eye/hair color, skin texture, nose position, etc.
  Traditionally, the feature extractor is trained in a supervised way. Therefore, the relevance of the representations hugely depends on 1) how close is the training dataset from the searched query images 2) the potential visual biases in the annotations (see a famous example here).
• Speed of search in the representation space - comparing each query image to every single image in the searched database in near real-time is challenging and expensive with large datasets.

In this talk, we will build a Visual Search Engine:

• We will introduce Self-Supervised Learning (SSL) in the context of computer vision and the data2vec approach. Labelling data can be a time-consuming and expensive process, especially if it requires specialized knowledge or expertise. SSL does not require labelled training data to learn good representations, hence it allows to lower the cost and time to build a model producing good representations for our visual search engine.
• As a concrete example for this talk, we will use the DINO's SSL method to build a feature extractor.
• We will compare the DINO feature extractor with supervised pre-trained feature extractors. We will show the main differences between the obtained representations: SSL ones are generally richer (more visual features are in the representation) whereas supervised learning introduces a natural semantic bias in the representations. In addition, we will present practical tools to understand the visual features encoded in the embeddings (activation maps, grad-cams, self-attention maps for transformers).
• We will present Milvus, a vector database built for scalable similarity search: it’s an open-source search engine tool (14.5k stars on Github) that is suitable for production use cases as it can be easily scaled and managed. Milvus uses Approximate Nearest Neighbors (ANN) methods to build vector indexes that improve retrieval efficiency by sacrificing accuracy within an acceptable range.
• We will use the Milvus Python API to index the image representation vectors: as a result, the images the most similar to a query image can be retrieved in a split second, even for datasets containing millions of vectors.

By the end of the session, participants will have learned how to build a Visual Search Engine using Milvus with pre-trained self-supervised and supervised models.