Multi-colour sketch-based image retrieval with an explicable feature embedding

Authors: Wang, S., Xia, Y., Xiang, N., Qian, K., Yang, X., You, L. and Zhang, J.

Journal: Engineering Applications of Artificial Intelligence

Volume: 135

ISSN: 0952-1976

DOI: 10.1016/j.engappai.2024.108757

Abstract:

Sketch-based image retrieval (SBIR) is a cross-domain matching problem that has been gaining continuous attention in the computer vision community. Currently, SBIR techniques mainly focus on dealing with black-and-white sketches and ignore the utilization of multi-colour information of images. This leads to insufficient retrieval performance since they cannot distinguish between images that have the same shape but different colours. To address this problem, a multi-colour sketch-based image retrieval (MCSBIR) method using a two-stage network architecture is proposed. A novel feature embedding for explicably describing the shape and colour information is designed. A triplet loss function is developed to learn the feature embedding, in which a new distance metric is proposed to separate the shape and colour features. In addition, the first multi-colour sketch-image dataset is built to achieve the MCSBIR task and a user interface is designed to visually present the MCSBIR method. The effectiveness of the MCSBIR method is demonstrated by comprehensive experiments.

Source: Scopus