Neural-Based Cross-Modal Search and Retrieval of Artwork
Authors: Gong, Y., Cosma, G. and Finke, A.
Journal: 2023 IEEE Symposium Series on Computational Intelligence, SSCI 2023
Pages: 264-269
ISBN: 9781665430654
DOI: 10.1109/SSCI52147.2023.10371948
Abstract:Creating an intelligent search and retrieval system for artwork images, particularly paintings, is crucial for documenting cultural heritage, fostering wider public engagement, and advancing artistic analysis and interpretation. Visual-Semantic Embedding (VSE) networks are deep learning models used for information retrieval, which learn joint representations of textual and visual data, enabling 1) cross-modal search and retrieval tasks, such as image-to-text and text-to-image retrieval; and 2) relation-focused retrieval to capture entity relationships and provide more contextually relevant search results. Although VSE networks have played a significant role in cross-modal information retrieval, their application to painting datasets, such as ArtUK, remains unexplored. This paper introduces BoonArt, a VSE-based cross-modal search engine that allows users to search for images using textual queries, and to obtain textual descriptions along with the corresponding images when using image queries. The performance of BoonArt was evaluated using the ArtUK dataset. Experimental evaluations revealed that BoonArt achieved 97 % Recall@10 for image-to-text retrieval, and 97.4 % Recall@10 for text-to-image Retrieval. By bridging the gap between textual and visual modalities, BoonArt provides a much-improved search performance compared to traditional search engines, such as the one provided by the ArtUK website. BoonArt can be utilised to work with other artwork datasets.
Source: Scopus
Neural-based cross-modal search and retrieval of artwork
Authors: Gong, Y., Cosma, G. and Finke, A.
Conference: 2023 IEEE Symposium Series on Computational Intelligence (SSCI)
Dates: 5-8 December 2023
Source: Manual