3D estimation of single-view 2d images using shape priors and transfer learning

Authors: Shoukat, M.A., Sargano, A.B., You, L. and Habib, Z.

Journal: Multimedia Tools and Applications

eISSN: 1573-7721

ISSN: 1380-7501

DOI: 10.1007/s11042-023-17960-9

Abstract:

Humans possess a natural ability to infer the three-dimensional (3D) structure of a scene by leveraging prior knowledge and visual understanding. Conversely, computers face significant challenges in 3D reconstruction, which has long been a subject of interest in computer graphics. However, recent advances in computer vision have introduced a range of techniques aimed at addressing this problem. Despite these efforts, extracting the necessary information from two-dimensional (2D) images for accurate 3D reconstruction remains difficult due to complex object geometries, noisy backgrounds, and occlusions. Drawing inspiration from human visual perception, this study proposes a technique that utilizes transfer learning to acquire discriminative features. Additionally, it introduces a memory component designed to store information related to the category, shape, and geometry of similar objects. This memory component plays a crucial role in compensating for missing information in the scene. By employing an intelligent fusion mechanism that integrates relevant computer-aided design (CAD) models, the proposed approach enhances the estimation of an accurate generic 3D model for a given input image. This mechanism proves especially effective in scenarios where objects are occluded or situated within complex environments. Moreover, incorporating features of new object categories into the designed memory component expands the model’s capabilities to encompass those categories. To assess the performance of the proposed approach, a set of experiments is conducted on the ObjectNet3D dataset, which comprises 3D shapes precisely aligned with real-world images. These experiments confirm that the proposed method improves the results of state-of-the-art methods.

Source: Scopus