Context-Aware Mixed Reality: A Learning-Based Framework for Semantic-Level Interaction

Authors: Chen, L., Tang, W., John, N.W., Wan, T.R. and Zhang, J.J.

http://eprints.bournemouth.ac.uk/32880/

https://onlinelibrary.wiley.com/journal/14678659

Journal: Computer Graphics Forum

Publisher: Wiley-Blackwell

Mixed Reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic-based interactive MR framework that is beyond current geometry-based approaches, offering a step-change in generating high-level context-aware interactions. Our critical insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object-specific behaviors, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real-world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material-aware prototype system for context-aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real-time semantic level interactions.

This data was imported from Scopus:

Authors: Chen, L., Tang, W., John, N.W., Wan, T.R. and Zhang, J.J.

http://eprints.bournemouth.ac.uk/32880/

Journal: Computer Graphics Forum

eISSN: 1467-8659

ISSN: 0167-7055

DOI: 10.1111/cgf.13887

© 2019 The Authors. Computer Graphics Forum published by Eurographics - The European Association for Computer Graphics and John Wiley & Sons Ltd Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic-based interactive MR framework that is beyond current geometry-based approaches, offering a step change in generating high-level context-aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object-specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real-world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material-aware prototype system for context-aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real-time semantic-level interactions.

This data was imported from Web of Science (Lite):

Authors: Chen, L., Tang, W., John, N.W., Wan, T.R. and Zhang, J.J.

http://eprints.bournemouth.ac.uk/32880/

Journal: COMPUTER GRAPHICS FORUM

Volume: 39

Issue: 1

Pages: 484-496

eISSN: 1467-8659

ISSN: 0167-7055

DOI: 10.1111/cgf.13887

The data on this page was last updated at 05:24 on October 24, 2020.