Towards Mixed Reality AI Docents: Egocentric Smart Glasses with Vision and LLM Interaction

초록

We present a novel AI-driven docent framework as a foundational step toward Mixed Reality (MR) museum interaction. Traditional docent systems, such as QR-based or audio devices, offer limited, one-way experiences that require active device handling. In contrast, our system enables egocentric and bidirectional interaction using smart glasses equipped with a camera, microphone, and speaker, integrated with BLE-based localization, image recognition, and large language models (LLMs) for contextual dialogue. This demonstration implements and evaluates two core components of our envisioned MR docent system: spatial awareness and conversational intelligence. While augmented reality (AR) display and visual wayfinding remain future directions, the current prototype supports hands-free, context-aware, and personalized interaction with nearby artworks while minimizing smartphone operation. We conducted a preliminary user study with 16 participants at a museum in Seoul, Korea. Results indicated high user satisfaction, particularly in the accuracy of image-based recognition, the usefulness of AI explanations, and the seamless integration of spatial context. Reported usability ratings averaged 6.5/7 across key metrics such as goal achievement, information clarity, and interface intuitiveness. © 2025 IEEE.

키워드