Скачать или смотреть GeMR: Multi-modal Interactive 3D Scene Composition in XR

This paper presents GeMR, a novel mixed-reality (MR) 3D authoring platform that enables real-time, interactive composition of 3D scenes in augmented and mixed reality environments. GeMR introduces two key innovations: a three-tier 3D asset generation workflow and a 3D sketch-to-shape retrieval algorithm.

The three-tier architecture addresses the traditionally time-consuming process of 3D asset creation by allowing users to interact with the generation pipeline at intermediate stages. This supports rapid validation and iterative refinement of user inputs, significantly accelerating the realization of imagined assets while enhancing interactivity.

Complementing this is our 3D sketch-to-shape retrieval system, which features an intuitive and robust sketching interface to query a library of both procedurally generated and artist-created 3D assets. By interpreting freehand 3D sketches, the system retrieves objects that closely match both the semantic category and geometric form of the user’s input, enabling precise and context-aware asset selection.

GeMR supports multimodal, intuitive input modalities—including speech, text, and freehand 3D sketching—making it accessible to both novice and expert users for creating, retrieving, modifying, and composing 3D assets and textures. By seamlessly integrating generative AI models with naturalistic interaction techniques, the platform serves as a flexible testbed for 3D content development and a powerful step toward democratizing 3D scene authoring.

We demonstrate GeMR’s effectiveness through diverse use cases, including scenario composition for human-robot collaboration. To evaluate its accessibility, we conducted a user study comparing 3D modeling experts and novices on a standardized scene creation task. Both groups reported equally high usability ratings and low perceived workload, underscoring the platform’s inclusive design. Our findings highlight that combining generative models with multimodal, intuitive inputs represents a significant leap forward in making 3D scene authoring more approachable, efficient, and broadly accessible.