Long-Term Planning Around Humans in Domestic Environments with 3D Scene Graphs

Long-term planning for robots operating in domestic environments poses unique challenges due to the interactions between humans, objects, and spaces. Recent advancements in trajectory planning have leveraged vision-language models (VLMs) to extract contextual information for robots operating in real-world environments. While these methods achieve satisfying performance, they do not explicitly model human activities. Such activities influence surrounding objects and reshape spatial constraints. This paper presents a novel approach to trajectory planning that integrates human preferences, activities, and spatial context through an enriched 3D scene graph (3DSG) representation. By incorporating activity-based relationships, our method captures the spatial impact of human actions, leading to more context-sensitive trajectory adaptation. Preliminary results demonstrate that our approach effectively assigns costs to spaces influenced by human activities, ensuring that the robot trajectory remains contextually appropriate and sensitive to the ongoing environment. This balance between task efficiency and social appropriateness enhances context-aware human-robot interactions in domestic settings. Future work includes implementing a full planning pipeline and conducting user studies to evaluate trajectory acceptability.

Streaming Network for Continual Learning of Object Relocations under Household Context Drifts

In most applications, robots need to adapt to new environments and be multi-functional without forgetting previous information. This requirement gains further importance in real-world scenarios where robots operate in coexistence with humans. In these complex environments, human actions inevitably lead to changes, requiring robots to adapt accordingly. To effectively address these dynamics, the concept of continual learning proves essential. It not only enables learning models to integrate new knowledge while preserving existing information but also facilitates the acquisition of insights from diverse contexts. This aspect is particularly relevant to the issue of context-switching, where robots must navigate and adapt to changing situational dynamics. Our approach introduces a novel approach to effectively tackle the problem of context drifts by designing a Streaming Graph Neural Network that incorporates both regularization and rehearsal techniques. Our Continual\_GTM model enables us to retain previous knowledge from different contexts, and it is more effective than traditional fine-tuning approaches. We evaluated the efficacy of Continual\_GTM in predicting human routines within household environments, leveraging spatio-temporal object dynamics across diverse scenarios.