Interactive Human-in-the-loop Coordination of Manipulation Skills Learned from Demonstration

Learning from demonstration (LfD) provides a fast, intuitive and efficient framework to program robot skills, which has gained growing interest both in research and industrial applications. Most complex manipulation tasks are long-term and involve a set of skill primitives. Thus it is crucial to have a reliable coordination scheme that selects the correct sequence of skill primitive and the correct parameters for each skill, under various scenarios. Instead of relying on a precise simulator, this work proposes a human-in-the-loop coordination framework for LfD skills that: builds parameterized skill models from kinesthetic demonstrations; constructs a geometric task network (GTN) on-the-fly from human instructions; learns a hierarchical control policy incrementally during execution. This framework can reduce significantly the manual design efforts, while improving the adaptability to new scenes. We show on a 7-DoF robotic manipulator that the proposed approach can teach complex industrial tasks such as bin sorting and assembly in less than 30 minutes.