Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bidirectional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to "affect" each other through physical actuation and digital computation. In the existing AR sketching, the relationship between virtual and physical worlds is only one-directional -- while physical interaction can affect virtual sketches, virtual sketches have no return effect on the physical objects or environment. In contrast, bi-directional sketching interaction allows the seamless coupling between sketches and actuated TUIs. In this paper, we employ tabletop-size small robots (Sony Toio) and an iPad-based AR sketching tool to demonstrate the concept. In our system, virtual sketches drawn and simulated on an iPad (e.g., lines, walls, pendulums, and springs) can move, actuate, collide, and constrain physical Toio robots, as if virtual sketches and the physical objects exist in the same space through seamless coupling between AR and robot motion. This paper contributes a set of novel interactions and a design space of bi-directional AR sketching. We demonstrate a series of potential applications, such as tangible physics education, explorable mechanism, tangible gaming for children, and in-situ robot programming via sketching.

Swarm Fabrication: Reconfigurable 3D Printers and Drawing Plotters Made of Swarm Robots

We introduce Swarm Fabrication, a novel concept of creating on-demand, scalable, and reconfigurable fabrication machines made of swarm robots. We present ways to construct an element of fabrication machines, such as motors, elevator, table, feeder, and extruder, by leveraging toio robots and 3D printed attachments. By combining these elements, we demonstrate constructing a X-Y-Z plotter with multiple toio robots, which can be used for drawing plotters and 3D printers. We also show the possibility to extend our idea to more general-purpose fabrication machines, which include 3D printers, CNC machining, foam cutters, line drawing devices, pick and place machines, 3D scanning, etc. Through this, we draw a future vision, where the swarm robots can construct a scalable and reconfigurable fabrication machines on-demand, which can be deployed anywhere the user wishes. We believe this fabrication technique will become a means of interactive and highly flexible fabrication in the future.