OrbTouch: using deformation as a medium for human-computer interaction

Home  /  Google Scholar  /  Github  /  LinkedIn

post-date: 09/06/2017

This mini-post explores the use of deformation as a medium for human computer interaction. The question being asked is the following: can we use the shape of an object, and how it changes in time, to encode information? Here I present a deep learning approach to this problem and show that we can use a balloon to control a computer. A detailed description of OrbTouch can be found in my recent paper entitled "OrbTouch: Recognizing Human Touch in Deformable Interfaces with Deep Neural Networks". The gist of this work is captured in the two videos below. In the first video, the real–time output from the software running on the embedded computer inside the controller. It uses convolutional neural networks to learn and then predict user-defined touch gestures. In the second movie, a user uses the gestures shown in Movie 1 to control a game of Tetris running on a host laptop. The controller communicates with its host via Bluetooth. Users can define their own arbitrary gestures and train OrbTouch to recognize them.