Type or swipe? Text input in virtual reality

Until now, inputting text in virtual spaces – when searching for a digital exhibition or using a business application, for instance – meant typing in every single letter individually on a virtual keyboard. Longer texts are often more difficult to input, however, as compared with a traditional computer keyboard, there is no haptic feedback in virtual reality and the movements of a finger can’t be captured with precision.

Florian Spiess, a doctoral candidate at the Department of Mathematics and Computer Science, and his team led by Professor Heiko Schuldt have developed a new method for inputting text in virtual spaces. With the help of a swipe keyboard, words can be entered with an effortless swipe gesture. The results were presented by the researchers at the IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

For a long time now, smartphone keyboards have offered the option of swiping from one letter to the next instead of tapping each individual letter to generate the desired word. This swipe function was harnessed by the researchers.

In virtual reality, users equipped with VR goggles see a swipe keyboard in front of them. Using the controller, they are able to connect the letters they want in a swipe gesture to form a word, drawing a line in three-dimensional space. The shape of this curve, which links together all of the individual letters, varies depending on the word. To type the word “hat” on a QWERTY keyboard, the hand moves from the “H” in the middle of the keyboard to the “A” on the left and back to the middle for the “T”.

The researchers were able to calculate the perfect graphs for a data set of the 100,000 most frequently used words in the English language. “We have to remember, though, that humans aren’t perfect,” explains Spiess. “In reality, the lines aren’t always as straight as the ones the computer calculates. So we need methods that take these inaccuracies into account.”

The system therefore identifies the entered word by comparing the swipe gesture with the calculated graph, as well as by registering the real position on the keyboard. For example, if the controller is located on the left side of the keyboard for a word’s initial letter, the program excludes all words beginning with letters on the right side of the keyboard.

The word calculated by the system with these two filters is displayed in an output window (see video). Up to four alternatives are also offered above the keyboard. With the word “to,” for example, the program also suggests “too” as an option since the graph for both words is identical.

User-friendly solution

In a preliminary user study, participants gave the new input format a positive rating for user-friendliness; however, at 13 words per minute, the speed of the swipe keyboard is currently only average compared with other text input methods. Accordingly, the researchers are planning to make improvements to the keyboard, for instance by adding an integrated spellchecker.

The swipe keyboard is already available free of charge on GitHub, a website for software development projects. “In the Department of Mathematics and Computer Science, we rely heavily on open source. I’ve spent a lot of time searching for a simple way of inputting text myself. We wanted to develop an application for ourselves, while at the same time finding a solution that everyone could benefit from,” says Spiess.

Original publication

Florian Spiess et. al.
Direct Interaction Word-Gesture Text Input in Virtual Reality
IEEE International Conference on Artificial Intelligence and Virtual Reality (2022), doi: 10.1109/AIVR56993.2022.00028.