Gloves that Translate Sign Language Into Text

Since 2012, several research teams have created gloves that can vocalize and translate sign languages into text. However, all these technologies have not succeeded in making the switch to industrial production; some only managed to transcribe a few words and others were weighed down by devices that were heavy and cumbersome or very expensive. The technology used in the design of the gloves presented in this article has solved the important problem faced by previous similar projects, namely hand movement variations from one person to another related to sign execution.

Researchers at the University of California in San Diego have created a glove that can transcribe American sign language into text in real time and control a virtual hand that imitates sign language gestures. The manufacturing cost of this glove is less than US $100. It is made of flexible electronic components that are inexpensive and available on the market.

In addition to transcribing sign language, the glove was created with other applications in mind, specifically as an accessory for augmented reality experience and for remote medical operations. It will allow a more intuitive immersion in virtual reality and will be more accurate than the joystick or mouse used for video games. It will also optimize virtual medical training sessions as it will accurately simulate the use of both hands.

Manufacturing and Operating the Prototype

To make the prototype, the team used leather sports gloves to which they attached nine stretchable sensors at the back, on the phalanges and metacarpals, two on each finger and one on the thumb. The sensors are made of thin silicon-based polymer strips coated with a thin layer of conductive carbon. They are attached to the glove with copper tape. A stainless steel wire connects each of the sensors to a printed circuit board attached to the back of the wrist.

The electrical resistance of the sensors changes depending on hand movements when closed or outstretched. The spatial positions of the nine phalanges and metacarpals enable transcription of the American sign language alphabet. A straight phalanx is coded “0” while a folded phalanx is coded “1”. The glove produces a nine-digit binary key for each letter. For example, the code for the letter “A” (straight thumb and all other fingers folded) is “011111111”, while the code for “B” (folded thumb and all other fingers straight) is “100000000”. The engineers equipped the glove with an accelerometer and a pressure sensor to distinguish letters such as “I” and “J”, whose finger position is the same and is differentiated by one small movement.

The circuit board converts the nine-digit key into a letter and sends it via Bluetooth to a smartphone or computer. This way, the glove can transmit the 26 letters of the sign language alphabet and decode them into text messages. The researchers also used the glove to control a virtual hand performing the signs associated with letters. The team is developing the next version of this glove, which will have the added sense of touch. The objective is to make a glove that can control either a virtual hand or a robot, and then send tactile sensations to the user’s own hand.

This study entitled “The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics” was published on July 12, 2017, in the PLOS journal. It was written by Timothy F. O’Connor, Mathew Fach, Rachel Miller, Samuel E. Root, Patrick P. Mercier and Darren J. Lipomi.