11th grader builds AI gloves that translate sign language for deaf people

Seeing how much deaf people struggle to communicate, an 11th-grade student in Quảng Trị, a province in central Vietnam, has built a pair of translating gloves to help those less fortunate study and gain knowledge.

"Tac-tac-tac, boom-boom-boom," the school drum signals the end of the day, and Trần Ngọc Long, an 11th grader at Lê Quý Đôn High School for the Gifted (in Đông Hà, formerly a city of Quảng Trị province, which has since been merged and enlarged) hurriedly slings on his backpack and heads to a room that holds a device, the long-cherished dream of this student from the "Sacred Land," as Quảng Trị is known.

No one would have thought that an 11th grader could program two artificial intelligence (AI) tools by himself and integrate them into a device of his own design - the translating glove - to help people with disabilities (the deaf and speech-impaired) communicate like anyone else.

Trần Ngọc Long and his AI glove.

"I want everyone to be able to learn"

A frequent visitor to the Quảng Trị Provincial School for Children with Disabilities, Long noticed that young people his own age could not communicate, study or take in knowledge, and it weighed heavily on his heart.

Driven by compassion, Trần Ngọc Long set out to build a communication aid to help those less fortunate than himself.

"I feel that people with disabilities are at a huge disadvantage. To learn and develop yourself, you have to be able to hear and speak. If I were unlucky enough to go blind, I could still seek out knowledge, but people who are born deaf and mute can't do that," Trần Ngọc Long said.

When he started the project more than half a year ago, the student had no idea where to begin; he was starting from scratch. At times he despaired and thought of giving up, but driven by his dream of helping people with disabilities, he kept working hard, diligently studying the technology to improve his product.

Training two AI tools

At 2 a.m., when Đông Hà lies in silence, a clicking sound comes from a bedroom somewhere: Trần Ngọc Long is hunched over his computer, fingers flying as he types lines of code. The student is training the AI to be built into his translating glove.

As a student specializing in physics, Long had no programming knowledge at all. Fortunately, his school's computer science teacher, Hồ Văn Lâm, guided him, sharing basic skills and materials so he could work with code on the computer.

"I started programming from zero. I'm truly grateful to my teacher for showing me how to code, so that I could build my own AI tool and train an existing AI program to fit the device and Vietnamese sign language data," Long said.

"My name is… I have a passion for art", a voice rings out from a phone in a classroom at the Quảng Trị Provincial School for Children with Disabilities. A student, a bright red scarf around the shoulders and an innocent, cheerful face, is wearing a glove fitted with sensor chips.

Born deaf and mute, the child moves both hands to form signs. With just a few gestures, the phone below plays audio introducing the young student and their passion.

This is made possible by the translating glove Long built. The product is a pair of electronic gloves that let deaf and mute people communicate in natural language with hearing people by converting sign language.

Long's translating gloves are fitted with sensors. He trained the artificial intelligence himself on a Vietnamese sign language dataset so that hand signs are converted into language accurately.

It packs a number of standout features and comes with a phone app that plays audio and displays the conversation.

Notably, the gloves enable two-way communication between hearing people and deaf and mute people in both English and Vietnamese. They turn the discrete words of sign language into complete sentences in natural language suited to the conversation.

Long's device is equipped with flex sensor chips, gyroscope sensors, magnetometers and accelerometers.

These measure differences in how far each finger bends and stretches, as well as voltage differences, to generate a data stream sent to a phone app. The artificial intelligence then turns this data into a complete spoken sentence.

Early on, however, the device suffered from errors in tracing motion trajectories in space. Long solved this by creating an AI tool to correct these sensor errors.

In addition, the student kept collecting Vietnamese sign language to add to the dataset, training the AI so the technology could produce sentences that accurately match the hand movements of deaf users.

Reaching for glory

At one point, Trần Ngọc Long wanted to abandon his research, because it was going nowhere and the time spent on the project was badly hurting his grades. He failed to earn the title of excellent student that semester, and the pressure left him in despair.

During trials at the provincial School for Children with Disabilities, Long's gloves did not work as expected, because the sensors could not produce accurate signs.

Trần Ngọc Long won first prize at the National Science and Engineering Fair with his translating glove project.

"Because I developed the device along the well-worn path of earlier products, the further I went the more wrong it got, and I couldn't build a device that met my goals. I was stuck and fell into a crisis, in despair, and the thought of giving up flickered in my mind," Long said.

Encouraged by his mentor, he boldly changed course and chose an entirely new approach. Trần Ngọc Long overcame the problem by integrating drift-correction technology based on a deep learning model to accurately trace trajectories on the IMU module, which is equipped with spatial motion sensors.

Teacher Hồ Văn Lâm (far left), who mentored Long on the translating glove project, and the principal of Lê Quý Đôn High School for the Gifted pose for a souvenir photo at the provincial science and engineering fair.

Little by little, Long's translating gloves worked better and better. The project "Gloves that convert sign language into natural language for deaf and mute people" won first prize at the National Science and Engineering Fair and advanced to the selection round for projects to compete at the International Science and Engineering Fair, held in the United States.

"I think success requires sacrifice and trade-offs. Even though I didn't earn the excellent student title last semester, I still feel very proud, and I hope that one day my product will help deaf people communicate easily, so they can study, gain knowledge and contribute to the country's development," Trần Ngọc Long said, wrapping up his story with the reporter.

Editorial information: See here

This article was translated from Vietnamese. Read the original