Smart camera that understands conversations without hearing sound.

 

NEW YORK: Phone and Amazon voice assistants Elsa and Siri help us by listening to voices, but now thanks to a visual system we can detect what is being said with just chin movements without hearing any sound.

Siri and Alexa may make a mistake in hearing the sound, but now Cornell University scientists have developed a system based on video cameras that is completely hands-free. The system is called SpinChin, which is actually an infrared camera.

Cheng Zheng of Cornell University and his student Rudong Zheng carry a camera around their necks. It keeps track of the movements of the chin and the muscles below it while staying in the upper chest. Now even if the speaker whispers or just moves his lips, he still feels it.

The chin and throat movements are first observed by a software and the algorithm tells what words are spoken silently. These instructions can then be sent where needed.

However, speakers on this system need some training. In the first phase, it was tested on 20 people. Ten of them were asked to count or speak simple words in English. 

The remaining ten were asked to say 44 simple words in Chinese. After some training, the camera began to recognize 90% of English and 91% of Chinese words very well.

If the same words were repeated while walking, its capacity was reduced because the movement of throat and chin was not being recorded properly in the video. ۔

The interesting thing about this whole system is that the high resolution camera installed in it records the words spoken only by looking at the muscles of the throat and the lower part of the chin, even if they are repeated in the mouth.