Punjabi University develops technology to convert spoken Punjabi into Indian Sign Language

In a significant advancement for inclusive communication, researchers at Punjabi University, Patiala, have developed an innovative system that automatically converts spoken Punjabi into Indian Sign Language (ISL). This transformative technology, funded by the Department of Science and Technology (DST), Government of India, bridges the communication gap between hearing-impaired and non-hearing-impaired individuals, fostering greater inclusion and accessibility.

The research project is led by Williamjeet Singh, Project Supervisor and Assistant Professor, Department of Computer Science and Engineering, and supported by Amandeep Singh, Research Project Assistant. The system integrates speech recognition, natural language processing, and synthetic animation to convert spoken words into ISL gestures in real time.

Williamjeet Singh told that the system is designed to empower the hearing-impaired community by enabling seamless communication in education, healthcare, and public services.

He said that communication is a vital part of human interaction, enabling the exchange of emotions, ideas, and information. While speech is the primary mode for most, hearing-impaired individuals rely on sign language, a structured form of gestural communication that uses hand movements and facial expressions. However, the lack of a universal sign language and limited knowledge of ISL among non-hearing-impaired individuals often create barriers, leading to exclusion and restricted opportunities for the hearing-impaired. He told that the newly developed system addresses these challenges by converting spoken Punjabi into ISL using advanced technologies. It employs the Hamburg Notation System (HamNoSys) to represent sign gestures and Signing Gesture Markup Language (SiGML) to generate animated 3D sign sequences. The system includes an ISL dictionary that captures both manual (hand gestures) and non-manual (facial expressions and body movements) features, ensuring accurate and natural sign representation.

Amandeep Singh said that the system leverages two approaches for sign language representation: synthetic animation and video-based sign presentation. A novel optimization technique enhances the accuracy of converting spoken words into structured ISL scripts. He told that rigorous testing has demonstrated the system’s efficiency, accuracy, and practical usability, offering a reliable platform for real-time communication. He told that available as a Progressive Web Application, the system supports multi-lingual speech input and is accessible on both web and mobile platforms. Beyond facilitating daily communication, it serves as an educational tool, enabling users to learn new signs and improve their communication skills.

He explained that through enabling real-time conversion of spoken Punjabi to ISL, they are making communication more accessible in critical areas like education, healthcare, and public services. The goal is to empower hearing-impaired individuals to engage fully in society. He told further that the research team envisions expanding the system to support additional languages, making it accessible to a broader audience through a dedicated mobile app.

Vice Chancellor Jagdeep Singh, while congratulating the research team, said that this innovation promises to transform the lives of millions of hearing-impaired individuals in India and beyond, offering new opportunities for inclusion and interaction.