A robotic twist to music

Varnit Jain and Anant Sharma, both students of UG course at  IIIT, Delhi, have built a music interpreting robot which can be controlled when a user's hand is occupied. The project christened  Jamoora, got accepted for The Student Innovation contest at UIST 2018 and was presented by the students at Berlin in October, 2018.

Jamoora is a performer in Indian traditional folk theatre who plays the role of a sidekick. The young innovators propose the concept of implicit control of Jamoora, a robotic puppet, using music and storytelling. The puppeteer controls the actions of Jamoora using keywords woven into a story being told by him while playing live music, which sets the emotional tone. They took inputs from Indian musical instruments, determine the corresponding raga — an organised sequence of musical notes associated with emotions and ambience, and then map the associated emotions to the body language of Jamoora. This integration of puppetry, music, and speech creates a rich medium of storytelling.

What makes it unique: Implicit control using music and explicit control using voice is a concept never explored before. They are using this concept by incorporating Indian classical music (ragas) for music detection.

Commands and operations this robot can take and do: The robot has four emotions — angry, sad, happy, serious and can perform commands like walking in all directions, picking up objects and a mixture of tasks by combining these movements.

Advantages: The robot can be programmed to perform a lot of functions. Raga detection can also be programmed to detect a variety of ragas and other western music as well. This robot can be used as an authoring tool for kids to author and perform their own stories.

Challenges: There were two major struggles while developing Jamoora. The first was taking voice and music input from a single audio source, which was solved using MIDI input for music detection. The second issue was portraying emotions using Jamoora. They used movement speed and head motions for the same.

The project was one of the 15 projects selected worldwide. The team also received a hardware kit, Makeblock Ultimate 2.0 Robot Kit, worth $350 and additional funding of $200.