Exploring the multimodality of Arts with the help of Robotics

Within the framework of the Ligeti Center, this project aims at combining art and science. It consists in the development of a robot capable of interpreting music to create a work of art. It combines the mechatronic field of engineering (building the robot's mechanism, designing the electronics for the sensors and actuators, and developing the programs to control the robot) with the theory of music analysis and interpretation. 

So far, a robot has been designed in the form of an arm that can move in a plane over a canvas. A brush is attached to the end effector. The end effector can rotate to adjust the orientation of the brush and can be raised to control the pressure of the brush on the canvas. Six compartments hold the water and paint (three primary colors, plus black and white), which can be mixed to create an infinite variety of colors. 

This project is broad and covers several aspects of musical interpretation. The first one is to extract some specific parameters of the music (such as BPM, loudness, centroid, genre, and mood) and map them to specific parameters of the robot control (such as speed, color, pressure, and shape). Another aspect is based on the analysis of human music interpretation, where drawings made by humans (mostly children) while listening to music are used to feed a machine learning algorithm to control the robot. Another aspect is the integration of dancers in the music interpretation loop to gesture control the robot. These different aspects will lead to different algorithms to control the robot, leading to different paintings that can then be compared. By comparing the different outputs and processes with those of humans, the aim is to gain a better understanding of human music interpretation. 

This project, initiated by the Haptic Lab (TUHH), integrates other clusters and undertakings. The music analysis and interpretation part will be improved with the support of the ArtSearch Lab, the XR Lab, the Prod Lab and the Inno Lab.  Workshops on music interpretation are organised in schools with the support and pedagogical expertise of colleagues from the HfMT. Public events for workshops and performances will be organised with the support of ligeti's transfer office. 

So far, a robot has been designed in the form of an arm that can move in a plane over a canvas. A brush is attached to the end effector. The end effector can rotate to adjust the orientation of the brush and can be lifted to control the pressure of the brush on the canvas. Six compartments hold the water and paint (three primary colors, plus black and white), which can be mixed to create an infinite variety of colors.