It is called SoundTracer. Developed by Olivier Lartillot in the Department of Musicology (IMV), it allows you to search for music in the library in a whole new way.
Imagine that you are looking for a particular type of music. All you have to do is to "simulate" the music by moving your hand. The app will then look up the music that fits with the hand gestures.
The app's developer, Olivier Lartillot gives a little demonstration. He finds his iPhone and opens the SoundTracer app. Then he moves his hand up and down in the air.
– The app is able to recognise a piece of music that contains the same movement or pitch, he explains, while the app looks for music that fits Lartillot's hand gestures. It makes a sound that rises and falls as Lartillot moves his hand up or down.
Find music that matches your hand gestures
A few seconds later, the app has found a piece of music to match the gestures Lartillot made. From the app the folk song Tjugedalaren can be heard. Two graphs are displayed on the screen, one at the top and one at the bottom. The top one shows the movements Lartillot has made with the mobile. The bottom one shows the music that is played.
– For humans It is natural to express or imitate music with hand gestures. This is something the researchers at IMV have been working with for many years. With the new app, it is possible to also use this knowledge to search for music, says Lartillot.
– The app lets you see drawings of your gestures, and the pitch of the music. When we play the piece of music, a red line moves across both graphs, so you can see how the two graphs match each other, explains Lartillot.
Thousands of melodies
The SoundTracer project is a collaboration between the Department of Musicology and the National Library of Norway. The researchers have gained access to a large collection of folk music audio files the National Library has in their catalogue. All together there are around 14,000 melodies. So far, Lartillot has added 50 of these to the app – but more will follow.
– These have been added manually. But I have now developed algorithms that mean this can be done automatically. It means that in the future, we can add thousands of new tunes, he explains.
While working on the prototype, the researchers have worked with folk music. It has presented some special challenges.
– Transferring songs is not that difficult. But the Hardanger fiddle is not so easy. The tones on the Hardanger fiddle are very complex, and it is therefore difficult to extract the pitch. I am currently working on automating this as well, explains Lartillot.
Further developing the app
At the moment, the app is based on vertical hand gestures — the higher you move your hand, the lighter the tone. However, future versions of the app will also include other features, such as rhythmic movements, said Lartillot.
The SoundTracer project concludes with a workshop at the National Library on Friday May the 4th. At the workshop, project manager Alexander Refsum Jensenius will introduce the project, while Olivier Lartillot will explain how he has worked to develop the app and its underlying systems. In addition, the world leading German researcher Meinard Müller will present an inspiring lecture on music informatics.
– The SoundTracer project is a good example of how basic research in the humanities can lead to technological innovation, concludes Jensenius.