Results Report for Research Council of Norway Project 183180, Sensing Music-Related Actions, January 2013.
1. Aims and background
As stated in our funding application for this project, our main aim has been to explore action-sound couplings in human-computer interaction. In our application, this was furthermore seen to necessitate developing sensor technologies for capturing human body motion and developing processing schemes for extracting semantically meaningful features from such motion data, but also to necessitate enhancing our basic understanding of sound-motion couplings in music and various everyday life contexts. Additionally, we envisaged developing media devices for testing our theories and for use in various music-related contexts.
Our project proposal emerged from several years of research on music and body motion, as well as on music technology, at the University of Oslo and within our network of international research cooperation. The basis for our project proposal was the conviction that there are many and close links between sound and body motion in music (people make music by body motion, and people make body motion to music), and that these links should be studied and exploited in various new music making, HCI, and everyday contexts.
2. Main results
The main results of the project can be summarized as follows:
- Basic understanding. We have documented that there are indeed many and robust links between sound and body motion in music, both in its production and its perception. Experiences of close relationships between sound and body motion seem to apply equally well to novices and experts, i.e. also people with little or no musical training seem to spontaneously associate body motions with musical sound, often tracing melodic contours and rhythmical patterns with their hands, head, torso and/or feet, as well as reflecting the overall affective features of the music (e.g. calm, agitated, fast, slow, etc.) in their body motion. Clearly, our research in this area has enhanced our knowledge of the music-related actions and our general understanding of music as a multimodal phenomenon.
- Methods and technologies. Music-related action is a relatively new area of research, and our project has been at the cutting edge internationally in developing methods (experiment and observation studies designs) and technologies for capturing and processing motion and sound data (various so-called motion capture technologies and associated schemes for formatting, representing, visualizing, classifying and interpreting such data). This methodological work has also included studying combinations of sound and motion data in musical applications such as in the interfaces for new digital musical instruments and interactive sonification applications (i.e. using sound output to represent various data). Our research project has thus (and in accordance with the stated aims of the project) resulted in both basic, theoretical insights with relevance for various musical, HCI, everyday, as well as therapeutic, contexts, and in the development of tools for further research and practical uses in this highly interdisciplinary area.
3. Main R&D activities
As suggested by the above mentioned results, our main research and development
activities have been concerned with the following:
- Enhancing our understanding of motion-sound couplings in production and perception of music and of sound in general.
- Developing means for motion capture by adapting and extending available technologies as well as developing new technologies.
- Developing means for processing and representing motion and sound data.
- Developing means for exploiting both innate and learned sound-action links in designing new musical instruments, in HCI, and in various everyday contexts.
4. Research environment and cooperation
The aims of our research required expertise within musicology, music psychology,
human movement science, and informatics, resulting in the following:
- Establishment of an interdisciplinary environment between the Departments of Informatics and Musicology (and towards the end of the project, also with the Department of Psychology) at the University of Oslo, as well as other national partners concerned with human movement.
- Establishment of a research infrastructure with lab facilities for human movement research with the Music, Mind, Motion, Machines lab (the fourMs lab), thanks to additional funding from the RCN and the University of Oslo. This also necessitated much work with evaluating, procuring, installing, and adapting existing technologies, as well as developing our own extensions to existing technologies, cf. the above mentioned method development work.
- Putting much effort into our international cooperation, as we were on the cutting edge of both theories and methods. In particular, we have had an extensive and very fruitful international collaboration concerning technologies with exchanges of software and general know-how. This included participating in the EU COST Action SID (Sonic Interaction Design), the international NIME (New Interfaces for Musical Expression) community, also arranging the large NIME conference in Oslo in 2011, and participating in the still ongoing EU project EPICS (Engineering Proprioception in Computing Systems). Additionally, we have had a bilateral agreement with McGill University in Montreal, and several more informal cooperation schemes with various European partners, entailing several research exchanges and workshops.
- Establishment of cooperation within Norway on human movement research (with NTNU and NIH), also resulting in our own developed software put to use in various medical diagnostic contexts.
- Additionally, our research effort and our international and national cooperation has in the course of the project resulted in our strongly interdisciplinary fourMs Centre of Excellence proposal to the RCN, a proposal that received very high scores from the international expert panel also in the final round of the competition, but which unfortunately was not selected as one of the new Centers of Excellence in November 2012. However, the work put into, and the very favorable assessments received for this CoE scheme, should be an impetus for similar research proposals in the future.
5. Project realization and resource use
The project was realized as a cooperative effort between the Departments of Informatics and of Musicology at the University of Oslo. The RCN provided funding for one PhD and one postdoctoral fellowship, funding for part-time participation by three senior staff from the involved departments, and for various running costs (travel, workshops, more minor equipment needs, etc.). Additionally, the Department of Informatics allocated one PhD fellowship to the project. The main efforts in realizing the project was organized as follows:
- Shifts between theoretical work and practical data collection as well as artistic implementations, with bi-weekly group meetings with all participants, and day-today work involving variable constellations of senior researchers, PhD fellows, and masters students.
- The research was organized in various joint sub-projects and paper publications, combining junior and senior researchers, a mode of working that was successful in creating a good research environment as well as assuring good quality output.
- Additional funding for equipment from the RCN and the University of Oslo enabled the procurement of state-of-the-art equipment and establishment of the fourMs lab, as mentioned above. However, this also entailed a considerable effort by all involved in developing the necessary methods and technologies to carry out cutting-edge research on music-related actions.
Considering the substantial amount of work required to develop the necessary methods and technologies, it could in hindsight have been a good idea to have applied for more funding resources for technical assistance, something that we certainly will take into consideration in future research grant applications.
6. Assessment of results and use of outcomes
The results of our research project are both in the more theoretical, basic science
domain of understanding sound-body motion relationships, and in the more
technological-practical domain of musical, HCI, everyday and therapeutic
applications, and can be summarized as follows:
- The basic perceptual and cognitive linking of sound and action in music has been well documented through this research project, both by our own experiments and observation studies, and by compilations of findings from the international research community. Based on the reception of our various publications by the international research community, it seems fair to say that our findings on music-related actions are both well founded and interesting by international standards (cf. the evaluation of our fourMs CoE application).
- Likewise, various methods and technologies developed in the course of the project seems to have been well received by the research community, and have been put to use both in musical and other contexts (e.g. medical diagnostics) in several places around the world.
7. Dissemination and expected use of results
Throughout the project, we have had a relatively high level of dissemination activity,
both to our peers and to the general public:
- We have had a fairly large number of public lectures/demos and concerts as well as had a fair amount of press coverage (see the pdf file attached to the RCN final report form listing our activities in these areas), and we have also worked closely with various musicians and other performance artists, both as a means of research dissemination and as a testing-ground for our findings and technologies.
- As can be seen from the publications lists (see the PDF files submitted to the RCN in our annual progress reports), we have presented our work for our peers at a fairly large number of international conferences and in international publications. Publications have also resulted in one awarded PhD (January 25, 2013), and one PhD thesis is to be submitted in April 2013.
- New hardware/software applications, resulting in better body motion input schemes, and better processing, recognition and mapping schemes, is now in use in several research and artistic environments around the world. (For details on this, see fourMs downloads).
- Elements from our research has been integrated in several bachelor and masters courses as well as masters theses at the University of Oslo (e.g. in the courses MUS2006, MUS4218, MUS4320), and in the annual EU-funded International Summer School in Systematic Musicology (ISSSM) for masters and PhD students.
8. Further research
In summary, we believe that the Sensing Music-Related Actions project has clearly demonstrated several close links between sound and action in music, as well as demonstrated the potential for exploiting these links in various artistic, everyday, HCI, educational and therapeutic contexts. As suggested by the international expert review of our fourMs CoE proposal, we also believe we have the qualifications for further and high quality research in the areas of the Sensing Music-Related Actions project and associated areas, research that we envisage focusing on the following main topics:
- Enhanced understanding of sound-action links in music and in human perception, cognition and behavior in general.
- Enhanced methods for representations, visualizations, and processing of sound-motion data.
- Enhanced methods for sonic design, enabling aesthetically better user experience of sound-motion links in the arts, HCI, and various everyday contexts.
This means that we shall in the future also work towards submitting a new, strongly interdisciplinary and internationally oriented CoE research proposal (or a similar type of integrated research proposal), based on the outcomes of the Sensing Music-Related Actions project.