Sensing Music-related Actions (completed)

Sensing Music-related Actions (SMA) is an interdisciplinary research project focusing on capturing and understanding more about body movements in a musical context. This is done through the combination of methods from musicology and informatics, and the researchers are working with advanced motion capture technologies and machine learning techniques.

About the project

This interdisciplinary project, combining scientific, technological and artistic methods, proposes three crucial topics for future development of multimodal devices:

  • the importance of action-sound couplings for both performance and perception of music
  • the control potential of human bodily movement
  • a move towards an active music experience

Research questions and methods

The basic research question that underlies this project is:

  • How can action-sound couplings be exploited in human-computer interaction?

A number of sub-questions are linked to this:

  • Which sensor technologies can be used to capture complex body movement?
  • Which machine-learning techniques can be used to extract semantic actions from a continuous stream of movement data?
  • How are actions and sounds coupled in everyday life?
  • How can such theories and technologies be used to control sound and music?
  • How can we create musical structures that can be controlled by the user?

Aims and objectives

Principal objective:

  • Exploring action-sound couplings in human-computer interaction.

Sub-goals:

  • Develop sensor technologies for capturing complex body movement.
  • Develop machine-learning techniques and segmentation methods for extracting semantic actions from a continuous stream of sensor data.
  • Develop theories of action-sound couplings in everyday life.
  • Develop prototypes of enactive media devices that allow for continuously controlling music based on the actions of the user.
  • Develop hypermusic structures that can be used in the enactive media devices.

Cooperation and financing

Sensing Music-related Actions is a joint research project of the departments of Musicology and Informatics, and has received external funding through the VERDIKTprogram of the The Research Council of Norway. The project runs from July 2008 until December 2012.

Publications

  • Alexander Refsum Jensenius (2013). Kinectofon: Performing with Shapes in Planes, In Wakefield Graham; Woon Seung Yeo; Haru Ji; Alexander Sigman & Kyogu Lee (ed.),  Proceedings of the International Conference on New Interfaces For Musical Expression.  Korea Advance Institute of Science and Technology.  Paper.  s 196 - 197
  • Jim Tørresen; Yngve Hafting & Kristian Nymoen (2013). A new wi-fi based platform for wireless sensor data collection. In Proceedings of the International Conference on New Interfaces For Musical Expression, In Kyogu Lee; Alexander Sigman; Haru Ji; Wakefield Graham & Woon Seung Yeo (ed.),  Proceedings of the International Conference on New Interfaces For Musical Expression.  Korea Advance Institute of Science and Technology.  22.  s 337 - 340
  • Alexander Refsum Jensenius (2013). Non-Realtime Sonification of Motiongrams, In Roberto Bresin (ed.),  Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden.  KTH.  ISBN 978-91-7501-831-7.  KAPITTEL.  s 500 - 505
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Mats Erling Høvin; Sverre Holm & Alexander Refsum Jensenius (2013). Filtering Motion Capture Data for Real-Time Applications, In Wakefield Graham; Woon Seung Yeo; Kyogu Lee; Alexander Sigman & Haru Ji (ed.),  Proceedings of the International Conference on New Interfaces For Musical Expression.  Korea Advance Institute of Science and Technology.  Kapittel.  s 142 - 147
  • Alexander Refsum Jensenius (2013). Non-Realtime Sonification of Motiongrams, In Roberto Bresin (ed.),  Proceedings of the Sound and Music Computing Conference 2013.  Logos Verlag Berlin.  ISBN 978-3-8325-3472-1.  Chapter.  s 500 - 505
  • Rolf Inge Godøy (2013). Quantal Elements in Musical Experience, In Rolf Bader (ed.),  Sound - Perception - Performance.  Springer.  ISBN 9783319001067.  4.  s 113 - 128
  • Kristian Nymoen; Arve Voldsund; Ståle Andreas van Dorp Skogstad; Alexander Refsum Jensenius & Jim Tørresen (2012). Comparing Motion Data from an iPod Touch to a High-End Optical Infrared Marker-Based Motion Capture System, In Brent Gillespie; Georg Essl; Sile O’Modhrain & Michael Gurevich (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  University of Michigan Press.  ISBN 978-0-9855720-1-3.  Artikkel i Proceedings.  s 88 - 91
  • Alexander Refsum Jensenius & Arve Voldsund (2012). The Music Ball Project: Concept, Design, Development, Performance, In Sile O’Modhrain; Georg Essl; Michael Gurevich & Brent Gillespie (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  University of Michigan Press.  ISBN 978-0-9855720-1-3.  Article.  s 300 - 303
  • Jim Tørresen; Øyvind N Hauback; Dan Overholt & Alexander Refsum Jensenius (2012). Development and evaluation of a ZigFlea-based wireless transceiver board for CUI32, In Michael Gurevich; Brent Gillespie; Georg Essl & Sile O’Modhrain (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  University of Michigan Press.  ISBN 978-0-9855720-1-3.  Peer-reviewed paper.  s 409 - 412
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Yago de Quay & Alexander Refsum Jensenius (2012). Developing the Dance Jockey system for musical interaction with the Xsens MVN suit, In Brent Gillespie; Michael Gurevich; Georg Essl & Sile O’Modhrain (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  University of Michigan Press.  ISBN 978-0-9855720-1-3.  Del av proceedings.  s 226 - 229
  • Kristian Nymoen; Jim Tørresen; Rolf Inge Godøy & Alexander Refsum Jensenius (2012). A Statistical Approach to Analyzing Sound Tracings, In Mitsuko Aramaki; Richard Kronland-Martinet; Kristoffer Jensen; Sølvi Ystad & Sanghamitra Mohanty (ed.),  Speech, Sound and Music Processing: Embracing Research in India. 8th International Symposium, CMMR 2011. 20th International Symposium, FRSM 2011.  Springer.  ISBN 978-3-642-31979-2.  Kapittel i bok.  s 120 - 145
  • Alexander Refsum Jensenius (2012). Evaluating how different video features influence the visual quality of resultant motiongrams, In Bob L. Sturm; Sofia Dahl & Jan Larsen (ed.),  Proceedings of the 9th Sound and Music Computing Conference - "Illusions".  Logos Verlag Berlin.  ISBN 9783832531805.  Paper.  s 467 - 472
  • Alexander Refsum Jensenius; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Arve Voldsund (2012). A Study of the Noise-Level in Two Infrared Marker-Based Motion Capture Systems, In Jan Larsen; Bob L. Sturm & Sofia Dahl (ed.),  Proceedings of the 9th Sound and Music Computing Conference - "Illusions".  Logos Verlag Berlin.  ISBN 9783832531805.  Paper.  s 258 - 263
  • Rolf Inge Godøy; Alexander Refsum Jensenius; Arve Voldsund; Kyrre Harald Glette; Mats Erling Høvin; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Jim Tørresen (2012). Classifying Music-Related Actions, In Emilios Cambouropoulos; Costas Tsougras; Panayotis Mavromatis & Konstantinos Pastiadis (ed.),  Proceedings of the ICMPC-­ESCOM 2012 Joint Conference: 12th Biennial International Conference for Music Perception and Cognition, 8th Triennial Conference of the European Society for the Cognitive Sciences of Music.  School of Music Studies, Aristotle University of Thessaloniki Thessaloniki, Hellas.  ISBN 978-960-99845-1-5.  Artikkel i Proceedings.  s 352 - 357
  • Alexander Refsum Jensenius & Kari Anne Vadstensvik Bjerkestrand (2012). Exploring micromovements with motion capture and sonification, In Anthony Brooks (ed.),  Arts and Technology, Second International Conference, ArtsIT 2011, Esbjerg, Denmark, December 7-8, 2011, Revised Selected Papers.  Springer.  ISBN 978-3-642-28081-8.  Chapter.  s 100 - 107
  • Ståle Andreas van Dorp Skogstad; Sverre Holm & Mats Erling Høvin (2012). Designing Digital IIR Low-Pass Differentiators With Multi-Objective Optimization, In Yuan Baozong; Tang Xiaofang & Ruan Qiuqi (ed.),  Proceedings of 2012 IEEE 11th International Conference on Signal Processing.  IEEE Press.  ISBN 978-1-4673-2194-5.  3.
  • Ståle Andreas van Dorp Skogstad; Sverre Holm & Mats Erling Høvin (2012). Digital IIR filters with minimal group delay for real-time applications, In Paul J. Küehn; Alaa Khamis & Yasser G. Hegazy (ed.),  Engineering and Technology (ICET), 2012 International Conference on.  IEEE conference proceedings.  ISBN 978-1-4673-4808-9.  proceedings.  s 1 - 6
  • Davide Rocchesso; Kristian Nymoen; Alexander Refsum Jensenius; Tim Redfern; Richard Widerberg & Rolf Nordahl (2011). Protocols, tools, and curricula, In Davide Rocchesso (ed.),  Explorations in Sonic Interaction Design.  Logos Verlag Berlin.  ISBN 978-3-8325-2858-4.  14.  s 67 - 72
  • Kristian Nymoen & Alexander Refsum Jensenius (2011). A Toolbox for Storing and Streaming Music-related Data, In Serena Zanolla; Amalia de Götzen; Frederico Avanzini & Sergio Canazza (ed.),  Proceedings of SMC 2011 8th Sound and Music Computing Conference “Creativity rethinks science”.  Padova University Press.  ISBN 9788897385035.  Artikkel i Proceedings.  s 427 - 430
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen & Mats Erling Høvin (2011). Comparing Inertial and Optical MoCap Technologies for Synthesis Control, In Amalia de Götzen; Frederico Avanzini; Serena Zanolla & Sergio Canazza (ed.),  Proceedings of SMC 2011 8th Sound and Music Computing Conference “Creativity rethinks science”.  Padova University Press.  ISBN 9788897385035.  Artikkel i Proceedings.  s 421 - 426
  • Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Alexander Refsum Jensenius (2011). SoundSaber - A Motion Capture Instrument, In Alexander Refsum Jensenius; Dan Overholt; Anders Tveit & Rolf Inge Godøy (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  Universitetet i Oslo.  ISBN 978-82-991841-6-8.  Artikkel.  s 312 - 315
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Yago de Quay & Alexander Refsum Jensenius (2011). OSC Implementation and Evaluation of the Xsens MVN suit, In Anders Tveit; Dan Overholt; Rolf Inge Godøy & Alexander Refsum Jensenius (ed.),  Proceedings of the International Conference on New Interfaces for Musical Expression.  Universitetet i Oslo.  ISBN 978-82-991841-6-8.  KAPITTEL.  s 300 - 303
  • Kristian Nymoen; Baptiste Caramiaux; Mariusz Kozak & Jim Tørresen (2011). Analyzing sound tracings: a multimodal approach to music information retrieval, In Douglas Eck; Cynthia Liem; Meinhard Müller & George Tzanetakis (ed.),  Proceedings of the 1st international ACM workshop on Music information retrieval with user-centered and multimodal strategies.  Association for Computing Machinery (ACM).  ISBN 978-1-4503-0986-8.  Article in proceedings.  s 39 - 44
  • Rolf Inge Godøy (2011). Sound-action awareness in music, In David Ian Clarke & Eric F. Clarke (ed.),  Music and consciousness: philosophical, psychological, and cultural perspectives.  Oxford University Press.  ISBN 978-0-19-955379-2.  13.  s 231 - 243
  • Rolf Inge Godøy (2011). Coarticulated gestural-sonorous objects in music, In Elaine King & Anthony Gritten (ed.),  New perspectives on music and gesture.  Ashgate.  ISBN 978-0-7546-6462-8.  3.  s 67 - 82
  • Kristian Nymoen; Alexander Refsum Jensenius; Jim Tørresen; Kyrre Harald Glette & Ståle Andreas van Dorp Skogstad (2010). Searching for Cross-Individual Relationships between Sound and Movement Features using an SVM Classifier, In Andrew Johnston; Kirsty Beilharz; Amy Yi-Chun Chen & Sam Ferguson (ed.),  NIME 2010 proceedings: New Interfaces for Musical Expression++, 15-18th June 2010, University of Technology Sydney.  Association for Computing Machinery (ACM).  ISBN 978-0-646-53482-4.  Artikkel.  s 259 - 262
  • Ståle Andreas van Dorp Skogstad; Alexander Refsum Jensenius & Kristian Nymoen (2010). Using IR Optical Marker Based Motion Capture for Exploring Musical Interaction, In Andrew Johnston; Kirsty Beilharz; Sam Ferguson & Amy Yi-Chun Chen (ed.),  NIME 2010 proceedings: New Interfaces for Musical Expression++, 15-18th June 2010, University of Technology Sydney.  Association for Computing Machinery (ACM).  ISBN 978-0-646-53482-4.  KAPITTEL.  s 407 - 410
  • Alexander Refsum Jensenius; Kyrre Harald Glette; Rolf Inge Godøy; Mats Erling Høvin; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Jim Tørresen (2010). fourMs, University of Oslo – Lab Report, In Dimitris Samaras & Robert Rowe (ed.),  Proceedings of the International Computer Music Conference, June 1-5 2010, New York.  International Computer Music Association.  ISBN 0-9713192-8-6.  KAPITTEL.  s 290 - 293
  • Ståle Andreas van Dorp Skogstad & Mats Erling Høvin (2007). Using Heuristic Search To Find Stable High-Order Single-Bit Delta Sigma Modulators, In Carl Crane; Rodney G. Roberts & Sabri Tosunoglu (ed.),  Proceedings of the 7th IEEE International Symposium on Computational Intelligence in Robotics and Automation, Jacksonville, IEEE Robotics and Automation Society.  IEEE Press.  ISBN 1-4244-0790-7.  FrAT3.  s 350 - 355
  • Ståle Andreas van Dorp Skogstad & Mats Erling Høvin (2006). A Proposed Stability Characterization and Verification Method for High-Order Single-Bit Delta-Sigma Modulators, In Thomas Johansson (ed.),  Proceedings 24th Norchip Conference.  IEEE conference proceedings.  ISBN 1-4244-0772-9.  I.  s 100 - 103

View all works in Cristin

  • Alexander Refsum Jensenius; Rolf Inge Godøy; Anders Tveit & Dan Overholt (ed.) (2011). Proceedings of the International Conference on New Interfaces for Musical Expression. Universitetet i Oslo.  ISBN 978-82-991841-6-8.  586 s.
  • Alexander Refsum Jensenius; Ståle Andreas van Dorp Skogstad; Anette Forsbakk & Alison Bullock Aarsten (ed.) (2011). Program book of the International Conference on New Interfaces for Musical Expression. Universitetet i Oslo og Norges musikkhøgskole.  92 s.

View all works in Cristin

  • Rolf Inge Godøy (2013). The Convergence of "Hard" and "Soft" in Music Technology.
  • Alexander Refsum Jensenius (2013). Musikk, kropp, bevegelse og bevegelsessporing.
  • Alexander Refsum Jensenius (2013). Kinectofon: Performing with Shapes in Planes.
  • Alexander Refsum Jensenius (2013). Non-Realtime Sonification of Motiongrams.
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Mats Erling Høvin; Sverre Holm & Alexander Refsum Jensenius (2013). Filtering Motion Capture Data for Real-Time Applications.
  • Rolf Inge Godøy (2013). Coarticulation in the production and perception of music.
  • Rolf Inge Godøy (2013). Understanding Coarticulation in Music.
  • Alexander Refsum Jensenius (2012). Musikk og kropp.
  • Alexander Refsum Jensenius (2012). Musikk, kropp og bevegelse.
  • Jim Tørresen; Øyvind N. Hauback; Dan Overholt & Alexander Refsum Jensenius (2012). Development and evaluation of a ZigFlea-based wireless transceiver board for CUI32.
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Yago de Quay & Alexander Refsum Jensenius (2012). Developing the Dance Jockey System for Musical Interaction with the Xsens MVN Suit.
  • Alexander Refsum Jensenius (2012). Evaluating how different video features influence the visual quality of resultant motiongrams.
  • Alexander Refsum Jensenius; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Arve Voldsund (2012). A Study of the Noise-Level in Two Infrared Marker-Based Motion Capture Systems.
  • Alexander Refsum Jensenius (2012). Musikk og bevegelse: Kroppens meningsbærende og meningsdannende funksjon i musikalsk utøving og sansing.
  • Kristian Nymoen; Arve Voldsund; Ståle Andreas van Dorp Skogstad; Alexander Refsum Jensenius & Jim Tørresen (2012). Comparing Motion Data from an iPod Touch to a High-End Optical Infrared Marker-Based Motion Capture System.
  • Arjun Chandra; Kristian Nymoen; Arve Voldsund; Alexander Refsum Jensenius; Kyrre Harald Glette & Jim Tørresen (2012). Enabling Participants to Play Rhythmic Solos Within a Group via Auctions.
  • Rolf Inge Godøy (2012). Thinking Shapes in Musical Experience.
  • Rolf Inge Godøy (2012). Postures, Trajectories, and Sonic Shapes.
  • Rolf Inge Godøy; Alexander Refsum Jensenius; Arve Voldsund; Kyrre Harald Glette; Mats Erling Høvin; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Jim Tørresen (2012). Classifying Music-Related Actions.
  • Alexander Refsum Jensenius (2012). Representation of music-related body movement.
  • Alexander Refsum Jensenius (2012). Motion capture in music composition and performance.
  • Rolf Inge Godøy (2012). Sonic Object Design.
  • Ståle Andreas van Dorp Skogstad; Sverre Holm & Mats Erling Høvin (2012). Digital IIR Filters With Minimal Group Delay for Real-Time Applications.
  • Ståle Andreas van Dorp Skogstad; Sverre Holm & Mats Erling Høvin (2012). Designing Digital IIR Low-Pass Differentiators With Multi-Objective Optimization.
  • Rolf Inge Godøy (2012). Continuity and discontinuity in music-related motion.
  •  (2012). Oslo iPhone Ensemble på åpningen av Realfagsbiblioteket.
  •  (2012). Sverm: Visning og felles utprøving av mikrolyd/mikrobevegelser.
  •  (2012). OMO performance: Squeeky.
  •  (2012). OMO plays HTC.
  • Kristian Nymoen; Even Bekkedal & Kyrre Harald Glette (2011). Motion Capture demo / SoundSaber.
  • Kristian Nymoen & Even Bekkedal (2011). SoundSaber demo.
  • Even Bekkedal & Kristian Nymoen (2011). Kinect-piano, demo.
  • Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Alexander Refsum Jensenius (2011). SoundSaber - A Motion Capture Instrument.
  • Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Yago de Quay & Alexander Refsum Jensenius (2011). OSC Implementation and Evaluation of the Xsens MVN suit.
  • Birgitta Burger; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Arve Voldsund (2011). Optical Motion Capture Technology Workshop.
  • Kristian Nymoen; Baptiste Caramiaux; Mariusz Kozak & Jim Tørresen (2011). Analyzing sound tracings: a multimodal approach to music information retrieval.
  • Mariusz Kozak; Kristian Nymoen & Rolf Inge Godøy (2011). The Effects of Spectral Features of Sound on Gesture Type and Timing.
  • Rolf Inge Godøy (2011). Sonic feature timescales and music-related actions.
  • Rolf Inge Godøy (2011). Images of sound, postures and trajectories in music. Keynote lecture, the Embodiment-Experiment Seminar, Department of Music, University of York, May 10th-11th, 2011.
  • Rolf Inge Godøy (2011). Sound-Action Timescales. Lecture at the International Summer School in Systematic Musicology, Jyväskylä, Finland, 08/08/11-18/08/11.
  • Rolf Inge Godøy (2011). Coarticulation in Music-Related Gestures.
  •  (2011). Oslo iPhone Ensemble @ Future Media Days.
  •  (2011). Oslo iPhone Ensemble på NMH.
  •  (2011). Oslo iPhone Ensemble på åpningen av Ole Johan Dals hus (IFI2).
  •  (2011). Where Art Thou? Dance Jockey.
  •  (2011). Dance Jockey.
  •  (2011). Oslo iPhone Ensemble på Researcher's Night.
  • Rolf Inge Godøy (2010). Musical Gestures: Sound, Movement, and Meaning - En bokpresentasjon.
  • Ståle Andreas van Dorp Skogstad; Alexander Refsum Jensenius & Kristian Nymoen (2010). Using IR Optical Marker Based Motion Capture for Exploring Musical Interaction.
  • Ståle Andreas van Dorp Skogstad (2010). Using Full Body Motion Capture Data as Input for Musical Sound Generation.
  • Kristian Nymoen (2010). Motion Capture: Motion Analysis and Musical Instrument.
  • Alexander Refsum Jensenius; Kyrre Harald Glette; Rolf Inge Godøy; Mats Erling Høvin; Kristian Nymoen; Ståle Andreas van Dorp Skogstad & Jim Tørresen (2010). fourMs, University of Oslo – Lab Report.
  • Rolf Inge Godøy (2010). Sound Shapes.
  • Rolf Inge Godøy (2010). Music-related Actions.
  • Kristian Nymoen; Alexander Refsum Jensenius; Jim Tørresen; Ståle Andreas van Dorp Skogstad & Kyrre Harald Glette (2010). Searching for Cross-Individual Relationships between Sound and Movement Features using an SVM Classifier.
  • Alexander Refsum Jensenius & Rolf Inge Godøy (2010). Input technologies for music-related actions.
  • Kyrre Harald Glette; Alexander Refsum Jensenius & Rolf Inge Godøy (2010). Extracting action-sound features from a sound-tracing study.
  • Rolf Inge Godøy; Alexander Refsum Jensenius & Kristian Nymoen (2009). Chunking by coarticulation in music-related gestures.
  • Rolf Inge Godøy; Alexander Refsum Jensenius & Kristian Nymoen (2009). Coarticulation of sound and movement in music.
  • Alexander Refsum Jensenius; Ståle Andreas van Dorp Skogstad; Kristian Nymoen; Rolf Inge Godøy; Jim Tørresen & Mats Erling Høvin (2009). Reduced displays of multidimensional motion capture data sets of musical performance.
  • Rolf Inge Godøy (2009). Chunking sound-actions in musical experience.
  • Rolf Inge Godøy & Alexander Refsum Jensenius (2009). Body Movement in Music Information Retrieval.
  • Rolf Inge Godøy (2009). Sound, Movement, Key-Frames and Inter-Frames.
  • Rolf Inge Godøy (2009). Anticipatory chunking of music-related actions.
  • Kyrre Harald Glette; Jim Tørresen & Ståle Andreas van Dorp Skogstad (2009). Machine-learning techniques for action-sound MIR analysis and classification.
  • Timothy Place; Trond Lossius; Alexander Refsum Jensenius; Nils Peters & Pascal Baltazar (2008). Addressing Classes by Differentiating Values and Properties in OSC.
  • Rolf Inge Godøy; Alexander Refsum Jensenius & Kristian Nymoen (2008). Production and perception of goal-points and coarticulations in music.
  •  (2008). Konsert med Oslo Laptop-orkester, dirigent: roboten "Anna".

View all works in Cristin

Published Aug. 21, 2012 11:42 AM - Last modified Apr. 6, 2018 7:57 PM