Computer Graphics Laboratory ETH Zurich

ETH

Scalable Music: Automatic Music Retargeting and Synthesis

S. Wenner, J.C. Bazin, A. Sorkine-Hornung, C. Kim, M. Gross

Proceedings of Eurographics (Girona, Spain,, May 6-10, 2013), Computer Graphics Forum, vol. 32, no. 2, pp. 345-354
[Abstract] [BibTeX] [PDF] [Video]

Abstract

In this paper we propose a method for dynamic rescaling of music, inspired by recent works on image retargeting, video reshuffling and character animation in the computer graphics community. Given the desired target length of a piece of music and optional additional constraints such as position and importance of certain parts, we build on concepts from seam carving, video textures and motion graphs and extend them to allow for a global optimization of jumps in an audio signal. Based on an automatic feature extraction and spectral clustering for segmentation, we employ length-constrained least-costly path search via dynamic programming to synthesize a novel piece of music that best fulfills all desired constraints, with imperceptible transitions between reshuffled parts. We show various applications of music retargeting such as part removal, decreasing or increasing music duration, and in particular consistent joint video and audio editing.

[Download Video]

@article{Wenner2013music,
author = {Simon Wenner and Jean-Charles Bazin and Alexander Sorkine-Hornung and Changil Kim and Markus Gross},
title = {Scalable Music: Automatic Music Retargeting and Synthesis},
journal = {Computer Graphics Forum (Proceedings of Eurographics 2013)},
volume = {32},
number = {2},
year = {2013},
month = may,
pages = {345-354}
}
[Download BibTeX]

Example of our structure-aware music retargeting in the context of video editing.

Results

We applied our algorithm to several applications such as music retargeting (rescaling, singing part removal, structural reshuffling) and video editing (action level control, synchronization). We tested it on a wide range of instrumental musical genres from heavy metal to traditional flamenco. Some representative results are shown in the below video.

Downloads

Download Paper
[PDF]
Download Video
[Video]