Han Han (LS2N) Unearthing the physicality of instrumental timbre - 30/04/24

Sound production in the physical world involves a resonator (an instrument body) and an exciter (a musician). Decoding and disentangling the effects of these causes enables a more accurate analysis of expressive musical performances, and paves ways to the making of more ergonomic digital musical instruments. In general, decoding timbre into physical information requires the presence of a physical model or evidence of player-instrument interaction acquired in the real world. The former is an idealized model and does not account for deviations from reality, while the latter requires a tremendous effort of data acquisition and annotations. We adopt both of the above approaches and investigates the extent to which “physicality” is implicit in an instrumental sound. Three work (in progress) will be presented, each lying at different levels of reality: synthetic, controlled reality, and reality.

The first work learns to extract physical parameters relevant to instrument making from synthetic, percussive drum sounds. It proposes a method of accelerating a perceptually relevant loss function that is otherwise expensive to compute. The second work learns to extract physical parameters of plucked string sounds recorded in a lab environment. It addresses the problem of transferring knowledge obtained from simulation to reality under data scarcity. The third and last work begins with recording an audiovisual dataset of isolated guqin playing techniques. It then aims to computationally identify the correlation between gestures and sounds.

Together, the three works demonstrate the distinct challenges encountered and the tailored approaches employed in tackling the problem of extracting physicality from sounds.