Ben Hayes

I am a PhD student in Artificial Intelligence and Music at Queen Mary University of London’s Centre for Digital Music. My research currently focuses on differentiable digital signal processing for deep learning based control over audio synthesisers and effects. I am particularly interested in how we might overcome the optimisation pathologies that currently limit this methodology. My broader research interests include audio synthesis, symmetry, meta-learning, timbre perception, and cross-modal interactions. I am jointly supervised by Dr Charalampos Saitis and Dr György Fazekas, and am very grateful to be funded by the UKRI Centre for Doctoral Training in Artificial Intelligence and Music.

Previously, I was Music Lead at the award-winning AI-driven generative music startup Jukedeck, and was a research intern with ByteDance’s Speech, Audio & Music Intelligence (SAMI) team. I also make music and taught undergraduate Electronic and Produced music at the Guildhall School of Music and Drama.

I am particularly open to collaborations with musicians and artists looking to apply artificial intelligence to their work, engineers interested in building new musical tools, and researchers working on related topics.

You can contact me at b.j.hayes (at) qmul.ac.uk

selected publications

ICLR

The Responsibility Problem in Neural Networks with Unordered Targets

Hayes, Ben, Saitis, Charalampos, and Fazekas, George

In The First Tiny Papers Track at ICLR 2023, Tiny Papers @ ICLR 2023, Kigali, Rwanda, May 5, 2023 2023

Abs PDF

We discuss the discontinuities that arise when mapping unordered objects to neural network outputs of fixed permutation, referred to as the responsibility problem. Prior work has proved the existence of the issue by identifying a single discontinuity. Here, we show that discontinuities under such models are uncountably infinite, motivating further research into neural networks for unordered data.
ICASSP

Sinusoidal Frequency Estimation by Gradient Descent

Hayes, Ben, Saitis, Charalampos, and Fazekas, George

In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2022

Abs PDF

Sinusoidal parameter estimation is a fundamental task in applications from spectral analysis to time-series forecasting. Estimating the sinusoidal frequency parameter by gradient descent is, however, often impossible as the error function is non-convex and densely populated with local minima. The growing family of differentiable signal processing methods has therefore been unable to tune the frequency of oscillatory components, preventing their use in a broad range of applications. This work presents a technique for joint sinusoidal frequency and amplitude estimation using the Wirtinger derivatives of a complex exponential surrogate and any first order gradient-based optimizer, enabling end to-end training of neural network controllers for unconstrained sinusoidal models.
JAES

Disembodied Timbres: A Study on Semantically Prompted FM Synthesis

Hayes, Ben, Saitis, Charalampos, and Fazekas, George

In Journal of the Audio Engineering Society 2022

Abs PDF

Disembodied electronic sounds constitute a large part of the modern auditory lexicon, but research into timbre perception has focused mostly on the tones of conventional acoustic musical instruments. It is unclear whether insights from these studies generalise to electronic sounds, nor is it obvious how these relate to the creation of such sounds. In this work, we present an experiment on the semantic associations of sounds produced by FM synthesis with the aim of identifying whether existing models of timbre semantics are appropriate for such sounds. We applied a novel experimental paradigm in which experienced sound designers responded to semantic prompts by programming a synthesiser, and provided semantic ratings on the sounds they created. Exploratory factor analysis revealed a five-dimensional semantic space. The first two factors mapped well to the concepts of luminance, texture, and mass. The remaining three factors did not have clear parallels, but correlation analysis with acoustic descriptors suggested an acoustical relationship to luminance and texture. Our results suggest that further enquiry into the timbres of disembodied electronic sounds, their synthesis, and their semantic associations would be worthwhile, and that this could benefit research into auditory perception and cognition, as well as synthesis control and audio engineering.
ISMIR

Neural Waveshaping Synthesis

Hayes, Ben, Saitis, Charalampos, and Fazekas, George

In Proceedings of the 22nd International Society for Music Information Retrieval Conference 2021

Abs PDF

We present the Neural Waveshaping Unit (NEWT): a novel, lightweight, fully causal approach to neural audio synthesis which operates directly in the waveform domain, with an accompanying optimisation (FastNEWT) for efficient CPU inference. The NEWT uses time-distributed multilayer perceptrons with periodic activations to implicitly learn nonlinear transfer functions that encode the characteristics of a target timbre. Once trained, a NEWT can produce complex timbral evolutions by simple affine transformations of its input and output signals. We paired the NEWT with a differentiable noise synthesiser and reverb and found it capable of generating realistic musical instrument performances with only 260k total model parameters, conditioned on F0 and loudness features. We compared our method to state-of-the-art benchmarks with a multi-stimulus listening test and the Fréchet Audio Distance and found it performed competitively across the tested timbral domains. Our method significantly outperformed the benchmarks in terms of generation speed, and achieved real-time performance on a consumer CPU, both with and without FastNEWT, suggesting it is a viable basis for future creative sound design tools.