How does music sound to artificial ears?
In recent years, a rapidly increasing presence of artificial intelligence can be observed in the field of music and sound processing, especially in the form of neural networks and so-called deep learning. Google generates the voice of its assistant, Spotify evaluates millions of songs to automatically compile personalized playlists, and artificial neural networks are composing pop songs and film music.
It is often difficult even for experts to understand what exactly is going on in these complex self-learning systems and how they get to these remarkable results. The project "Immersions" renders the inside of sound processing networks audible. The basic idea is essentially the following: one starts with some sound snippet (e.g. a drum loop). This snippet is modified by an optimization procedure in such a way that it activates a certain area of the neural network. The modified clip can then be used as the basis of a new optimization with another region in the net as a target. This produces sounds that the neural network generates in a freely associative manner. At the same time, the processes in the network are visualized so that one can watch the system while it is processing information.
One of the most important properties of neural networks is that they process information in different degrees of abstraction. Accordingly, activation in a certain area of the network can be associated, for example, with simple short tones or noises, while in another area it can be associated with more complex musical phenomena such as phrase, rhythm or key. What exactly a neural network "listens to" depends both on the data based on which it has learned and on the task for which it is specialized. This means that music sounds different for different networks.
To allow all this to be experienced directly, I have developed a setup that allows the generation and control of sounds in real-time. The material for the live performance is not songs or pre-programmed loops, but hallucinations of neural networks.
Independent study project.
Supported by Prof. Dr. Christoph Seibert.