What makes a good team player? The ability to invent, listen, and give feedback in due time. Such ability is key to social organization: it is by cooperating productively that humans accomplish shared goals.

In this context, artificial intelligence (AI) research strives to orient virtual agents towards the same goals as humans. Yet, not all human endeavors are reducible to goal-seeking analysis. Art offers a counter example: humans are intrinsically motivated to produce objects of beauty, even when their artworks are not "rewarding" in the utilitarian sense. This simple observation poses a significant challenge for AI: that is, to interact creatively with humans in the absence of an explicit goal. The need for co-creativity between humans and AI is particularly evident in the case of music.

As of today, musicians interact with software on a daily basis: not only for audio playback, but also for score writing, mixing, and production. However, the computer remains largely absent from music rehearsals as well as free-form creative sessions. This is because most available tools for digital music are designed as single-user rather than cooperative; goal-seeking rather than curiosity-driven; and responsive rather than improvisational. To address this technological gap, the COLLAGE research project proposes a new direction in cyber-human co creativity, with application to musical improvisation. Its main originality is to model not only the evolution of each improvised stream, but also their interactions. COLLAGE will estimate, at any point in time, who in the band is attempting to lead versus who is attempting to follow.

In free improvisation, these social roles evolve quickly and have strong acoustical correlates: hence, the use of AI in COLLAGE will serve to decipher the “co-generative scheme” underlying musical performance without composer nor conductor. Specifically, the PhD candidate will reuse and extend state-of-the-art methods in multiview representation learning, sequence modeling, and neural audio synthesis. In the short term, COLLAGE will be evaluated on a dataset of improvised duets in terms of its ability to identify modes of interaction: reactivity, imitation, confrontation, indifference, to name a few. In the longer term, COLLAGE will be integrated as a machine listening frontend for cyber-human musicianship as part of the ERC REACH project.

In summary, creative co-improvisation is omnipresent in human social interactions, yet remains largely under-discussed in AI research, for lack of a methodological framework. The vision behind COLLAGE is to develop this framework from the standpoint of cyber-human musicianship. As such, it constitutes a challenging yet foundational case study towards the understanding of multi-human-multi-robot interactions in their fullest generality.

PhD student: Orian SHARONI

PhD supervisors: Pr Gérard ASSAYAG (Director), Dr Vincent LOSTANLEN (Co-director)

Research laboratory: Sciences et Technologies de la Musique et du Son (STMS) UMR 9912 - Tutelles : Ircam, CNRS, Sorbonne Université, Ministère de la Culture et de la Communication