Our aim is to develop techniques to both (i) recognize or at least detect textual elements related to performance (words, phrases, etc.) in texts between the 5th and 15th century, and (ii) identify performances (musicians, instruments) in images, using automatic and human-in-the-loop processes. The first challenge is to incorporate different data sources lacking explicit contextual linking. Because of the complexity of the subject — the variety of sources, languages, and artistic contexts — we would like to continue implementing more complete transcultural and diachronic knowledge graphs. Since, knowledge graph embedding techniques have not been exploited in the medieval musicology domain, we will explore different approaches using purely knowledge graph embedding, visual embedding or combined embedding to calculate the similarity between items. Another major challenge in Deep Learning is the need for vast amounts of labeled data for training the images. In this project, we will use Generative Adversarial Networks (GAN) and Neural Network Style Transfer (NNST) for the generation of large training databases and introduce a pioneering approach towards the generation and completion of historical images containing musical instruments. Furthermore, we will introduce the idea of using GAN for domain transfer between these two domains, i.e., historical representations to “real-life” representations and vice versa. This will be useful for data augmentation as well as image enhancement and completion. Natural Language Processing (NLP) techniques will extract references to musical performances in texts: multilingual for medieval languages and multiple versions of descriptions are a novel challenge. We expect to identify approximately 30,000 scenes within the 300 million accessible (3,000 are now available in the Musiconis meta-database). For texts, a similar or greater volume of data is expected.

 

PhD student: Imad Bekkouch 

PhD supervisor: Frédéric Billiet

Research laboratory: IReMus - Institut de Recherche en Musicologie