English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose a novel system that takes as an input body movements of a musicianplaying a musical instrument and generates music in an unsupervised setting.Learning to generate multi-instrumental music from videos without labeling theinstruments is a challenging problem. To achieve the transformation, we built apipeline named 'Multi-instrumentalistNet' (MI Net). At its base, the pipelinelearns a discrete latent representation of various instruments music fromlog-spectrogram using a Vector Quantized Variational Autoencoder (VQ-VAE) withmulti-band residual blocks. The pipeline is then trained along with anautoregressive prior conditioned on the musician's body keypoints movementsencoded by a recurrent neural network. Joint training of the prior with thebody movements encoder succeeds in the disentanglement of the music into latentfeatures indicating the musical components and the instrumental features. Thelatent space results in distributions that are clustered into distinctinstruments from which new music can be generated. Furthermore, the VQ-VAEarchitecture supports detailed music generation with additional conditioning.We show that a Midi can further condition the latent space such that thepipeline will generate the exact content of the music being played by theinstrument in the video. We evaluate MI Net on two datasets containing videosof 13 instruments and obtain generated music of reasonable audio quality,easily associated with the corresponding instrument, and consistent with themusic audio content.

Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements