Overview
This is the demo page of our work: Self-Representation Alignment in Diffusion Models for Video Background Music Generation. Since the BGM909 authors only provide the extracted video features and their manually edited videos themselves are not released, we also provide the generated music samples without the videos.
We propose a novel diffusion U-Net architecture enhanced by explicitly aligning the weaker representation from the earlier layer conditioned on higher noise with the better representation from the later layer conditioned on lower noise during training, which we use for the challenging multimodal task of video background music generation. Our diffusion U-Net Self-Representation Alignment (U-SRA) method employs an alignment loss to effectively guide our diffusion model to learn good internal representations, thereby enabling it to produce higher-quality symbolic music conditioned on video features. We employ a cross-attention mechanism to synchronize music with video segments. Experiments on a benchmark video-music dataset demonstrate that the proposed U-SRA approach for video background music generation surpasses the state-of-the-art across key evaluation metrics, including retrieval-based video–music alignment and overall music quality.
U-SRA(5-15)
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Sample 6
Diff-BGM
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Sample 6