Model Architecture
The complete proposed pipeline is a four-stage encoder–adapter–decoder system.
Audio feature extraction
A frozen Music2Latent encoder transforms the raw waveform into a sequence of 64-dimensional continuous latents produced at ≈10 Hz.
Projection adapter
A trainable projection layer maps the continuous latents to the token-embedding space of the language model.
Conditioning strategy
The projected latents are prepended as a prefix and are seen through cross-attention only once, keeping generation cost independent of audio length.
Autoregressive decoder
A Perceiver AR network predicts DADA-GP tokens autoregressively while attending to both the prefix and its own latent queries.