|
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight
and sound observed at a source viewpoint, can we synthesize the sound of
that scene from an unseen target viewpoint? We propose a neural rendering
approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to
synthesize the sound of an arbitrary point in space by analyzing the input
audio-visual cues. To benchmark this task, we collect two first-of-their-kind
large-scale multi-view audio-visual datasets, one synthetic and one real. We
show that our model successfully reasons about the spatial cues and synthesizes
faithful audio on both datasets. To our knowledge, this work represents the
very first formulation, dataset, and approach to solve the novel-view acoustic
synthesis task, which has exciting potential applications ranging from AR/VR to
art and design. Unlocked by this work, we believe that the future of novel-view
synthesis is in multi-modal learning from videos.
|
