HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness

We present HOI-Swap that seamlessly swaps the in-contact object from the video, given one user-provided reference object image, with hand-object interaction (HOI) awareness.

Notice how the generated hand needs to adjust to the shapes of the swapped-in objects and how the reference object may require automatic re-posing to fit the video context.

We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made in video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits—especially when object swapping results in object shape or functionality changes. To bridge this gap, we present HOI-Swap, a novel diffusion-based video editing framework trained in a self-supervised manner. Designed in two stages, the first stage focuses on object swapping in a single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based on changes in the object’s properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping a new sequence from the stage-I edited frame based on sampled motion points and (2) conditioning video generation on the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.

HOI-Swap involves two stages, each trained separately in a self-supervised manner. In stage I, an image diffusion model is trained to inpaint the masked object region with a strongly augmented version of the original object image. In stage II, one frame is selected from the video to serve as the anchor. The remaining video is then warped using this anchor frame, several points sampled within it, and optical flow extracted from the video. A video diffusion model is trained to reconstruct the full video sequence from the warped sequence. During inference, the stage-I model swaps the object in one frame. This edited frame then serves as the anchor for warping a new video sequence, which is subsequently taken as input for the stage-II model to generate the complete video.

Results

Reference object image

Source (original video)

HOI-Swap generated video

More Results

Source (original video)

Reference object A

Reference object B

Reference object C

Source (original video)

HOI-Swap generated A

HOI-Swap generated B

HOI-Swap generated C

Source (original video)

Reference object A

Reference object B

Reference object C

Source (original video)

HOI-Swap generated A

HOI-Swap generated B

HOI-Swap generated C