Space-Time Video Completion

Y. Wexler, E. Shechtman and M. Irani

NEW: A journal version (almost final) from TPAMI, March 2007, can be found here (pdf, 30MB).
A shorter version from CVPR 2004 is here (pdf, 3.5MB).

Abstract

We present a method for space-time completion of large space-time holes in video sequences of complex dynamic scenes. The missing portions are filled-in by sampling spatio-temporal patches from the available parts of the video, while enforcing global spatio-temporal consistency between all patches in and around the hole. This is obtained by posing the task of video completion and synthesis as a global optimization problem with a well-defined objective function.The consistent completion of static scene parts simultaneously with dynamic behaviors leads to realistic looking video sequences. Space-time video completion is useful for a variety of tasks, including, but not limited to:

Sophisticated video removal (of undesired static or dynamic objects) by completing the appropriate static or dynamic background information.
Correction of missing/corrupted video frames in old movies
Synthesis of new video frames to add a visual story, modify it, or generate a new one. Some examples of these are shown in the paper.
Video texture synthesis
Full field-of-view stabilized video.

NEW: An implementation inspired by this work for filling images is available here.

Sample videos:

NEW: These are new results -- improved since the video demo in CVPR'2004. See details in the journal version.
NEW: Added raw uncompressed data for some of the following examples. These videos can be used in publications provided the above paper is cited.

Removal of a beach umbrella

Raw data may be found here (zip, 9MB).
Removal of the jogging lady

Raw data may be found here (zip, 5MB).
Reconstruction of three missing frames in a sequence
Removal of the waving lady

Raw data may be found here (zip, 32MB).
Retouching of old Chaplin movie with simulated large "coffee stain"

Full field-of-view of stabilized video.
The input was taken from this nice work

Last changed on March 27, 2006

Removal of a beach umbrella Raw data may be found here (zip, 9MB).
Removal of the jogging lady Raw data may be found here (zip, 5MB).
Reconstruction of three missing frames in a sequence
Removal of the waving lady Raw data may be found here (zip, 32MB).
Retouching of old Chaplin movie with simulated large "coffee stain"
Full field-of-view of stabilized video. The input was taken from this nice work