Space-Time Behavior Based Correlation*
– OR –
How to tell if two underlying motion fields are similar without computing them?

Eli Shechtman and Michal Irani

*Patent Pending*


One possible application of our space-time correlation is to detect "behaviors of interest'' in a video database. A behavior-of-interest can be defined via one (or more) example video clip (a "video query''). Such video queries serve as space-time correlation templates and are correlated with the long video using our method.

In the examples below, the correlation result videos show a colored representation of the "correlation volume" around the highest peaks of detection, superimposed on the original videos. Red denotes high correlation values; Blue denotes low values.

Press on the images to download/view the video clips (in MPEG-I format).


Example #1 - Detecting instances of walking people


Walk Template

The 'walk' template is correlated twice - once as is, and once with its mirror reflection, to allow detections of walks in both directions.

Long beach video

Correlation result

Note that no background-foreground segmentation was required. The behavioral-consistency between the template and the underlying video segment is invariant to differences in spatial appearance of the foreground moving objects and of their backgrounds. It is sensitive only to the underlying motions.


Example #2 - A ballet footage
Performed by the "Birmingham Royal Ballet" from the "London Dance" website. Original full video can be found also here (WMV format, 400KB).


Turn Template
Long ballet video
(3.8MB, with sound)
Correlation result

Most of the turns of the two dancers (a man and a woman) were detected, despite the variability in scale relative to the template (up to 20%). Note that this example contains very fast moving parts (frame-to-frame).


Example #3 - Detecting dives into a pool
From the 2004 Olympic Games website (part of the 'Aquatics' clip).


Dive Template
Long pool video
(21MB, with sound)
Correlation result

Despite the numerous simultaneous activities (a variety of swim styles, flips under the water, splashes of water), and despite the severe noise, the space-time correlation was able to separate most of the dives from other activities.
One dive is missed due to partial occlusion by the Olympic logo at the bottom right of the frame. There is also one false detection, due to a similar motion pattern occurring in the water.


Example #4 - Detecting multiple actions



Five small video queries of different activities were provided (including 'flowing water' activity). These were performed by different people&backgrounds (T1-T4) than in the longer video. A short sub-clip from the right-most fountain was used as the fountain-query (T5).

Long video of multiple activities
Correlation result

The result video shows the highest peaks detected in each of the five correlation surfaces. Space-time ellipses are displayed around each peak, with its corresponding activity color. All activities were correctly detected, including the flowing water in all three fountains.

BibTeX - journal paper

  author    = {Eli Shechtman and Michal Irani},
  title     = {Space-Time Behavior Based Correlation –OR– How to tell if two underlying motion fields are similar without
computing them?},
  booktitle = {In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)},
  volume    = {29},
  number    = {11},
  pages     = {2045--2056},
  month     = {November},
  year      = {2007},
  ee        = {},
BibTeX - conference paper
  author    = {Eli Shechtman and Michal Irani},
  title     = {Space-Time Behavior Based Correlation},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  volume    = {1},
  pages     = {405--412},
  month     = {June},
  location  = {San-Diego},
  year      = {2005},
  ee        = {},

eXTReMe Tracker