(a)  To find “object-like” regions among the proposals, we look for regions that have (1) appearance cues typical to objects in general, and (2) differences in motion patterns relative to their surroundings.  We define a function S(r) = A(r) +M(r), that scores a region r according to its static intra-frame appearance score A(r) and dynamic inter-frame motion score M(r).

(b)  Given the scored regions, we next identify groups of key-segments that may represent a foreground object in the video.  We perform a form of spectral clustering with the highest scored regions as input to produce multiple binary inlier/outlier partitions of the data.  Each cluster (inlier set) is a hypothesis h of a foreground object’s key-segments.  We automatically rank the clusters based on the average object-like score S(r) of its member regions.  If that scoring is successful, the clusters among the highest ranks will correspond to the primary foreground object(s), since they are likely to contain frequently appearing object-like regions.

(c)  We build foreground and background color Gaussian Mixture Models and extract a set of shape exemplars for each hypothesis.

(d)  We next devise a space-time Markov Random Field (MRF) that uses these models to guide a pixel-wise segmentation for the entire video.  We compute color fg/bg estimates using the GMMs, and estimate fg location priors with the shape exemplars.  The main idea is to use the key-segments detected across the sequence, projecting their shapes into other frames via local shape matching.  The spatial extent of that projected shape then serves as a location and scale prior in which we prefer to label pixels as foreground. Since we have multiple key-segments and many possible local shape matches, many such projected shapes are aggregated together, essentially “voting” for the location/scale likelihoods.

(e)  We compute the foreground object segmentation by minimizing the energy function for the space-time MRF using graph cuts.