Observe locally, Infer Globally: a Space-Time MRF for Detecting Abnormal Activities with Incremental Updates

Jaechul Kim and Kristen Grauman

Problem Statement

Detection of abnormal activities in an unsupervised manner.
Statistical approach: abnormality is defined as unseen or rarely occurring events.

Main Challenges

How to discriminate a truly abnormal event from noisy normal observations?
How to incrementally update the statistical model of abnormality when visual context in a scene changes over time?

Overview of the Algorithm

Figure 1. System overview

Algorithm Details

- Motion representation

Histogram of optical flows (HOFs) are used at each local region in the video frame.

Figure 2. Histogram of optical flows

- Learning of motion patter

Mixture of probabilistic principal component analyzers (MPPCA) are adopted to encode HOF patterns at each local region.
EM is used to learn the initial MPPCA model. (Afterwards, the MPPCA model is incrementally updated as new data come in.)

- Inference of abnormal activities in space-time MRF

Given the probabilities of current observations at individual local regions (i.e., node potential functions in MRF graph) as well as at pair-wise neighbor local regions (i.e., pair-wise potential functions in MRF graph), both of which are assigned by the learned MPPCA model, we infer whether observation at each local region is normal or abnormal.
For inference, belief propagation is used to find the maximum a posterior (MAP) for the given observations.

- Incremental update

As new data stream in, MPPCA parameters are updated using a closed-form equation.
Parameters defining MRF graphs are also updated according to the updated MPPCA parameters.

Results

Algorithm was tested for over two hours of surveillance videos at a subway station.
For every input frame, 10 most recent frames are used to build space-time MRF and MAP inference is carried out using belief propagation.

- Examples of the detected abnormal events: [demo videos]

Figure 3. Snapshots of the detected abnormal events in subway station

- Advantages of the space-time MRF

     (a) Crowded but otherwise normal scene          (b) A person gets off the train,                         (c) A person abruptly stops walking,
                                                                             and then gets on the train very soon.                and changes his direction.

                     Figure 4. Examples showing advantages of space-time MRF formulation for abnormal events detection

Advantages from localization by dividing the clip into local regions

Robust to noise in the crowded scene: As seen in Figure 4 (a), our system does not produce false alarms in the noisy crowded scene because noise is confined at each local region level, not being accumulated in the entire clip.
Enhanced detection resolution: As seen in Figure 4 (b), our system can detect abnormal events happening at the small scale.

Advantages from localization by dividing the clip into local regions

Pure local methods (i.e., no space-time association between local events) fail to detect abnormal activities with irregular temporal orderings as shown in Figure 4 (c), while our system can do.

- An effect of incremental learning

(a) Frame number 75,381 (b) Frame number 98,104

Figure 5. An effect of incremental learning. A movement from the right entrance to the gate is at first detected as “abnormal” at (a) because many of the most recent motions were along another path.
Later, the same type of movement is correctly detected as “normal” at (b) since several similar observations are accumulated in between (a) and (b).

- Quntitative results

Numbers in parentheses denote count for each abnormal activity in the ground truth. The first number in the slash (/) denotes the entrance gate video result; the second is for the exit gate video result.
Please note that there are only minor differences in detection accuracy and false alarm rates between the batch baseline and our incremental scheme, while our method can update the model parameters in an online fashion whenever new data come in.

Conclusion

We proposed a space-time MRF for detecting abnormal activities that combines the advantages of both local and global approaches: not only can the method localize abnormalities even in the crowded scenes, but it can also capture irregular interactions between local activities in a global sense
We demonstrated incremental real-time updates to adapt the system to visual context changes over time.

Publication

Jaechul Kim and Kristen Grauman, Observe locally, Infer Globally: a Space-Time MRF for Detecting Abnormal Activities with Incremental Updates, In Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2009. [pdf] [data] [demo]