Temporally Coherent Bayesian Models for Entity Discovery in Videos by Tracklet Clustering

A video can be represented as a sequence of tracklets, each spanning 10-20 frames, and associated with one entity (eg. a person). The task of \emph{Entity Discovery} in videos can be naturally posed as tracklet clustering. We approach this task by leveraging \emph{Temporal Coherence}(TC): the fundamental property of videos that each tracklet is likely to be associated with the same entity as its temporal neighbors. Our major contributions are the first Bayesian nonparametric models for TC at tracklet-level. We extend Chinese Restaurant Process (CRP) to propose TC-CRP, and further to Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to jointly model short temporal segments. On the task of discovering persons in TV serial videos without meta-data like scripts, these methods show considerable improvement in cluster purity and person coverage compared to state-of-the-art approaches to tracklet clustering. We represent entities with mixture components, and tracklets with vectors of very generic features, which can work for any type of entity (not necessarily person). The proposed methods can perform online tracklet clustering on streaming videos with little performance deterioration unlike existing approaches, and can automatically reject tracklets resulting from false detections. Finally we discuss entity-driven video summarization- where some temporal segments of the video are selected automatically based on the discovered entities.