Saying What You're Looking For: Linguistics Meets Video Search

Daniel Paul Barrett, Andrei Barbu, N. Siddharth, Jeffrey Mark Siskind

Research output: Contribution to journalArticlepeer-review


We present an approach to searching large video corpora for clips which depict a natural-language query in the form of a sentence. Compositional semantics is used to encode subtle meaning differences lost in other approaches, such as the difference between two sentences which have identical words but entirely different meaning: The person rode the horse versus The horse rode the person. Given a sentential query and a natural-language parser, we produce a score indicating how well a video clip depicts that sentence for each clip in a corpus and return a ranked list of clips. Two fundamental problems are addressed simultaneously: detecting and tracking objects, and recognizing whether those tracks depict the query. Because both tracking and object detection are unreliable, our approach uses the sentential query to focus the tracker on the relevant participants and ensures that the resulting tracks are described by the sentential query. While most earlier work was limited to single-word queries which correspond to either verbs or nouns, we search for complex queries which contain multiple phrases, such as prepositional phrases, and modifiers, such as adverbs. We demonstrate this approach by searching for 2,627 naturally elicited sentential queries in 10 Hollywood movies.
Original languageEnglish
Pages (from-to)2069-2081
Number of pages13
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number10
Early online date2 Dec 2015
Publication statusPublished - 1 Oct 2016


Dive into the research topics of 'Saying What You're Looking For: Linguistics Meets Video Search'. Together they form a unique fingerprint.

Cite this