Image Description using Visual Dependency Representations

Describing the main event of an image involves
identifying the objects depicted and
predicting the relationships between them.
Previous approaches have represented images
as unstructured bags of regions, which makes
it difficult to accurately predict meaningful
relationships between regions. In this paper,
we introduce visual dependency representations
to capture the relationships between
the objects in an image, and hypothesize that
this representation can improve image description.
We test this hypothesis using a
new data set of region-annotated images, associated
with visual dependency representations
and gold-standard descriptions. We describe
two template-based description generation
models that operate over visual dependency
representations. In an image description
task, we find that these models outperform
approaches that rely on object proximity
or corpus information to generate descriptions
on both automatic measures and on human
judgements.