Resolving Linguistic Ambiguities by Visual Context

Abstract

Resolving linguistic ambiguities is a crucial type of inference in several aspects of language, communication in the first place, and in the grounding of language in perception. This task is frequently called for in human communication and, in many cases, it cannot be solved without additional information about an associated context. In this paper we focus on the contextual effects of visual scenes on semantics, investigated using neural computational simulation. Specifically, provided with a sentence, admitting two or more candidate resolutions for a prepositional phrase attachment, and an image that depicts the content of the sentence, we address the problem of selecting the interpretation of the sentence matching the context provided by visual perception, choosing the correct resolution depending on the image’s content. From the neuro-computational point of view, our model is based on Nengo, the implementation of Neural Engineering Framework (NEF), whose basic semantic component is the so-called Semantic Pointer Architecture (SPA), a biologically plausible way of representing concepts by dynamic neural assemblies. We evaluated the ability of our model in resolving linguistic ambiguities on the LAVA (Language and Vision Ambiguities) dataset, a corpus of sentences with a wide range of ambiguities, associated with visual scenes.