Neural Semantic Pointers in Context

Abstract

Resolving linguistic ambiguities is a task frequently called for in human communication. In many cases, such task cannot be solved without additional information about an associated context, which can be often captured from the visual scene referred by the sentence. This type of inference is crucial in several aspects of language, communication in the first place, and in the grounding of language in perception. This paper focuses on the contextual effects of visual scenes on semantics, investigated using neural computational simulation. Specifically, here we address the problem of selecting the interpretation of sentences with an ambiguous prepositional phrase, matching the context provided by visual perception. More formally, provided with a sentence, admitting two or more candidate resolutions for a prepositional phrase attachment, and an image that depicts the content of the sentence, it is required to choose the correct resolution depending on the image's content. From the neuro-computational point of view, our model is based on Nengo, the implementation of Neural Engineering Framework (NEF), whose basic semantic component is the so-called Semantic Pointer Architecture (SPA), a biologically plausible way of representing concepts by dynamic neural assemblies. We evaluated the ability of our model in resolving linguistic ambiguities on the LAVA (Language and Vision Ambiguities) dataset, a corpus of sentences with a wide range of ambiguities, associated with visual scenes.