Christiansen, M.H., Chater, N. & Seidenberg, M.S. (Eds.) (1999). Connectionist models of human language processing: Progress and prospects. Special issue of Cognitive Science, Vol. 23(4), 415-634.

Connectionist Models of Human Language Processing:
Progress and Prospects

Introduction

This Special Issue on Connectionist Models of Human Language Processing provides an opportunity for an appraisal both of specific connectionist models and of the status and utility of connectionist models of language in general. This introduction provides the background for the papers in the Special Issue. The development of connectionist models of language is traced, from their intellectual origins, to the state of current research. Key themes that arise throughout different areas of connectionist psycholinguistics are highlighted, and recent developments in speech processing, morphology, sentence processing, language production, and reading are described. We argue that connectionist psycholinguistics has already had a significant impact on the psychology of language, and that connectionist models are likely to have an important influence on future research.

Part I: Progress

A critical property of many localist models of speech perception is the parallel activation of multiple lexical representations during the perception of spoken words. Here, we examine how a distributed model of speech perception accommodates this property. We used statistical analyses of vector spaces to show that coactivation of more than one distributed representation is inherently noisy, and depends on parameters such as sparseness and dimensionality. Furthermore, the characteristics of coactivation vary considerably, depending on the organization of distributed representations within the mental lexicon. This view of lexical access is supported by analyses of phonological and semantic word representations, which allow the model to simulate in detail a recent set of experiments on coactivation in speech perception (Gaskell & Marslen-Wilson, submitted)

The acquisition of English noun and verb morphology is modelled using a single system connectionist network. The network is trained to produce the plurals and past tense forms of a large corpus of monosyllabic English nouns and verbs. The developmental trajectory of network performance is analysed in detail and is shown to mimic a number of important features of the acquisition of English noun and verb morphology in young children. These include an initial error-free period of performance on both nouns and verbs followed by a period of intermittent overregularization of irregular nouns and verbs. Errors in the model show evidence of phonological conditioning and frequency effects. Furthermore, the network demonstrates a strong tendency to regularize denominal verbs and deverbal nouns and masters the principles of voicing assimilation. Despite their incorporation into a single-system network, nouns and verbs exhibit some important differences in their profiles of acquisition. The simulations generate several empirical predictions that can be used to evaluate further the suitability of this type of cognitive architecture in the domain of inflectional morphology.

We suggest that the mathematical theory of dynamical systems provides a revealing general framework for modeling the representations and mechanism underlying syntactic processing. We show how a particular dynamical model, the Visitation Set Gravitation (VSG) model of Tabor, Juliano, and Tanenhaus (Language and Cognitive processes, 1997), develops syntactic representations and models a set of contingent frequency effects in parsing that are problematic for other models. We also present new simulations showing how the model accounts for semantic effects in parsing and propose a new account of the distinction between syntactic and semantic incongruity. The results help flesh out a resolution of the connectionists vs. symbolists debate by showing how symbolic structures useful in parsing arise in emergent properties of connectionist dynamical systems.

Theories of language production have long been expressed as connectionist models. We outline the issues and challenges that must be addressed by connectionist models of lexical access and grammatical encoding, and review three recent models. The models illustrate the value of an interactive activation approach to lexical access in production, the need for sequential output in both phonological and grammatical encoding, and the potential for accounting for structural effects on errors and structural priming from learning.

A connectionist approach to word reading, based on the principles of distributed representation, graded learning of statistical nature, and interactivity in processing, has led to the development of explicit computational models which account for a wide range of data on normal skilled reading and on patterns of reading impairment due to brain damage. There have, however, been recent empirical challenges to these models, and the approach in general, relating to the influence of orthographic length on the naming latencies of both normal and dyslexic readers. The current work presents a simulation which generates sequential phonological output in response to written input, and which can refixate the input when encountering difficulty. The normal model reads both words and nonwords accurately, and exhibits an effect of orthographic length and a frequency-by-consistency interaction in its naming latencies. When subject to peripheral damage, the model exhibits an increased length effect which interacts with word frequency, characteristic of letter-by-letter reading in pure alexia. Although the model is far from a fully adequate account of all the relevant phenomena, it suggests how connectionist models may be extended to provide deeper insight into sequential processes in reading.

Part II: Prospects

This article provides an overview of a probabilistic constraints approach to language acquisition and processing. This framework offers an alternative to how these issues have been construed within generative linguistics. The generative approach attempts to characterize knowledge of language (i.e., competence grammar) and then asks how this knowledge is acquired and used. Our approach is performance oriented: the goal is to explain how people comprehend and produce utterances and how children acquire this skill. Using language is thought to involve exploiting multiple probabilistic constraints over various types of linguistic and nonlinguistic information. Children begin accumulating this information at a young age. The constraint satisfaction processes that are central to language use are in the child the bootstrapping processes that provide entry to the language. Framing questions about acquisition in terms of models of skilled language use has important consequences for arguments concerning language learnability and holds out the possibility of a unified theory of acquisition and use.

This article describes an approach to connectionist language research which relies on the development of grammar formalisms rather than computer models. From formulations of the fundamental theoretical commitments of connectionism and of generative grammar, it is argued that these two paradigms are mutually compatible. Integrating the basic assumptions of the paradigms results in formal theories of grammar that centrally incorporate a certain degree of connectionist computation. Two such grammar formalisms - Harmonic Grammar (Legendre, Miyata and Smolensky, 1990ab) and Optimality Theory (Prince and Smolensky, 1991, 1993) - are briefly introduced to illustrate grammar-based research and more traditional model-based research are argued to be complementary, suggesting a significant role for both strategies in the spectrum of connectionist language research.

The emphasis in the connectionist sentence-processing literature on distributed representation and emergence of grammar from such systems seems to have prevented connectionists and symbolists alike from recognizing the often close relations between their respective systems. This paper argues that simply recurrent network (SRN) models proposed by Jordan (1990) and Elman (1990) are more directly related to stochastic Part-of-Speech (POS) Taggers than to parsers or grammars as such, while recursive auto-associative memory (RAAM) of the kind pioneered by Pollack and incorporated in many hybrid connectionist parsers since may be useful for grammar induction from a network-based conceptual structure as well as for structure-building.

These observations suggest some interesting new directions for connectionist sentence processing research, including more efficient devices for representing finite state machines, and acquisition devices based on a distinctively connectionist grounded conceptual structure.

Home | People | Research | Links | Contact | Publications | Presentations