Lending direction to neural networks

Richard S. Zemel, Christopher K.I. Williams, Michael C. Mozer

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

We present a general formulation for a network of stochastic directional units. This formulation is an extension of the Boltzmann machine in which the units are not binary, but take on values on a cyclic range, between 0 and 2π radians. This measure is appropriate to many domains, representing cyclic or angular values (e.g., wind direction, days of the week, phases of the moon). The state of each unit in a directional-unit Boltzmann machine (DUBM) is described by a complex variable, where the phase component specifies a direction; the weights are also complex variables. We associate a quadratic energy function, and corresponding probability, with each DUBM configuration. The probability distribution for the state of a given unit conditioned on the state of the rest of the network is a circular version of the Gaussian probability distribution, known as the von Mises distribution. In a mean-field approximation to a stochastic DUBM the phase component of a unit's state represents its mean direction, and the magnitude component specifies the degree of certainty associated with this direction. This combination of a value and a certainty provides additional representational power in a unit. We present a proof that the settling dynamics for a mean-field DUBM cause convergence to a free-energy minimum. Finally, we describe a learning algorithm and simulations that demonstrate a mean-field DUBM's ability to learn interesting mappings.
Original languageEnglish
Pages (from-to)503-512
Number of pages10
JournalNeural Networks
Issue number4
Publication statusPublished - 1995

Keywords / Materials (for Non-textual outputs)

  • Free-energy minimization


Dive into the research topics of 'Lending direction to neural networks'. Together they form a unique fingerprint.

Cite this