How to pretend that correlated variables are independent by using difference observations

In many areas of data modelling it is the case that observations at different locations (e.g. time frames or pixel locations) are augmented by differences of nearby observations (e.g. -features in speech recognition, Gabor jets in image analysis). These augmented observations are then often modelled as being independent—how can this make sense? We provide two interpretations, showing (1) that the likelihood of data generated from an autoregressive (AR) process can be computed in terms of “independent” augmented observations, and (2) that the augmented observations can be given a coherent treatment in terms of the Products of Experts model (Hinton, 1999). In automatic speech recognition it is often the case that Hidden Markov mod- 1 els (HMMs) are used on observation vectors that are augmented by difference observations (so-called features), see Furui (1986). Under the HMM each observation vector is modelled as being conditionally independent given the hidden state. How can this make sense, as close-by differences are clearly not independent? A similar difficulty arises in image analysis tasks such as texture segmentation, see e.g. Dunn and Higgins (1995). Here derivative features obtained e.g. from Gabor filters or wavelet analysis are modelled as being independent at different locations, despite the fact that these features will have been computed sharing some pixels in common. In this paper we present two solutions to this problem. In section 1 we show that if the data is generated from a vector autoregressive (AR) model then the likelihood can be expressed in terms of “independent” difference observations. In section 2 we show that the local models at each location can be combined using a Product of Experts model (Hinton, 1999) to provide a well-defined joint model for the data, and that this can be related to AR models. Section 3 discusses how these interpretations are affected if the local models are conditional on a hidden state variable, as is the case e.g. for HMMs.