Layerwise Systematic Scan: Deep Boltzmann Machines and Beyond

Heng Guo, Kaan Kara, Ce Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

For Markov chain Monte Carlo methods, one of the greatest discrepancies between theory and system is the scan order — while most theoretical development on the mixing time analysis deals with random updates, real-world systems are implemented with systematic scans. We bridge this gap for models that exhibit a bipartite structure, including, most notably, the Restricted/Deep Boltzmann Machine. The de facto implementation for these models scans variables in a layer wise fashion. We show that the Gibbs sampler with a layer-wise alternating scan order has its relaxation time (in terms of epochs) no larger than that of a random-update Gibbs sampler (in terms of variable updates). We also construct examples to show that this bound is asymptotically tight. Through standard inequalities, our result also implies a comparison on the mixing times.
Original languageEnglish
Title of host publication21st International Conference on Artificial Intelligence and Statistics
Place of PublicationPlaya Blanca, Lanzarote, Canary Islands
PublisherPMLR
Pages178-187
Number of pages10
Volume84
Publication statusPublished - 11 Apr 2018
EventThe 21st International Conference on Artificial Intelligence and Statistics - Playa Blanca, Lanzarote, Canary Islands, Lanzarote, Spain
Duration: 9 Apr 201811 Apr 2018
Conference number: 21
http://www.aistats.org/

Publication series

NameProceedings of Machine Learning Research
PublisherPMLR
Volume84
ISSN (Electronic)2640-3498

Conference

ConferenceThe 21st International Conference on Artificial Intelligence and Statistics
Abbreviated titleAISTATS 2018
CountrySpain
CityLanzarote
Period9/04/1811/04/18
Internet address

Fingerprint Dive into the research topics of 'Layerwise Systematic Scan: Deep Boltzmann Machines and Beyond'. Together they form a unique fingerprint.

Cite this