Asymptotic Analysis of the LMS Algorithm with Momentum

Laszlo Gerencser; Balazs Csanad Csaji; Sotirios Sabanis

doi:10.1109/CDC.2018.8619576

Asymptotic Analysis of the LMS Algorithm with Momentum

Laszlo Gerencser, Balazs Csanad Csaji, Sotirios Sabanis

School of Mathematics

Research output: Contribution to conference › Paper › peer-review

Abstract

A widely studied filtering algorithm in signal processing is the least mean square (LMS) method, due to B. Widrow and T. Hoff, 1960. A popular extension of the LMS algorithm, which is also important in deep learning, is the LMS method with momentum, originated by S. Roy and J.J. Shynk back in 1988. This is a fixed gain (or constant step-size) version of the LMS method modified by an additional momentum term that is proportional to the last correction term. Recently, a certain equivalence of the two methods has been rigorously established by K. Yuan, B. Ying and A.H. Sayed, assuming martingale difference gradient noise. The purpose of this paper is to present the outline of a significantly simpler and more transparent asymptotic analysis of the LMS algorithm with momentum under the assumption of stationary, ergodic and mixing signals.

Original language	English
Pages	3062-3067
DOIs	https://doi.org/10.1109/CDC.2018.8619576
Publication status	Published - 21 Jan 2019
Event	2018 IEEE Conference on Decision and Control (CDC) - Miami Beach, FL Duration: 17 Dec 2018 → 19 Dec 2018

Conference

Conference	2018 IEEE Conference on Decision and Control (CDC)
Period	17/12/18 → 19/12/18

Access to Document

10.1109/CDC.2018.8619576

GCS_CDC18_LMS_MomentumAccepted author manuscript, 169 KB

https://ieeexplore.ieee.org/document/8619576/

Fingerprint Dive into the research topics of 'Asymptotic Analysis of the LMS Algorithm with Momentum'. Together they form a unique fingerprint.

Cite this

@conference{2aeabbf567664f00bc48f1a796507e99,

title = "Asymptotic Analysis of the LMS Algorithm with Momentum",

abstract = "A widely studied filtering algorithm in signal processing is the least mean square (LMS) method, due to B. Widrow and T. Hoff, 1960. A popular extension of the LMS algorithm, which is also important in deep learning, is the LMS method with momentum, originated by S. Roy and J.J. Shynk back in 1988. This is a fixed gain (or constant step-size) version of the LMS method modified by an additional momentum term that is proportional to the last correction term. Recently, a certain equivalence of the two methods has been rigorously established by K. Yuan, B. Ying and A.H. Sayed, assuming martingale difference gradient noise. The purpose of this paper is to present the outline of a significantly simpler and more transparent asymptotic analysis of the LMS algorithm with momentum under the assumption of stationary, ergodic and mixing signals.",

author = "Laszlo Gerencser and Csaji, {Balazs Csanad} and Sotirios Sabanis",

year = "2019",

month = jan,

day = "21",

doi = "10.1109/CDC.2018.8619576",

language = "English",

pages = "3062--3067",

note = "2018 IEEE Conference on Decision and Control (CDC) ; Conference date: 17-12-2018 Through 19-12-2018",

}

TY - CONF

T1 - Asymptotic Analysis of the LMS Algorithm with Momentum

AU - Gerencser, Laszlo

AU - Csaji, Balazs Csanad

AU - Sabanis, Sotirios

PY - 2019/1/21

Y1 - 2019/1/21

N2 - A widely studied filtering algorithm in signal processing is the least mean square (LMS) method, due to B. Widrow and T. Hoff, 1960. A popular extension of the LMS algorithm, which is also important in deep learning, is the LMS method with momentum, originated by S. Roy and J.J. Shynk back in 1988. This is a fixed gain (or constant step-size) version of the LMS method modified by an additional momentum term that is proportional to the last correction term. Recently, a certain equivalence of the two methods has been rigorously established by K. Yuan, B. Ying and A.H. Sayed, assuming martingale difference gradient noise. The purpose of this paper is to present the outline of a significantly simpler and more transparent asymptotic analysis of the LMS algorithm with momentum under the assumption of stationary, ergodic and mixing signals.

AB - A widely studied filtering algorithm in signal processing is the least mean square (LMS) method, due to B. Widrow and T. Hoff, 1960. A popular extension of the LMS algorithm, which is also important in deep learning, is the LMS method with momentum, originated by S. Roy and J.J. Shynk back in 1988. This is a fixed gain (or constant step-size) version of the LMS method modified by an additional momentum term that is proportional to the last correction term. Recently, a certain equivalence of the two methods has been rigorously established by K. Yuan, B. Ying and A.H. Sayed, assuming martingale difference gradient noise. The purpose of this paper is to present the outline of a significantly simpler and more transparent asymptotic analysis of the LMS algorithm with momentum under the assumption of stationary, ergodic and mixing signals.

U2 - 10.1109/CDC.2018.8619576

DO - 10.1109/CDC.2018.8619576

M3 - Paper

SP - 3062

EP - 3067

T2 - 2018 IEEE Conference on Decision and Control (CDC)

Y2 - 17 December 2018 through 19 December 2018

ER -