The Practicality of Stochastic Optimization in Imaging Inverse Problems

Junqi Tang, Karen Egiazarian, Mohammad Golbabaee, Michael Davies

Research output: Contribution to journalArticlepeer-review

Abstract / Description of output

In this work we investigate the practicality of stochastic gradient descent and its variants with variance-reduction techniques in imaging inverse problems. Such algorithms have been shown in the large-scale optimization and machine learning literature to have optimal complexity in theory, and to provide great improvement empirically over the deterministic gradient methods. However, in some tasks such as image deblurring, many of such methods fail to converge faster than the deterministic gradient methods, even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism for the practitioners to efficiently characterize whether it is beneficial for an inverse problem to be solved by stochastic optimization techniques or not. Using standard tools in numerical linear algebra, we derive conditions on the spectral structure of the inverse problem for being a suitable application of stochastic gradient methods. Particularly, if the Hessian matrix of an imaging inverse problem has a fast-decaying eigenspectrum, then our theory suggests that the stochastic gradient methods can be more advantageous than deterministic methods for solving such a problem. Our results also provide guidance on choosing appropriately the partition minibatch schemes, showing that a good minibatch scheme typically has relatively low correlation within each of the minibatches. Finally, we present numerical studies which validate our results.
Original languageEnglish
Pages (from-to)1471-1485
Number of pages15
JournalIEEE Transactions on Computational Imaging
Volume6
Early online date21 Oct 2020
DOIs
Publication statusE-pub ahead of print - 21 Oct 2020

Keywords / Materials (for Non-textual outputs)

  • Imaging Inverse Problems
  • Stochastic Optimization
  • Large-scale Optimization

Fingerprint

Dive into the research topics of 'The Practicality of Stochastic Optimization in Imaging Inverse Problems'. Together they form a unique fingerprint.

Cite this