Projects per year
Abstract / Description of output
We consider nonconvex stochastic optimization problems where the objective functions have superlinearly growing and discontinuous stochastic gradients. In such a setting, we provide a nonasymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2021). In particular, we establish nonasymptotic error bounds for the TUSLA algorithm in Wasserstein1 and Wasserstein2 distances. The latter result enables us to further derive nonasymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which supports our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) SGD, may fail to find the minimizer of the objective functions due to the superlinear growth and the discontinuity of the corresponding stochastic gradient, while the TUSLA algorithm converges rapidly to the optimal solution.
Original language  English 

Journal  IMA Journal of Numerical Analysis 
Publication status  Accepted/In press  28 Apr 2023 
Keywords / Materials (for Nontextual outputs)
 math.OC
 cs.LG
 cs.NA
 math.NA
 math.PR
 stat.ML
Fingerprint
Dive into the research topics of 'Nonasymptotic estimates for TUSLA algorithm for nonconvex learning with applications to neural networks with ReLU activation function'. Together they form a unique fingerprint.Projects
 1 Finished

TRAIN@Ed: Transnational Research and Innovation Network at Edinburgh
Gorjanc, G., Bell, C., Duncan, A., Farrington, S., Florian, L., Forde, M., Hickey, J., Lacka, E., Ma, T., Mcneill, G., MedinaLopez, E., Rosser, S., Rossi, R., Sabanis, S., Szpruch, L., Tenesa, A., Wake, D., Williamson, B. & Yang, Y.
EU government bodies, NonEU industry, commerce and public corporations
1/11/19 → 19/04/23
Project: Research