Analysis of Set-Valued Stochastic Approximations: Applications to Noisy Approximate Value and Fixed point Iterations

The main aim of this paper is the development of Lyapunov function based sufficient conditions for stability (almost sure boundedness) and convergence of stochastic approximation algorithms (SAAs) with set-valued mean-fields, a class of model-free algorithms that have become important in recent times. In this paper we provide a complete analysis of such algorithms under three different, yet related set of sufficient conditions, based on the existence of an associated global/local Lyapunov function. Unlike previous Lyapunov function based approaches, we provide a simple recipe for explicitly constructing the Lyapunov function needed for analysis. Our work builds on the works of Abounadi, Bertsekas and Borkar (2002), Munos (2005) and Ramaswamy and Bhatnagar (2016). An important motivation to the flavor of our assumptions comes from the need to understand approximate dynamic programming and reinforcement learning algorithms, that use deep neural networks (DNNs) for function approximations and parameterizations. These algorithms are popularly known as deep reinforcement learning algorithms. As an important application of our theory we provide a complete analysis of the stochastic approximation counterpart of approximate value iteration (AVI), an important dynamic programming method designed to tackle Bellman's curse of dimensionality. Although motivated by the need to understand deep reinforcement learning algorithms our theory is more generally applicable. It is further used to develop the first SAA for finding fixed points of contractive set-valued maps and provide a comprehensive analysis of the same.