Fitted Q-Learning in Mean-field Games

In the literature, existence of equilibria for discrete-time mean field games has been in general established via Kakutani's Fixed Point Theorem. However, this fixed point theorem does not entail any iterative scheme for computing equilibria. In this paper, we first propose a Q-iteration algorithm to compute equilibria for mean-field games with known model using Banach Fixed Point Theorem. Then, we generalize this algorithm to model-free setting using fitted Q-iteration algorithm and establish the probabilistic convergence of the proposed iteration. Then, using the output of this learning algorithm, we construct an approximate Nash equilibrium for finite-agent stochastic game with mean-field interaction between agents.