Multiscale replay: A robust algorithm for stochastic variational inequalities with a Markovian buffer

We introduce the Multiscale Experience Replay (MER) algorithm for solving a class of stochastic variational inequalities (VIs) in settings where samples are generated from a Markov chain and we have access to a memory buffer to store them. Rather than uniformly sampling from the buffer, MER utilizes a multi-scale sampling scheme to emulate the behavior of VI algorithms designed for independent and identically distributed samples, overcoming bias in the de facto serial scheme and thereby accelerating convergence. Notably, unlike standard sample-skipping variants of serial algorithms, MER is robust in that it achieves this acceleration in iteration complexity whenever possible, and without requiring knowledge of the mixing time of the Markov chain. We also discuss applications of MER, particularly in policy evaluation with temporal difference learning and in training generalized linear models with dependent data.

Estimating stationary mass, frequency by frequency

Suppose we observe a trajectory of length $n$ from an $\alpha$-mixing stochastic process over a finite but potentially large state space. We consider the problem of estimating the probability mass placed by the stationary distribution of any such process on elements that occur with a certain frequency in the observed sequence. We estimate this vector of probabilities in total variation distance, showing universal consistency in $n$ and recovering known results for i.i.d. sequences as special cases. Our proposed methodology carefully combines the plug-in (or empirical) estimator with a recently-proposed modification of the Good--Turing estimator called \textsc{WingIt}, which was originally developed for Markovian sequences. En route to controlling the error of our estimator, we develop new performance bounds on \textsc{WingIt} and the plug-in estimator for $\alpha$-mixing stochastic processes. Importantly, the extensively used method of Poissonization can no longer be applied in our non i.i.d. setting, and so we develop complementary tools -- including concentration inequalities for a natural self-normalized statistic of mixing sequences -- that may prove independently useful in the design and analysis of estimators for related problems.

Variational Learning Algorithms For Channel Estimation in RIS-assisted mmWave Systems

We consider the problem of estimating the channel in reconfigurable intelligent surface (RIS) assisted millimeter wave (mmWave) systems. We propose two variational expectation maximization (VEM) based algorithms for channel estimation in RIS-aided wireless systems. The first algorithm is a structured mean field-based sparse Bayesian learning (SM-SBL) algorithm that exploits the doubly-structured sparsity and the individual sparsity of the elements of the channel. To exploit the sparsities, we propose a column-wise coupled Gaussian prior. We next design the factorized mean field-based algorithm based on the prior we propose. This algorithm called the factorized mean field SBL (FM-SBL) algorithm, addresses the time complexities of the SM-SBL algorithm without sacrificing channel estimation accuracy. We show using extensive numerical investigations that the i) proposed SM-SBL and FM-SBL algorithms outperform several existing algorithms and ii) FM-SBL has lower time complexity than the SM-SBL algorithm.