Sequential Stochastic Combinatorial Optimization Using Hierarchal Reinforcement Learning

Reinforcement learning (RL) has emerged as a promising tool for combinatorial optimization (CO) problems due to its ability to learn fast, effective, and generalizable solutions. Nonetheless, existing works mostly focus on one-shot deterministic CO, while sequential stochastic CO (SSCO) has rarely been studied despite its broad applications such as adaptive influence maximization (IM) and infectious disease intervention. In this paper, we study the SSCO problem where we first decide the budget (e.g., number of seed nodes in adaptive IM) allocation for all time steps, and then select a set of nodes for each time step. The few existing studies on SSCO simplify the problems by assuming a uniformly distributed budget allocation over the time horizon, yielding suboptimal solutions. We propose a generic hierarchical RL (HRL) framework called wake-sleep option (WS-option), a two-layer option-based framework that simultaneously decides adaptive budget allocation on the higher layer and node selection on the lower layer. WS-option starts with a coherent formulation of the two-layer Markov decision processes (MDPs), capturing the interdependencies between the two layers of decisions. Building on this, WS-option employs several innovative designs to balance the model's training stability and computational efficiency, preventing the vicious cyclic interference issue between the two layers. Empirical results show that WS-option exhibits significantly improved effectiveness and generalizability compared to traditional methods. Moreover, the learned model can be generalized to larger graphs, which significantly reduces the overhead of computational resources.

Adaptive Cell Range Expansion in Multi-Band UAV Communication Networks

This paper leverages stochastic geometry to model, analyze, and optimize multi-band unmanned aerial vehicle (UAV) communication networks operating across low-frequency and millimeter-wave (mmWave) bands. We introduce a novel approach to modeling mmWave antenna gain in such networks, which allows us to better capture and account for interference in our analysis and optimization. We then propose a simple yet effective user-UAV association policy, which strategically biases users towards mmWave UAVs to take advantage of lower interference and wider bandwidths compared to low-frequency UAVs. Under this scheme, we analytically derive the corresponding association probability, coverage probability, and spectral efficiency. We conclude by assessing our proposed association policy through simulation and analysis, demonstrating its effectiveness based on coverage probability and per-user data rates, as well as the alignment between analytical and simulation results.