Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectories

Mobile Edge Computing (MEC) broadens the scope of computation and storage beyond the central network, incorporating edge nodes close to end devices. This expansion facilitates the implementation of large-scale "connected things" within edge networks. The advent of applications necessitating real-time, high-quality service presents several challenges, such as low latency, high data rate, reliability, efficiency, and security, all of which demand resolution. The incorporation of reinforcement learning (RL) methodologies within MEC networks promotes a deeper understanding of mobile user behaviors and network dynamics, thereby optimizing resource use in computing and communication processes. This paper offers an exhaustive survey of RL applications in MEC networks, initially presenting an overview of RL from its fundamental principles to the latest advanced frameworks. Furthermore, it outlines various RL strategies employed in offloading, caching, and communication within MEC networks. Finally, it explores open issues linked with software and hardware platforms, representation, RL robustness, safe RL, large-scale scheduling, generalization, security, and privacy. The paper proposes specific RL techniques to mitigate these issues and provides insights into their practical applications.

HARQ Retransmissions in C-V2X: A BSM Latency Analysis

Cellular vehicular-to-everything (C-V2X) systems offer the potential for improving road safety, in part through the exchange of periodic basic safety messages (BSMs) between nearby vehicles. The reliability and latency of these messages is a key metric. Hybrid automatic repeat request (HARQ) retransmissions are one technique used to this end. However, HARQ may come at the expense of consuming the limited available wireless resources, especially in highly congested scenarios. This paper studies BSM transmission latency and reliability when HARQ retransmissions are used with the semi-persistent scheduling (SPS) in C-V2X transmission mode 4. We do so through extensive system-level simulations that closely follow the SPS process. Furthermore, we provide an analytical model for the tail behavior of the BSM latency distribution with HARQ retransmissions that is a good approximation to the simulation results. Our study reveals the impact of several deployment settings (e.g., bandwidth configurations and vehicle density).

Study of BSM Inter-Packet Gap Tails in C-V2X Networks

Cellular vehicle-to-everything (C-V2X) enables safety-critical connected vehicular service by exchanging basic safety messages (BSMs) among nearby vehicular users (VUEs). Timely transmission of BSMs is crucial to avoid stale information at VUEs. However, successive packet losses can lead to large inter-packet gaps (IPGs), reducing the BSMs' reliability. This paper investigates the tail behavior of IPG and information age (IA) distributions in C-V2X mode 4, a decentralized resource allocation method based on semi-persistent scheduling (SPS). We study the improvements and trade-offs introduced by SAE one-shot transmission to decrease the number of successive BSM losses at destination VUEs. The study employs high-fidelity system-level simulations that closely follow the SPS process of CV2X mode 4 to evaluate the performance of interleaved one-shot SPS transmissions. The numerical results demonstrate significant improvement in the IPG and IA tail distributions in various simulation scenarios. Additionally, we propose an accurate analytical model to characterize the IPG tail behavior of C-V2X BSM transmissions. The proposed model is validated by comparing its results with those obtained using the system-level simulations. Our validation shows that the proposed model generates analytical results that coincide with the asymptotic slopes of IPG distribution in different BSM transmission modes.

Interleaved One-shot Semi-Persistent Scheduling for BSM Transmissions in C-V2X Networks

Cellular vehicle-to-everything (C-V2X) networks are regarded as one of the main pillars to enable efficient and sustainable Intelligent Transportation Systems (ITS) safety applications and services. Such services rely on the concept of exchanging periodic status updates (i.e., basic safety messages (BSMs)) between nearby vehicular users (VUEs). Hence, it is essential to ensure small inter-packet gaps (IPGs) between successive BSMs from nearby VUEs. Large IPGs, due to successive packet losses, can result in stale information at a VUE. In this paper, we study the tail behavior of the IPG and the information age (IA) distributions using C-V2X transmission mode 4 (a decentralized resource allocation method based on semi-persistent scheduling (SPS)). Specifically, we investigate improvements and trade-offs introduced by the SAE-specified concept of one-shot transmissions. We use a high-fidelity system-level simulator that closely follows the SPS process of C-V2X transmission mode 4 to evaluate the performance of the interleaved one-shot SPS transmissions. Our numerical results show that the tails of the IA and IPG complementary cumulative distribution functions (CCDFs) are significantly improved when one-shot transmissions are enabled in various simulation scenarios.

Faithful Edge Federated Learning: Scalability and Privacy

Federated learning enables machine learning algorithms to be trained over a network of multiple decentralized edge devices without requiring the exchange of local datasets. Successfully deploying federated learning requires ensuring that agents (e.g., mobile devices) faithfully execute the intended algorithm, which has been largely overlooked in the literature. In this study, we first use risk bounds to analyze how the key feature of federated learning, unbalanced and non-i.i.d. data, affects agents' incentives to voluntarily participate and obediently follow traditional federated learning algorithms. To be more specific, our analysis reveals that agents with less typical data distributions and relatively more samples are more likely to opt out of or tamper with federated learning algorithms. To this end, we formulate the first faithful implementation problem of federated learning and design two faithful federated learning mechanisms which satisfy economic properties, scalability, and privacy. Further, the time complexity of computing all agents' payments in the number of agents is $\mathcal{O}(1)$. First, we design a Faithful Federated Learning (FFL) mechanism which approximates the Vickrey-Clarke-Groves (VCG) payments via an incremental computation. We show that it achieves (probably approximate) optimality, faithful implementation, voluntary participation, and some other economic properties (such as budget balance). Second, by partitioning agents into several subsets, we present a scalable VCG mechanism approximation. We further design a scalable and Differentially Private FFL (DP-FFL) mechanism, the first differentially private faithful mechanism, that maintains the economic properties. Our mechanism enables one to make three-way performance tradeoffs among privacy, the iterations needed, and payment accuracy loss.

Fictitious GAN: Training GANs with Historical Models

Generative adversarial networks (GANs) are powerful tools for learning generative models. In practice, the training may suffer from lack of convergence. GANs are commonly viewed as a two-player zero-sum game between two neural networks. Here, we leverage this game theoretic view to study the convergence behavior of the training process. Inspired by the fictitious play learning process, a novel training method, referred to as Fictitious GAN, is introduced. Fictitious GAN trains the deep neural networks using a mixture of historical models. Specifically, the discriminator (resp. generator) is updated according to the best-response to the mixture outputs from a sequence of previously trained generators (resp. discriminators). It is shown that Fictitious GAN can effectively resolve some convergence issues that cannot be resolved by the standard training approach. It is proved that asymptotically the average of the generator outputs has the same distribution as the data samples.