Federated Learning and Proximal Policy Optimization for Content Caching in D2D Networks

Abstract

Device-to-device (D2D) communication networks face challenges in efficient content caching under limited bandwidth, stringent delay requirements, and dynamic user mobility. To address these issues, we propose a federated learning (FL)-enabled hierarchical edge caching framework that integrates three key components: a mobility-aware variant of density-based spatial clustering of applications with noise (DBSCAN) to form stable device clusters based on velocity and content similarity; a deep deterministic policy gradient (DDPG)-based mechanism to select the most capable master user equipment (MUE) in each cluster using link quality, battery level, and processing power; and a two-timescale deep reinforcement learning (DRL) strategy that combines deep Q-learning (DQL) for short-term caching decisions with proximal policy optimization (PPO) for long-term policy refinement and cache eviction. The proposed framework optimizes a multi-objective reward that jointly accounts for content delivery delay, cache hit ratio (CHR), mobility cost, and throughput. Simulation results over networks with 500–1000 mobile devices show that our method achieves a CHR of 94.6%, reduces average content delivery delay from 168 ms to 155 ms as cache capacity increases, and maintains higher throughput than classical caching policies and single-timescale DRL baselines. These results validate the framework’s adaptability and effectiveness in dynamic D2D environments.