In this paper, we investigate decentralized online convex optimization with compressed communication, where $n$ learners over a network jointly minimize a sequence of global loss functions based only on their local losses and compressed messages from their neighbors. The best known regret bounds for this problem are $O...
Si-Fan Yang, Wen-Hao Yang, Wei Jiang et al.· 0 citations
Adaptive optimization methods have received considerable attention in recent years due to their strong empirical performance. However, they generally require maintaining a preconditioning matrix, which incurs prohibitive computational overhead in large-scale applications. To address these limitations, we investigate th...
Si-Fan Yang, Yuanyu Wan, Yi-Bo Wang et al.· 0 citations
Stochastic recursive momentum (STORM) achieves fast convergence for nonconvex optimization via the variance reduction effect, but existing analyses rely on the strong average smoothness assumption. In this paper, we study the convergence of STORM for different objectives without average smoothness. We first revisit the...
Wei Jiang, Yi-Bo Wang, Wen-Hao Yang et al.· 0 citations
A momentum-based estimator with mini-batches is employed to track the function values of each level, which are subsequently used to construct momentum gradient estimators, and a batch-free method that incorporates a first-order approximation and a clipping technique for function value estimation is developed.
Wei Jiang, Rui Yan, Si-Fan Yang et al.· 0 citations
Agnostic federated learning (AFL) seeks a model that performs reliably across $m$ heterogeneous workers, but communication remains a bottleneck. We improve communication efficiency by reducing the number of synchronization rounds via faster convergence and the communication cost per round via compression. We first prop...
Hao-Min Bai, Jun-Yan Sun, Si-Fan Yang et al.· 0 citations
A framework that reduces dynamic regret minimization to switching regret minimization is presented, which can derive dynamic regret bounds by using off-the-shelf algorithms with switching regret guarantees and all the findings match the minimax optimal results for these three types of losses.
Yi-Bo Wang, Wen-Hao Yang, Si-Fan Yang et al.· 1 citation
This paper solves the problem of sign-based variance reduction methods' failure to obtain optimal convergence rates for both nonconvex stochastic and finite-sum optimization, and proposes tracking the global gradient at the server through unbiased compression of recursive gradient increments.
This paper studies projection-free algorithms for stochastic constrained multi-level compositional optimization and proposes variance-reduced projection-free algorithms and establishes complexity guarantees under both the Frank-Wolfe gap and the gradient mapping criteria.
Wei Jiang, Si-Fan Yang, Wen-Hao Yang et al.· 3 citations
Traditional variance reduction methods (e.g., SPIDER, SARAH, STORM) have been extensively investigated for improving the convergence rates of stochastic optimization. These techniques typically maintain a sequence of estimators for a single function (or gradient) across iterations. However, what if we need to track mul...
Wei Jiang, Si-Fan Yang, Yi-Bo Wang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.