An efficient proximal point algorithm (PPA) is developed to solve the stationary dual problem and the SNA subroutine is incorporated into the inexact PPA to solve jump-sparse signal recovery and computed tomography (CT) image restoration.
Abstract
Many image-processing problems can be formulated as composite cardinality optimization (CCOP), whose objective is the sum of two convex terms and a cardinality function composed with a linear operator. The composite cardinality term creates major algorithmic challenges: the separability of the cardinality function is lost and convergence analysis often requires surjectivity-type assumptions on the linear operators. To overcome these challenges, we consider the stationary dual formulation of CCOP, which has more favorable structure consisting of two convex terms and a separable cardinality function. In this paper, we develop an efficient proximal point algorithm (PPA) to solve the stationary dual problem. The efficiency of our PPA stems from two aspects. Firstly, the key step of its subproblem solver minimizes a smooth convex function over a low-dimensional subspace by the classic semismooth Newton algorithm (SNA), which has global convergence and local superlinear rate under suitable conditions. Secondly, implementable inexact criteria are proposed for early termination of the SNA subroutine. These components form the basic framework of our inexact PPA. Under suitable conditions, it enjoys global convergence and local linear convergence rate. In particular, we provide examples in which the convergence assumptions are automatically satisfied. Finally, the SNA subroutine is incorporated into our inexact PPA to solve jump-sparse signal recovery and computed tomography (CT) image restoration. Numerical results demonstrate the time efficiency and solution accuracy of our proposed method.
We study online optimization for a broad class of structured non-convex non-smooth problems where each loss is a composition of a difference-of-convex function with a smooth mapping, and the feasible region is defined by constraint functions of the same kind. We propose a time-smoothed proximal linear algorithm and a local-regret measure based on a proximal residual mapping. We show that this residual is a proper stationarity measure for the original problem: its fixed-point condition implies first-order stationarity. Our analysis relies on a tangent-cone characterization for a feasible region described by composite difference-of-convex constraints, which is of independent interest and allows each update to be computed via a convex optimization oracle, despite the non-convexity of the problem. We establish a local-regret bound and a bound on the total number of inner convex subproblems. We also derive an error bound connecting the proximal residual to the distance to stationarity, providing a quantitative certificate of approximate stationarity.
Jingwei Ji, Jong-Shi Pang, Renyuan Xu· 0 citations
Monotonic optimisation is a broad class of non-convex problems formulated in terms of monotone functions. Such problems are commonly solved via the polyblock outer-approximation algorithm (POA), a branch-and-bound method that iteratively refines a rectangular outer-approximation of the feasible set. POA scales poorly, however: the number of vertices needed to describe the approximation can grow exponentially, leading to large memory requirements and increasingly expensive subroutines. To address these limitations, we propose three algorithmic improvements: a generalised anchor selection that yields an optimal balanced monotonicity cut, a relaxed optimality condition that guarantees finite termination without continuity assumptions, and a vectorised variant that processes multiple nodes concurrently. We further develop an efficient tree-based implementation, which accelerates POA's core subroutines while storing the outer-approximation compactly. Numerical experiments show that these improvements yield order-of-magnitude speed-ups over standard POA and solve problems on which existing variants fail. Finally, we introduce polyblocks, an open-source Python package implementing the proposed algorithms alongside a framework for developing new POA variants, available at https://github.com/RashwanA/polyblocks.
This work presents two different types of proximal gradient methods, with line search and without line search, for solving unconstrained set-valued optimization problems under the lower set-less ordering relation induced by a solid cone that is convex, pointed, and closed. The objective mapping of the problem involves finitely many functions, with each one being the sum of a continuously differentiable function and a convex function that is proper and closed. We present an approach to characterize weakly minimal points of the problem with the help of weakly efficient points of a family of vector optimization problems. Thereafter, we establish a stationarity condition along with its connection with weakly minimal points of the problem under study. Based on the stationary condition, the concept of a descent direction at a non-stationary point is discussed. In view of the line search-based method, we formulate an Armijo-type line search condition and establish the existence of such a step-size. For the proposed methods, global convergence is established under mild assumptions. The convergence analysis of the proximal gradient method with line search provides a theoretical advancement over the convergence results previously established for the steepest descent method in set-valued optimization problems. In addition, we analyze the computational complexity of the proposed methods and show that both methods achieve a convergence rate of $\mathcal{O}(1/\sqrt{k})$. Numerical results are reported to test the performance of the methods in practice.
Ravi Raushan, Debdas Ghosh, Anshika et al.· 0 citations
In this paper, we propose a balanced augmented Lagrangian method based on accelerated stochastic ADMM (b-ASADMM) to efficiently solve structured separable nonconvex optimization problems subject to linear constraints. The objective function in this problem comprises potentially nonsmooth and smooth functions, where the smooth function is an average of multiple nonconvex smooth functions. The involved smooth subproblem is tackled by an accelerated stochastic gradient method based on weighting of stochastic item and pre-variable. The involved nonsmooth subproblem is solved under incorporation of Bregman distance to avoid the case that subproblem does not have a closed-form solution due to the complicated quadratic term or other hindering. The involved balanced augmented Lagrangian method advances the original ALM by balancing its subproblems and improving its implementation. In contrast to most deterministic and stochastic ADMMs, our dual variable allows a more flexible and larger step-size region. By standard smoothness assumption, we establish the global convergence and iteration complexity of the generated sequence. Furthermore, we provide a linear convergence rate of b-ASADMM under a local error bound condition and the weakly convex property of the nonsmooth component. Numerical experiments on the graph-guided fused Lasso problem and the smooth clipped absolute deviation penalty problem are conducted to verify the effectiveness of b-ASADMM.
Convex-concave saddle-point problems are ubiquitous across diverse domains, including machine learning, image processing, economics, and equilibrium problems. Primal-dual algorithms provide a highly effective and powerful framework for convex-concave saddle-point problems. Convex combination has become a crucial acceleration technique for primal-dual algorithms, and the integration of this technique has recently made these algorithms a highly active research topic. The choice of the convex combination parameter often has a significant impact on both the theoretical analysis and the numerical performance of the corresponding algorithms. However, the requirements on this parameter imposed by theory are sometimes inconsistent with those suggested by numerical experiments. For instance, theoretical analysis often requires the parameter to be small, while numerical experiments tend sometimes to favor larger values. To address this inconsistency and further advance primal-dual algorithms with convex combination, we develop a novel strategy based on two convex combinations, integrate it into a primal-dual framework, and propose a new primal-dual algorithm with linesearch, termed NPDAL-n, for general convex-concave saddle-point problems. The proposed two convex combinations in NPDAL-n ensure that the permissible range of the convex combination parameters is mainly determined by theoretical considerations, with little regard for numerical performance. Through rigorous Lyapunov energy descent analysis, we establish the global convergence and a sublinear ergodic convergence rate of $\mathcal{O}(1/N)$ for NPDAL-n under standard assumptions. When the primal function is strongly convex, we develop an accelerated version of NPDAL-n that achieves an optimal $\mathcal{O}(1/N^2)$ rate.
This paper proposes a general line-search Newton framework for unconstrained optimization that avoids repeated Hessian regularization by exploiting the Newton direction only when it is well-defined and suitable and provides the first Newton-type algorithm together with a comprehensive convergence analysis for this important class of nonconvex optimization problems.