arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Qihang Lin

78 papers · 2010–2026

2010: 42011: 12012: 22013: 22014: 52015: 82016: 82017: 42018: 62019: 72020: 42021: 12022: 52023: 42024: 62025: 82026: 3
2010peak 8 in 20152026

Frequent co-authors

Papers

showing 1–25 of 78 RSS

Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learning

Hassan Rafique, Mingrui Liu, Qihang Lin +1 more

math.OC · · cited 144× · arXiv ↗ · more like this ·

Min-max problems have broad applications in machine learning, including learning with non-decomposable loss and learning with robustness to data distribution. Convex-concave min-max problem is an active topic of research with efficient algorithms and sound theoretical foundations developed. However, it remains a challenge to design provably efficient algorithms for non-convex min-max problems with or without smoothness. In this paper, we study a family of non-convex min-max problems, whose objective function is weakly convex in the variables of minimization and is concave in the variables of maximization. We propose a proximally guided stochastic subgradient method and a proximally guided stochastic variance-reduced method for the non-smooth and smooth instances, respectively, in this family of problems. We analyze the time complexities of the proposed methods for finding a nearly…

A Unified Analysis of Stochastic Momentum Methods for Deep Learning

Yan Yan, Tianbao Yang, Zhe Li +2 more

cs.LG · · cited 143× · arXiv ↗ · more like this ·

Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochastic gradient (SG) method, and the stochastic momentum methods including two famous variants, i.e., the stochastic heavy-ball (SHB) method and the stochastic variant of Nesterov's accelerated gradient (SNAG) method. We propose a framework that unifies the three variants. We then derive the convergence rates of the norm of gradient for the non-convex optimization problem, and analyze the generalization performance through the uniform stability approach. Particularly, the convergence analysis of the training objective exhibits that SHB and SNAG have no advantage over SG.…

Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization

Tianbao Yang, Qihang Lin, Zhe Li

math.OC · · cited 134× · arXiv ↗ · more like this ·

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of two stochastic momentum methods, namely stochastic heavy-ball method and the stochastic variant of Nesterov's accelerated gradient method. We hope that the basic convergence results developed in this paper can serve the reference to the convergence of stochastic momentum methods and also serve the baselines for comparison in future development of stochastic momentum methods. The novelty of convergence analysis presented in this paper is a unified framework, revealing more insights about the similarities and differences between different stochastic momentum methods and…

Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing

Xi Chen, Qihang Lin, Dengyong Zhou

Computer Science Applications · International Conference on Machine Learning · · cited 108× · OpenAlex ↗ · more like this ·

In real crowdsourcing applications, each label from a crowd usually comes with a certain cost. Given a pre-fixed amount of budget, since different tasks have different ambiguities and different workers have different expertises, we want to find an optimal way to allocate the budget among instance-worker pairs such that the overall label quality can be maximized. To address this issue, we s-tart from the simplest setting in which all workers are assumed to be perfect. We formulate the problem as a Bayesian Markov Decision Process (MDP). Using the dynamic programming (DP) algorithm, one can obtain the optimal allocation policy for a given budget. However, DP is computationally intractable. To solve the computational challenge, we propose a novel approximate policy which is called optimistic knowledge gradient. It is practically efficient while theoretically its consistency can be…

RSG: Beating Subgradient Method without Smoothness and Strong Convexity

Tianbao Yang, Qihang Lin

math.OC · · cited 92× · arXiv ↗ · more like this ·

In this paper, we study the efficiency of a {\bf R}estarted {\bf S}ub{\bf G}radient (RSG) method that periodically restarts the standard subgradient method (SG). We show that, when applied to a broad class of convex optimization problems, RSG method can find an $ε$-optimal solution with a lower complexity than the SG method. In particular, we first show that RSG can reduce the dependence of SG's iteration complexity on the distance between the initial solution and the optimal set to that between the $ε$-level set and the optimal set {multiplied by a logarithmic factor}. Moreover, we show the advantages of RSG over SG in solving three different families of convex optimization problems. (a) For the problems whose epigraph is a polyhedron, RSG is shown to converge linearly. (b) For the problems with local quadratic growth property in the $ε$-sublevel set, RSG has an…

An Accelerated Proximal Coordinate Gradient Method

Qihang Lin, Zhaosong Lu, Lin Xiao

Artificial Intelligence · Neural Information Processing Systems · · cited 85× · OpenAlex ↗ · more like this ·

We develop an accelerated randomized proximal coordinate gradient (APCG) method, for solving a broad class of composite convex optimization problems. In particular, our method achieves faster linear convergence rates for minimizing strongly convex functions than existing randomized proximal coordinate gradient methods. We show how to apply the APCG method to solve the dual of the regularized empirical risk minimization (ERM) problem, and devise efficient implementations that avoid full-dimensional vector operations. For ill-conditioned ERM problems, our method obtains improved convergence rates than the state-of-the-art stochastic dual coordinate ascent (SDCA) method.

Generalized Inverse Classification

Michael T. Lash, Qihang Lin, W. Nick Street +2 more

cs.LG · · cited 68× · arXiv ↗ · more like this ·

Inverse classification is the process of perturbing an instance in a meaningful way such that it is more likely to conform to a specific class. Historical methods that address such a problem are often framed to leverage only a single classifier, or specific set of classifiers. These works are often accompanied by naive assumptions. In this work we propose generalized inverse classification (GIC), which avoids restricting the classification model that can be used. We incorporate this formulation into a refined framework in which GIC takes place. Under this framework, GIC operates on features that are immediately actionable. Each change incurs an individual cost, either linear or non-linear. Such changes are subjected to occur within a specified level of cumulative change (budget). Furthermore, our framework incorporates the estimation of features that change as a consequence of direct…

DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization

Lin Xiao, Adams Wei Yu, Qihang Lin +1 more

math.OC · · cited 61× · arXiv ↗ · more like this ·

Machine learning with big data often involves large optimization models. For distributed optimization over a cluster of machines, frequent communication and synchronization of all model parameters (optimization variables) can be very costly. A promising solution is to use parameter servers to store different subsets of the model parameters, and update them asynchronously at different machines using local datasets. In this paper, we focus on distributed optimization of large linear models with convex loss functions, and propose a family of randomized primal-dual block coordinate algorithms that are especially suitable for asynchronous distributed implementation with parameter servers. In particular, we work with the saddle-point formulation of such problems which allows simultaneous data and model partitioning, and exploit its structure by doubly stochastic coordinate optimization with…

An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimization

Qihang Lin, Zhaosong Lu, Lin Xiao

math.OC · · cited 59× · arXiv ↗ · more like this ·

We consider the problem of minimizing the sum of two convex functions: one is smooth and given by a gradient oracle, and the other is separable over blocks of coordinates and has a simple known structure over each block. We develop an accelerated randomized proximal coordinate gradient (APCG) method for minimizing such convex composite functions. For strongly convex functions, our method achieves faster linear convergence rates than existing randomized proximal coordinate gradient methods. Without strong convexity, our method enjoys accelerated sublinear convergence rates. We show how to apply the APCG method to solve the regularized empirical risk minimization (ERM) problem, and devise efficient implementations that avoid full-dimensional vector operations. For ill-conditioned ERM problems, our method obtains improved convergence rates than the state-of-the-art stochastic dual…

Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling

Xi Chen, Qihang Lin, Dengyong Zhou

cs.LG · · cited 55× · arXiv ↗ · more like this ·

In crowd labeling, a large amount of unlabeled data instances are outsourced to a crowd of workers. Workers will be paid for each label they provide, but the labeling requester usually has only a limited amount of the budget. Since data instances have different levels of labeling difficulty and workers have different reliability, it is desirable to have an optimal policy to allocate the budget among all instance-worker pairs such that the overall labeling accuracy is maximized. We consider categorical labeling tasks and formulate the budget allocation problem as a Bayesian Markov decision process (MDP), which simultaneously conducts learning and decision making. Using the dynamic programming (DP) recurrence, one can obtain the optimal allocation policy. However, DP quickly becomes computationally intractable when the size of the problem increases. To solve this challenge, we propose a…

Sparse Latent Semantic Analysis

Xi Chen, Yanjun Qi, Bing Bai +2 more

Artificial Intelligence · Figshare · · cited 43× · DOI ↗ · more like this ·

Latent semantic analysis (LSA), as one of the most popular unsupervised dimension reduction tools, has a wide range of applications in text mining and information retrieval. The key idea of LSA is to learn a projection matrix that maps the high dimensional vector space representations of documents to a lower dimensional latent space, i.e. so called latent topic space. In this paper, we propose a new model called Sparse LSA, which produces a sparse projection matrix via the ℓ1 regularization. Compared to the traditional LSA, Sparse LSA selects only a small number of relevant words for each topic and hence provides a compact representation of topic-word relationships. Moreover, Sparse LSA is computationally very efficient with much less memory usage for storing the projection matrix. Furthermore, we propose two important extensions of Sparse LSA: group structured Sparse LSA and…

Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure

Yan Yan, Yi Xu, Qihang Lin +2 more

cs.LG · · cited 40× · arXiv ↗ · more like this ·

Previous studies on stochastic primal-dual algorithms for solving min-max problems with faster convergence heavily rely on the bilinear structure of the problem, which restricts their applicability to a narrowed range of problems. The main contribution of this paper is the design and analysis of new stochastic primal-dual algorithms that use a mixture of stochastic gradient updates and a logarithmic number of deterministic dual updates for solving a family of convex-concave problems with no bilinear structure assumed. Faster convergence rates than $O(1/\sqrt{T})$ with $T$ being the number of stochastic gradient updates are established under some mild conditions of involved functions on the primal and the dual variable. For example, for a family of problems that enjoy a weak strong convexity in terms of the primal variable and has a strongly concave function of the dual variable, the…

On Degrees of Freedom of Projection Estimators with Applications to Multivariate Nonparametric Regression

Xi Chen, Qihang Lin, Bodhisattva Sen

math.ST · · cited 25× · arXiv ↗ · more like this ·

In this paper, we consider the nonparametric regression problem with multivariate predictors. We provide a characterization of the degrees of freedom and divergence for estimators of the unknown regression function, which are obtained as outputs of linearly constrained quadratic optimization procedures, namely, minimizers of the least squares criterion with linear constraints and/or quadratic penalties. As special cases of our results, we derive explicit expressions for the degrees of freedom in many nonparametric regression problems, e.g., bounded isotonic regression, multivariate (penalized) convex regression, and additive total variation regularization. Our theory also yields, as special cases, known results on the degrees of freedom of many well-studied estimators in the statistics literature, such as ridge regression, Lasso and generalized Lasso. Our results can be readily used to…

Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network

Adams Wei Yu, Lei Huang, Qihang Lin +2 more

cs.LG · · cited 24× · arXiv ↗ · more like this ·

In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient. We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi-layer perceptron, convolution neural networks and recurrent neural networks. The results indicate the block-normalized gradient can help accelerate the training of neural networks. In particular, we observe that the normalized gradient methods having constant step size with occasionally decay, such as SGD with momentum, have better performance in the deep…

Hybrid Predictive Model: When an Interpretable Model Collaborates with a Black-box Model

Tong Wang, Qihang Lin

cs.LG · · cited 19× · arXiv ↗ · more like this ·

Interpretable machine learning has become a strong competitor for traditional black-box models. However, the possible loss of the predictive performance for gaining interpretability is often inevitable, putting practitioners in a dilemma of choosing between high accuracy (black-box models) and interpretability (interpretable models). In this work, we propose a novel framework for building a Hybrid Predictive Model (HPM) that integrates an interpretable model with any black-box model to combine their strengths. The interpretable model substitutes the black-box model on a subset of data where the black-box is overkill or nearly overkill, gaining transparency at no or low cost of the predictive accuracy. We design a principled objective function that considers predictive accuracy, model interpretability, and model transparency (defined as the percentage of data processed by the…

Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimization

Yan Yan, Yi Xu, Qihang Lin +2 more

math.OC · · cited 18× · arXiv ↗ · more like this ·

Epoch gradient descent method (a.k.a. Epoch-GD) proposed by Hazan and Kale (2011) was deemed a breakthrough for stochastic strongly convex minimization, which achieves the optimal convergence rate of $O(1/T)$ with $T$ iterative updates for the {\it objective gap}. However, its extension to solving stochastic min-max problems with strong convexity and strong concavity still remains open, and it is still unclear whether a fast rate of $O(1/T)$ for the {\it duality gap} is achievable for stochastic min-max optimization under strong convexity and strong concavity. Although some recent studies have proposed stochastic algorithms with fast convergence rates for min-max problems, they require additional assumptions about the problem, e.g., smoothness, bi-linear structure, etc. In this paper, we bridge this gap by providing a sharp analysis of epoch-wise stochastic gradient descent ascent…

A Smoothing Stochastic Gradient Method for Composite Optimization

Qihang Lin, Xi Chen, Javier Pena

math.OC · · cited 17× · arXiv ↗ · more like this ·

We consider the unconstrained optimization problem whose objective function is composed of a smooth and a non-smooth conponents where the smooth component is the expectation a random function. This type of problem arises in some interesting applications in machine learning. We propose a stochastic gradient descent algorithm for this class of optimization problem. When the non-smooth component has a particular structure, we propose another stochastic gradient descent algorithm by incorporating a smoothing method into our first algorithm. The proofs of the convergence rates of these two algorithms are given and we show the numerical performance of our algorithm by applying them to regularized linear regression problems with different sets of synthetic data.

Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity

Jason D. Lee, Qihang Lin, Tengyu Ma +1 more

math.OC · · cited 16× · arXiv ↗ · more like this ·

We study distributed optimization algorithms for minimizing the average of convex functions. The applications include empirical risk minimization problems in statistical machine learning where the datasets are large and have to be stored on different machines. We design a distributed stochastic variance reduced gradient algorithm that, under certain conditions on the condition number, simultaneously achieves the optimal parallel runtime, amount of communication and rounds of communication among all distributed first-order methods up to constant factors. Our method and its accelerated extension also outperform existing distributed algorithms in terms of the rounds of communication as long as the condition number is not too large compared to the size of data in each machine. We also prove a lower bound for the number of rounds of communication for a broad class of distributed first-order…

Distributed formation control for leader‐follower quadrotor unmanned aerial vehicles system based on practical sliding mode and event‐triggered mechanisms

X. Y. Lin, Qihang Lin, Chenglong Yin +1 more

Computer Networks and Communications · International Journal of Robust and Nonlinear Control · · cited 13× · DOI ↗ · more like this ·

Abstract This article investigates the distributed formation trajectory tracking problem of a homogeneous quadrotor unmanned aerial vehicles (UAVs) system using a leader‐follower consensus protocol in three‐dimensional space. To enable the follower UAVs in the swarm to track a single leader UAV and form a desired formation under undirected topology, a six‐degree‐of‐freedom dynamic model with bounded time‐varying disturbances is established, which is divided into a tracking subsystem and a stabilizing subsystem. A practical sliding mode control (PSMC) based event‐triggered algorithm is proposed to achieve distributed formation control for a leader‐follower quadrotor UAVs system and reduce the network transmission resources. To address the potential issue of inter‐UAVs collisions during formation, an artificial potential field (APF) method is introduced in the tracking subsystem, and an…

Streamlining sustainable design in building information modeling BIM-based PV design and analysis tools

Qihang Lin, Karen Kensek, Marc Schiler +1 more

Artificial Intelligence · Architectural Science Review · · cited 13× · DOI ↗ · more like this ·

The current BIM-based PV system design process largely depends on transferring a model to special PV simulation software. Critical geometric information is often lost during this process. An automatic PV system design and management tool was developed, which fully integrates the existing and future information of PV systems in a building information modeling (BIM) platform. This tool, PV Link, first analyzes the solar radiation on a building geometry. Then, it automatically places the PV panel based on the solar analysis and user-defined variables like PV size, roof geometry, and rooftop equipment. Finally, the solar radiation on each panel and the electricity produced are calculated and results shown in a building information model. Several case studies show it can design systems efficiently. With the information-rich BIM model, real-time analysis and management of the PVs are feasible…

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Yi Xu, Qihang Lin, Tianbao Yang

math.OC · · cited 10× · arXiv ↗ · more like this ·

In this paper, a new theory is developed for first-order stochastic convex optimization, showing that the global convergence rate is sufficiently quantified by a local growth rate of the objective function in a neighborhood of the optimal solutions. In particular, if the objective function $F(\mathbf w)$ in the $ε$-sublevel set grows as fast as $\|\mathbf w - \mathbf w_*\|_2^{1/θ}$, where $\mathbf w_*$ represents the closest optimal solution to $\mathbf w$ and $θ\in(0,1]$ quantifies the local growth rate, the iteration complexity of first-order stochastic optimization for achieving an $ε$-optimal solution can be $\widetilde O(1/ε^{2(1-θ)})$, which is optimal at most up to a logarithmic factor. To achieve the faster global convergence, we develop two different accelerated stochastic subgradient methods by iteratively solving the original problem approximately in a local region around a…

Adaptive SVRG Methods under Error Bound Conditions with Unknown Growth Parameter

Yi Xu, Qihang Lin, Tianbao Yang

Artificial Intelligence · Neural Information Processing Systems · · cited 9× · OpenAlex ↗ · more like this ·

Error bound, an inherent property of an optimization problem, has recently revived in the development of algorithms with improved global convergence without strong convexity. The most studied error bound is the quadratic error bound, which generalizes strong convexity and is satisfied by a large family of machine learning problems. Quadratic error bound have been leveraged to achieve linear convergence in many first-order methods including the stochastic variance reduced gradient (SVRG) method, which is one of the most important stochastic optimization methods in machine learning. However, the studies along this direction face the critical issue that the algorithms must depend on an unknown growth parameter (a generalization of strong convexity modulus) in the error bound. This parameter is difficult to estimate exactly and the algorithms choosing this parameter heuristically do not…

Stochastic subGradient Methods with Linear Convergence for Polyhedral Convex Optimization

Tianbao Yang, Qihang Lin

cs.LG · · cited 8× · arXiv ↗ · more like this ·

In this paper, we show that simple {Stochastic} subGradient Decent methods with multiple Restarting, named {\bf RSGD}, can achieve a \textit{linear convergence rate} for a class of non-smooth and non-strongly convex optimization problems where the epigraph of the objective function is a polyhedron, to which we refer as {\bf polyhedral convex optimization}. Its applications in machine learning include $\ell_1$ constrained or regularized piecewise linear loss minimization and submodular function minimization. To the best of our knowledge, this is the first result on the linear convergence rate of stochastic subgradient methods for non-smooth and non-strongly convex optimization problems.

Doubly Stochastic Primal-Dual Coordinate Method for Empirical Risk Minimization and Bilinear Saddle-Point Problem

Adams Wei Yu, Qihang Lin, Tianbao Yang

Artificial Intelligence · arXiv (Cornell University) · · cited 8× · OpenAlex ↗ · more like this ·

We proposed a doubly stochastic primal-dual coordinate (DSPDC) optimization algorithm for empirical risk minimization, which can be formulated as a bilinear saddle-point problem. In each iteration, our method randomly samples a block of coordinates of the primal and dual solutions to update. The convergence of our method is established in both the distance from the current iterate to the optimal solution and the primal-dual objective gap. We show that the proposed method has a lower overall complexity than existing coordinate methods when either the data matrix has a factorized structure or the proximal mapping on each block is computationally expensive, e.g., involves an eigenvalue decomposition. Furthermore, we give a theoretical lower bound on the iteration complexity of a general family of primal-dual (block) coordinate methods for bilinear saddle-point problems, which also includes…

Path Planning for Wheeled Robots Based on the Fusion of Improved A* and TEB Algorithms

Weiqiang Zai, Qihang Lin, Shiyuan Wang +1 more

Computer Vision and Pattern Recognition · · cited 6× · DOI ↗ · more like this ·

In this paper, we propose a hierarchical path planning method to address the issues of unsmooth path curves in autonomous navigation for wheeled robots using the original A * algorithm, and the adverse impact on robots caused by jumping changes in output commands of the original TEB algorithm. Firstly, we evaluate the curvature using the curvature smoothing function in the A * algorithm, followed by introducing interpolation to replace some of the unsmoothed path nodes to enhance path smoothness. Secondly, we add jerk constraints on multiple nodes in the TEB algorithm to reduce the impact of acceleration jumps in the output on the mechanical structure of the robot body, thereby improving the operation efficiency of the local path planning algorithm. Finally, we integrate the improved A * algorithm with the TEB algorithm to enable the robot to follow global path points better while…