arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Lin Shang

281 papers · 1950–2026

1967: 01968: 01969: 01970: 01971: 01972: 01973: 01974: 01975: 01976: 01977: 01978: 01979: 01980: 01981: 01982: 01983: 01984: 01985: 01986: 01987: 01988: 01989: 01990: 01991: 01992: 01993: 01994: 01995: 01996: 01997: 01998: 01999: 02000: 12001: 12002: 22003: 22004: 02005: 82006: 32007: 72008: 52009: 82010: 32011: 112012: 72013: 182014: 142015: 52016: 132017: 32018: 92019: 152020: 192021: 222022: 142023: 122024: 172025: 312026: 30
1967peak 31 in 20252026

Frequent co-authors

Papers

showing 1–25 of 281 RSS

Rainbow Plots, Bagplots, and Boxplots for Functional Data

Rob J. Hyndman, Han Lin Shang

Statistics and Probability · Journal of Computational and Graphical Statistics · · cited 294× · DOI ↗ · more like this ·

We propose new tools for visualizing large amounts of functional data in the form of smooth curves. The proposed tools include functional versions of the bagplot and boxplot, which make use of the first two robust principal component scores, Tukey's data depth and highest density regions. By-products of our graphical displays are outlier detection methods for functional data. We compare these new outlier detection methods with existing methods for detecting outliers in functional data, and show that our methods are better able to identify outliers. An R-package containing computer code and datasets is available in the online supplements.

Methods for Scalar‐on‐Function Regression

Philip T. Reiss, Jeff Goldsmith, Han Lin Shang +1 more

Statistics and Probability · International Statistical Review · · cited 182× · DOI ↗ · more like this ·

Summary Recent years have seen an explosion of activity in the field of functional data analysis (FDA), in which curves, spectra, images and so on are considered as basic functional data units. A central problem in FDA is how to fit regression models with scalar responses and functional data points as predictors. We review some of the main approaches to this problem, categorising the basic model types as linear, non‐linear and non‐parametric. We discuss publicly available software packages and illustrate some of the procedures by application to a functional magnetic resonance imaging data set.

Binary particle swarm optimisation for feature selection: A filter based approach

Liam Cervante, Bing Xue, Mengjie Zhang +1 more

Artificial Intelligence · Zenodo (CERN European Organization for Nuclear Research) · · cited 170× · DOI ↗ · more like this ·

Based on binary particle swarm optimisation (BPSO) and information theory, this paper proposes two new filter feature selection methods for classification problems. The first algorithm is based on BPSO and the mutual information of each pair of features, which determines the relevance and redundancy of the selected feature subset. The second algorithm is based on BPSO and the entropy of each group of features, which evaluates the relevance and redundancy of the selected feature subset. Different weights for the relevance and redundancy in the fitness functions of the two proposed algorithms are used to further improve their performance in terms of the number of features and the classification accuracy. In the experiments, a decision tree (DT) is employed to evaluate the classification accuracy of the selected feature subset on the test sets of four datasets. The results show that with…

A multi-objective particle swarm optimisation for filter-based feature selection in classification problems

Bing Xue, Liam Cervante, Lin Shang +2 more

Artificial Intelligence · Connection Science · · cited 122× · DOI ↗ · more like this ·

Feature selection has the two main objectives of minimising the classification error rate and the number of features. Based on binary particle swarm optimisation (BPSO), we develop two novel multi-objective feature selection frameworks for classification, which are multi-objective binary PSO using the idea of non-dominated sorting (NSBPSO) and multi-objective binary PSO using the ideas of crowding, mutation and dominance (CMDBPSO). Four multi-objective feature selection methods are then developed by applying mutual information and entropy as two different filter evaluation criteria in each of the proposed frameworks. The proposed algorithms are examined and compared with a single objective method on eight benchmark data sets. Experimental results show that the proposed multi-objective algorithms can evolve a set of solutions that use a smaller number of features and achieve better…

Point and interval forecasts of mortality rates and life expectancy: A comparison of ten principal component methods

Han Lin Shang, Heather Booth, Rob J. Hyndman

Demography · Demographic Research · · cited 114× · DOI ↗ · more like this ·

Using the age- and sex-specific data of 14 developed countries, we compare the point and interval forecast accuracy and bias of ten principal component methods for forecasting mortality rates and life expectancy. The ten methods are variants and extens

Grouped functional time series forecasting: An application to age-specific mortality rates

Han Lin Shang, Rob J Hyndman

stat.AP · · cited 91× · arXiv ↗ · more like this ·

Age-specific mortality rates are often disaggregated by different attributes, such as sex, state and ethnicity. Forecasting age-specific mortality rates at the national and sub-national levels plays an important role in developing social policy. However, independent forecasts at the sub-national levels may not add up to the forecasts at the national level. To address this issue, we consider reconciling forecasts of age-specific mortality rates, extending the methods of Hyndman et al. (2011) to functional time series, where age is considered as a continuum. The grouped functional time series methods are used to produce point forecasts of mortality rates that are aggregated appropriately across different disaggregation factors. For evaluating forecast uncertainty, we propose a bootstrap method for reconciling interval forecasts. Using the regional age-specific mortality rates in Japan…

MULTI-OBJECTIVE EVOLUTIONARY ALGORITHMS FOR FILTER BASED FEATURE SELECTION IN CLASSIFICATION

Bing Xue, Liam Cervante, Lin Shang +2 more

Computational Theory and Mathematics · International Journal of Artificial Intelligence Tools · · cited 89× · DOI ↗ · more like this ·

Feature selection is a multi-objective problem with the two main conflicting objectives of minimising the number of features and maximising the classification performance. However, most existing feature selection algorithms are single objective and do not appropriately reflect the actual need. There are a small number of multi-objective feature selection algorithms, which are wrapper based and accordingly are computationally expensive and less general than filter algorithms. Evolutionary computation techniques are particularly suitable for multi-objective optimisation because they use a population of candidate solutions and are able to find multiple non-dominated solutions in a single run. However, the two well-known evolutionary multi-objective algorithms, non-dominated sorting based multi-objective genetic algorithm II (NSGAII) and strength Pareto evolutionary algorithm 2 (SPEA2) have…

BINARY PSO AND ROUGH SET THEORY FOR FEATURE SELECTION: A MULTI-OBJECTIVE FILTER BASED APPROACH

Bing Xue, Liam Cervante, Lin Shang +2 more

Computational Theory and Mathematics · International Journal of Computational Intelligence and Applications · · cited 66× · DOI ↗ · more like this ·

Feature selection is a multi-objective problem, where the two main objectives are to maximize the classification accuracy and minimize the number of features. However, most of the existing algorithms belong to single objective, wrapper approaches. In this work, we investigate the use of binary particle swarm optimization (BPSO) and probabilistic rough set (PRS) for multi-objective feature selection. We use PRS to propose a new measure for the number of features based on which a new filter based single objective algorithm (PSOPRSE) is developed. Then a new filter-based multi-objective algorithm (MORSE) is proposed, which aims to maximize a measure for the classification performance and minimize the new measure for the number of features. MORSE is examined and compared with PSOPRSE, two existing PSO-based single objective algorithms, two traditional methods, and the only existing BPSO and…

A Survey on Unbalanced Classification: How Can Evolutionary Computation Help?

Wenbin Pei, Bing Xue, Mengjie Zhang +3 more

Artificial Intelligence · IEEE Transactions on Evolutionary Computation · · cited 49× · DOI ↗ · more like this ·

Unbalanced classification is an essential machine learning task, which has attracted widespread attention from both the academic and industrial communities due mainly to its broad applications. Evolutionary computation (EC) has contributed greatly to unbalanced classification. However, to the best of our knowledge, there have not been any comprehensive investigations on the strengths and weaknesses of alternative EC methods in addressing various challenging problems in unbalanced classification. This article reviews the literature which utilize EC techniques for unbalanced classification, with the aim of revealing the contributions of EC to unbalanced classification, providing an overview of recent advances, and identifying limitations of existing works. In addition, we present a series of real-world applications, and identify open challenges as well as possible research directions for…

Mortality and life expectancy forecasting for a group of populations in developed countries: A multilevel functional data method

Han Lin Shang

stat.AP · · cited 45× · arXiv ↗ · more like this ·

A multilevel functional data method is adapted for forecasting age-specific mortality for two or more populations in developed countries with high-quality vital registration systems. It uses multilevel functional principal component analysis of aggregate and population-specific data to extract the common trend and population-specific residual trend among populations. If the forecasts of population-specific residual trends do not show a long-term trend, then convergence in forecasts may be achieved. This method is first applied to age- and sex-specific data for the United Kingdom, and its forecast accuracy is then further compared with several existing methods, including independent functional data and product-ratio methods, through a multi-country comparison. The proposed method is also demonstrated by age-, sex- and state-specific data in Australia, where the convergence in forecasts…

Dynamic linear models with R

Han Lin Shang

Statistics and Probability · Journal of Applied Statistics · · cited 44× · DOI ↗ · more like this ·

Dynamic linear models with R, by Giovanni Petris, Sonia Petrone and Patrizia Campagnoli, New York, Springer, 2009, xiii+251 pp., £44.99 or US$59.95 (softback), ISBN 978-0-387-77237-0 This book focu...

Bayesian bandwidth estimation for a nonparametric functional regression model with mixed types of regressors and unknown error density

Han Lin Shang

stat.ME · · cited 37× · arXiv ↗ · more like this ·

We investigate the issue of bandwidth estimation in a nonparametric functional regression model with function-valued, continuous real-valued and discrete-valued regressors under the framework of unknown error density. Extending from the recent work of Shang (2013, Computational Statistics & Data Analysis), we approximate the unknown error density by a kernel density estimator of residuals, where the regression function is estimated by the functional Nadaraya-Watson estimator that admits mixed types of regressors. We derive a kernel likelihood and posterior density for the bandwidth parameters under the kernel-form error density, and put forward a Bayesian bandwidth estimation approach that can simultaneously estimate the bandwidths. Simulation studies demonstrated the estimation accuracy of the regression function and error density for the proposed Bayesian approach. Illustrated by a…

Bootstrap Prediction Bands for Functional Time Series

Efstathios Paparoditis, Han Lin Shang

math.ST · · cited 33× · arXiv ↗ · more like this ·

A bootstrap procedure for constructing prediction bands for a stationary functional time series is proposed. The procedure exploits a general vector autoregressive representation of the time-reversed series of Fourier coefficients appearing in the Karhunen-Loeve representation of the functional process. It generates backward-in-time, functional replicates that adequately mimic the dependence structure of the underlying process in a model-free way and have the same conditionally fixed curves at the end of each functional pseudo-time series. The bootstrap prediction error distribution is then calculated as the difference between the model-free, bootstrap-generated future functional observations and the functional forecasts obtained from the model used for prediction. This allows the estimated prediction error distribution to account for the innovation and estimation errors associated with…

Improving Generalisation of Genetic Programming for Symbolic Regression with Structural Risk Minimisation

Qi Chen, Bing Xue, Lin Shang +1 more

Artificial Intelligence · · cited 33× · DOI ↗ · more like this ·

Generalisation is one of the most important performance measures for any learning algorithm, no exception to Genetic Programming (GP). A number of works have been devoted to improve the generalisation ability of GP for symbolic regression. Methods based on a reliable estimation of generalisation error of models during evolutionary process are a sensible choice to enhance the generalisation of GP. Structural risk minimisation (SRM), which is based on the VC dimension in the learning theory, provides a powerful framework for estimating the difference between the generalisation error and the empirical error. Despite its solid theoretical foundation and reliability, SRM has seldom been applied to GP. The most important reason is the difficulty in measuring the VC dimension of GP models/programs. This paper introduces SRM, which is based on an empirical method to measure the VC dimension of…

Clustering and Forecasting Multiple Functional Time Series

Chen Tang, Han Lin Shang, Yanrong Yang

stat.ME · · cited 30× · arXiv ↗ · more like this ·

Modelling and forecasting homogeneous age-specific mortality rates of multiple countries could lead to improvements in long-term forecasting. Data fed into joint models are often grouped according to nominal attributes, such as geographic regions, ethnic groups, and socioeconomic status, which may still contain heterogeneity and deteriorate the forecast results. Our paper proposes a novel clustering technique to pursue homogeneity among multiple functional time series based on functional panel data modelling to address this issue. Using a functional panel data model with fixed effects, we can extract common functional time series features. These common features could be decomposed into two components: the functional time trend and the mode of variations of functions (functional pattern). The functional time trend reflects the dynamics across time, while the functional pattern captures…

A fast wrapper feature subset selection method based on binary particle swarm optimization

Xing Liu, Lin Shang

Artificial Intelligence · · cited 29× · DOI ↗ · more like this ·

Although many particle swarm optimization (PSO) based feature subset selection methods have been proposed, most of them seem to ignore the difference of feature subset selection problems and other optimization problems. We analyze the search process of a PSO based wrapper feature subset selection algorithm and find that characteristics of feature subset selection can be used to optimize this process. We compare wrapper and filter ways of evaluating features and define the domain knowledge of feature subset selection problems and we propose a fast wrapper feature subset selection algorithm based on PSO employed the domain knowledge of feature subset selection problems. Experimental results show that our method can work well, and the new algorithm can improve both the running time and the classification accuracy.

Model confidence sets and forecast combination: An application to age-specific mortality

Han Lin Shang, Steven Haberman

stat.AP · · cited 25× · arXiv ↗ · more like this ·

Model averaging combines forecasts obtained from a range of models, and it often produces more accurate forecasts than a forecast from a single model. The crucial part of forecast accuracy improvement in using the model averaging lies in the determination of optimal weights from a finite sample. If the weights are selected sub-optimally, this can affect the accuracy of the model-averaged forecasts. Instead of choosing the optimal weights, we consider trimming a set of models before equally averaging forecasts from the selected superior models. Motivated by Hansen, Lunde and Nason (2011), we apply and evaluate the model confidence set procedure when combining mortality forecasts. The proposed model averaging procedure is motivated by Samuels and Sekkel (2017) based on the concept of model confidence sets as proposed by Hansen et al. (2011) that incorporates the statistical significance…

Web usage mining based on fuzzy clustering in identifying target group

Jianxi Zhang, Pei-Ying Zhao, Lin Shang +1 more

Information Systems · · cited 25× · DOI ↗ · more like this ·

Data mining focuses on the techniques of non-trivial extraction of implicit, previously unknown, and potentially useful information from very large amounts of data. Web mining can be defined simply as the application of data mining techniques to Web data. Web usage mining (WUM) is an important category in Web mining. Web usage mining is an important and fast developing area of Web mining where a lot of research has been done already. We improved the fuzzy clustering algorithm to find groups which share common interests and behaviors by analyzing the data collected in Web servers.

Statistically tested comparisons of the accuracy of forecasting methods for age-specific and sex-specific mortality and life expectancy

Han Lin Shang

Demography · Population Studies · · cited 24× · DOI ↗ · more like this ·

Although there are continuing developments in the methods for forecasting mortality, there are few comparisons of the accuracy of the forecasts. The subject of the statistical validity of these comparisons, which is essential to demographic forecasting, has all but been ignored. We introduce Friedman's test statistics to examine whether the differences in point and interval forecast accuracies are statistically significant between methods. We introduce the Nemenyi test statistic to identify which methods give results that are statistically significantly different from others. Using sex-specific and age-specific data from 20 countries, we apply these two test statistics to examine the forecast accuracy obtained from several principal component methods, which can be categorized into coherent and non-coherent forecasting methods.

Binary particle swarm optimisation and rough set theory for dimension reduction in classification

Liam Cervante, Bing Xue, Lin Shang +1 more

Computational Theory and Mathematics · · cited 24× · DOI ↗ · more like this ·

Dimension reduction plays an important role in many classification tasks. In this work, we propose a new filter dimension reduction algorithm (PSOPRSE) using binary particle swarm optimisation and probabilistic rough set theory. PSOPRSE aims to maximise a classification performance measure and minimise a newly developed measure reflecting the number of attributes. Both measures are formed by probabilistic rough set theory. PSOPRSE is compared with two existing PSO based algorithms and two traditional filter dimension reduction algorithms on six discrete datasets of varying difficulty. Five continues datasets including a large number of attributes are discretised and used to further examine the performance of PSOPRSE. Three learning algorithms, namely decision trees, nearest neighbour algorithms and naive Bayes, are used in the experiments to examine the generality of PSOPRSE. The…

Forecasting age distribution of death counts: An application to annuity pricing

Han Lin Shang, Steven Haberman

stat.AP · · cited 23× · arXiv ↗ · more like this ·

We consider a compositional data analysis approach to forecasting the age distribution of death counts. Using the age-specific period life-table death counts in Australia obtained from the Human Mortality Database, the compositional data analysis approach produces more accurate one- to 20-step-ahead point and interval forecasts than Lee-Carter method, Hyndman-Ullah method, and two naïve random walk methods. The improved forecast accuracy of period life-table death counts is of great interest to demographers for estimating survival probabilities and life expectancy, and to actuaries for determining temporary annuity prices for various ages and maturities. Although we focus on temporary annuity prices, we consider long-term contracts which make the annuity almost lifetime, in particular when the age at entry is sufficiently high.

Dynamic principal component regression: Application to age-specific mortality forecasting

Han Lin Shang

stat.AP · · cited 22× · arXiv ↗ · more like this ·

In areas of application, including actuarial science and demography, it is increasingly common to consider a time series of curves; an example of this is age-specific mortality rates observed over a period of years. Given that age can be treated as a discrete or continuous variable, a dimension reduction technique, such as principal component analysis, is often implemented. However, in the presence of moderate to strong temporal dependence, static principal component analysis commonly used for analyzing independent and identically distributed data may not be adequate. As an alternative, we consider a \textit{dynamic} principal component approach to model temporal dependence in a time series of curves. Inspired by Brillinger's (1974) theory of dynamic principal components, we introduce a dynamic principal component analysis, which is based on eigen-decomposition of estimated long-run…

Developing Interval-Based Cost-Sensitive Classifiers by Genetic Programming for Binary High-Dimensional Unbalanced Classification [Research Frontier]

Wenbin Pei, Bing Xue, Lin Shang +1 more

Artificial Intelligence · IEEE Computational Intelligence Magazine · · cited 21× · DOI ↗ · more like this ·

Cost-sensitive learning is a popular approach to addressing the problem of class imbalance for many classification algorithms in machine learning. However, most cost-sensitive algorithms are dependent on manually designed cost matrices. Unfortunately, in many cases, it is often not easy for humans, even experts, to accurately specify misclassification costs for different mistakes due to the lack of domain knowledge related to actual situations in some complex unbalanced problems. As a result, these cost-sensitive algorithms cannot be directly applied. This paper proposes a new genetic programmingbased approach to developing cost-sensitive classifiers that are independent of manually designed cost matrices. The proposed method is able to construct classifiers and learn cost intervals automatically and simultaneously. In the proposed method, a tree representation, terminal sets and…