Mathematics · Statistics and Probability
On Optimal Correlation-Based Prediction
Matteo Bottai, Taeho Kim, Benjamin Lieberman, George Luta, Edsel A. Peña
See on the mapAbstract
This note examines, at the population-level, the approach of obtaining predictors h˜(X) of a random variable Y, given the joint distribution of (Y,X), by maximizing the mapping h↦κ(Y,h(X)) for a given correlation function κ(·,·). Commencing with Pearson’s correlation function, the class of such predictors is uncountably infinite. The least-squares predictor h* is an element of this class obtained by equating the expectations of Y and h(X) to be equal and the variances of h(X) and E(Y|X) to be also equal. On the other hand, replacing the second condition by the equality of the variances of Y and h(X), a natural requirement for some calibration problems, the unique predictor h** that is obtained has the maximum value of Lin’s (1989 Lin, L. (1989), “A Concordance Correlation Coefficient to Evaluate Reproducibility,” Biometrics, 45, 255–268. DOI: 10.2307/2532051.[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]) concordance correlation coefficient (CCC) with Y among all predictors. Since the CCC measures the degree of agreement, the new predictor h** is called the maximal agreement predictor. These predictors are illustrated for three special distributions: the multivariate normal distribution; the exponential distribution, conditional on covariates; and the Dirichlet distribution. The exponential distribution is relevant in survival analysis or in reliability settings, while the Dirichlet distribution is relevant for compositional data.