Social Sciences · Public Administration
CMSC 35900 (Spring 2009) Large Scale Learning Lecture: 5 Nearest Neighbor Rules
Instructors Sham Kakade, Greg Shakhnarovich
See on the mapAbstract
In this lecture we will go over some basic asymptotic properties of the nearest neighbor rules for classification and regression. 1 Nearest neighbor properties We consider a set of labeled data points (x1, y1),..., (xn, yn) drawn i.i.d. from a joint distribution p(x, y) = p(x)p(y|x) over X × Y. We will for now assume that X = R d, and that Y = {1,..., M} which means simply M-class classification. We will denote by x (i) the i-th nearest neighbor (i-NN) of x among x1,..., xn, and by y (i) the label of that i-NN. A basic property of the NN is its convergence to the “query ” point (the point in neighbors of which we are interested), as n → ∞. We start with the result from [3]: Lemma 1.1. (convergence of the nearest neighbor) Let x0, x1,..., xn be an i.i.d. sequence of random variables in R d. 1 Then, x (1) → x0 with probability 1. Proof. Let Br (x) be the (closed) ball of radius r centered at x Br (x) � {z ∈ R d: D(z, x) ≤ r}, for some metric D defined on Rd. We will first consider a point x such that for any r> 0, P (Br (x)) � Pr [z ∈ Br (x)] = p(z)dz> 0. (1)