Scribd Logo
Upload

Welcome to Scribd!

  • 36 views

    Live 1 - AI - K Nearest Neighbors

    Uploaded by

    Vỹ Trần
    The document discusses the k-nearest neighbors (kNN) machine learning algorithm. kNN can be used for both classification and regression tasks using a similarity measure to assign labels or values to new data based on closest training examples. It requires a value for k neighbors, a labeled training set, and a distance metric. The algorithm finds the k closest training examples in feature space and assigns the new data the most common label of its neighbors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Live 1 - AI - K Nearest Neighbors

    Uploaded by

    Vỹ Trần
    36 views21 pages
    The document discusses the k-nearest neighbors (kNN) machine learning algorithm. kNN can be used for both classification and regression tasks using a similarity measure to assign labels or values to new data based on closest training examples. It requires a value for k neighbors, a labeled training set, and a distance metric. The algorithm finds the k closest training examples in feature space and assigns the new data the most common label of its neighbors.

    Original Title

    Live 1 - AI - K nearest neighbors

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses the k-nearest neighbors (kNN) machine learning algorithm. kNN can be used for both classification and regression tasks using a similarity measure to assign labels or values to new data based on closest training examples. It requires a value for k neighbors, a labeled training set, and a distance metric. The algorithm finds the k closest training examples in feature space and assigns the new data the most common label of its neighbors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    36 views21 pages

    Live 1 - AI - K Nearest Neighbors

    Uploaded by

    Vỹ Trần
    The document discusses the k-nearest neighbors (kNN) machine learning algorithm. kNN can be used for both classification and regression tasks using a similarity measure to assign labels or values to new data based on closest training examples. It requires a value for k neighbors, a labeled training set, and a distance metric. The algorithm finds the k closest training examples in feature space and assigns the new data the most common label of its neighbors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 21

    Artificial Intelligence

    k - Nearest Neighbors

    Pham Viet Cuong


    Dept. Control Engineering & Automation, FEEE
    Ho Chi Minh City University of Technology
    k - Nearest Neighbors
    ü A type of supervised ML algorithm
    ü Can be used for both classification and regression
    ü Lazy learning algorithm

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 2
    k - Nearest Neighbors
    ü Uses ‘feature similarity’ to predict the values of new datapoints
    ü The new data point will be assigned a value based on how closely it
    matches the points in the training set

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 3
    k - Nearest Neighbors
    ü The kNN requires
    v An integer k
    v A set of labeled examples (training data)
    v A metric to measure “closeness”
    ü Example 1: Classification
    v 2D
    v 2 classes
    v k=3
    v Euclidean distance
    v 2 sea bass, 1 salmon

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 4
    k - Nearest Neighbors
    ü Example 2: Classification
    v 2D
    v Three classes
    v k=5
    v Euclidean distance

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 5
    k - Nearest Neighbors
    ü Example 3: Classification
    v Three-class 2D problem
    v non-linearly separable
    v k=5
    v Euclidean distance

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 6
    k - Nearest Neighbors
    ü Example 4: Classification
    v Three-class 2D problem
    v non-linearly separable
    v k=5
    v Euclidean distance

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 7
    k - Nearest Neighbors
    ü Algorithm
    v Step 1: Load training data and test data
    v Step 2: Choose k
    v Step 3:
    Ø Calculate distance between test data and other data points
    Ø Identify k nearest neighbors
    Ø Use class labels of nearest neighbors to determine the class label
    of test data (e.g., by taking majority vote)
    v Step 4: End

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 8
    k - Nearest Neighbors
    ü How to choose k?
    v If infinite number of samples available, the larger is k the better
    v In practice: # samples is finite
    v Rule of thumb: k = sqrt(n), n: number of examples
    v k = 1: for efficiency, but can be sensitive to “noise”

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 9
    k - Nearest Neighbors
    ü How to choose k?
    v Larger k may improve performance, but too large k destroys locality
    v Smaller k: higher variance (less stable)
    v Larger k: higher bias (less precise)

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 10
    k - Nearest Neighbors
    ü How well does KNN work?
    v If we have lots of samples, kNN works well

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 11
    k - Nearest Neighbors
    ü Minkowski distance

    ü Mahattan distance

    ü Euclidean distance
    ü Chebyshev distance

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 12
    k - Nearest Neighbors
    ü Best distance?

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 13
    k - Nearest Neighbors
    ü Euclidian distance

    v Euclidean distance treats each feature as equally important


    v However some features (dimensions) may be much more
    discriminative than others

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 14
    k - Nearest Neighbors
    ü Euclidian distance

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 15
    k - Nearest Neighbors
    ü Feature nomalization
    v Linearly scale to 0 mean, variance 1

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 16
    k - Nearest Neighbors
    ü Feature weighting
    v Scale each feature by its importance for classification

    wi

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 17
    k - Nearest Neighbors
    ü Computational complexity
    v Basic kNN algorithm stores all examples
    v Very expensive for a large number of samples

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 18
    k - Nearest Neighbors
    ü kNN - a lazy learning algorithm
    v Defers data processing until it receives a request to classify unlabeled
    data
    v Replies to a request for information by combining its stored training
    data
    v Discards the constructed answer and any intermediate results
    v Lazy algorithms have fewer computational costs than eager
    algorithms during training but greater storage requirements and
    higher computational costs on recall

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 19
    k - Nearest Neighbors
    ü Advantages
    v Can be applied to the data from any distribution
    v Very simple and intuitive
    v Good classification if the number of samples is large enough
    v Uses local information, which can yield highly adaptive behavior
    v Very easy for parallel implementations
    ü Disadvantages
    v Choosing k may be tricky
    v Test stage is computationally expensive
    v Need large number of samples for accuracy
    v Large storage requirements
    v Highly susceptible to the curse of dimensionality
    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 20
    k - Nearest Neighbors
    ü Sources:
    v https://www.csd.uwo.ca/courses/CS4442b/L3-ML-knn.pdf
    v http://research.cs.tamu.edu/prism/lectures/pr/pr_l8.pdf
    v http://web.iitd.ac.in/~bspanda/KNN%20presentation.pdf
    v V. B. Surya Prasath et. al., Effects of Distance Measure Choice on KNN
    Classifier Performance - A Review, Big Data. 7. 10.1089/big.2018.0175.

    Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 21

    You might also like