# AI News, Open Machine Learning Course. Topic 7. Unsupervised Learning: PCA and Clustering

- On Sunday, September 30, 2018
- By Read More

## Open Machine Learning Course. Topic 7. Unsupervised Learning: PCA and Clustering

Useful links The main feature of unsupervised learning algorithms, when compared to classification and regression methods, is that input data are unlabeled (i.e.

Second, it is difficult to evaluate the quality of an unsupervised algorithm due to the absence of an explicit goodness metric as used in supervised learning.

More generally speaking, all observations can be considered as an ellipsoid in a subspace of an initial feature space, and the new basis set in this subspace is aligned with the ellipsoid axes.

In the general case, the resulting ellipsoid dimensionality matches the initial space dimensionality, but the assumption that our data lies in a subspace with a smaller dimension allows us to cut off the “excessive” space with the new projection (subspace).

Let’s take a look at the mathematical formulation of this process: In order to decrease the dimensionality of our data from n to k with k ≤ n, we sort our list of axes in order of decreasing dispersion and take the top-k of them.

In terms of matrices where X is the matrix of observations, the covariance matrix is as follows: Quick recap: matrices, as linear operators, have eigenvalues and eigenvectors.

Formally, a matrix M with eigenvector w and eigenvalue λ satisfy this equation: The covariance matrix for a sample X can be written as a product of a transposed matrix X and X itself.

Now let’s see how PCA will improve the results of a simple model that is not able to correctly fit all of the training data: Let’s try this again, but, this time, let’s reduce the dimensionality to 2 dimensions: The accuracy did not increase significantly in this case, but, with other datasets with a high number of dimensions, PCA can drastically improve the accuracy of decision trees and other ensemble methods.

It would be nice to describe these things more concretely, and, when a new point comes in, assign it to the correct group.” This general idea encourages exploration and opens up a variety of algorithms for clustering.

But, there is a problem — the optimum is reached when the number of centroids is equal to the number of observations, so you would end up with every single observation as its own separate cluster.

an example is MiniBatch K-means, which takes portions (batches) of data instead of fitting the whole dataset and then moves centroids by taking the average of the previous steps.

The implemetation of the algorithm using scikit-learn has its benefits such as the possibility to state the number of initializations with the n_init function parameter, which enables us to identify more robust centroids.

The matrices are updated sequentially with the following rules: Spectral clustering combines some of the approaches described above to create a stronger clustering method.

This matrix describes a full graph with the observations as vertices and the estimated similarity value between a pair of observations as edge weights for that pair of vertices.

The algorithm is fairly simple: The process of searching for the nearest cluster can be conducted with different methods of bounding the observations: The 3rd one is the most effective in computation time since it does not require recomputing the distances every time the clusters are merged.

The results can be visualized as a beautiful cluster tree (dendogram) to help recognize the moment the algorithm should be stopped to get optimal results.

External metrics use the information about the known true split while internal metrics do not use any external information and assess the goodness of clusters based only on the initial data.

The Rand Index can be calculated using the following formula: RI = 2(a + b)/n(n — 1) In other words, it evaluates a share of observations for which these splits (initial and clustering result) are consistent.

It is defined by the [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory) function and interprets a sample split as a discrete distribution (likelihood of assigning to a cluster is equal to the percent of objects in it).

Values close to zero mean the splits are independent, and those close to 1 mean they are similar (with complete match at AMI = 1).

Homogeneity, completeness, V-measure Formally, these metrics are also defined based on the entropy function and the conditional entropy function, interpreting the sample splits as discrete distributions: where K is a clustering result and C is the initial split.

Therefore, h evaluates whether each cluster is composed of same class objects, and c measures how well the same class objects fit the clusters.

Let a be the mean of the distance between an object and other objects within one cluster and b be the mean distance from an object to an object from the nearest cluster (different from the one the object belongs to).

With the help of silhouette, we can identify the optimal number of clusters k (if we don’t know it already from the data) by taking the number of clusters that maximizes the silhouette coefficient.

To conclude, let’s take a look at how these metrics perform with the MNIST handwritten numbers dataset: Full versions of assignments are announced each week in a new run of the course (October 1, 2018).

- On Thursday, January 17, 2019

**k means clustering example HD**

This is not my work! Please give credits to the original author: To calculate means from cluster centers: For example, if a cluster ..

**How to Perform K-Means Clustering in R Statistical Computing**

In this video I go over how to perform k-means clustering using r statistical computing. Clustering analysis is performed and the results are interpreted.

**K Means Clustering in R**

This video tutorial shows you how to use the means function in R to do K-Means clustering. You will need to know how to read in data, subset data and plot items ...

**How K Means Clustering Algorithm Works Visually C#**

K-means clustering is a method of vector quantization, originally from signal processing, that is popular for cluster analysis in data mining. k-means clustering ...

**Machine Learning in R - Classification, Regression and Clustering Problems**

Learn the basics of Machine Learning with R. Start our Machine Learning Course for free: ...

**How to run cluster analysis in Excel**

A step by step guide of how to run k-means clustering in Excel. Please note that more information on cluster analysis and a free Excel template is available at ...

**Technical Course: Cluster Analysis: K-Means Algorithm for Clustering**

K-Means Algorithm for clustering by Gaurav Vohra, founder of Jigsaw Academy. This is a clip from the Clustering module of our course on analytics. Jigsaw ...

**Mod-01 Lec-08 Rank Order Clustering, Similarity Coefficient based algorithm**

Manufacturing Systems Management by Prof. G. Srinivasan, Department of Management, IITmadras. For more details on NPTEL visit

**Clustering using PROC FASTCLUS in SAS**

In this video you will learn how to do cluster analysis using PROC FASTCLUS in SAS For Training & Study packs on Analytics/Data Science/Big Data, Contact us ...

**Understanding the Basics of Cluster Analysis| Cluster Analysis Tutorial | Introduction to Clustering**

Learn the basics of Cluster Analysis using real-life examples. Know more about the objective of cluster analysis, the methodology used and interpreting results ...