80

I understand how an artificial neural network (ANN), can be trained in a supervised manner using backpropogation to improve the fitting by decreasing the error in the predictions. I have heard that an ANN can be used for unsupervised learning but how can this be done without a cost function of some sort to guide the optimization stages? With k-means or the EM algorithm there is a function for which each iteration searches to increase.

  • How can we do clustering with an ANN and what mechanism does it use to group data points in the same locality?

(and what extra capabilities are brought with adding more layers to it?)

gung - Reinstate Monica
  • 145,122
Vass
  • 1,675
  • Are you interested in NN unsupervised learning in general, or specifically in unsupervised clustering with neural networks? – Denis Tarasov Mar 03 '15 at 16:44
  • 1
    @DenisTarasov, I am interested primarily in unsupervised clustering with NN, but do not know much about NN unsupervised NN learning in general. It would be great if an answer would include a bit of the NN unsupervised learning in general before discussing the specific application. – Vass Mar 03 '15 at 17:02
  • 1
    Self Organizing Map (SOM) is a type of network used for clustering. – Cagdas Ozgenc Mar 03 '15 at 17:16
  • unsupervised learning in ANN – It extracts statistical properties from the training set. – Unsupervised learning is more difficult but is seen as biologically plausible
    • Requires no teacher.
     – yonas Apr 03 '17 at 01:29

3 Answers3

78

Neural networks are widely used in unsupervised learning in order to learn better representations of the input data. For example, given a set of text documents, NN can learn a mapping from document to real-valued vector in such a way that resulting vectors are similar for documents with similar content, i.e. distance preserving. This can be achieved using, for example, auto-encoders - a model that is trained to reconstruct the original vector from a smaller representation (hidden layer activations) with reconstruction error (distance from the ID function) as cost function. This process doesn't give you clusters, but it creates meaningful representations that can be used for clustering. You could, for instance, run a clustering algorithm on the hidden layer's activations.

Clustering: There are a number of different NN architectures specifically designed for clustering. The most widely known is probably self organizing maps. A SOM is a NN that has a set of neurons connected to form a topological grid (usually rectangular). When some pattern is presented to an SOM, the neuron with closest weight vector is considered a winner and its weights are adapted to the pattern, as well as the weights of its neighbourhood. In this way an SOM naturally finds data clusters. A somewhat related algorithm is growing neural gas (it is not limited to predefined number of neurons).

Another approach is Adaptive Resonance Theory where we have two layers: "comparison field" and "recognition field". Recognition field also determines the best match (neuron) to the vector transferred from the comparison field and also have lateral inhibitory connections. Implementation details and exact equations can readily found by googling the names of these models, so I won't put them here.

mkt
  • 18,245
  • 11
  • 73
  • 172
Denis Tarasov
  • 1,574
24

You want to look into self-organizing maps. Kohonen (who invented them) wrote a book about them. There are packages for this in R (som, kohonen), and there are implementations in other languages such as MATLAB.

gung - Reinstate Monica
  • 145,122
  • can you go into some detail about how the NN can do this and elaborate on the theory? possibly also explain the effect of using a deep NN (DNN)? – Vass Mar 03 '15 at 17:22
  • 1
    I'm afraid I have very little expertise here, @Vass. I don't think adding extra layers will do much, other than slow it down. Someone else will have to give you the theory, I just wanted to get you started. – gung - Reinstate Monica Mar 03 '15 at 17:25
  • 2
    A student in our lab experimented with clustering using SOM. It took forever to run and the results were very disappointing compared to other approaches (in our case standard graph clustering algorithms). I have always been puzzled by the fact that the standard 2D target domain (topological grid) seems to be a highly arbitrary space. More worryingly is is very simplistic and essentially needs to compress the data into a space described by just two variables. – micans Mar 04 '15 at 11:39
  • @micans makes some good points, but the data is not simply compressed into a space described by just two variables, because each node is also associated with a prototype. Also if the running is slow, it may well be an implementation issue. Statistically, other methods than SOM should achieve better classification results. As for the topology issue, the brain seems to be organised as layers of 2D topology, but it achieves great results (or so I'd like to think). – Tom Anderson May 08 '17 at 03:30
1

Maybe the Clustering with Neural Network and Index (CNNI) model is what you are looking for.

https://doi.org/10.31219/osf.io/ejxm6

CNNI uses a Neural Network to cluster data points. Training of the Neural Network mimics supervised learning, with an internal clustering evaluation index acting as the loss function. It successively adjusts the weights of the Neural Network to reduce the loss (improve the value of the index).

Structure of CNNI:

The structure of CNNI is simple: a Neural Network for supervised learning plus an internal clustering evaluation index. The index acts as the loss function, because there is no target output associated with each input data point in clustering scenario.

The number of neurons in the input layer of CNNI equals to the dimension of the data points given to the network. The number of neurons in the output layer of CNNI equals to $ K $ (number of clusters we want to classify). By comparing the values of each output neuron, label of one data point is obtained (e.g., find out the maximum of output neurons).

Training of CNNI has some difference from other supervised learning Neural Networks. We need to compute each data point's label according to the Neural Network's current state, then calculate the value of the clustering evaluation index, according to the labels of all data points. Adjustment of the weights of the Neural Network is based on the value of the index.

An experiment shows CNNI equipped with MMJ-SC, achieves the first parametric (inductive) clustering model that can deal with non-convex shaped (non-flat geometry) data, which implies it is the first general-purpose parametric (inductive) clustering model.

Mike Mathcook
  • 51