1

So, I have created a neural network using back propagation and sigmoid function. It seems to work for XOR and images with size of 28×28. However, When I input it 100×100 image the mean-square-error is 0.3 ish.

I am using 1 hidden layer. I basically have two questions:

  1. Is it possible to use neural network not deep neural network to learn of an image the size of 100×100? If so, could you give me a detailed explanation, I have posted the code that I wrote below, sorry in advance as it is not the best or the cleanest code.
  2. When passing error gradient in a deep neural network for 2 hidden layers do you pass the hidden gradient calculated by the output layer to the 2nd hidden layer and this 2nd hidden layer then calculates the gradient descent/hidden error gradient for the 1st hidden layer is this correct? Whilst updating the weights.

Here is the link!

kjetil b halvorsen
  • 77,844
Sad.coder
  • 11
  • A1: It is possible for some classification problems. Having a single hidden layer limits the complexity of function that the NN can represent. The number of nodes in the hidden layer will affect the complexity too. The outcome will also depend on the number of classes. I didn't look at your code in detail, but it seems you have 1 output node. Usually the output layer has as many nodes, as number of classes in the data set. I don't have a formal way to determine what is a good size for the hidden layer, but I'd try with 150.

    A2: Yes, you are correct - that's why it's back propagation.

     – Iliyan Bobev Sep 05 '16 at 19:00
  • If I try it with 150 nodes in hidden layer it outputs 0.2 as mse and gets stuck in it. Thank you for clearing Q2 – Sad.coder Sep 05 '16 at 19:12
  • Did you also bump the number of nodes in the output layer to the number of classes? – Iliyan Bobev Sep 05 '16 at 19:15
  • Nope do I bump it to 2 – Sad.coder Sep 05 '16 at 19:25
  • As I have increased the nodes in the output it seems to have better results – Sad.coder Sep 05 '16 at 19:34

  • [ 0.225061 0.82384 0.774939 0.029444 0.979184 0.647941 0.17616 0.647941 0.82384 0.029257 0.979252 0.774939 0.352059 0.774939 0.647941 0.0290735 0.979318 0.82384 ] 0.0474061 0.0474061 0.0474061 0.000142677 7.17506e-05 0.0474061

     – Sad.coder Sep 05 '16 at 19:35
  • Thats for 6 output – Sad.coder Sep 05 '16 at 19:35
  • Thanks @IliyanBobev I changed it to outputs for each classes and it works as in I get the values for mse to below 0.001. – Sad.coder Sep 05 '16 at 23:09
  • But why can't there be only a output learning different outputs for each input? – Sad.coder Sep 05 '16 at 23:15

2 Answers2

0

The answer is stat in the above comment:

A1: It is possible for some classification problems. Having a single hidden layer limits the complexity of function that the NN can represent. The number of nodes in the hidden layer will affect the complexity too. The outcome will also depend on the number of classes. I didn't look at your code in detail, but it seems you have 1 output node. Usually the output layer has as many nodes, as number of classes in the data set. I don't have a formal way to determine what is a good size for the hidden layer, but I'd try with 150.

A2: Yes, you are correct - that's why it's back propagation

this seems to work for me. Thanks @iliyan bobev

gung - Reinstate Monica
  • 145,122
Sad.coder
  • 11
0

First question: Yes, why should it not be possible ?

Second question: I would propose to tackle the backpropagation topic first on the paper by understanding why the weight-update-rule looks the way it does. For this you need only partial derivation as mathematical tools: deriving the error function with respect to the net weights. You will see that for hidden layers that error is actually not given directly (because there is no present expected output as for the output layer). The backpropagation algorithm is simply way of propagating that output error backwards and give the hidden-layers the possibility of knowing how much their weights are influencing the decision made at the output layer in order to update them accordingly.

Generally the architecture of classical feedforward networks depends heavily on the problem that needs to be solved (furthermore problems are devided into two types, namely regression and classification problems).

So lets say your task is to classify images of handwritten images provided as 28x28 matrices of gray-scale values (like the famous MNIST dataset) there is no strict specification on how much hidden layers your net has to have or how the output layer should look like.

As was already stated here: The output layer for some digit-classification problem may have 10 neurons in the output-layer (whereby neuron with highest output would point to the currently made net-decision). But there is no rule that this is the only way to go.

efendi
  • 1