1

I'm working with neural networks.

I have a number of inputs to the network which is far greater than the number of training instances ($\sim 7k$ inputs vs $700$ samples, $1$ order of magnitude, number of outputs $2k$).

My question, as a newcomer to neural nets, is this:

Should I diminish the size of the input layer (maybe through an autoencoder) ?

Dimensional reduction was suggested to me, but I didn't really understand why.

Haitao Du
  • 36,852
  • 25
  • 145
  • 242
Tommaso Guerrini
  • 1,259

1 Answers1

2

A NN's inputs can be equal to number of features in data, and have less relationship with number of data points. For example, we can some DNA data, where we only have 1000 people / instances, but each instance has millions of features in DNA. In such setting, it is perfectly OK to build a NN with millions of inputs.

For your question about dimension reduction: the key idea of using NN is letting the model to figure out the feature engineering / necessary transformation. So, the model can automatically do the feature reduction to us. It is not very common to run feature reduction (say PCA) first then feed into NN, unless there are some computational resource constraints.

Also, output is 2K seems too much for me. Are you trying to predict a discrete outcome with 2K possible values? In most cases, people predicting a much less possible values, such as binary Yes/No. (There are some reasons on why it is hard to do to predict 2K possible values, which I will not explain here.)

Haitao Du
  • 36,852
  • 25
  • 145
  • 242
  • Sorry for bothering you, it's not that clear to me one thing:

    are you saying that the number of inputs ($7k$ in my case) is not necessarily equal to the number of features one extract from those inputs? Or are you plainly saying that a high number of inputs/features is not an issue whichever the number of data?

    I'm forecasting $2k$ products daily sales. For now I'm considering $340$ products (read output) and a corresponding $\sim 1k$ inputs. Forecasting performance minimizing WeightedMape out performs every other algorithm, so I was doubting the suggestion given to me

     – Tommaso Guerrini Feb 16 '17 at 16:06
  • By the way (I wrote more here if you are interested: http://stats.stackexchange.com/questions/260444/from-bayesian-networks-to-neural-networks) – Tommaso Guerrini Feb 16 '17 at 16:07
  • @TommasoGuerrini I am trying to say "high number of inputs/features is not an issue whichever the number of data" – Haitao Du Feb 16 '17 at 16:07
  • Sorry for bothering you again: couldn't I have a pretty high probability of overfitting? – Tommaso Guerrini Mar 08 '17 at 15:00
  • 1
    @TommasoGuerrini yes, you could. And you can (and should) use regularization on your model. – Haitao Du Mar 08 '17 at 15:01
  • That's why I thought about reducing dimensionality of the input through an autoencoder. – Tommaso Guerrini Mar 08 '17 at 15:03
  • 1
    @TommasoGuerrini adding regularization and reducing input dimensions are different things. that is why I said, the model will work well with huge number of inputs (with regularization.) – Haitao Du Mar 08 '17 at 15:04