Neural Network for Multiple Output Regression

Question

I have a dataset containing 34 input columns and 8 output columns.

One way to solve the problem is to take the 34 inputs and build individual regression model for each output column.

I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used Multilayer Perceptron but that needs multiple models just like linear regression. Can Sequence to Sequence be a viable option?

I am using TensorFlow. I have code but I think it is more important to understand what I am missing out in terms of the multilayer perceptron theory.

I understand that in MLP if you have one output node it will provide one output. If you have 10 output nodes then it is a multi class problem. You pick the class with the highest probability out of the 10 outputs. But in my case it is certain there will be 8 outputs for same input.

Lets say, for a set of inputs you will get the 3D coordinate of something (X,Y,Z). Like, Inputs = {1,10,5,7} Output = {1,2,1}. So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model. So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though.

You seem to have some problems understanding multilayer perceptron NN model, and also TensorFlow - your statements about these are incorrect. However, it is not clear why you have that misunderstanding, which means an answer cannot help you fix this. Predicting e.g. 8 regression outputs in a single NN model is trivially easy in most NN frameworks, no need for sequences in your case. So I think it may be important to look at what your last paragraph is based on in order to help you - could you add some detail of what you have seen or tried in order to come to those thoughts? — Neil Slater, Feb 11 '17 at 08:36
Would it be possible for you provide an answer how to get 8 regression outputs using one single NN model? Thanks. — sjishan, Feb 11 '17 at 16:18
Probably, if you explain a few things by editing your question: 1) In what framework? 2) What is your code (or design, if you have no code) so far? 3) What is preventing you from doing this yourself? I need 1 and 2 in order to reply with something you can use. I need 3 in order to understand what your problem is and explain the solution. — Neil Slater, Feb 11 '17 at 16:45
So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model.
So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though. — sjishan, Feb 11 '17 at 17:07

score 17 · Answer 1 · edited Dec 18 '19 at 07:15

What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the nature of the problem.

Any neural network framework is able to do something like that.

The key to do that is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (34,) and the output (8,).

Update: the usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras; the network is not an MLP but it should be Ok to illustrate the idea.

Probably worth adding a line about usual cost function for regression (mean square error) and point at TensorFlow regression example - although I just spent 10 minutes looking for one now and didn't see anything . . . (examples skip from linear regression to MNIST classifiers, but no basic MLP regression models). — Neil Slater, Feb 11 '17 at 21:16

score 4 · Answer 2 · edited Mar 16 '19 at 15:58

You can implement this very simply in Python.
Your X will be the collection of training x,y,z coordinates.
Your Y will be the collection of testing x,y,z coordinates.

from sklearn import cross_validation                     
from sklearn.neural_network import MLPRegressor   

model = MLPRegressor(solver='lbfgs',alpha=0.001,hidden_layer_sizes=(150,))
cross_validation.cross_val_score(model, X, Y,scoring='mean_squared_error')

score 0 · Answer 3 · edited Feb 09 '21 at 04:25

This is much easier than you would think - you can simply set your output layer to be a vector instead of a single scalar. Of course there's no magic around here and I advice you to prep your data (perform batch normalization so all outputs will be values between 0 and 1).

If you're using Keras, the way for doing this is by adding a dense layer as the final output layer:

model.add(Dense(8, activation='linear'))

Neural Network for Multiple Output Regression

3 Answers3

Linked