6

enter image description here

I have just started studying Linear Regression using Neural Network and I believe this is a naive question.

I have a neural network for a Linear Regression Model with 3 independent variables. I have 1 hidden layer with 3 neurons with linear activations for each neuron. The last neuron is just another linear activation function that sums the three outputs from the hidden layer and compare the predicted value with the actual value.

I am wondering which weights are the final weights between the weights before the hidden layer and the weights after the hidden layer that I can use for the parameters of the Linear Regression Model. Hope to hear some explanations.

xabzakabecd
  • 3,455
  • 1
    What do you mean by “neural network for a Linear Regression Model”? A neural network is a different model than linear regression, you meant “for regression task”? What do you mean by “that I can use for the parameters of the Linear Regression Model”? You cannot use weights from one model in another model. – Tim Apr 02 '22 at 07:20
  • I used a neural network with backpropagation to find weights for each independent variable for regression task. – xabzakabecd Apr 02 '22 at 07:40

3 Answers3

9

A Neural network with one layer using linear activation is a linear regression. If you have a multi-layer neural network that only uses linear activations, it will reduce to linear regression. In other cases, it is a different algorithm. The same applies to logistic regression: it is the simplest, single-layer neural network with sigmoid activation, but if you add more layers, it is a different algorithm. You cannot use neural networks to "find" parameters of linear or logistic regression unless you mean the special cases mentioned above.

Firebug
  • 19,076
  • 6
  • 77
  • 139
Tim
  • 138,066
  • You can use neural networks in infinite width to create a kernel with so-called "neural tangent kernel" – qwr Apr 04 '22 at 02:18
8

This neural network does regression, but what it does is not equivalent to linear regression with the original features. So, you cannot just get the weights of either layer and build a linear regression model with those.

In order to emulate linear regression with a neural network, you should not have a hidden layer, but just one output layer.

gunes
  • 57,205
  • Does the same principle applies to Logistic Regression using Neural Network, too? – xabzakabecd Apr 02 '22 at 08:00
  • 3
    If you use only one neuron with sigmoid, yes. The loss should be cross entropy for being equivalent to a logistic regression. – gunes Apr 02 '22 at 08:36
3

Since your network has linear activation functions it can be written as a linear regression (it couldn't be if it used other activation functions). If write $w_{1i}$ for the weight of $x_i$ in neuron $h_1$ and $v_i$ for the weight of the output of $h_i$ in the final output, and $\beta_i$ for the coefficient of $x_i$ in a linear regression, then $$\beta_1=\sum_{j=1}^3 v_jw_{j1}$$ $$\beta_2=\sum_{j=1}^3 v_jw_{j2}$$ $$\beta_3=\sum_{j=1}^3 v_jw_{j3}$$

The estimates will be the same as the OLS estimates if you use in-sample squared error as your loss function and if you actually find the globally optimal $w$ and $v$. I don't know whether back-propagation guarantees finding the global optimum in this very simple case; it doesn't in more complicated cases.

Thomas Lumley
  • 38,062