Does each gate have one, or two matrices in LSTM?

Question

According to these equations in wikimedia:

each gate has two weight matrices:W,U respectively,but according this: from:http://colah.github.io/posts/2015-08-Understanding-LSTMs/

each gate has only one weight matrix:W,and in the example code:

z = np.row_stack((h_prev, x))
f = sigmoid(np.dot(W_f, z) + b_f)
i = sigmoid(np.dot(W_i, z) + b_i)

from:http://blog.varunajayasiri.com/numpy_lstm.html,apparently:

input and previous hidden status are stacked together to multiply same weight matrix

So,

Does each gate have one, or two matrices in LSTM? or I don't understand them correctly
if two,what does U matrix(wikimedia) mean?

Frames Catherine White · Accepted Answer · 2017-12-05T09:05:38.307

4

The area the same thing.

$W\cdot [a; b] = U\cdot a + V\cdot b$
if $W=[U\; V]$

For $[a;b]$ being the vertical concatenation of the vectors (considered as column matrices) and $[U\; V]$ the horizontal concatenation of the matrices respectively.

The difference is purely notation.

Consider how matrix multiplication works. or we are looking at it as a block matrix

$$[U\; V]\cdot [a;b] = U\cdot a + V\cdot b $$

edited Dec 05 '17 at 09:05

answered Dec 05 '17 at 08:58

Frames Catherine White

3,154

,thanks,so the example is wrong? – Alex Luya Dec 05 '17 at 09:14
No, both are correct. They are just two different ways of writing the same thing. They are mathematically equivalent. – Frames Catherine White Dec 05 '17 at 09:19
so wikimeida cuts a tall weight matrix to two short matrices? – Alex Luya Dec 05 '17 at 09:23
Exactly. (or equivalently colah stacks to short matrices into one tall one) – Frames Catherine White Dec 05 '17 at 09:24

Does each gate have one, or two matrices in LSTM?

1 Answers1