Equivalent for np.add.at in tensorflow

Question

How do I convert a np.add.at statement into tensorflow?

np.add.at(dW, self.x.ravel(), dout.reshape(-1, self.D))

Edit

self.dW.shape is (V, D), self.D.shape is (N, D) and self.x.size is N

Expand on your question. Are you trying to understand what the `add.at` does? Or trying get something in `tensorflow` that does the same thing and at the same speed? `add.at` is use to speed up iterative problems where the standard buffer `add` produces the wrong result. — hpaulj, Nov 02 '16 at 21:26
Search `SO` for '[numpy] add.at' to see how `add.at` has been used solve various `numpy` problems. — hpaulj, Nov 02 '16 at 21:35
Do you know if there are duplicate indices in `self.x`? That's when `at` makes a difference. — hpaulj, Nov 02 '16 at 22:25
I do understand what the code does. I am looking for an tensorflow alternative. — Amey Agrawal, Nov 03 '16 at 00:54
If anybody, who is open to do it in pytorch, comes here looking for a solution, check this out: https://stackoverflow.com/a/65584479/3337089 — Nagabhushan S N, Jan 05 '21 at 18:36

score 2 · Accepted Answer · answered Nov 02 '16 at 21:33

2

For np.add.at, you probably want to look at tf.SparseTensor, which represents a tensor by a list of values and a list of indices (which is more suitable for sparse data, hence the name).

So for your example:

np.add.at(dW, self.x.ravel(), dout.reshape(-1, self.D))

that would be (assuming dW, x and dout are tensors):

tf.sparse_add(dW, tf.SparseTensor(x, tf.reshape(dout, [-1])))

This is assuming x is of shape [n, nDims] (i.e. x is a 'list' of n indices, each of dimension nDims), and dout has shape [n].

answered Nov 02 '16 at 21:33

Lars Mennen

424
4
4

Actually in the original numpy code, dout is of shape (N, D). self.x.reshape(-1) gives array of shape (N, ). While dW has shape (V, D). – Amey Agrawal Nov 03 '16 at 00:52
What is the shape of self.x? (before reshape) Your edit mentions it is of size N, but in case it is of shape N, why do you do .ravel()? – Lars Mennen Nov 03 '16 at 08:25
Actually the shape of x is different but it's size is N. – Amey Agrawal Nov 03 '16 at 08:29
You could do tf.reshape(x, [-1]) to get a shape [N] tensor then. – Lars Mennen Nov 03 '16 at 21:17
If this sparse tensor is modeled on the scipy csr sparse matrix, duplicate indices are summed. That's the kind of behavior that the unbuffered `add.at` is supposed to produce. – hpaulj Nov 04 '16 at 08:15
Thank you! Sorry for the late response. – Amey Agrawal Dec 04 '16 at 04:32

score 0 · Answer 2 · answered Nov 03 '16 at 00:16

Here's an example of what np.add.at does:

In [324]: a=np.ones((10,))
In [325]: x=np.array([1,2,3,1,4,5])
In [326]: b=np.array([1,1,1,1,1,1])
In [327]: np.add.at(a,x,b)
In [328]: a
Out[328]: array([ 1.,  3.,  2.,  2.,  2.,  2.,  1.,  1.,  1.,  1.])

If instead I use +=

In [331]: a1=np.ones((10,))
In [332]: a1[x]+=b
In [333]: a1
Out[333]: array([ 1.,  2.,  2.,  2.,  2.,  2.,  1.,  1.,  1.,  1.])

note that a1[1] is 2, not 3.

If instead I use an iterative solution

In [334]: a2=np.ones((10,))
In [335]: for i,j in zip(x,b):
     ...:     a2[i]+=j
     ...:     
In [336]: a2
Out[336]: array([ 1.,  3.,  2.,  2.,  2.,  2.,  1.,  1.,  1.,  1.])

it matches.

If x does not have duplicates then += works just fine. But with the duplicates, the add.at is required to match the iterative solution.

Equivalent for np.add.at in tensorflow

2 Answers2

Linked