3

Suppose that I created this model using Keras:

model = Sequential()
model.add(Dense(32, activation='tanh', input_dim=1))
model.add(Dense(10, activation='tanh'))
model.add(Dense(1, activation='linear'))

The input and output dimensions of this model are both 1. Now, suppose I want to add one more layer, which is the sum of the first derivative and second derivative (with respect to the input) of the model above, and use it as my new output layer. Is that possible in Keras? I did quite a lot of Googling but couldn't find anything.

user1691278
  • 1,243
  • 1
  • 12
  • 20

1 Answers1

4

You can calculate gradient in Tensorflow with tf.gradients and in Keras with K.gradients:

first = K.gradients(model.outputs, model.inputs)
second = K.gradients(first, model.inputs)

Here is the code that calculates the gradient in your model:

from tensorflow.python.keras import Model, Input
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.layers import Dense, Lambda

def deriative(inps):
    i, o = inps[0], inps[1]
    grad1 = K.gradients(o, i)[0]
    grad2 = K.gradients(grad1, i)[0]
    return K.concatenate([grad1, grad2])


inps = Input(shape=(1,))
dense1 = Dense(32, activation='tanh')(inps)
dense2 = Dense(10, activation='tanh')(dense1)
dense3 = Dense(1, activation='linear')(dense2)

output = Lambda(deriative)([inps, dense3])

new_model = Model(inputs=inps, outputs=output)
new_model.compile('adam', 'mse')
print(new_model.summary())
Amir
  • 15,137
  • 10
  • 74
  • 113
  • I tried your code, but I got an error `AttributeError: 'NoneType' object has no attribute 'op'` from line `output = Lambda(deriative)([inps, dense3])`. – tryingtosolve Dec 25 '18 at 19:29
  • 1
    @tryingtosolve I added some more code. It compiles successfully on my own system with Tensorflow version 1.12 – Amir Dec 25 '18 at 19:34
  • 1
    Interesting. When I copy your code (i.e. when I use tensorflow), it works perfectly. However, when I use Keras, such as `from keras.layers import Dense, Activation, Lambda`, I get the same error. – tryingtosolve Dec 25 '18 at 20:52
  • Sorry, one more question: `K.concatenate([grad1, grad2])` returns a tensor of dimension 2. What if I want to return the sum of the two? I tried `K.sum(K.concatenate([grad1, grad2]), keepdims = True` but it gave me an error saying `An operation has None for gradient` – tryingtosolve Dec 25 '18 at 21:03
  • Did you try K.sum([grad1, grad2])? – Amir Dec 25 '18 at 21:10
  • 1
    I did. I just tired `K.sum([grad1, grad2], axis= 0)`. I got the same error. Does tensorflow recognize `K.gradients` as a differentiable function? – tryingtosolve Dec 25 '18 at 21:13
  • You can simply `grad1 + grad2`, but this error should be from somewhere else. – Daniel Möller Dec 26 '18 at 10:57