1

So I'm trying to build a seq2seq encoder-decoder network for a translation task. I've been stuck now for a while, I have no idea how to fix the error I'm getting and would appreciate any help.

My code looks like this:

input_shape = shape[1]
encoder_hidden_units = input_shape
dec_hidden = input_shape
inputs = tf.random.normal([shape[0], shape[1], emb_dim])

embedding = layers.Embedding(input_dim=input_shape, output_dim=emb_dim, mask_zero=True, batch_size=64)
model = keras.Sequential()
#encoder
model.add(embedding)
encoder = layers.Bidirectional(layers.GRU(encoder_hidden_units, dropout=dropout, return_sequences=False, return_state=False))
model.add(encoder)

#decoder
model.add(layers.Embedding(input_dim=enc_dim, output_dim=17, mask_zero=True, batch_size=64))
decoder = layers.GRU(dec_hidden, dropout=dropout, input_shape=shape, return_sequences=False, return_state=False)
model.add(decoder)
opt = keras.optimizers.Adam()
model.compile(loss='categorical_crossentropy', optimizer=opt)
model.summary()
model.fit(np.array(x), np.array(y), epochs=max_epoch, batch_size=64, verbose=0)

x and y are my training set/labels with shape (812,17)

The error message I get is:

ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

I tried adding a Dense layer with uniform activation, changing the loss function and the optimizer, getting the gradients as follows:

    with tf.GradientTape() as tape:
    loss = 'categorical_crossentropy'
vars = [1,2,3]
grads = tape.gradient(loss, vars)
processed_grads = [process_gradient(g) for g in grads]
opt.apply_gradients(zip(processed_grads, vars))

(I have to admit that I have no idea what to put in 'vars', that's something I found online) My data is paddded with 0s, so I thought this might be the problem? But that wouldn't make sense to me, as it's not the first time I build a network with padded data this way. It might be something very obvious that I'm doing wrong, I'm new to working with networks.. I would be happy for any comment/advice!

1

This error means that an operation in your graph is not differentiable. In this case, it's one of the operations within the embedding layers. For this reason, it's generally said that embedding layers can only be used as the first layer in a network.

This doesn't mean you can't do what you want, however, just that you'll have to code it differently. For example, since you're already set up with the GradientTape, you might be able to simply change your architecture into two models with the embedding layers as first layers, then calculate the gradients together:

with tf.GradientTape() as tape:
  y_pred = model2(model1(X))
  loss = loss_function(y_true,y_pred)
  grads = tape.gradient(loss,model1.trainable_variables+model2.trainable_variables)  
opt.apply_gradients(zip(grads,model1.trainable_variables+model2.trainable_variables))

You might also look into more robust solutions using tf.stop_gradient.

And as you know that you already know, don't sweat about zero-padding, that's normal practice.

2
  • Thanks a lot for taking the time to answer me! I have some follow up questions: Do I put this code before or after fitting the models? If I first try to fit the models, I get this error: tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[28,12] = 34 is not in [0, 17) Anf if I put it before fitting, I get AttributeError: 'int' object has no attribute 'dtype' for the line y_pred = model2(model1(x)) Do you have any ideas on why this is? And again, thank you for your help! I appreciate it. – maeven Aug 11 '20 at 12:22
  • The gradient tape code block is an alternative to fit; it's like one fitting step. It watches the models while the input tensor X goes through them, a loss function says how wrong the models were, gradients are computed thereon, then the optimizer applies them; the backprop learning process. Using tape let's you quickly get your hands into the fitting process, in this case so that one fitting can consider multiple models. Also, no need to compile when using tape. Again, though, I don't know if this will achieve what you want, just a way to redefine "beginning of a model." – TheLoneDeranger Aug 11 '20 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.