So I'm trying to build a seq2seq encoder-decoder network for a translation task. I've been stuck now for a while, I have no idea how to fix the error I'm getting and would appreciate any help.
My code looks like this:
input_shape = shape encoder_hidden_units = input_shape dec_hidden = input_shape inputs = tf.random.normal([shape, shape, emb_dim]) embedding = layers.Embedding(input_dim=input_shape, output_dim=emb_dim, mask_zero=True, batch_size=64) model = keras.Sequential() #encoder model.add(embedding) encoder = layers.Bidirectional(layers.GRU(encoder_hidden_units, dropout=dropout, return_sequences=False, return_state=False)) model.add(encoder) #decoder model.add(layers.Embedding(input_dim=enc_dim, output_dim=17, mask_zero=True, batch_size=64)) decoder = layers.GRU(dec_hidden, dropout=dropout, input_shape=shape, return_sequences=False, return_state=False) model.add(decoder) opt = keras.optimizers.Adam() model.compile(loss='categorical_crossentropy', optimizer=opt) model.summary() model.fit(np.array(x), np.array(y), epochs=max_epoch, batch_size=64, verbose=0)
x and y are my training set/labels with shape (812,17)
The error message I get is:
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.
I tried adding a Dense layer with uniform activation, changing the loss function and the optimizer, getting the gradients as follows:
with tf.GradientTape() as tape: loss = 'categorical_crossentropy' vars = [1,2,3] grads = tape.gradient(loss, vars) processed_grads = [process_gradient(g) for g in grads] opt.apply_gradients(zip(processed_grads, vars))
(I have to admit that I have no idea what to put in 'vars', that's something I found online) My data is paddded with 0s, so I thought this might be the problem? But that wouldn't make sense to me, as it's not the first time I build a network with padded data this way. It might be something very obvious that I'm doing wrong, I'm new to working with networks.. I would be happy for any comment/advice!