How to design a neural network to predict arrays from arrays
What you are trying to build is called a De-noising autoencoder. The goal here is to be able to reconstruct a noise-less sample by artificially introducing noise in a dataset, feed it to an encoder, then try to regenerate it without noise using a decoder.
This can be done with any form of data, including image and text.
I would recommend reading more about this. There are various concepts that ensure proper training of the model including understanding the requirement of a bottleneck in the middle to ensure proper compression and information loss, else the model just learns to multiply by 1 and returns the output.
Here is a sample piece of code. You can read more about this type of architecture here, written by the author of Keras himself.
from tensorflow.keras import layers, Model, utils, optimizers
#Encoder
enc = layers.Input((99,))
x = layers.Dense(128, activation='relu')(enc)
x = layers.Dense(56, activation='relu')(x)
x = layers.Dense(8, activation='relu')(x) #Compression happens here
#Decoder
x = layers.Dense(8, activation='relu')(x)
x = layers.Dense(56, activation='relu')(x)
x = layers.Dense(28, activation='relu')(x)
dec = layers.Dense(99)(x)
model = Model(enc, dec)
opt = optimizers.Adam(learning_rate=0.01)
model.compile(optimizer = opt, loss = 'MSE')
model.fit(x_train, y_train, epochs = 20)
Please beware, Autoencoders assume that the input data has some underlying structure to it and therefore can be compressed into a lower-dimensional space, which the decoder can use to regenerate the data. Using randomly generated sequences as data may not show any good results because its compression is not going to work without massive loss of information which itself has no structure to it.
As most of the other answers suggest, you are not using the activations properly. Since the goal is to regenerate a 99-dimensional vector with continuous values, it would make sense NOT to use sigmoid, instead, work with tanh as it compresses (-1,1) or no final layer activation, and not gates (0-1) the values.
Here is a Denoising autoencoder with conv1d and deconv1d layers. The issue here is that the input is too simple. See if you can generate more complex parametric functions for input data.
from tensorflow.keras import layers, Model, utils, optimizers
#Encoder with conv1d
inp = layers.Input((99,))
x = layers.Reshape((99,1))(inp)
x = layers.Conv1D(5, 10)(x)
x = layers.MaxPool1D(10)(x)
x = layers.Flatten()(x)
x = layers.Dense(4, activation='relu')(x) #<- Bottleneck!
#Decoder with Deconv1d
x = layers.Reshape((-1,1))(x)
x = layers.Conv1DTranspose(5, 10)(x)
x = layers.Conv1DTranspose(2, 10)(x)
x = layers.Flatten()(x)
out = layers.Dense(99)(x)
model = Model(inp, out)
opt = optimizers.Adam(learning_rate=0.001)
model.compile(optimizer = opt, loss = 'MSE')
model.fit(x_train, y_train, epochs = 10, validation_data=(x_test, y_test))
Epoch 1/10
188/188 [==============================] - 1s 7ms/step - loss: 2.1205 - val_loss: 0.0031
Epoch 2/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0032 - val_loss: 0.0032
Epoch 3/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0032 - val_loss: 0.0030
Epoch 4/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0031 - val_loss: 0.0029
Epoch 5/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0030 - val_loss: 0.0030
Epoch 6/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0029 - val_loss: 0.0027
Epoch 7/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0028 - val_loss: 0.0029
Epoch 8/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0028 - val_loss: 0.0025
Epoch 9/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0028 - val_loss: 0.0025
Epoch 10/10
188/188 [==============================] - 1s 5ms/step - loss: 0.0026 - val_loss: 0.0024
utils.plot_model(model, show_layer_names=False, show_shapes=True)
To make the model work you need to do some changes
First of all, your problem is a regression problem not a classification problem. So you need to change the loss from crossentropy to the mean squared error (mse)
Then you need to change the output of your last layer to output raw values
EDIT: On second thought because I saw wrong the kind of input, as suggested by @desertnaut it's better to use the raw output of the model
Anyway it is better to use an auto-encoder as suggested by @AkshaySehgal because in this way you are forcing the de-noising, making the net learning in the compressed space the underlying function
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=(99,)))
model.add(tf.keras.layers.Dense(768, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(768, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(99))
model.compile(optimizer = 'adam',
loss = 'mean_squared_error',
metrics = ['mse'])
model.fit(x_train, y_train, epochs = 20)
output:
Epoch 1/20
188/188 [==============================] - 2s 9ms/step - loss: 28.7281 - mse: 28.7281
Epoch 2/20
188/188 [==============================] - 2s 9ms/step - loss: 1.6866 - mse: 1.6866
Epoch 3/20
188/188 [==============================] - 2s 9ms/step - loss: 0.5031 - mse: 0.5031
Epoch 4/20
188/188 [==============================] - 2s 9ms/step - loss: 0.3126 - mse: 0.3126
Epoch 5/20
188/188 [==============================] - 2s 9ms/step - loss: 0.2186 - mse: 0.2186
Epoch 6/20
188/188 [==============================] - 2s 9ms/step - loss: 0.1420 - mse: 0.1420
Epoch 7/20
188/188 [==============================] - 2s 9ms/step - loss: 0.1334 - mse: 0.1334
Epoch 8/20
188/188 [==============================] - 2s 9ms/step - loss: 0.1193 - mse: 0.1193
Epoch 9/20
188/188 [==============================] - 2s 9ms/step - loss: 0.1174 - mse: 0.1174
Epoch 10/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0813 - mse: 0.0813
Epoch 11/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0334 - mse: 0.0334
Epoch 12/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0592 - mse: 0.0592
Epoch 13/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0162 - mse: 0.0162
Epoch 14/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0255 - mse: 0.0255
Epoch 15/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0208 - mse: 0.0208
Epoch 16/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0365 - mse: 0.0365
Epoch 17/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0236 - mse: 0.0236
Epoch 18/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0155 - mse: 0.0155
Epoch 19/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0204 - mse: 0.0204
Epoch 20/20
188/188 [==============================] - 2s 9ms/step - loss: 0.0145 - mse: 0.0145
<tensorflow.python.keras.callbacks.History at 0x7f60d19256d8>
If you need I also built the model in keras using colab. You can check the results directly from my notebook