... | ... | @@ -14,13 +14,13 @@ The number of neurons within the hidden layers typically follows a bow tie trend |
|
|
|
|
|
## Hints and Tips
|
|
|
|
|
|
Hint: Linear auto-encoders are equivalent to PCA. If you do not use a nonlinear activation function, no need to bother with NN training.
|
|
|
* Linear auto-encoders are equivalent to PCA. If you do not use a nonlinear activation function, no need to bother with NN training.
|
|
|
|
|
|
Hint: If the reconstruction loss cannot be improved, you may got stuck in a local minima. Play around to pay pass it. If still not possible, a greedy training approach (like Boltzmann machines) can be used to initialize the network weights for a better performance. Alternatively, you can recursively train the layers of the autoencoder. First, train a simple AE instead of a deep one. Once you are fine with the reconstruction loss, get the latent representation as a second data set and create a second AE model, further reduce the number of dimensions. Once the training of the second model is done, use its latent space for the third model and so on. Once it is finished, you can combined the trained layers together, giving a big AE.
|
|
|
* If the reconstruction loss cannot be improved, you may got stuck in a local minima. Play around to pay pass it. If still not possible, a greedy training approach (like Boltzmann machines) can be used to initialize the network weights for a better performance. Alternatively, you can recursively train the layers of the autoencoder. First, train a simple AE instead of a deep one. Once you are fine with the reconstruction loss, get the latent representation as a second data set and create a second AE model, further reduce the number of dimensions. Once the training of the second model is done, use its latent space for the third model and so on. Once it is finished, you can combined the trained layers together, giving a big AE.
|
|
|
|
|
|
Hint: If you want to build an autoencoder for high dimensional datasets like images, you will need a **convolutional autoencoder**. In such a case, you can trade the spatial dimensions of the data (width and height of an image) with the image features (depth; number of channels) in the encoder. The decoder is to be working in the opposite direction. For such a purpose, you can use transpose convolutional layers in TensorFlow (e.g., layers.Conv2DTranspose). This is illustrated in DDE 2 lecture, CNN week.
|
|
|
* If you want to build an autoencoder for high dimensional datasets like images, you will need a **convolutional autoencoder**. In such a case, you can trade the spatial dimensions of the data (width and height of an image) with the image features (depth; number of channels) in the encoder. The decoder is to be working in the opposite direction. For such a purpose, you can use transpose convolutional layers in TensorFlow (e.g., layers.Conv2DTranspose). This is illustrated in DDE 2 lecture, CNN week.
|
|
|
|
|
|
Hint: If you are dealing with series, you can use RNN layers as an AE as well. Herein, AE is to be built as a seq-to-seq model, where the encoder acts like a sequence-to-vector RNN and the decoder functions as a vector-to-sequence RNN. This can be achieved via “return_sequences=False” ,RepeatVector layer and TimeDistributed layer in TF. See the notebook from RNN week for an explicit illustration.
|
|
|
* If you are dealing with series, you can use RNN layers as an AE as well. Herein, AE is to be built as a seq-to-seq model, where the encoder acts like a sequence-to-vector RNN and the decoder functions as a vector-to-sequence RNN. This can be achieved via “return_sequences=False” ,RepeatVector layer and TimeDistributed layer in TF. See the notebook from RNN week for an explicit illustration.
|
|
|
|
|
|
|
|
|
## Additional resources |
|
|
\ No newline at end of file |