- Learning Rate Decay
- Adaptive Learning Rate Optimizers
- Systematic evaluation of CNN advances on the ImageNet, Author: Dmytro Mishkin, Nikolay Sergievskiy, Jiri Matas
- Early Stopping
Number of Hidden Units / Layers
Typically, 3-layer NNs performs better than 2-layer NNs in practice. But deeper NNs (i.e. 4-, 5-, 6-layer NNs) don’t help a lot, which is for sure contrary to CNNs. People have found that the depth is one of the most important components for CNNs. And the deeper the CNNs are, the better they perfrom as recognision systems. (Reference: cs231n - Andrej Karpathy)
- Practical recommendations for gradient-based training of deep architectures, Author: Yoshua Bengio
- Deep Learning - 11.4 - Choosing Hyperparameters, Author: Ian Goodfellow、Yoshua Bengio、Aaron Courville
- Neural Networks and Deep Learning - Chap.3 - How to choose a neural network’s hyper-parameters, Author: Michael Nielsen
- Efficient BackProp, Author: Yann Lecun
- How to Generate a Good Word Embedding?, Author: Siwei Lai, Kang Liu, Liheng Xu, Jun Zhao
- Systematic evaluation of CNN advances on the ImageNet, Author: Dmytro Mishkin、Nikolay Sergievskiy、Jiri Matas
- Visualizing and Understanding Recurrent Networks, Author: Andrej Karpathy、Justin Johnson、Li Fei-Fei
LSTM vs. GRU
The evaluation clearly demonstrated the superiority of the gated units; both the LSTM unit and GRU, over the traditional tanh unit. This was more evident with the more challenging task of raw speech signal modeling. However, we could not make concrete conclusion on which of the two gating units was better.
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Author: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
The GRU outperformed the LSTM on all tasks with the exception of language modelling.
Our consistent finding is that depth of at least two is beneficial. However, between two and three layers our results are mixed. Additionally, the results are mixed between the LSTM and the GRU, but both significantly outperform the RNN.
Visualizing and Understanding Recurrent Networks, Author: Andrej Karpathy, Justin Johnson, Li Fei-Fei
Which of these variants is best? Do the differences matter? Greff, et al. (2015) do a nice comparison of popular variants, finding that they’re all about the same. Jozefowicz, et al. (2015) tested more than ten thousand RNN architectures, finding some that worked better than LSTMs on certain tasks.
Understanding LSTM Networks, Author: Chris Olah
|Application||Cell||Num. of Layers||Size||Vocabulary||Embedding Size||Learning Rate||Reference|
|Large Vocabulary Speech Recognition||LSTM||5, 7||600, 1000||82K, 500K||-||-||paper|
|Speech Recognition||LSTM||1, 3, 5||250||-||-||0.001||paper|
|seq2seq||LSTM||4||1000||Origin: 160K, Target: 80K||1000||-||paper|
|Image Caption Generator||LSTM||-||512||-||512||(fixed)||paper|
|Image Generation||LSTM||-||256, 400, 800||-||-||-||paper|
|Text Summarization (seq2seq)||GRU||-||200||Origin: 119K, Target: 68K||100||0.001||paper|