Deep Learning Resources


Learning Rate

Minibatch Size


  • Early Stopping
    • Recently, TensorFlow has abandoned Monitor function, but uses SessionRunHooks instead. SessionRunHook is a part of tf.train, which has been developing. It will be the most important API to implement Early Stopping.
    • Reference to API - Train Hooks

Number of Hidden Units / Layers

Typically, 3-layer NNs performs better than 2-layer NNs in practice. But deeper NNs (i.e. 4-, 5-, 6-layer NNs) don’t help a lot, which is for sure contrary to CNNs. People have found that the depth is one of the most important components for CNNs. And the deeper the CNNs are, the better they perfrom as recognision systems. (Reference: cs231n - Andrej Karpathy)




The evaluation clearly demonstrated the superiority of the gated units; both the LSTM unit and GRU, over the traditional tanh unit. This was more evident with the more challenging task of raw speech signal modeling. However, we could not make concrete conclusion on which of the two gating units was better.

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, Author: Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio

The GRU outperformed the LSTM on all tasks with the exception of language modelling.

An Empirical Exploration of Recurrent Network Architectures

Our consistent finding is that depth of at least two is beneficial. However, between two and three layers our results are mixed. Additionally, the results are mixed between the LSTM and the GRU, but both significantly outperform the RNN.

Visualizing and Understanding Recurrent Networks, Author: Andrej Karpathy, Justin Johnson, Li Fei-Fei

Which of these variants is best? Do the differences matter? Greff, et al. (2015) do a nice comparison of popular variants, finding that they’re all about the same. Jozefowicz, et al. (2015) tested more than ten thousand RNN architectures, finding some that worked better than LSTMs on certain tasks.

Understanding LSTM Networks, Author: Chris Olah

Application Cell Num. of Layers Size Vocabulary Embedding Size Learning Rate Reference
Large Vocabulary Speech Recognition LSTM 5, 7 600, 1000 82K, 500K - - paper
Speech Recognition LSTM 1, 3, 5 250 - - 0.001 paper
seq2seq LSTM 4 1000 Origin: 160K, Target: 80K 1000 - paper
Image Caption Generator LSTM - 512 - 512 (fixed) paper
Image Generation LSTM - 256, 400, 800 - - - paper
Question Answering LSTM 2 500 - 300 - paper
Text Summarization (seq2seq) GRU - 200 Origin: 119K, Target: 68K 100 0.001 paper