Affiliations: [a] School of Computing Science and Artificial Intelligence, Changzhou University, Changzhou, Jiangsu, China | [b] School of Computer and Computing Science, Zhejiang University City College, Hangzhou, Zhejiang, China | [c] China National Air Separation Engineering Co., Ltd, Hangzhou, Zhejiang, China
Correspondence:
[*]
Corresponding author: Guanlin Chen, School of Computer and Computing Science, Zhejiang University City College, Hangzhou, Jiangsu 310015, China. E-mail: [email protected].
Abstract: Recently, AWD-LSTM (ASGD Weight-Dropped LSTM) has achieved good result in the language model, and many AWD-LSTM based models have obtained state-of-the-art perplexities. However, in fact, large-scale neural language models have been shown to be prone to overfitting. In AWD-LSTM original paper, the author decided to adopt the way of retraining calling finetune to get a better result. In this paper, we present a simple yet effective parameter rollback mechanism for neural language models. And we introduce the parameter rollback averaged stochastic gradient descent (PR-ASGD), wherein the parameter “step” in ASGD will decrease according to a certain probability. Using this strategy, we achieve better word level perplexities on Penn Treebank: 56.26 based on AWD-LSTM model and 53.57 based on AWD-LSTM-MoS (AWD-LSTM Mixture of Softmaxes) model.
Keywords: Optimizer, language model, machine learning
