Affiliations:
College of Electronic and Optical Engineering & College of Flexible Electronics, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu Province, China
Correspondence:
[*]
Corresponding author. Shujuan Yu, College of Electronic and Optical Engineering & College of Flexible Electronics, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu Province, China. E-mail: [email protected].
Abstract: Reading Comprehension models have achieved superhuman performance on mainstream public datasets. However, many studies have shown that the models are likely to take advantage of biases in the datasets, which makes it difficult to efficiently reasoning when generalizing to out-of-distribution datasets with non-directional bias, resulting in serious accuracy loss. Therefore, this paper proposes a pre-trained language model based de-biasing framework with positional generalization and hierarchical combination. In this work, generalized positional embedding is proposed to replace the original word embedding to initially weaken the over-dependence of the model on answer distribution information. Secondly, in order to make up for the influence of regularization randomness on training stability, KL divergence term is introduced into the loss function to constrain the distribution difference between the two sub models. Finally, a hierarchical combination method is used to obtain classification outputs that fuse text features from different encoding layers, so as to comprehensively consider the semantic features at the multidimensional level. Experimental results show that PLM-PGHC helps learn a more robust QA model and effectively restores the F1 value on the biased distribution from 37.51% to 81.78%.
Keywords: Natural language processing, machine reading comprehension, pre-trained language model, de-biasing framework
