基于回译和集成学习的维汉神经机器翻译方法

兰州理工大学学报 ›› 2022, Vol. 48 ›› Issue (5): 99-106.

• 自动化技术与计算机技术 • 上一篇下一篇

基于回译和集成学习的维汉神经机器翻译方法

冯笑^1,2,3, 杨雅婷^*1,2,3, 董瑞^1,2,3, 艾孜麦提·艾尼瓦尔^1,2,3, 马博^1,2,3

1.中国科学院新疆理化技术研究所, 新疆乌鲁木齐 830011;
2.中国科学院大学, 北京 100049;
3.新疆民族语音处理实验室, 新疆乌鲁木齐 830011

收稿日期:2021-04-16 出版日期:2022-10-28 发布日期:2022-11-21
通讯作者: 杨雅婷(1984-),女,新疆奇台人,博士,研究员. Email:yangyt@ms.xjb.ac.cn
基金资助:
国家自然科学基金(U2003303),新疆高层次引进人才项目(新人社函[2017]699号),中国科学院西部之光人才培养计划A类资助项目(2017-XBQNXZ-A-005),中科院创新青年促进会资助项目(2017472,科发人函字[2019]26号)

Uyghur-Chinese neural machine translation method based on back translation and ensemble learning

FENG Xiao^1,2,3, YANG Ya-ting^1,2,3, DONG Rui^1,2,3, AZMAT Anwar^1,2,3, MA Bo^1,2,3

1. Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi 830011, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China;
3. Xinjiang Laboratory of Minority Speech and Language Information Processing,Urumqi 830011, China

Received:2021-04-16 Online:2022-10-28 Published:2022-11-21

摘要/Abstract

摘要： 从高效利用现有资源的角度出发,针对维汉平行语料匮乏导致维汉神经机器翻译效果欠佳的问题,提出一个基于回译和集成学习的方法.首先,利用回译和大规模汉语单语语料构造出维汉伪平行语料,并利用伪平行语料进行训练得到中间模型;其次,使用自助采样法对原始平行语料进行N次重采样,得到N个近似同一分布但具有差异性的子数据集;基于N个子数据集分别对中间模型进行微调,得到N个具有差异性的子模型;最后,将这些子模型集成.在CWMT2015和CWMT2017的测试集上的实验证明,该方法比基线系统的BLEU值分别提升了2.37和1.63.

关键词: 神经机器翻译, 回译, 集成学习, 中间模型, 微调, 灾难性遗忘

Abstract: From the perspective of efficient utilization of existing resources, a method based on back-translation and ensemble learning is proposed to solve the problem of the poor performance of Uyghur-Chinese neural machine translation caused by the lack of parallel corpus. Firstly, Uyghur and Chinese pseudo-parallel corpora are constructed by using back translation and large-scale Chinese monolingual corpora, and the intermediate model is obtained by using pseudo parallel corpora training. Secondly, the bootstrap is used to resample the original parallel corpus for N times, and N sub-datasets with similar distribution but different characteristics are obtained. The intermediate model were fine-tuned based on N sub-data sets, and N sub-models with differences were obtained. Finally, integrate these sub-models. Experiments on the test sets of CWMT2015 and CWMT2017 show that theBLEU(Bilingual Evaluation Understudy) value of this method are 2.37 and 1.63 higher than that of the baseline system, respectively.

Key words: neural machine translation, back translation, ensemble learning, intermediate model, fine tuning, catastrophic forgetting

中图分类号:

TP391

冯笑, 杨雅婷, 董瑞, 艾孜麦提·艾尼瓦尔, 马博. 基于回译和集成学习的维汉神经机器翻译方法[J]. 兰州理工大学学报, 2022, 48(5): 99-106.

FENG Xiao, YANG Ya-ting, DONG Rui, AZMAT Anwar, MA Bo. Uyghur-Chinese neural machine translation method based on back translation and ensemble learning[J]. Journal of Lanzhou University of Technology, 2022, 48(5): 99-106.

参考文献

[1] ABUDUKELIMU H,LIU Y,CHEN X,et al.Learning distributed representations of uyghur words and morphemes [M]//Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data.Cham:Springer,2015:202-211.
[2] 哈里旦木·阿布都克里木,刘洋,孙茂松.神经机器翻译系统在维吾尔语-汉语翻译中的性能对比 [J].清华大学学报(自然科学版),2017,57(8):878-883.
[3] SENNRICH R,HADDOW B,BIRCH A.Improving neural machine translation models with monolingual data[EB/OL].[2021-02-15].https://arxiv.org/abs/1511.06709.
[4] PONCELAS A,SHTERIONOV D,WAY A,et al.Investigating backtranslation in neural machine translation [EB/OL].[2021-02-15].https://arxiv.org/abs/1804.06189.
[5] FADAEE M,MONZ C.Back-translation sampling by targeting difficult words in neural machine translation [EB/OL].[2021-02-15].https://arxiv.org/abs/1808.09006.
[6] HOANG V C D,KOEHN P,HAFFARI G,et al.Iterative back-translation for neural machine translation [C]//Proceedings of the 2nd Workshop on Neural Machine Translation and Generation.Melbourne,Australia:[s.n.],2018:18-24.
[7] COTTERELL R,KREUTZER J.Explaining and generalizing back-translation through wake-sleep [EB/OL].[2021-02-15].https://arxiv.org/abs/1806.04402.
[8] PONCELAS A,POPOVIC M,SHTERIONOV D,et al.Combining SMT and NMT back-translated data for efficient NMT [EB/OL].[2021-02-15].https://arxiv.org/abs/1909.03750.
[9] LUO G X,YANG Y T,DONG R,et al.A joint back-translation and transfer learning method for low-resource neural machine translation [J/OL].[2020-02-15].https://doaj.org/article/3ee33e9edcd04597aa86ce4dd19ea6e7.
[10] ZHOU Z H.Ensemble methods:foundations and algorithms [M].Boca Raton:CRC Press,2012.
[11] BREIMAN L.Bagging predictors [J].Machine Learning,1996,24(2):123-140.
[12] 李睿,张九蕊,毛莉.基于AdaBoost的弱分类器选择和整合算法 [J].兰州理工大学学报,2012,38(2):87-90.
[13] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need [EB/OL].[2021-02-15].https://arxiv.org/abs/1706.03762.
[14] SENNRICH R,HADDOW B,BIRCH A.Edinburgh neural machine translation systems for WMT 16 [EB/OL].[2021-02-15].https://arxiv.org/abs/1606.02891.
[15] SENNRICH R,BIRCH A,CURREY A,et al.The university of Edinburgh's neural MT systems for WMT17 [EB/OL].[2021-02-15].https://arxiv.org/abs/1708.00726.
[16] 李北,王强,肖桐,等.面向神经机器翻译的集成学习方法分析 [J].中文信息学报,2019,33(3):42-51.
[17] 张新路,李晓,杨雅婷,等.面向维汉神经机器翻译的双向重排序模型分析 [J].北京大学学报(自然科学版),2020,56(1):31-38.
[18] WANG Y,WU L,XIA Y,et al.Transductive ensemble learning for neural machine translation [C]//Proceedings of the AAAI Conference on Artificial Intelligence.New York:[s.n.],2020:6291-6298.
[19] SUTSKEVER I,VINYALS O,LE Q V.Sequence to sequence learning with neural networks [EB/OL].[2021-02-15].https://arxiv.org/abs/1409.3215.
[20] BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate [EB/OL].[2021-02-15].https://arxiv.org/abs/1409.0473v4.
[21] GEHRING J,AULI M,GRANGIER D,et al.Convolutional sequence to sequence learning [C]//International Conference on Machine Learning.[S.l.]:PMLR,2017:1243-1252.
[22] BA J L,KIROS J R,HINTON G E.Layer normalization [EB/OL].[2021-02-15].https://arxiv.org/abs/1607.06450.
[23] OCH F J,GILDEA D,KHUDANPUR S,et al.A smorgasbord of features for statistical machine translation [C]//Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics.Boston:[s.n.],2004:161-168.
[24] IMAMURA K,SUMITA E.Ensemble and reranking:Using multiple models in the NICT-2 neural machine translation system at WAT2017 [C]//Proceedings of the 4th Workshop on Asian Translation (WAT2017).Taipei,China:[s.n.],2017:127-134.
[25] JOHNSON R W.An introduction to the bootstrap [J].Teaching Statistics,2010,23(2):49-54.
[26] SENNRICH R,HADDOW B,BIRCH A.Neural machine translation of rare words with subword units [EB/OL].[2021-02-15].https://arxiv.org/abs/1508.07909.
[27] ZHANG J J,ZONG C Q.Exploiting source-side monolingual data in neural machine translation [C]//Proceedings of EMNLP2016.Austin,Texas:[s.n.],2016:1535-1545.
[28] KINGMA D P,BA J.Adam:A method for stochastic optimization [EB/OL].[2021-02-15].https://arxiv.org/abs/1412.6980.
[29] GAL Y,GHAHRAMANI Z.A theoretically grounded application of dropout in recurrent neural networks [J].Advances in Neural Information Processing Systems,2016,29:1019-1027.
[30] PAPINENI K,ROUKOS S,WARD T,et al.BLEU:a method for automatic evaluation of machine translation [C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.Philadelphia:[s.n.],2002:311-318.
[31] ZHANG Y,VOGEL S,WAIBEL A.Interpreting BLEU/NIST scores:How much improvement do we need to have a better system [C/OL].[2021-02-15].https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.368.1133& repl & type=pdf.

基于回译和集成学习的维汉神经机器翻译方法

Uyghur-Chinese neural machine translation method based on back translation and ensemble learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

编辑推荐

Metrics

本文评价

[1]	赵荣珍, 赵楠. 结合扰动集成RBF的故障识别方法[J]. 兰州理工大学学报, 2022, 48(6): 40-45.
[2]	赵荣珍, 马森财, 吴耀春. 云模型和集成极限学习机相结合的滚动轴承故障诊断方法[J]. 兰州理工大学学报, 2021, 47(4): 33-39.