Gaze prediction algorithm based on hypercomplex wavelet convolutional network

Abstract

Abstract: Gaze based prediction algorithms has a wide range of applications in object recognition, video compression, object tracking and so on. For existing gaze prediction models, the accuracy of gaze prediction is low due to the lack of feature details, single scale, and serious background information interference. This paper proposes a gaze prediction algorithm based on hypercomplex wavelet convolutional network. Firstly, aiming at the problem of loss of detailed features, the hypercomplex wavelet transform is used to extract the detailed features of the image in the frequency domain and fused with the spatial features extracted by the convolutional network. Then, through the atrous spatial pyramid pooling module, the feature maps obtained from different receptive fields are fused to effectively solve the problem of single feature scale. Finally, the proposed algorithm introduces a residual convolutional attention module, which combines spatial and channel attention mechanisms to effectively suppress the interference of background information and improve the accuracy of gaze prediction. On the SALICON datasets, CC, sAUC and SIM evaluation metrics, the performance of the proposed algorithm reaches 0.884 7, 0.769 3 and 0.778 0. On the CAT2000 datasets, the performance of the proposed algorithm is 0.735 5, 0.870 1, and 0.664 5. The experimental results show that the proposed algorithm has a good ability to predict fixation points.

Key words: gaze prediction, hypercomplex wavelet transform, spatial features, convolutional neural network

CLC Number:

TP393

LI Ce, ZHU Zi-zhong, XU Da-you, GAO Wei-zhe, JIN Shan-gang. Gaze prediction algorithm based on hypercomplex wavelet convolutional network[J]. Journal of Lanzhou University of Technology, 2021, 47(5): 76-84.

References

[1] 姜波.基于卷积神经网络的注视点检测 [D].大连:大连理工大学,2017.
[2] ITTI L,KOCH C,NIEBUR E.A model of saliency-based visual attention for rapid scene analysis [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1998,20(11):1254-1259.
[3] HOU X,ZHANG L.Saliency detection:a spectral residual approach [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Minneapolis,MN,USA:IEEE,2007:1-8.
[4] TORRALBA A,OLIVA A,CASTELHANO M S,et al.Contextual guidance of eye movements and attention in real-world scenes:the role of global features in object search [J].Psychological Review,2006,113(4):766-786.
[5] GUO C,MA Q,ZHANG L.Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Anchorage,AK,USA:IEEE,2008:1-8.
[6] 肖利梅,曹洁,李策.基于多尺度相位谱的显著性运动目标检测 [J].兰州理工大学学报,2011,37(4):90-94.
[7] LI J,LEVINE M D,AN X,et al.Visual saliency based on scale-space analysis in the frequency domain [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(4):996-1010.
[8] JUDD T,EHINGER K,DURAND F,et al.Learning to predict where humans look [C]//Proceedings of IEEE International Conference on Computer Vision.Kyoto,Japan:IEEE,2010:2106-2113.
[9] VIG E,DORR M,COX D.Large-scale optimization of hierarchical features for saliency prediction in natural images [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Columbus:IEEE,2014:2798-2805.
[10] KÜMMERER M,WALLIS T S A,GATYS L A,et al.Understanding low-and high-level contributions to fixation prediction[C]//Proceedings of IEEE International Conference on Computer Vision.Venice,Italy:IEEE,2017:4799-4808.
[11] CORNIA M,BARALDI L,SERRA G,et al.A deep multi-level network for saliency prediction [C]//Proceedings of IEEE International Conference on Pattern Recognition.Cancun,Mexico:IEEE,2016:3488-3493.
[12] HE S,TAVAKOLI H R,BORJI A,et al.Understanding and visualizing deep visual saliency models [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Long Beach,CA,USA:IEEE,2019:10206-10215.
[13] CORNIA M,BARALDI L,SERRA G,et al.Predicting human eye fixations via an LSTM-based saliency attentive model [J].IEEE Transactions on Image Processing,2018,27(10):5142-5154.
[14] CHAN W L,CHOI H,BARANIUK R G.Multiscale image disparity estimation using the quaternion wavelet transform [C]//Proceedings of IEEE International Conference on Image Processing.Atlanta,GA,USA:IEEE,2007:1229-1232.
[15] KRONER A,SENDEN M,DRIESSEN K,et al.Contextual encoder-decoder network for visual saliency prediction [CP/OL].(2019-08-10) [2020-01-05].https://github.com/alexanderkroner/saliency.
[16] WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional block attention module [C]//European Conference on Computer Vision.Munich,Germany:[s.n.],2018:3-19.
[17] JIANG M,HUANG S S,DUAN J Y,et al.SALICON:Saliency in context [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Boston:IEEE,2015:1072-1080.
[18] BORJI A,ITTI L.CAT2000:A large scale fixation dataset for boosting saliency research [C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.[S.l.]:IEEE,2015:1-4.
[19] LIU N,HAN J.A deep spatial contextual long-term recurrent convolutional network for saliency detection [J].IEEE Transactions on Image Processing,2018,27(7):3264-3274.