Visual saliency is widely applied to computer vision. Image saliency detection has been extensively studied, while there are only a few effective methods of computing saliency for videos owing to its high challenge. Inspired by image saliency methods, this paper proposes a unified spatiotemporal feature extraction and optimization framework for video saliency. First, the spatiotemporal feature descriptor is constructed via region covariance. Then, initial saliency map is computed by the local contrast of the descriptor. Finally, a local spatiotemporal optimization framework considering the previous and next frames of the current one is modeled to obtain the final saliency map. Extensive experiments on two public datasets demonstrate that the proposed algorithm not only outperforms the state-of-the-art methods, but also is of great extendibility.
田畅,姜青竹,吴泽民,刘涛,胡磊. 基于区域协方差的视频显著度局部空时优化模型[J]. 电子与信息学报, 2016, 38(7): 1586-1593.
TIAN Chang, JIANG Qingzhu, WU Zemin, LIU Tao, HU Lei. A Local Spatiotemporal Optimization Framework for Video Saliency Detection Using Region Covariance. JEIT, 2016, 38(7): 1586-1593.
BORJI A, CHENG M, JIANG H, et al. Salient object detection: A survey[OL]. http://arxiv.org/abs/1411.5878, 2014.
[2]
BORJI A, CHENG M, JIANG H, et al. Salient object detection: A benchmark[J]. IEEE Transactions on Image Processing, 2015, 24(12): 5706-5722. doi: 10.1109/ TIP.2015.2487833.
[3]
ROTHER C, KOLMOGOROV V, and BLAKE A. Grabcut: Interactive foreground extraction using iterated graph cuts[J]. ACM Transactions on Graphics, 2004, 23(1): 309-314. doi: 10.1145/1186562.1015720.
[4]
DING Y, XIAO J, and YU J. Importance filtering for image retargeting[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, USA, 2011: 89-96. doi: 10.1109/CVPR.2011.5995445.
[5]
MAHADEVAN V and VASOONCEIOS N. Saliency-based discriminant tracking[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, USA, 2009: 1007-1013. doi: 10.1109/CVPR.2009.5206573.
[6]
SHARMA G, JURIE F, and SCHMID C. Discriminative spatial saliency for image classification[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Rhode Island, USA, 2012: 3506-3513. doi: 10.1109/ CVPR.2012.6248093.
[7]
HADIZADEH H and BAJIC I. Saliency-aware video compression[J]. IEEE Transactions on Image Processing, 2014, 23(1): 19-33. doi: 10.1109/TIP.2013.2282897.
[8]
CHENG M, ZHANG G, MIERA N, et al. Global contrast based salient region detection[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, USA, 2011: 409-416. doi: 10.1109/ CVPR.2011.5995344.
[9]
PERAZZI F, KRAHENBUHL P, PRITCH Y, et al. Saliency filters: Contrast based filtering for salient region detection[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Rhode Island, USA, 2012: 733-740. doi: 10.1109/CVPR.2012.6247743.
[10]
YANG C, ZHANG L, LU H, et al. Saliency detection via graph-based manifold ranking[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, USA, 2013: 3166-3173. doi: 10.1109/CVPR. 2013.407.
[11]
ZHU W, LIANG S, WEI Y, et al. Saliency optimization from robust background detection[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, USA, 2014: 2814-2821. doi: 10.1109/ CVPR. 2014.360.
[12]
QIN Y, LU H, XU Y, et al. Saliency detection via cellular automata[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, 2015: 111-119. doi: 10.1109/CVPR.2015.7298606.
JIANG Yuwen, TAN Leyi, and WANG Shoujue. Saliency detected model based on selective edges prior[J]. Journal of Electronics & Information Technology, 2015, 37(1): 130-136. doi: 10.11999/JEIT140119.
LV Jianyong and TANG Zhenmin. An improved graph-based manifold ranking for salient object detection[J]. Journal of Electronics & Information Technology, 2015, 37(11): 2555-2563. doi: 10.11999/JEIT150619.
[15]
MAHADEVAN V and VASCONCELOS N. Spatiotemporal saliency in dynamic scenes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(1): 171-177. doi: 10.1109/TPAMI.2009.112.
[16]
ZHOU F, KANG S B, and COHEN M F. Time-mapping using space-time saliency[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, USA, 2014: 3358-3365. doi: 10.1109/CVPR.2014.429.
[17]
HUANG C, CHANG Y, YANG Z, et al. Video saliency map detection by dominant camera motion removal[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(8): 1336-1349. doi: 10.1109/TCSVT.2014.2308652.
[18]
LIU Z, ZHANG X, LUO S, et al. Superpixel-based spatiotemporal saliency detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(9): 1522-1540. doi: 10.1109/TCSVT.2014.2308642.
[19]
WANG W, SHEN J, and SHAO L. Consistent video saliency using local gradient flow optimization and global refinement[J]. IEEE Transactions on Image Processing, 2015, 24(10): 1-12. doi: 10.1109/TIP.2015.2460013.
[20]
MUTHUSWAMY K and RAJAN D. Particle filter framework for salient object detection in videos[J]. IET Computer Vision, 2015, 9(3): 428-438. doi: 10.1049/ iet-cvi.2013.0298.
[21]
KIM H, KIM Y, SIM J, et al. Spatiotemporal saliency detection for video sequences based on random walk with restart[J]. IEEE Transactions on Image Processing, 2015, 24(8): 2552-2564. doi: 10.1109/TIP.2015.2425544.
[22]
WANG W, SHEN J, and PORIKLI F. Saliency-aware geodesic video object segmentation[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, 2015: 3395-3402. doi: 10.1109/CVPR.2015.7298961.
[23]
KIM W and HAN J. Video saliency detection using contrast of spatiotemporal directional coherence[J]. IEEE Signal Processing Letters, 2014, 21(10): 1250-1254. doi: 10.1109/ LSP.2014.2332213.
[24]
ERDEM E and ERDEM A. Visual saliency estimation by nonlinearly integrating features using region covariances[J]. Journal of Vision, 2013, 13(4): 1-20. doi: 10.1167/13.4.11.
[25]
KOCAK A, CIZMECILERr K, ERDEM A, et al. Top down saliency estimation via superpixel-based discriminative dictionaries[C]. British Machine Vision Conference (BMVC), Nottingham, UK, 2014: 1-10. doi: 10.5244/C.28.73.
[26]
CHANG K, LIU Y, CHEN H, et al. Fusing generic objectness and visual saliency for salient object detection[C]. IEEE Conference on Computer Vision (ICCV), Barcelona, Spain, 2011: 914-921. doi: 10.1109/ICCV.2011.6126333.
[27]
LI J, TIAN Y, DUAN L, et al. Estimating visual saliency through single image optimization[J]. IEEE Signal Processing Letters, 2013, 20(9): 845-848. doi: 10.1109/LSP. 2013.2268868.
[28]
TUAEL O, PORIKLI F, and MEER P. Region covariance: A fast descriptor for detection and classification[C]. European Conference on Computer Vision, Graz, Austria, 2006: 589-600. doi: 10.1007/11744047_45.
[29]
ACHANTA R, SHAJI A, SMITH K, et al. Slic superpixels compared to state-of-the-art superpixel methods[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2274-2281. doi: 10.1109/TPAMI.2012.120.
[30]
BROX T and MALIK J. Large displacement optical flow: Descriptor matching in variational motion estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(3): 500-513. doi: 10.1109/TPAMI. 2010.143.
[31]
HONG X, CHANG H, and SHAN S. Sigma set: A small second order statistical region descriptor[C]. IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, 2009: 1802-1809. doi: 10.1109/CVPR. 2009.5206742.
[32]
TSAI D, FLAGG M, and REHG J M. Motion coherent tracking with multilabel MRF optimization[C]. British Machine Vision Conference (BMVC), Aberystwyth, UK, 2010: 1-11. doi: 10.5244/C.24.56.
