Youzhao Yang
ABSTRACT
Single image deraining is an important problem in many computer vision tasks since rain streaks can severely hamper and degrade the visibility of images. In this paper, we propose a novel network named Recurrent Hierarchy Enhancement Network (ReHEN) to remove rain streaks from rainy images stage by stage. Unlike previous deep convolutional network methods, we adopt a Hierarchy Enhancement Unit (HEU) to fully extract local hierarchical features and generate effective features. Then a Recurrent Enhancement Unit (REU) is added to keep the useful information from HEU and benefit the rain removal in the later stages. To focus on different scales, shapes, and densities of rain streaks adaptively, Squeeze-and-Excitation (SE) block is applied in both HEU and REU to assign different scale factors to high-level features. Experiments on five synthetic datasets and a real-world rainy image set show that the proposed method outperforms the state-of-the-art methods considerably. The source code is available at https://github.com/nnUyi/ReHEN.
- Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et almbox. 2016. Tensorflow: a system for large-scale machine learning.. In OSDI, Vol. 16. 265--283.Google Scholar
Digital Library
- Peter C Barnum, Srinivasa Narasimhan, and Takeo Kanade. 2010. Analysis of rain and snow in frequency space. International Journal of Computer Vision , Vol. 86, 2--3 (2010), 256.Google ScholarDigital Library
- Jie Chen, Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau, and He Li. 2018. Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6286--6295.Google ScholarCross Ref
- Yi-Lei Chen and Chiou-Ting Hsu. 2013. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE International Conference on Computer Vision. 1968--1975.Google ScholarDigital Library
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).Google Scholar
- Liang-Jian Deng, Ting-Zhu Huang, Xi-Le Zhao, and Tai-Xiang Jiang. 2018. A directional global sparse model for single image rain removal. Applied Mathematical Modelling , Vol. 59 (2018), 662--679.Google ScholarCross Ref
- David Eigen, Dilip Krishnan, and Rob Fergus. 2013. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE International Conference on Computer Vision. 633--640.Google ScholarDigital Library
- Zhiwen Fan, Huafeng Wu, Xueyang Fu, Yue Huang, and Xinghao Ding. 2018. Residual-Guide Network for Single Image Deraining. In Proceedings of the 26th ACM International Conference on Multimedia. 1751--1759.Google ScholarDigital Library
- Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1715--1723.Google ScholarCross Ref
- Kshitiz Garg and Shree K Nayar. 2004. Detection and removal of rain from videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 528--535.Google ScholarCross Ref
- Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. 2017. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision. 1717--1725.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarCross Ref
- Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132--7141.Google Scholar
- Quan Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics letters , Vol. 44, 13 (2008), 800--801.Google Scholar
- Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, and Yao Wang. 2018. FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors. arXiv preprint arXiv:1803.07487 (2018).Google Scholar
- Li-Wei Kang, Chia-Wen Lin, and Yu-Hsiang Fu. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing , Vol. 21, 4 (2012), 1742--1755.Google ScholarDigital Library
- Jin-Hwan Kim, Jae-Young Sim, and Chang-Su Kim. 2015. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing , Vol. 24, 9 (2015), 2658--2670.Google ScholarDigital Library
- Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et almbox. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 105--114.Google ScholarCross Ref
- Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, and Deyu Meng. 2018b. Video Rain Streak Removal By Multiscale Convolutional Sparse Coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6644--6653.Google ScholarCross Ref
- Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. 2017. Single Image Deraining using Scale-Aware Multi-Stage Recurrent Network. arXiv preprint arXiv:1712.06830 (2017).Google Scholar
- Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018a. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. In The European Conference on Computer Vision (ECCV). 262--277.Google Scholar
- Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. 2016. Rain streak removal using layer priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2736--2744.Google ScholarCross Ref
- Jiaying Liu, Wenhan Yang, Shuai Yang, and Zongming Guo. 2018. Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3233--3242.Google ScholarCross Ref
- Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision . 3397--3405.Google ScholarDigital Library
- Danilo P Mandic and Jonathon Chambers. 2001. Recurrent neural networks for prediction: learning algorithms, architectures and stability .John Wiley & Sons, Inc.Google Scholar
- Jinshan Pan, Sifei Liu, Deqing Sun, Jiawei Zhang, Yang Liu, Jimmy Ren, Zechao Li, Jinhui Tang, Huchuan Lu, Yu-Wing Tai, et almbox. 2018. Learning Dual Convolutional Neural Networks for Low-Level Vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 3070--3079.Google ScholarCross Ref
- Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2536--2544.Google ScholarCross Ref
- Weihong Ren, Jiandong Tian, Zhi Han, Antoni Chan, and Yandong Tang. 2017. Video desnowing and deraining based on matrix decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4210--4219.Google ScholarCross Ref
- Varun Santhaseelan and Vijayan K Asari. 2015. Utilizing local phase information to remove rain from video. International Journal of Computer Vision , Vol. 112, 1 (2015), 71--89.Google ScholarDigital Library
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , Vol. 13, 4 (2004), 600--612.Google ScholarDigital Library
- Wei Wei, Lixuan Yi, Qi Xie, Qian Zhao, Deyu Meng, and Zongben Xu. 2017. Should we encode rain streaks in video as deterministic or stochastic. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2516--2525.Google ScholarCross Ref
- Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2016. Joint rain detection and removal via iterative region dependent multi-task learning. CoRR, abs/1609.07769 , Vol. 2 (2016), 3.Google Scholar
- Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1357--1366.Google ScholarCross Ref
- Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014).Google Scholar
- He Zhang and Vishal M. Patel. 2018. Density-Aware Single Image De-Raining Using a Multi-Stream Dense Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 695--704.Google Scholar
- He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2017. Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957 (2017).Google Scholar
- Lei Zhu, Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. 2017. Joint bilayer optimization for single-image rain streak removal. In Proceedings of the IEEE International Conference on Computer Vision. 2526--2534.Google ScholarCross Ref
Index Terms
- Single Image Deraining via Recurrent Hierarchy Enhancement Network
Recommendations
Recurrent wavelet structure-preserving residual network for single image deraining
Highlights- A novel Recurrent Wavelet Structure-preserving Residual Network (RWSRNet) is proposed to pay attention to both the low-frequency and high-frequency parts of ...
AbstractThe combination of deep learning and image prior has been widely used in single image deraining since 2017. Recent studies have demonstrated an excellent deraining effect on the high-frequency part of rain images, but less attention ...
Single image deraining via a recurrent multi-attention enhancement network
AbstractSingle image deraining has attracted considerable attention because rain streaks can severely degrade image quality and affect performance in computer vision tasks. While recurrent networks have achieved promising results, they still have ...
Highlights- Propose multi-attention enhancement deraining network to recover rainy images.
- Integrate multi-attentions to enhance effective features.
- The proposed network achieves superior performance on public rain datasets.
Recurrent multi-level residual and global attention network for single image deraining
AbstractDeraining is an essential preprocess for many computer vision tasks, e.g., vision-based autonomous driving. The existing methods usually depend on the prior information or specified network structures and correspondingly suffer from high ...
Comments