DTDN: Dual-task De-raining Network

ABSTRACT

Removing rain streaks from rainy images is necessary for many tasks in computer vision, such as object detection and recognition. It needs to address two mutually exclusive objectives: removing rain streaks and reserving realistic details. Balancing them is critical for de-raining methods. We propose an end-to-end network, called dual-task de-raining network (DTDN), consisting of two sub-networks: generative adversarial network (GAN) and convolutional neural network (CNN), to remove rain streaks via coordinating the two mutually exclusive objectives self-adaptively. DTDN-GAN is mainly used to remove structural rain streaks, and DTDN-CNN is designed to recover details in original images. We also design a training algorithm to train these two sub-networks of DTDN alternatively, which share same weights but use different training sets. We further enrich two existing datasets to approximate the distribution of real rain streaks. Experimental results show that our method outperforms several recent state-of-the-art methods, based on both benchmark testing datasets and real rainy images.

References

1. Peter C Barnum, Srinivasa Narasimhan, and Takeo Kanade. 2010. Analysis of rain and snow in frequency space. International journal of computer vision , Vol. 86, 2--3 (2010), 256.
2. Jérémie Bossu, Nicolas Hautière, and Jean-Philippe Tarel. 2011. Rain or snow detection in image sequences through use of a histogram of orientation of streaks. International journal of computer vision , Vol. 93, 3 (2011), 348--367.
3. Yi-Lei Chen and Chiou-Ting Hsu. 2013. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. IEEE International Conference on Computer Vision (2013), 1968--1975.
4. Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. 2017a. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing , Vol. 26, 6 (2017), 2944--2956.
5. Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017b. Removing rain from single images via a deep detail network. IEEE Conference on Computer Vision and Pattern Recognition (2017), 1715--1723.
6. Kshitiz Garg and Shree K Nayar. 2004. Detection and removal of rain from videos. IEEE Conference on Computer Vision and Pattern Recognition , Vol. 1 (2004), I--I.
7. Kshitiz Garg and Shree K Nayar. 2005. When does a camera see rain? IEEE International Conference on Computer Vision , Vol. 2 (2005), 1067--1074.
8. Kshitiz Garg and Shree K Nayar. 2007. Vision and rain. International Journal of Computer Vision , Vol. 75, 1 (2007), 3--27.
9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems (2014), 2672--2680.
10. Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided image filtering. IEEE transactions on pattern analysis and machine intelligence , Vol. 35, 6 (2013), 1397--1409.
11. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694--711.
12. Li-Wei Kang, Chia-Wen Lin, and Yu-Hsiang Fu. 2012. Automatic single-image-based rain streaks removal via image decomposition. IEEE Transactions on Image Processing , Vol. 21, 4 (2012), 1742.
13. Jin Hwan Kim, Chul Lee, Jae Young Sim, and Chang Su Kim. 2014. Single-image deraining using an adaptive nonlocal means filter. IEEE International Conference on Image Processing (2014), 914--917.
14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
15. Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018. Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining. arXiv preprint arXiv:1807.05698 (2018).
16. Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, and Michael S. Brown. 2016. Rain Streak Removal Using Layer Priors. IEEE Conference on Computer Vision and Pattern Recognition (2016), 2736--2744.
17. Yu Luo, Yong Xu, and Hui Ji. 2015. Removing Rain from a Single Image via Discriminative Sparse Coding. IEEE International Conference on Computer Vision (2015), 3397--3405.
18. Varun Santhaseelan and Vijayan K Asari. 2015. Utilizing local phase information to remove rain from video. International Journal of Computer Vision , Vol. 112, 1 (2015), 71--89.
19. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
20. Abhishek Kumar Tripathi and Sudipta Mukhopadhyay. 2014. Removal of rain from videos: a review. Signal, Image and Video Processing , Vol. 8, 8 (2014), 1421--1430.
21. Zhou Wang and Alan C Bovik. 2002. A universal image quality index. IEEE signal processing letters , Vol. 9, 3 (2002), 81--84.
22. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , Vol. 13, 4 (2004), 600--612.
23. Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. IEEE Conference on Computer Vision and Pattern Recognition (2017), 1357--1366.
24. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. European Conference on Computer Vision (2014), 818--833.
25. He Zhang and Vishal M Patel. 2018. Density-aware single image de-raining using a multi-stream dense network. arXiv preprint arXiv:1802.07412 (2018).
26. He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2017. Image de-raining using a conditional generative adversarial network. arXiv preprint arXiv:1701.05957 (2017).
27. Xiaopeng Zhang, Hao Li, Yingyi Qi, Wee Kheng Leow, and Teck Khim Ng. 2006. Rain removal in video by combining temporal and chromatic properties. IEEE International Conference on Multimedia and Expo (2006), 461--464.
28. Lei Zhu, Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. 2017. Joint bilayer optimization for single-image rain streak removal. IEEE International Conference on Computer Vision (2017), 2526--2534.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Image processing

  • Machine learning

    • Machine learning approaches

      • Neural networks