Karasawa Takumi
ABSTRACT
Recently, researchers have actively conducted studies on mobile robot technologies that involve autonomous driving. To implement an automatic mobile robot (e.g., an automated driving vehicle) in traffic, robustly detecting various types of objects such as cars, people, and bicycles in various conditions such as daytime and nighttime is necessary. In this paper, we propose the use of multispectral images as input information for object detection in traffic. Multispectral images are composed of RGB images, near-infrared images, middle-infrared images, and far-infrared images and have multilateral information as a whole. For example, some objects that cannot be visually recognized in the RGB image can be detected in the far-infrared image. To train our multispectral object detection system, we need a multispectral dataset for object detection in traffic. Since such a dataset does not currently exist, in this study we generated our own multispectral dataset. In addition, we propose a multispectral ensemble detection pipeline to fully use the features of multispectral images. The pipeline is divided into two parts: the single-spectral detection model and the ensemble part. We conducted two experiments in this work. In the first experiment, we evaluate our single-spectral object detection model. Our results show that each component in the multispectral image was individually useful for the task of object detection when applied to different types of objects. In the second experiment, we evaluate the entire multispectral object detection system and show that the mean average precision (mAP) of multispectral object detection is 13% higher than that of RGB-only object detection.
- Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip Torr. 2014. BING: Binarized normed gradients for objectness estimation at 300fps Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Google Scholar
Digital Library
- Navneet Dalal and Bill Triggs. 2005. Histograms of oriented gradients for human detection Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Google ScholarDigital Library
- Piotr Dollar, Zhuowen Tu, Pietro Perona, and Serge Belongie. 2009. Integral Channel Features. In Proceedings of the British Machine Vision Conference.Google ScholarCross Ref
- Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision Vol. 111, 1 (2015), 98--136. Google ScholarDigital Library
- Pedro Felzenszwalb, David McAllester, and Deva Ramanan. 2008. A discriminatively trained, multiscale, deformable part model Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. 2010. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, Vol. 32, 9 (2010), 1627--1645. Google ScholarDigital Library
- Ross Girshick. 2015. Fast r-cnn Proceedings of the IEEE International Conference on Computer Vision. Google ScholarDigital Library
- Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Google ScholarDigital Library
- Alejandro González, Zhijie Fang, Yainuvis Socarras, Joan Serrat, David Vázquez, Jiaolong Xu, and Antonio M López. 2016. Pedestrian detection at day/night time with visible and FIR cameras: A comparison. Sensors, Vol. 16, 6 (2016), 820.Google ScholarCross Ref
- Alejandro González, Gabriel Villalonga, Jiaolong Xu, David Vázquez, Jaume Amores, and Antonio M López. 2015. Multiview random forest of local experts combining rgb and lidar data for pedestrian detection IEEE Intelligent Vehicles Symposium.Google Scholar
- P Govardhan and Umesh Chandra Pati. 2014. NIR image based pedestrian detection in night vision with cascade classification and validation. In Proceedings of International Conference on Advanced Communication Control and Computing Technologies.Google ScholarCross Ref
- Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon. 2015. Multispectral pedestrian detection: Benchmark dataset and baseline Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Eun Som Jeon, Jong-Suk Choi, Ji Hoon Lee, Kwang Yong Shin, Yeong Gon Kim, Toan Thanh Le, and Kang Ryoung Park. 2015. Human detection based on the generation of a background image by using a far-infrared light camera. Sensors, Vol. 15, 3 (2015), 6763--6788.Google ScholarCross Ref
- Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016. SSD: Single shot multibox detector. In Proceedings of European Conference on Computer Vision.Google ScholarCross Ref
- David G Lowe. 1999. Object recognition from local scale-invariant features Proceedings of the IEEE international conference on Computer vision. Google ScholarDigital Library
- Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Joseph Redmon and Ali Farhadi. 2017. YOLO9000: Better, Faster, Stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks Proceedings of Advances in neural information processing systems. Google ScholarDigital Library
- Yainuvis Socarrás, Sebastian Ramos, David Vázquez, Antonio M López, and Theo Gevers. 2011. Adapting pedestrian detection from synthetic to far infrared images Proceedings of the IEEE International Conference on Computer Vision, Workshop on Visual Domain Adaptation and Dataset Bias.Google Scholar
- Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders. 2013. Selective search for object recognition. International journal of computer vision Vol. 104, 2 (2013), 154--171. Google ScholarDigital Library
- Maurice Velte. 2015. Semantic image segmentation combining visible and near-infrared channels with depth information. Ph.D. Dissertation. bibinfoschoolBonn-Rhein-Sieg University of Applied Sciences.Google Scholar
- Jörg Wagner, Volker Fischer, Michael Herman, and Sven Behnke. 2016. Multispectral pedestrian detection using deep fusion convolutional neural networks Proceedings of European Sympousium on Artificial Neural Networks.Google Scholar
- C Lawrence Zitnick and Piotr Dollár. 2014. Edge boxes: Locating object proposals from edges. Proceedings of European Conference on Computer Vision.Google Scholar
Index Terms
- Multispectral Object Detection for Autonomous Vehicles
Recommendations
Design Guidelines on Deep Learning–based Pedestrian Detection Methods for Supporting Autonomous Vehicles
Invited TutorialIntelligent transportation systems (ITS) enable transportation participants to communicate with each other by sending and receiving messages, so that they can be aware of their surroundings and facilitate efficient transportation through better decision ...
Motion planning of autonomous vehicles in a non-autonomous vehicle environment without speed lanes
Planning is one of the key problems for autonomous vehicles operating in road scenarios. Present planning algorithms operate with the assumption that traffic is organised in predefined speed lanes, which makes it impossible to allow autonomous vehicles ...
Nighttime vehicle light detection on a moving vehicle using image segmentation and analysis techniques
This study proposes a vehicle detection system for identifying the vehicles by locating their headlights and rear-lights in the nighttime road environment. The proposed system comprises of two stages for detecting the vehicles in front of the camera-...
Comments