Automatic Radar-Camera Dataset Generation for Sensor-Fusion Applications

Arindam Sengupta, Atsushi Yoshizawa, Siyang Cao

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


Withheterogeneous sensors offering complementary advantages in perception, there has been a significant growth in sensor-fusion based research and development in object perception and tracking using classical or deep neural networks based approaches. However, supervised learning requires massive labeled data-sets, that require expensive manual labor to generate. This paper presents a novel approach that leverages YOLOv3 based highly accurate object detection from camera to automatically label point cloud data obtained from a co-calibrated radar sensor to generate labeled radar-image and radar-only data-sets to aid learning algorithms for different applications. To achieve this we first co-calibrate the vision and radar sensors and obtain a radar-to-camera transformation matrix. The collected radar returns are segregated by different targets using a density based clustering scheme and the cluster centroids are projected onto the camera image using the transformation matrix. The Hungarian Algorithm is then used to associate the radar cluster centroids with the YOLOv3 generated bounding box centroids, and are labeled with the predicted class. The proposed approach is efficient, easy to implement and aims to encourage rapid development of multi-sensor data-sets, which are extremely limited currently, compared to the optical counterparts. The calibration process, software pipeline and the data-set generation is described in detail. Furthermore preliminary results from two sample applications for object detection using the data-sets are also presented.

Original languageEnglish (US)
Pages (from-to)2875-2882
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number2
StatePublished - Apr 1 2022


  • Calibration and identification
  • data sets for robotic vision
  • neural and fuzzy control
  • object detection
  • segmentation and categorization
  • sensor fusion

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'Automatic Radar-Camera Dataset Generation for Sensor-Fusion Applications'. Together they form a unique fingerprint.

Cite this