Improving the Efficiency of CMOS Image Sensors through In-Sensor Selective Attention

Tianyi Zhang, Kishore Kasichainula, Dong Woo Jee, Injune Yeo, Yaoxin Zhuo, Baoxin Li, Jae Sun Seo, Yu Cao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Inspired by the selective attention mechanism in human vision, we propose to introduce a saliency-based processing step in the CMOS image sensor, to continuously select pixels corresponding to salient objects and feedback such information to the sensor, instead of blindly passing all pixels to the sensor output. To minimize the overhead of saliency detection in this feedback loop, we propose two techniques: (1) saliency detection with low-precision, down-sampled grayscale images, and (2) Optimization of the loss function and model structure. Finally, we pad the minimum number of pixels around the selected pixels to maintain the accuracy of object detection (OD). Our method is experimented with two types of OD algorithms on three representative datasets. At the similar OD accuracy with the full image, our proposed selective feedback method successfully achieves 70.5% reduction in the volume of output pixels for BDD100K, which translates to 4.3× and 3.4× reduction in power consumption and latency, respectively.

Original languageEnglish (US)
Title of host publicationISCAS 2023 - 56th IEEE International Symposium on Circuits and Systems, Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665451093
DOIs
StatePublished - 2023
Event56th IEEE International Symposium on Circuits and Systems, ISCAS 2023 - Monterey, United States
Duration: May 21 2023May 25 2023

Publication series

NameProceedings - IEEE International Symposium on Circuits and Systems
Volume2023-May

Conference

Conference56th IEEE International Symposium on Circuits and Systems, ISCAS 2023
Country/TerritoryUnited States
CityMonterey
Period5/21/235/25/23

Keywords

  • image sensor
  • latency
  • object detection
  • power consumption
  • saliency detection
  • selective attention

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Improving the Efficiency of CMOS Image Sensors through In-Sensor Selective Attention'. Together they form a unique fingerprint.

Cite this