Trying to See Low Exposure Images using CNN

Trying to See Low Exposure Images using CNN

Kalpesh Shinde

Information Technology Vidyavardhinis College of Engineering and Technology Vasai, Palghar

Raviraj Hirve

Information Technology Vidyavardhinis College of Engineering and Technology Vasai, Palghar

Mayur Kale

Information Technology Vidyavardhinis College of Engineering and Technology Vasai, Palghar

Abstract Due to low photon count ratio imaging is very difficult in low light. In long exposure images there is a blur and in short exposure image there is noise present. There are various types of techniques given like image enhancement, denoising, deblurring but they are limited in difficult conditions. Considering the new developments in learning- based pipeline in low light image processing we have selected a dataset of raw short exposure low light images along the side which there are reference of long exposure images. Based on fully convolutional network with end-to-end training and with the dataset selected a pipeline for low light images processing is developed. This pipeline network processes on raw sensor data and improves the old image processing pipeline which does not give good results on the data presented. Our result is genuine on new dataset presented.

Keywords: CNN, Short exposure, Contrast Enhancement, Resolution.

1. INTRODUCTION

   Short exposure images are the images of moving objects which often introduces noise, blur, and the image therefore is not clear. If the brightness of the images are to be increased it introduces noise. In case of fast imaging in low light in photography is a challenge, various techniques have been introduced by various researchers like enhancement of low light images, deblurring, denoising. But in these techniques there are certain conditions like image need to be captured in dim environments with noise level moderate. But we are interested in short exposure and extreme low light imaging.

2. RELATED WORK

   Many researches have been done on different methods for Long exposure Image to Short Exposure Images, image denoising and enhancement of low light Imaging, here some prominent works done in the field arediscussed.

   1. Low Light Image Enhancement

      In [1] for Low Light Image enhancement they have used a Retinex Model in the model the observed low light image is represented as pixel wise multiplication between enhanced image recovered from observed low light image and illumination map. The method works as follows the model first estimates the illumination map and based on element-wise division between observed low light image and illumination map. To estimate each element in illumination map Max-RGB technique is used. After max-RGB estimation the image is processed

      for texture removal then edge recovery through guided filter then passed through Simplified Retinex Model to acquire enhanced image. Various techniques have been applied to improve the contrast of low light images. The most common is histogram equalization it balances the histogram of entire image. Another widely used techniques is gamma correction where the brightness of dark regions is increased by compressing bright pixels. The advanced methods like inverse dark channel prior, the wavelet transform the retinex model and illumination map estimation. But there is a drawback that these methods assume that the images are already containing good representation of image content. The method for image denoising is not explicit or different and apply the general denoising technique in the postprocess. Therefore we consider extreme low light imaging in which color distortion and noise is present which is very difficult to operate in existing enhancement pipeline

   2. Image Dehazing

      Haze is natural phenomenon where the visibility of the sky is altered or disturbed by dust, smoke and other particles. Hazy images causes different visibility issues for tourists, traffic users and in hilly regions where the haze and fog are very common. In [6] method for single image dehazing using convolutional neural network is used. The images used are outdoor on which particular filters are applied to find haze in image. The images which are hazy contain a small value in only one color alpha channel from red, green, blue RGB channel. The pixel intensity is mainly represented by air light depth map. By estimating these low value points of haze transmission map it is useful to obtain high quality dehazed image. To achieve a high quality dehazed image an end to end encoder decoder training model is used.equalization function is known then the original histogram can be recovered. The calculation in this method is not computationally difficult, but there is an disadvantage while decreasing the usable signal it may increase background noise. Histogram equalization is suitable for scientific images like the x-ray, satellite, thermal images but it produces unrealistic effects in images. It can also introduce undesirable effects if applied to images with low color depth.

   3. Contrast Enhancement

   Contrast enhancement (CE) refers to the image enhancement on contrast by adjusting the dynamic range of pixel intensity distribution [1]. CE plays an important role in the improvement of visual quality for computer vision, pattern recognition and digital image processing. In real applications, we usually encounter digital images with poor contrast or abnormal brightness, which may result from different factors, such as the inexperience of taking photographs and the inherent deficiency of imaging devices. The capturing scenes with low or high illuminance intensity may also lead to reduced contrast quality. Despite of visual quality degradation, low contrast might hinder the further applications of a digital image, including image analysis and understanding, object recognition and digital printing, etc. As such, it is essential to enhance the contrast of such distorted images before further applications.

3. METHODOLOGY

   A. Low Light Image Enhancement Using CNN

   We used an end to end trainable network so that the images captured in low light will be converted to proper exposed images.

   We have selected See-in-the-dark (SID) dataset which consist of raw Short exposure images paired with Long exposure images.

   1. Bayer Array

      — Color filter array (CFA) where RGB color filters are arranged on square grid of photosensors.

      — RGB checkerboard pattern are generated

      — 50% green 25% blue 25% red

   2. Network Architecture

      The network architecture:

      — Images have to be static. i.e. not in motion

      — Pipeline is still not efficient for real time video capture and processing.

      1. Contrast Enhancement:

         Contrast is an important element in evaluation of image quality. Contrast is nothing but the difference between luminance of two adjacent surfaces. In simple words Contrast is a property which helps an object stand out from other objects and its background. To determine the contrast find the difference in color and brightness of the required object with another objects. Contrast Enhancement of images whose brightness is distorted which have global intensity relatively high or low is applied.

      2. Increasing Resolution:

         We have used Deep CNN with Residual Net and Skip Connection where Skip Connection in an architecture of neural network skips some layer and the output of the previous layer as passed onto input of the layer after skipped. The feature extraction

         network is nothing but combination of Deep CNN and Skip Connections. For image reconsruction network parallelized 1*1 CNN which is called Network in Network.

      3. Dataset:

   We have used a Dataset, See-in-the-dark (SID) consisting of raw SEIs paired with reference LEIs. An end to end trainable network is used to enable images captured in low light to be converted ti properly exposed images.

   Burst of Images collected for evaluating denoising techniques. Luminance (lux)

   Outdoor scenes 0.2< l< 5 Indoor scenes 0.03 < l < 0.3 Exposure 1/10 < E < 1/10

   Ground Truth 10 < E < 30 seconds Sony Images 4240 * 2832 pixels

4. RESULTS

   1. Low Light Image Enhancement Using CNN

      Above is the comparison between the image produced by

      Convolutional Neural Network. The qualitative analysis is of the image produced by CNN compared to ground truth is given is table below

      We have tried to denoise the the given output that got by CNN to remove the gaussian noise from the output to improve the quality of the image and we also succeeded in doing so clear from the below tabl which cpmares PSNR and MSSSIM compared to the long exposure ground ground truth from given dataset.

      PSNR:

      The term peak signal-to-noise ratio (PSNR) is an expression for the ratio between the maximum possible value (power) of a signal and the power of distorting noise that affects the quality of its representation

      MSSSIM:

      SSIM i.e. Structural Similarity Index Measure is a method which predicts the correct quality of television pictures and also predicts other images and videos. It also measures the similarity between two images.

      	PSNR	MSSSIM
      CNN Denoised	18.2029 17.0549	0.8143 0.7493

   2. Contrast, Dehazing and Resolution Enhancement

   CNN Final

   Image

   uqi(higher is better)

   psnr(higher is better)

   psnrb(higher is better)

   rmse(lower is better)

   mse(lower is better)

   msssim(lower is better)

   rase(lower is better)

   sam(lower is better)

   scc(lower is better)

   ssim(higher is better)

   0.3264
   14.689
   14.924
   46.995
   2208.5
   0.7279
   0.3378
   4761.4
   0.1954
   (0.361, 0.733)

   0.6336
   17.3624
   18.2029
   34.5479
   1193.55
   0.8143
   0.1447
   3417.10
   0.2801
   (0.6193, 0.7893)

   uqi(higher is better)

   psnr(higher is better)

   psnrb(higher is better)

   rmse(lower is better)

   mse(lower is better)

   msssim(lower is better)

   rase(lower is better)

   sam(lower is better)

   scc(lower is better)

   ssim(higher is better)

   0.3264
   14.689
   14.924
   46.995
   2208.5
   0.7279
   0.3378
   4761.4
   0.1954
   (0.361, 0.733)

   0.6336
   17.3624
   18.2029
   34.5479
   1193.55
   0.8143
   0.1447
   3417.10
   0.2801
   (0.6193, 0.7893)

   As discussed above the denoised image is dehazed removing any hazing effect the given image might have.

   After adjusting the contrast and resolution the qualitative analysis of the CNN and post processed image is mntioned in the table above.

   The figure mentioned below shows the output of different methods to incease the contras

5. CONCLUSION

   A deep learning network to solve the problem of low light imaging achieve results better than traditional and recent denoising pipelines. We tried an array of various components as part of their network but U-Net outperforms every single one.

6. REFERENCES

1. Zhuang Feng, Shijie Hoo Low-Light Image enhancement by refining illumination map with self-guided filtering.

2. H. C. Burger, C. J. Schuler, and S. Harmeling. Image denoising: Can plain neural networks compete with BM3D? In CVPR, 2012.

3. Junyuan Xie, Linli Xu,Enhong Chen. Image Denoising and inpainting with Deep Neural Networks.

4. Chen Chen, Qifeng Chen, Minh N. Do and Vladlen Koltun. Learning to See in the dark. CVPR 2018.

5. J. Anaya and A. Barbu. RENOIR – A dataset for real lowlight image noise reduction. arXiv:1409.8230,2014.

6. Huzaifa Rashid Single Image Dehazing Using CNN. In ELSEVIER