A Dynamic 2-D FFT Video Focus Assessment Algorithm for Real Time Video Stabilized Goggles using Raspberry Pi

A Dynamic 2-D FFT Video Focus Assessment Algorithm for Real Time Video Stabilized Goggles using Raspberry Pi

S. Nagajyothi

Department of electronics and communication Madanapalle institute of technology and science Madanapalle, A.P, India

Dr. S. A. K. Jilani

Department of electronics and communication Madanapalle institute of technology and science Madanapalle, A.P, India

Mr. S. Javeed Hussain Department of electronics and communication Madanapalle institute of technology and science

Madanapalle, A.P, India

Abstract—This paper describes the design and implementation of video stabilization using raspberry pi and USB camera. The video stabilization is an important video enhancement technique, that describes how to remove the effect of camera motion from a video stream using dynamic 2-D FFT video focus assessment algorithm (DVFA) and Mat lab simulink tool box for obtaining stabilized video. The stabilized video will be displayed in goggles. Hardware components such as raspberry pi (model b), USB camera and 52 inch 4:3 virtual Mobile theatre video glasses display are used. The proposed method works in real time application using mat lab and raspberry pi. The results are tested while reading news paper in moving vehicle.

Keywords—- Raspberry pi, display, video stabilization, usb camera.

1. INTRODUCTION

   There are many people throughout the world .who are having hobby of reading News papers and watching videos on a handheld device, such as a mobile phones while they are travelling from one place to another place. When a hand held devices undergoes vibrations or unwanted motions while vehicle is moving on a non-smooth road or terrain it records a jumpy video created by small, random movements. In such conditions it is very difficult to read news paper or to watch a video. The poor quality of such recording may cause the video not to be useful. In order to develop applications on different platforms simple and efficient algorithms must be developed. It is nothing but video stabilization.

   Video stabilization methods can be classified into three different categories. They are

   1. Electronic Image Stabilization (EIS)

   2. Optical Image Stabilization (OIS)

   3. Digital Image Stabilization (DIS).

   A method becoming more and more common in the end is combination of all three image stabilization methods. If the effect of the stabilization is increased it will give best results.

   1. Electronic Image Stabilization (EIS)

      In electronic image stabilization the data from accelerometers, gyros or other motion registration devices gives a CPU enough information to accurately stabilize the image. Unfortunately this includes blurring in the image as it is moved after being digitally stored. It is also a lens dependent method, where the sensor is tuned to fit the built-in lens. Adding a lens, in the form of an objective lens, to the camera may cause over or under compensation, which makes the shakiness worse.

   2. Optical Image Stabilization

      Optical image stabilization is very similar to electronic image stabilization, but instead of compensating for the motion on the stored image, it moves the lens directly. This method gives the best result, but it requires lenses and a measure of physical space to implement. The use of optical image stabilization will not blur the image, as would be the case in EIS and DIS, because the stabilization is performed on the lens directly.

   3. Digital Image Stabilization

   Digital image stabilization provides the cheapest solution of all three since it does not require any additional hardware. And is a simple feature to add in regular moderately priced products. It is also easy to implement in existing products with no physical alterations. This characteristic makes the method a good alternative for use in mobile equipment. A number of different methods of DIS exist and they all use the information extracted from the image stream to estimate the motion. Generally, DIS gives the least accurate result of all image stabilization methods. OIS and EIS must be applied during the recording phase; DIS can also be implemented in a post- processing scenario. The ranges of related products are wide, and the use of them is in all video applications, not only in mobile equipment. Some are for editing and some are designed for real time applications.

   In this paper we had developed a real time based efficient dynamic 2-D FFT video focus assessment (DVFA) algorithm to stabilize the video. In order to display the obtained stabilized video we are using goggles. To implement (DVFA) algorithm and stabilization process, Here we are using raspberry pi and simulink tools that run on mat lab software on a computer.

2. LITERATURE SURVEY

In the last two decades, the researchers have developed many video stabilization algorithms. For any video stabilization technique the inter frame motion may be modeled using 2D or 3D motion model. Video stabilization is a field where lot of research has been happening based on various technologies. These research attempts can be broadly sub divided into

1. Hardware based approaches.

2. Software based approaches.

   1. Video Stabilization Using Point Feature Matching technique

      Abdullah, L.M.; Tahir, N.M.; Samad, M[1]. proposed an algorithm to stabilize jumpy videos directly without the need to estimate camera motion. A stable output video will be attained without the effect of jumpy that caused by shaking of the handheld camera during video recording. Firstly, significant points from each frame from the input video is identified and processed followed by optimizing and stabilize the video. Optimization includes the quality of the video stabilization and less affected area after the process of stabilization. The output of using such method showed good result in terms of stabilization and unnecessary distortion from the output videos recorded in different circumstances. Initial results showed that the proposed technique is suitable to be used and generate a stabilized video.

   2. Real-time Video Stabilization For Moving Platforms

      Eddy Vermeulen [2], describes a method and practical implementation to stabilize video sequences obtained from a camera mounted on a moving platform. For implementation of this method we use a twofold camera motion. The first one is from the motion path of the vehicle itself which has to be followed as closely as possible. The second one is due to mechanical vibrations of the mounted sensor and environmental influences (e.g. wind). The second motion disturbance has to be filtered out as much as possible from the camera image. The first step of this method need an electronic video stabilization, specifically related to the defense and surveillance markets. The second step describes a stabilization process and produces stabilized video for moving platforms. The third step handles the physical implementation. Finally, some future impacts of technology and market trends are used and generate a stabilized video.

   3. Implementation and Validation of Video Stabilization using Simulink

      L.Garlin Delphina and VPS Naidu[3],Proposed Implementation and Validation of Video Stabilization using Simulink in which Aerial Vehicles carry visual and infrared cameras to capture outside scenes on real-time basis. The usefulness of these images or videos that these flying machines relay to the operators in military applications is

      directly related to how much of information they can take out from the frames being captured on real-time basis. Generally the captured video looks jumpy and not attractive due to vibrations of the platform. In order to overcome this problem of oscillations and low frequency vibrations in the recorded images, implementation of a Digital Image Stabilization (DIS) algorithm is required. It improves the visual quality of recorded images or video. Furthermore, the video processing algorithms are needed to perform video stabilization on real time basis. Hence, we use GCBP matching using TSS Algorithm and a Simulink tool box is used. It reduces the computational overhead by realizing the stabilization technique on real-time basis using binary Boolean functions.

   4. Video Stabilization Using SIFT Features, Fuzzy Clustering, and Kalman Filtering

To achieve the goal of a stabilized video. Kevin L. Veon [4], developed a video stabilization algorithm involving scale- invariant feature transform (SIFT) features, fuzzy set theory, and Kalman filtering. We use SIFT features to match significant points between frames. The significant points in the image which are clear and distinct enough will be accurately matched between frames with little or clear does not match the frames will be removed. SIFT is widely regarded in the computer vision community as one of the most accurate and robust types of features. SIFT features provide information about scale, location, and point of reference. The most important information SIFT provides is the point of reference of the features, which is un available in other feature types. By looking at how these features have moved, they can be used to measure the motion of the camera, though both local and global motion is provided with SIFT features. We use Fuzzy set theory to separate local and global motion. Fuzzy set theory simplifies some design decisions with regards to clustering and provides information about point of reference and separates when uncertainty exists .We use Kalman filtering to estimate the desired motion of the camera in video stabilization process.

1. IMPLEMENTATION

   The proposed system is meant for video stabilization by using dynamic 2-D FFT video focus assessment algorithm (DVFA), raspberry pi model b, and USB camera and 52 inch 4:3 virtual display theatre glasses. The main objective of this proposed method is to display the stabilized video in goggles.

   First we will capture the video using USB camera and by using dynamic video focus assessment (DVFA) algorithm we can stabilize the video and interfacing is done between USB camera and Raspberry pi to obtain a stabilized video. Then the stabilized video will be displayed in 52 inch 4:3 virtual display theatre glasses. In order implement the proposed method, hardware and software are required and specified below.

   1. Raspberry pi

   2. USB web camera

   3. 52 inch 4:3 virtual display theatre glasses

   4. Mat lab software

   5. Simulink tools

      Figure 1 Block Diagram of Proposed Method

      1. Hardware implementation

         In order to design the proposed system we need four hardware devices such as raspberry pi model b, power supply, USB camera and 52 inch 4:3 virtual display theatre glasses.

         3.3 USB camera

         Figure 3 power usage of raspberry pi

         1. Raspberry pi model b

            Although raspberry pi is as small as the size of the credit card, it works as if a normal computer. It is based on a Broadcom BCM2835 System on chip which includes ARM1176JZFyS700 MHz processor as its core and it was initially with 256 MB of RAM, and later upgraded to 512MB. It is designed on a single board with all the essential peripherals required for running an operating system. Raspberry pi board is equipped with a wide variety of connectors, 2 USB ports, 26 GPIO pins ,UART, Serial peripheral interface bus, Power supply Ethernet port ,it uses 3.5mm jack for audio out ,HDMI ,Composite Video. As raspberry pi comes without a display unit, in order to display any processed video as output we can use goggles, NTSC/TFT display, and HDTV standard TV screen.

            Figure 2 Raspberry pi model b

         2. Power supply

         Raspberry pi uses a Micro USB connection to power itself. For this a standard modem phone charger with a USB connector is needed and it is required at least of 700mA at 5volts.

         USB camera is the main component in this video stabilization process. In this project tech-com SSD 352 USB web camera is used. Some of the features and specifications of the USB camera are.

         Features:

         1. It supports up to 40 mega pixel interpolated the frame rate up to 30fps.

         2. True Plug & play easy USB Interface.

         3. High-quality CMOS sensor.

         4. Clear and sharp still-picture & motion video capturing.

         5. Ideally designed to work well with both Laptops & Desktop computers.

         6. Adjustable lens for accurate image shooting.

         7. Includes variety of image control.

         Figure 4 Tech-com SSD 352 USB camera

         1. 52 inch 4:3 virtual display theatre glasses

            As raspberry pi comes without a display unit, In order to display the stabilized video we need a display unit. It is nothing but 52 inch 4:3 virtual display theatre glasses Comes with 4GB internal memory can expanded upto1-32GB Micro SD card. Theatre video glasses are the portable virtual gadget that is perfect to watch 52 inch cinema display at anytime. Some of the features and specifications of goggle are.

            1. It is capable of handling a wide variety of media files.

            2. It can play everything from movies to photos to eBooks to music.

            3. It has 4GB of internal memory for storing all your favorite media files, as well as a micro SD / TF card slot that can accept up to a 32GB.

            4. It is lightweight glasses and comfortable enough to wear for extended periods of time and the battery will last more than long enough for a movie.

         Figure5 52 inch 4:3 virtual display theatre glasses

      2. software implementation

         At First, we will develop program in Mat lab, Simulink which supports the Raspberry Pi packages. Where Simulink provides an different set of block libraries in order to design not only image, and video processing systems but also communication and control systems. Simulink let users design, simulate, implement, and test projects. The proposed Dynamic 2-D FFT video focus assessment (DVFA) algorithm is simulated using Mat lab software. The Proposed method implementation using simulink blocks are represented below,

         1. First step includes capturing the video from USB camera and identifying the focus states of an video for the special calibration by using the video focus assessment process.

            Figure 6 video focus assessment models

         2. Once ROI is defined in the video, by performing FFT operation we can calculate how much high frequency components there are on the video image. If there is a more high frequency data in the video then we will obtain a clear image which means in focus state. On the other hand, if there is a lot of low frequency content in the video then we will obtain a very blurry image. By using Relative Focus window we can plot the difference between the ratios they are nothing but high spatial frequency content ratio and low spatial frequency content ratio within the ROI. This ratio is an indication of the relative focus adjustment of the video camera. When this ratio is high, the video is in focus state. When this ratio is low, the video is out of focus state. Although it is possible to judge the relative focus of a camera with respect to the video using 2-D filters, the approach used in this project enables us to see the relationship between the high spatial frequency content of the video and its focus.

            Figure 7 FFT and relative focus

         3. Third part includes Video Focus Identification and Assessment in which video stabilization takes place depending on whether video frames are in focus state or not by using the ratio of the high spatial frequency content to the low spatial frequency content within a region of interest (ROI). When this ratio is high within ROI, the video is in focus state. When this ratio is low within ROI, the video is out of focus state. When in focus is established in the video with high spatial frequency within ROI it removes unwanted camera motions in the video and produces a stabilized video as output.

            Figure 8 focus metric state

      3. Flow Chart of DVFA algorithm

   A

   Start

   Acquiring image from the web camera

   When in focus metrics state ratio is high within ROI the video is said to be in focus state

   Converting the image into RGB by using color space conversion

   When in focus metrics state ratio is low within ROI the video is said to be in out of focus state

   Adjust the contrast of an image

   NO YES

   Due counter free running block the ratio will be high in focus state within ROI

   Identifying focus metrics in the video in order to stabilize the video

   In counter free running block the counter over flows back to zero after it has reached the maximum value possible for the specified no of bits.

   Identifying the standard shapes like rectangles on the images by over writing pixel values

   Performing segmentation operations by using morphological operators

   If u= =1

   By using ratio of high spatial frequency content and ratio of low spatial frequency content within the region of interest (ROI) we can identify focus metrics in the video

   When in focus state is established within ROI it removes unwanted translation camera motions from the video and produces a stabilized video

   The obtained stabilized video will be displayed on 52 inch 4:3 virtual display theatre glasses

   A

   Stop

2. EXPERMENTAL RESULTS

   To test the efficiency of dynamic 2-D FFT video focus assessment (DVFA) algorithm, the image shown in figure (9) represents unstabilized video. By performing DVFA algorithm on real time basis we had obtained stabilized video as shown in figure (10). The stabilized video that obtained is displayed in goggles as shown in figure (11).

3. CONCLUSION

In previous methods in order to stabilize the video they had used either simulink or mat lab for programming and designing but they have not used any hardware. The present method works in real time using raspberry pi as hardware and mat lab and simulink as software for programming and designing. By using raspberry pi simulink support package and dynamic 2-D FFT video focus assessment algorithm (DVFA). By using this algorithm we have designed video stabilization method to identify particular target location in the particular video frame, depending on ratio of high spatial frequency content within ROI, it determines whether the video is in focus state or out of focus state. if in focus state is established within ROI it removes unwanted translational camera motions and generate a stabilized video. The stabilized video is displayed on goggles. This makes people so easy to watch the videos and read news paper even though vehicle is moving. Finally we have designed a simply way to implement this method.

Figure 9 Unstabilized video

ACKNOWLEDGMENT

I extended my sincere thanks to professor Dr.S.A.K.Jilani, Associate professor Mr.S.Javeed Hussain, Department of Electronics and Communication Engineering and also my earnest thanks to Dr.A.R.Reddy head of the department of ECE for their support to carry out successfully of this research work.

Figure 10 Stabilized video

Figure 11 stabilized video will be displayed in goggle

REFERENCES

1. Abdullah, L.M.; Tahir, N.M.; Samad, M., "Video stabilization based on point feature matching technique," Control and System Graduate Research Colloquium (ICSGRC), 2012 IEEE , vol., no., pp.303,307, 16- 17 July 2012

2. Eddy Vermeulen Real-time Video Stabilization For Moving Platforms 21st Bristol UAV Systems Conference April 2007

3. L. Garlin Delphina and VPS Naidu Implementation and Validation of Video Stabilization using Simulink MSDF Lab, FMCD CSIR –

   National Aerospace Laboratories Bangalore, India. May 2013

4. A Thesis on Video Stabilization Using SIFT Features, Fuzzy Clustering, and Kalman Filtering Presented to The Faculty of Engineering and Computer Science University of Denver In Partial Fulfillment of the Requirements for the Degree Master of Science by Kevlin. Apirl 2011

5. A paper on contextual point matching for video stabilization presented to the university of California, Irvine in partial fulfillment of M.sc by ling han meng and joseph geumlek. May 2010

6. A thesis on Video Stabilization For A Hand-Held Camera Based On 3d Motion Model presented to faculty of Computer Science and Information Engineering, National Taiwan University, Taiwan by J.M. wang, H.P. chou.2009

7. Benny, Akesson , Henrik, Jansson Video Stabilizationfor Mobile Equipment for partial fulfillment of master of science thesis in blekinge tekniska hogskola university . November 2003.

8. A thesis onvideo stabilization with a depth camera presented to national university of Singapore in partial fulfillment of degree in master of technology by Shuaicheng Liu, Yinting Wang, in may 2012.