Research on Pretreatment of Metal Surface Defect Detection Based on Photometric Stereo Method

Research on Pretreatment of Metal Surface Defect Detection Based on Photometric Stereo Method

Pengyu Guan1

College of Automation and Electrical Engineering Tianjin University of Technology and Education Tianjin, China

Shuaipeng Yuan3

Optical Department

Fitow (Tianjin) Detection Technology Co. Ltd.

Tianjin, China

Genghuang Yang2

College of Automation and Electrical Engineering Tianjin University of Technology and Education Tianjin, China

AbstractIn order to improve the efficiency of metal surface defect detection, an image preprocessing method based on photometric stereo method was proposed. Firstly, the objects are illuminated from eight different directions to collect images. The collected images are used to calculate the light reflectance image and gradient image of the object based on the principle of photometric stereo method. Secondly, the albedo image is processed by edge extraction to obtain the detection area, and the average curvature of the object surface is calculated by gradient image. Finally, Finally, the curvature of each point of the image is converted into gray value, and the image with high contrast between defect and background is obtained. The results show that the eight-light-source photometric stereo method can accurately construct the gradient surface information, which is conducive to the detection of metal surface defects.

Keywords Defect detection; photometric stereo method; gradient information; average curvature; edge extraction

1. INTRODUCTION

   Defect detection is an important technology to ensure product quality and production efficiency. The traditional defect detection mainly focuses on manual detection [1]. It is difficult to ensure high detection accuracy while maintaining efficient detection during manual detection. Long time detection will greatly reduce the concentration of workers [2]. And it is difficult to detect all defective products. Automatic inspection equipment based on machine vision uses cameras to obtain product image information instead of human eyes [3]. The detection algorithm is used to identify defects, realizing high-precision and efficient detection. It has been fully applied in many industries such as automobiles, textiles, and semiconductors [4].

   At present, there are many types of metal surface defects with different characteristics [5]. Some defects show different characteristic information from different angles, which is easy to cause false detection and missed detection [6]. At the same time, the dirt on the tested metal surface will be similar to the defect characteristics of some angles, which increases the difficulty of defect detection. Therefore, metal surface defect detection technology cannot be widely used, which greatly reduces the work efficiency and enterprise automation level.

   In this paper, a pretreatment method for metal surface defect detection based on photometric stereo method is proposed. Firstly, the objects are illuminated from eight different directions to collect images. The collected images are used to calculate the light reflectance image and gradient image of the object. Secondly, the image synthesized by photometric stereo method is processed. Finally, a gray image with high contrast is obtained, which provides a high-quality image to be detected for later defect detection.

2. CLASSICAL PHOTOMETRIC STEREO METHOD In vision detection, images with clear features, uniform

   brightness, and high contrast between defect location and background area can effectively improve the detection accuracy of the detection algorithm. Photometric stereo method is a 3D reconstruction method improved by Woodham according to SFS (Shape from shading) algorithm [7]. The gradient image, light reflectance image and height field image of the object to be photographed are calculated based on the surface albedo and the brightness of the image. The image is selected according to the defect type during detection. The light reflectance image has uniform brightness, which can better present the defect location features, but does not contain depth information, and is easy to lose the depth features of the defect location. The processed gradient image is sensitive to depth information, and is easy to detect defects with depth.

   The main principle of photometric stereo method is to analyze object images under different illumination conditions, and to determine the normal direction of each pixel point [8]. Taking a single pixel point as an example, a diffuse reflection model of illumination on object surface shown in Fig. 1 is established.

   t

   L1 L1

   L1

   i1 x y z

   t

   L2 L2

   L2

   i 2

   x y z

   t

   L3 L3

   L3

   i3

   x y z N

   t L4 L4 L4 ix

   i 4

   x y z N

   (4)

   t

   i L5 L5

   L5

   iy

   i5

   x y z N

   Fig. 1. Light diffuse reflection model

   t L6 L6

   L6 iz

   i 6

   x y z

   7 7 7

   In Fig. 1, diffuse reflection occurs when light source J illuminates point I of the object surface. Lj, Ni, Vi are the light

   ti 7

   t

   Lx Ly

   L8 L8

   Lz

   L8

   source direction unit vector, surface unit normal vector and

   i8

   x y z

   camera direction unit vector, respectively. i is the reflectivity of the surface of the object. Thus the formula (1) can be obtained.

   Matrix equation (4) is a set of overdetermined linear

   equations, which can be solved by least square method to obtain the surface unit normal vector Ni of point i and

   tij i (Ni

   Lj )l j

   (1)

   reflectivity i

   (LLT )1 (LT t )

   (5)

   Where, tij is the gray value of point i under the illumination

   condition of light source j, lj is the light intensity. In the

   i ij

   (LLT )1 (LT t )

   formula tij can be directly obtained from images, Lj is the three-dimensional unit vector obtained according to the

   i

   N ij

   i

   (6)

   placement position of the light source, i and Ni are the surface information of the object to be sought. Without changing the

   positions of the camera and the object to be photographed,

   The gradients p and q in the x and y directions are:

   T 1 T

   only change the position of the light source so that the light source can light the same point i of the object at least three different positions. When the light source lights the point i of the object to be photographed at each position, an equation as shown in Formula (1) will be obtained. After obtaining at least three equations, the reflectivity of the object at point i can be

   (Lx Lx ) (Lxtij )

   p

   z z z ij

   (LT L )1 (LT t )

   (LT L )1 (LT t )

   q y y y ij

   z z z ij

   (LT L )1 (LT t )

   (7)

   (8)

   solved i and surface unit normal vector Ni. The solution formula is shown in (2) and (3).

   1

   When each pixel in the image is brought into Formula

   (4~6), the surface gradient image and reflectivity image of the object to be detected can be obtained. The direction gradient

   i tij Lj

   N 1 t L1

   (2)

   (3)

   corresponding to each pixel (x, y) is (p, q).

   B. Extract Detection Area

   i ij j

   i

3. DETECTION PRINCIPLE

   A. Eight-light-source Photometric Stereo Method

   The calculation of object surface reflectivity and unit normal vector using photometric stereo method requires at least three image with illumination in different directions, and the unknown parameters can be obtained by simultaneously establishing three equations. However, in practical applications, the light sources in three directions are not enough to light all the surfaces of an object. In image synthesis, the brightness of some areas is insufficient. In order to get more accurate results, more light sources are used to light the surface of an object. In this paper, aiming at the characteristics of industrial production, a photometric stereo

   During defect detection, an image is divided into detection area and non detection area. In the detection area, different detection requirements are divided according to the object to be detected. Therefore, the light reflectance image is segmented to find the detection area in the image, which can effectively improve the detection efficiency and detection accuracy. Edge extraction can accurately find out the contour of the detected object, which is convenient to segment the object and background. Canny edge detection includes three steps: filtering noise reduction, edge detection and edge filtering.

   The first step is image filtering and noise reduction. Gaussian filter is used to smooth the image, and two- dimensional Gaussian function G (x, y) is used to convolve the image. G (x, y) is as follows:

   x2 y2

   method with eight-light-sources is designed. The brightness of the i-th pixel in the image when the eight light sources are

   e

   G(x, y) 1 2 2

   2 2

   (9)

   illuminated separately is shown in the following formula:

   Where, is the coefficient of Gaussian filtering, the noise suppression effect improves with increasing value of , and the easier it is to eliminate defect features.

   The second step is to calculate the detection edge, the gradient amplitude of each point in the image and the position of each edge are calculated. Move the image mask in the

   image, and calculate the mask value of each position separately. When the calculated value exceeds the set threshold value, the current position is considered as the image edge.

   The third step is to filter the extracted edges. First, all the edges in the maximum position are filtered by non maximum suppression, and then use double threshold detection to determine whether the filtered edges are real edges.

   After the edge of the detection area is extracted, the edge contour of the object is filled to obtain the detection area. The final detection image is obtained by extracting the image processed by photometric stereo method from this area.

   C. Calculate The Average Curvature of The Image

   Photometric stereo method can synthesize gradient image and reflectance image of object surface, but gradient image cannot effectively identify defects, and reflectance image does not contain depth information, so defects with depth are easy to be missed. Therefore, the gradient image synthesized by photometric stereo method is processed to obtain the average curvature image of the object surface to be measured. The surface gradient of the object to be measured can be obtained by the following formula:

   collision defect. The groove part is prone to false detection during detection. Aluminum valve plate sample is shown in Figure 2.

   Fig. 2. Aluminum valve plate sample

   f (r, c) (u(r, c), v(r, c))

   (10)

   The camera model used in the experiment is MV-CH120-

   Where r is the row coordinate in the image and c is the

   column coordinate in the image. u(r, c) is the row gradient

   component of point (r, c) in the image, which can be obtained by formula (7), and v (r, c) is the column gradient component of point (r, c) in the image, which can be obtained by formula (8). The average curvature H of the image is calculated as follows:

   10GM, the lens model is MVL-KF3528M-12MP, and the

   illuminant uses a customized illuminant composed of eight lights. The distance from the lens to the sample is 280mm, and the distance from the end of the illuminant to the sample is 300mm, which can build an optical environment as shown in Figure 3.

   (1 f 2 ) f

   (1 f 2 ) f

   • ( f

   • f ) f

   3

   H x yy y xx xy yx xy (11)

   x y

   2(1 f 2 f 2 )2

   The second derivative in formula (11) is expressed as follows:

   2 f (r, c)

   r 2

   fxx

   2 f (r, c)

   f 2

   1 2

   yy c

   (12)

   fxy

   f (r, c)

   Fig. 3. Photometric stereo optical environment

   f yx

   rc

   The aluminum valve plate can be illuminated and

   2 f (r, c)

   cr

   After the average curvature of the image is calculated, the average curvature image is converted to the gray image used in image processing. The contrast between the defect location and the background area is high, which makes it easy to find the defect location features and separate the defects from the background.

4. EXPERIMENTAL RESULTS AND ANALYSIS

   1. Establishment of Experimental Environment

      The metal surface tested in the experiment is the aluminum valve plate on the automobile gearbox, the detection area is the machining surface, and the detection defect type is

      photographed through the optical environment shown in the figure. When shooting, the angles of the light source center point in the top view and side view required by the algorithm relative to the aluminum valve plate to be photographed are recorded, and convert the angle obtained into the light source direction unit normal vector. The top view light source layout is shown in Figure 4, and the top view angle description of the light source is shown in Figure 5. The right side of the central horizontal line is the starting point of the light source angle (0

      °), counterclockwise rotation is the positive direction, and the side view angle is shown in Figure 6. With the central axis of the camera as the reference, the angle between the lighting direction and the reference line is the required angle.

      Fig. 4. Top view of light source

      Fig. 5. Description of light source top view angle

      Fig. 6. Side view of light source

   2. Analysis of Experimental Results

      According to the experimental environment, the angle of the light source relative to the photographic object can be measured. The angle between light source 1 and aluminum valve plate is 247 °, the angle between light source 2 and aluminum valve plate is 202 °, the angle between light source 3 and aluminum valve plate is 157 °, the angle between light source 4 and aluminum valve plate is 112 °, the angle between light source 5 and aluminum valve plate is 67 °, the angle between light source 6 and aluminum valve plate is 22 °, the angle between light source 7 and aluminum valve plate is 337

      °, and the angle between light source 8 and aluminum valve plate is 292 °. In the process of successively lighting up light source 1 to light source 8, one light source camera is lit each time to take an image, and a total of 8 images are taken, as shown in Figure 7.

      Fig. 7. Photometric stereo imaging in eight directions

      Figures 7 is taken when one light source is on and other light sources are off. Only one side of the light source is bright and clear in each image. The area in the image has been photographed 8 times. Each point in the area has 8 corresponding characteristics. The reflectance image output by photometric stereo method can be obtained by calculating each point in the image through the formula. As shown in

      Figure 8.

      1. Reflectance image (b) Local amplification of defect

   location

   Fig. 8. Reflectance image synthesized by photometric stereo method

   The reflectivity image can directly see the defect position, but the contrast between the defect position and the background is not obvious. Therefore, the gradient image generated by photomtric stereo synthesis is processed to enhance the contrast between the defect position and the background. First, calculate the average curvature of the gradient image, and then propose the detection area by edge extraction to obtain the average curvature image containing only the detection area, as shown in Figure 9.

   (a) Average curvature image (b) Local amplification of defect location Fig. 9. Average curvature image of the detection area

   It can be seen from Figure 9 that the defect position in the average curvature image has a high contrast with the background, and the dirt in most images is eliminated, providing high-quality images to be detected for subsequent defect detection.

5. CONCLUSION

Because of the low contrast between the metal surface defect and the background, it is difficult for the detection algorithm to accurately extract the defect. This paper proposes a metal surface defect pretreatment method based on photometric stereo method, which is suitable for practical production. The experimental results show that the contrast between defect and background can be effectively improved by using photometric stereo method to process images, which has certain theoretical significance and practical value for automatic detection.

REFERENCES

[1] Y. L. Cao. Photometric-Stereo-Based Defect Detection System for Metal Parts. Sensors, 2022, vol.22, no.21, pp. 8374-8374.

[2] Q. Zheng, B. Shi and G. Pan. Summary study of data-driven photometric stereo methods. Virtual Reality & Intelligent Hardware, 2020, vol.2, no.3, pp. 213-221.

[3] Chen Q. Error Analysis of Photometric Stereo with Near QuasiPoint

[5] K. Christian and B. Ernst. Photometric stereo-based high-speed inline battery electrode inspection. tm – Technisches Messen, 2021, vol.88, no.7-8, pp.423-432.

[6] M. Ren. Multi-scale measurement of high-reflective surfaces by integrating near-field photometric stereo with touch trigger probe. CIRP Annals – Manufacturing Technology, 2020, vol.69, no.1, pp. 489- 492.

[7] R. J. Woodham. Photometric method for determining surface orientation from multiple images. Optical Engineering, 1980, vol.19,

[4]

cal

Lights. Computer Graphics Forum, 2022, vol.41, no.6, pp.149-165.

A. Lyes and A. Saliha. Improved photometric stereo based on lo

search. Multimedia Tools and Applications, 2022, vol.81, no.21, pp.31181-31195.

no.1, pp.139-144.

[8] C. WoHler and K. Hafezi . A general framework for three- dimensional surface reconstruction by self-consistent fusion of shading and shadow features. Pattern Recognition, 2005, vol.38, no.7, pp.965- 983.