Baby Face Predictor using Parents Facial Features

Baby Face Predictor using Parents Facial Features

A. Enoch

T. Charan Reddy

M D. Nasrulla

1. Tech, Assistant Professor B. Tech, Student B. Tech, Student

   Department of ECE

   Department of ECE

   Department of ECE

   ASIST, Paritala, A. P. ASIST, Paritala, A.P. ASIST, Paritala, A. P.

   S. Venkateswara Rao	K. Harsha Vardhan	V. Venkatesh
   B. Tech,Student	B. Tech, Student	B. Tech, Student
   Department of ECE	Department of ECE	Department of ECE
   ASIST, Paritala, A. P	ASIST, Paritala, A. P.	ASIST, Paritala, A.P.

   Fig. 1. Baby Face Generator algorithm

   AbstractThis paper presents a method to generate a compos-ite baby face from two parent images. Face detection, facial land-mark identification, and feature extraction are first implemented to collect the relevant information from each parent image. Using this information, a facial morphing algorithm including the partitioning of features into triangles and quadrilaterals based on feature keypoints, warping triangles and quadrilaterals using affine transformations and homography respectively, and cross-dissolving the warped images is implemented. The composite adult features are then combined with the base baby features and placed onto the base baby face at the corresponding locations.

   1. INTRODUCTION

      The identification, analysis, and modification of images containing human faces remains an active research area in- volving many image processing techniques. One subset of this research area, the automatic detection of faces and extraction of facial features has a wide range of applications from facial expression analysis to privacy protection in Google Street View [1]. Another subset of this research area, facial morphing, also has a wide range of applications including facial recognition and comically combining the faces of two individuals. The goal of this project was to leverage these two research areas for the entertaining application of intelligently morphing two parent images to generate a baby face.

      The algorithm used for generating the baby faces, seen in Fig. 1, involves three major components: facial feature extraction, base baby selection, and facial morphing. Sections II through IV discuss these methods and implementations in detail. The final system implementation included a MATLAB Graphical User Interface (GUI) seen in Fig. 2, allowing users to select the parent images either from a file or webcam snapshot and run the morphing code with ease.

   2. FACIAL FEATURE EXTRACTION

      The first step to extracting the facial features for each parent image involved facial detection using the Viola-Jones face detector implemented in OpenCV [2]. For each parent

      Fig. 2.

      P6 17

      P2 6 1 7 P3 18 P7

      	P8
      4	3	5
      9	8	10
      13P4	2	P514
      11	16	12

      15

      Fig. 3. Face bounding box (green), detected landmarks (red), and calculated keypoints(blue)

      image, the facial landmarks (eye corners, mouth corners, and nose tip) were detected using flandmark, an open source facial landmark detector [3]. The bounding box and facial landmarks were then used to calculate additional facial keypoints seen in Fig 3. Details of the keypoint calculations are given in Appendix A. These keypoints were then used in combination with edge detectors to extract each facial feature described in Sections II-A through II-D.

      1. Nose

         The extraction of the nose relied solely on the trapezoidal region formed by connecting keypoints 4, 5, 7, and 6 in a counterclockwise fashion. This choice followed from two observations: (1) the landmarks and keypoints surrounding the nose were detected with high accuracy and (2) the nose has

         Iris Center Point

         Left Point Eye Center PointRight Point

         1. Original eye (b) Eye after morphological recon- (a) Original eye (b) Combined Eye Ellipse Mask struction

            (c) Eye edges (d) Extracted Iris

            (e) Eye mask (f) Eye mask with reflections re-

            moved

            Fig. 5. Extraction of iris and eye color

            (c) Lower Ellipse with Dividing Line (d) Upper Ellipse with Dividing Line Fig. 6. Extraction of eye

            two ellipse-shaped masks. Because the eye often has a dif-ferent distance from the corner points to the upper and lower lids of the eyes, using two ellipses accommodates for this difference by allowing for different minor-axis radii. The major-axis radius for both ellipses was calculated us-

            l

            l

            ing half the distance between 0:98Xleft corner; Y eft corner and 1:02X right corner ; Yright corner . The minor-axis radius for the upper ellipse was calculated as 1:2 r iris +(y eye center y iris center

            y

            y

            and the minor-axis radius for the lower ellipse was calculated

            as1:2r

            iris

            (y

            eye center

            iris center where riris is the calculated

            line between the ellipses was

            no sharp edges other than the nostrils so edge detection adds limited new information. Results of this nose extraction can be seen in Fig. 4b and Fig. 4g.

      2. Eyes

         Feature extraction of the eyes included extracting the shape of the eye as well as the eye color. The first step in extracting each eye involved finding the iris center location and radius. The algorithm for detecting the iris, based on [4], involved a combination of morphological reconstruction [5], edge detection using a Canny edge detector [6], and circle detection using a circle Hough transform. The first step in the algorithm involves extracting a large bounding box containing the eye of interest and converting the region to grayscale. Morphological reconstruction is then used to fill the reflections in the eyes and edges are found using the Canny edge detector. A circle Hough transform is computed on the edge image to detect the three most prominent circles [7]. The center location of each circle is then compared to the center point between the two corner locations of the eye. The iris circle is chosen to be the circle whose center point minimizes the distance between the two center locations.

         To determine the eye color, a mask of the circle computed for the iris was applied to the eyes to extract the entire iris. Additionally, a reflection mask was constructed using a thresholded difference between the original eye and the eye after morphological reconstruction and overlaid on the iris mask. Using this mask, the HSV colors of the eye were extracted. The approach is illustrated in Fig. 5.

         Using the corner landmarks, iris center, and iris radius, the mask of the eye shape was calculated by combining

         radius of the iris. The boundary

         calculated as the line connecting the two eye corners. An example of creating this mask for the left eye is shown in Fig.

         6. Results of the eye extraction can be seen in Fig. 4c-d and Fig. 4h-i.

      3. Mouth

         A two-fold process was implemented to extract the mouth of each parent image. First, the method described in Section II-B of using two ellipse masks was applied to extract a general mask for the mouth. The second portion of the algorithm to extract the mouth involved edge detection on a lip-enhanced imae. Transforming from RGB color space into the lip-enhanced color space described by 14 2G R 0:5B greatly increased the success of the lip edge detector because it clearly differentiates between skin and lip color as seen in Fig. 7a [8]. Edge detection using a Laplacian of Gaussian operator was then performed on the lip-enhanced image as seen in Fig. 7b. The edges closest to the mouth landmarks, P 4 and P 5, were found and if the edge closest to P 4 was connected to the edge closest to P 5, this edge was identified as the lip edge. If this condition was not met, the edge mask was discarded and the two ellipse mask was the only mask used.

         To create the edge-based mask for the mouth, the edges of the lips were filled and then slightly dilated to ensure the lips were not cut off. Finally, the two ellipse mask was overlaid on the mouth edge mask to form the final mouth mask. Results of the mouth extraction can be seen in Fig. 4e and Fig. 4j.

         (a) Original Emma Watson (b) Emma Watson nose (c) Emma Watson left eye (d) Emma Watson right eye (e) Emma Watson mouth

         1. Original Ben Affleck (g) Ben Affleck nose (h) Ben Affleck left eye (i) Ben Affleck right eye (j) Ben Affleck mouth

            Fig. 4. Extracted facial features

            P4 P5

            1. Original facial bounding box of (b) Skin regions detected Hillary Rhoda

         (a) Lip-enhanced image (b) Edges detected

      4. Skin

      Fig. 8. Extraction of

      (c) Mouth mask based on lip edges (d) Combined mouth masks

      Fig. 7. Extraction of mo

      The skin color was the final feature extracted from each parent image and was based on segmenting skin colors by six planes in HSV color space [9]. Using this HSV color space segmentation, skin regions were successfully identified and used to calculate the mean HSV values for each parent skin color. A resulting skin mask can be seen in Fig. 8.

   3. BASE BABY SELECTION

      A baby database including nine babies of varying skin and eye colors was curated and compared to the adult facial features to select the baby of closest resemblance to the parent images. Because the flandmark library was based on an adult

      frontal face model and baby facial geometry differs signifi-cantly from adult facial geometry, the facial landmarks were manually identified. The facial keypoints were then calculated

      1. A (b) B

         (c) A0 (d) B0

         Fig. 9. Partitioned facial features of Hillary Rhoda

         using slightly modified facial keypoint equations adjusted for baby facial geometry. Finally, the features were extracted using the methods described in Sections II-A to II-D.

         (e) Cross-dissolving of A0

         and B0 , I

         To select the base baby to be morphed with the parent images, the primary consideration was skin color. This com-parison was calculated by linearly combining the HSV values of each parents skin color to form a composite skin color and finding the baby skin color closest to this composite skin color in HSV color space. Additionally, if both parents had green or blue eyes, a baby with blue or green eyes was selected.

   4. FEATURE MORPHING

      The combined facial features (i.e. eyes, nose, and mouth) of two parents and a base baby are generated using a morphing algorithm based on mesh warping [10], [11]. Four phases are involved in this algorithm: image partitioning, warping, cross-dissolving, and generating a composite baby image.

      1. Image Partitioning

         Each facial feature is first partitioned into triangles and quadrilaterals based on the extracted facial keypoints as shown in Fig. 9.

      2. Warping

      When the facial features of the two parents are referred to as images A and B, the image A is to be warped to A0 and the image B to B0 so that A0 and B0 have the same shape as shown in Fig. 10a-d. The shape of A0 and B0 is calculated using linear interpolation of the corresponding grid point locations of A and

      1. In the warping process, an affine transformation is applied to map the coordinates of the triangles in the original image to the triangles in the warped image, while homography is used for the quadrilaterals. The equation for coordinate transformation is

         x = T x

         wherex and x are homogeneous coordinates in the form [x y 1]T of the original and warped images, respectively.

         Fig. 10. Morphing of two parent features, A and B

         The implementation maps the coordinates of pixels in the target image to the coordinates in the original image and assigns the interpolated color of the original image to the target

         pixel using

         x = T -1x:

         This guarantees that every pixel in the warped image is assigned a value.

      2. Cross-Dissolving

         After the two feature images are warped to the same inter- mediate grid locations, they are cross-dissolved by calculating the weighted average of the two images using

         I[x; y] = A0 [x; y] + (1 )B0 [x; y]

         where 0 << 1. The cross-dissolve of the two parent features in Fig. 10c and Fig. 10d is shown in Fig. 10e.

      3. Generating a Composite Baby Image

         The composite adult features are then morphed with the re-spective features of the selected base baby using the morphing steps (image partitioning, warping, and cross-dissolving) pre-viously described. This is required to make the final combined features look like those of a baby since certain aspects of baby features differ from adult facial features. For example, babies

         1. Composite adult feature, I (b) Base babys feature, C

      (a) weight mask of left eye (b) weight mask of left eye

      (c) Warped composite adult feature, I0 (d) Warped base babys feature, C0

      (c) weight mask of right eye (d) weight mask of right eye

      (e) Cross-dissolved feature of I0 and C0 , F

      Fig. 11. Morphing between composite adult feature and base babys feature

      have larger irises and flatter nose bridges. Results of this final warping step can be seen in Fig 11.

      Next, the final composite features, F , are pasted onto the base baby at the same center location as the extracted baby features. Smoothing is employed at the boundary of the features in order to create a smooth transition from the composite features to the outer areas in the base babys face. A binary mask around F , BW , is created, open filtered and eroded to attain a smoother and smaller binary mask. A Gaussian filter is applied to the eroded binary mask to create weight mask . Depending on the facial feature, either an average or a Gaussian filter is applied to the inverted binary mask of BW to create weight mask . Fig. 12 illustrates the weight masks used in the smoothing process.

      The feature to be placed onto the base baby face is calculated from the following equation.

      Feature = IBaby + (1 )( F + (1 )C0 ) whereIBaby is the base baby image.

      Smoothing with the warped baby feature, C0 , is included

      because the original base baby features can be too different from the composite features to apply a gradual transition along the contour of the features. For instance, the base baby may have much wider nose than the composite nose, making the side of the base babys nose appear in the final image if gradual transition is applied. To accommodate for this, more surrounding regions are included in the extracted features.

   5. RESULTS

      Our proposed Baby Face Generator program was tested on 30 different adult images. The sample results are presented in Fig. 13. From qualitative examination, the generated baby images have features that pleasantly resemble the two parents, especially the shape of nose and mouth, eye color, and skin color.

      (e) weight mask of nose (f) weight mask of nose

      (g) weight mask of mouth (h) weight mask of mouth

      Fig. 12. Weight masks used in smoothing boundaries of each feature

      Significant artifacts, however, can occur if a parents face is turned by a considerable angle. Nonetheless, the program can be constrained to only accep forward facing input faces since it is intended to interact with the parent users directly.

   6. CONCLUSION

In conclusion, an automatic Baby Generator Program has been developed. Facial features extraction and facial morphing algorithms have been employed to produce a satisfying com-posite baby face of two parent images. Further improvements could be achieved by using a larger database of base baby images and adding the detection of parents ethnicity to en-hance the selection of base baby images. Additionally, an age progression algorithm could be implemented to first generate a baby face of each parent before morphing the two together. This could potentially reduce artifacts introduced by the base baby and the age-variant features.

ACKNOWLEDGMENT

The authors would like to thank Professor Bernd Girod, Professor Gordon Wetzstein, Huizhong Chen, and Jean-

Volume 4, Issue 18 Published by, www.ijert.org 8