Facial Expression based Song Recommendation: A Survey

Facial Expression based Song Recommendation: A Survey

Armaan Khan

Department of Computer Science and Engineering, MIT School of engineering, MIT-ADT University, Pune, 412201, India

Ankit Kumar

Department of Computer Science and Engineering, MIT School of engineering, MIT-ADT University, Pune, 412201, India

Abhishek Jagtap

Department of Computer Science and Engineering, MIT School of engineering, MIT-ADT University, Pune, 412201, India

Abstract Listening to Music is a very important aspect of life for a lot of people. There has been a lot of research and improvement being done in music search or music organization factors which directly leads us to the problem of finding or simplifying the process of selecting a particular song that you want to listen. One of the choices is that the song is being recommended which nowadays is getting more and more popular as it helps to select songs for a variety of occasions. As music is a great source of entertainment for humans and is used for relaxing, working, managing stress, and maintaining balance in mental and physical workloads. This paper will talk about the recommendation system where the user would be able to choose a particular song based on their emotion or we can say their facial expressions as humans to act differently according to their moods and facial expressions tells a lot about the mood of a person. When we combine the artificial intelligence approach with a generalized music approach then the resultant song recommendation system will help out people to get song recommendations with just a look at their facial expressions.

Keywords: Convolutional Neural Networks; Deep Learning; Face Recognition, Song Recommendation

1. INTRODUCTION

   A lot of research has been done with respect to music- driven influence on the physiological and emotional state of a human. We feel various emotions based on the type of song that we listen to. Listening to music impact our thoughts and feelings which in turn make an impact on our physical as well as mental health, which is why the music well-being topic is gaining popularity. As it affects our mental and physical health a lot of research is done on knowing the impact of music on memory.

   Along with mental health music also impact our decision- making, as human our mod changes with the kind of song we listen to. For example, if we are listening to a happy song then our mood at that time tends to be good meanwhile if we are listening to a sad song then our mood may not be that great at that time. Apart from this music also music is also an important medium of entertainment almost every person used to listen to songs for the entertainment.

   We also have a lot of music apps where we can go and choose the songs according to our mood and then listen to them. But the problem with this approach is that we have to manually select the song that we want to listen to. So it would

   be easy if we have a system that automatically plays a song based on the emotion detected on our face without the user manually selecting a song to listen to. The human body acts differently according to the mood and different music has a different effect on the human body. Music recommendation can be applied to various areas such as support of intellectual and physical work, studying, sports, re-laxing, stress and tiredness destruction, music therapy, and many others.

   The application takes an image as an input and based on the facial emotion pre-sent in the input image it predicts that emotion and then based on that emotion a song is chosen from the database that song is specific for a particular emotion. There is a playlist of songs for each different emotion. In this paper, we would compare this application with other researches done on this topic before.

2. LITERATURE SURVEY

   Nowadays everything is being automated due to which people get their work done by doing only a few gestures. Our motto with this application is also similar where instead of creating a playlist from thousands of songs, the user would be able to get song recommended on the basis of facial expression. S. Gilda [1] stated in their paper a way to get songs recommended using 3 different mod-ules, first, Emotion module which classifies their mood, Second, Music classification module which classifies the music and third, recommendation module which recommends music based on similarity of mood and type of song. Here the input is taken as an image and predicts the four different types of emotions. They have classified only some basic emotions.

   A better approach is used by S. Matilda [2] where instead of an image, a webcam is used to capture the facial emotion of the user. The emotion of the user was classified using the fishes face algorithm. They used a database of songs to recommend the songs. There were only 2 facial expressions that were captured according to the paper were Happy and Sad. Using a live webcam for facial emotion recognition is a better choice but the computational cost increases very much and also the accuracy comes down.

   The paper by Vinay. p [3] focus on the extraction of appearance and geometric-based features from the input, these features help us in the study of shape and the pixel

   intensity of the face. For this Support Vector Machine (SVM) was used. They got an accuracy of 85-90% for the real-time images while they got 98-100% accuracy for static images. there are many studies done to understand hu-man emotion and facial expressions tell a lot about them. So getting higher accuracy models is required, to get better accuracy using inbuilt libraries is a good choice. In the paper Deny John Samuel [4], the combination of OpenCV and Support Vector Machine (SVM) is used. OpenCV is used to extract the features from the image and SVM was used to predict the recognized emotion. The paper recognizes 4 different types of emotions those were: Happy, Anger, Surprise, Neutral.

   In [5] A. S. Bhat proposed an automatic way to identify the mood and tone in the music by analyzing spectral and harmonic features of music notes and human emotions.it classified songs according to their mood with a Thayers model.it identifies features of songs like spectra of beat, roughness be-fore classification. It classified songs with an efficiency of 94.4% at best

   In [6] Renuka R Londhe proposed a paper on the subject of the classification of emotions from their facial expression by studying the changes in the curvatures of the face and how the intensities varied for different pixels of images. The author used Artificial Neural Networks (ANN), which were used to classify emotions. They also proposed various techniques for playlist creation based on emotion. Zeng [7] proposed two major categories for facial feature extraction, in their paper which included 1) Appearance-based feature extraction and 2) geo-metric-based feature extraction, Appearance-based extraction included extraction of some essential features of the face such as mouth, eyes, and Eyebrows. And geometric-based included classification based on geometric features on the faces. Zhang [8] used a modified version of the AdaBoost algorithm which was based on OpenCV. They also used two different methods for face detection timer and dual thread. The result showed that the dual thread method was simpler and faster.

   Parul Tambe [9] proposed a system that automated the interactions between the users and music player, which would then run and learn the preference emotion and different activities of the user and will recommend songs based on the users current emotion. They also collected different facial expression which were then used to determine the emotion of the user based on the users facial expressions. Anukritine

   [10] came up with an algorithm that provides an inventory of songs from the users playlist in accordance with the users emotion. The algorithm which was designed was focused on having less computational time and also thus reducing the cost included in using various hardware. The main idea was to classify the emotions into five different classes i.e., Joy, sadness, anger, surprise, and fear which then provided a very good and efficient method for mu-sic recommendation.

3. BENEFITS AND LIMITAION

   Benefits

   1. Ease of use

   2. Mixed mood detection.

   3. Improved accuracy

   4. Contain a separate playlist for each emotion Limitations:

   1. Limited emotions were covered.

   2. Right now emotion-based music recommenders are less accurate and a little time-consuming

4. CONCLUSION

   The aim of this paper was to explore the field and research done in regard to automatic song recommendations based on the facial expression of a person. This type of recommendation system will be very useful for people because of its de-pendency on the users emotions rather than the users past history. recent development in different algorithms for emotion detection promises a very wide range of possibilities. This system can reduce the manual work of creating a playlist by a user and automatically create a playlist for the user and he can spend that time listening to music. it will also help in reducing the time it takes for a user to search for a song according to his current mood. It can be also concluded that the current system for recommendation is not up to the mark. this system can be used for better personalization of songs and get a better recommendation.

5. REFERENCES

1. Gilda, Shlok, et al. "Smart music player integrating facial emotion recognition and music mood recommendation." 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). IEEE, 2017.

2. Florence, S. Metilda, and M. Uma. "Emotional Detection and Music Recom-mendation System based on User Facial Expression." IOP Conference Series: Materials Science and Engineering. Vol. 912. No. 6. IOP Publishing, 2020.

3. Vinay p, Raj p, Bhargav S.K., et al. Facial Expression Based Music Recommendation System 2021 International Journal of Advanced Research in Computer and Communication Engineering, DOI: 10.17148/IJARCCE.2021.10682

4. Deny John Samuvel, B. Perumal and Muthukumaran Elangovan, "Music rec-ommendation system based on facial emotion recognition", 2020.

5. A. S. Bhat, V. S. Amith, N. S. Prasad and D. M. Mohan, "An Efficient Classifica-tion Algorithm for Music Mood Detection in Western and Hindi Music Using Audio Feature Extraction," 2014 Fifth International Conference on Signal and Image Processing, 2014, pp. 359-364, doi: 10.1109/ICSIP.2014.63.

6. Londhe RR and Pawar DV 2012 Analysis of facial expression and recognition based on statistical approach International Journal of Soft Computing and Engi-neering 2.

7. Zeng Z, Pantic M, Roisman GI and Huang TS 2008 A survey of affect recogni-tion methods Audio, visual, and spontaneous expressions IEEE transactions on pattern analysis and machine intelligence 31 39-58.

8. Xianghua Fan, Fuyou Zhang, Haixia Wang and Xiao Lu, "The system of face detection based on OpenCV," 2012 24th Chinese Control and Decision Confer-ence (CCDC), 2012, pp. 648-651, doi: 10.1109/CCDC.2012.6242980

9. Parul Tambe, Yash Bagadia, Taher Khalil and Noor UlAin Shaikh 2015 Advanced Music Player with Integrated Face Recognition Mechanism International Journal of Advanced Research in Computer Science and Software Engineering

10. Dureha A 2014 An accurate algorithm for generating a music playlist based on facial expressions International Journal of Computer Applications 100 33-9