A Review on Clothes Matching and Recommendation Systems based on user Attributes

A Review on Clothes Matching and Recommendation Systems based on user Attributes

Atharv Pandit

Department of Information Technology

D. J. Sanghvi College of Engineering Mumbai, India

Kunal Goel

Department of Information Technology

D. J. Sanghvi College of Engineering Mumbai, India

Manav Jain

Department of Information Technology

D. J. Sanghvi College of Engineering Mumbai, India

Neha Katre

Department of Information Technology

1. J. Sanghvi College of Engineering Mumbai, India

   AbstractDressing appropriately is very important when going out in the real world. Wearing clothes properly that show some level of style and wearing them such that they adhere to the norms of social standards uplifts the confidence of the person and creates a very good impression. The study focuses on helping the user to find optimized matching pair of clothes taking into account intricate details like style, patterns, colors, textures, etc. also keeping in mind users attributes like age, skin tone, favorite color etc. It aims to help the user choose clothes that are fashionable and organize their closet. It tries to help the user to wear clothes that are suitable to occasions and helps user to buy clothes that would suit their style. In this paper, an in depth study is performed of various systems that are developed for the various features that must be kept in mind for making a robust system that finds matching clothes of the user as well as makes recommendations. Systems developed to make recommendations of clothes using various approaches have been studied and their merits and demerits high-lighted. Systems that are used for clothes detection have also been studied to make the system user- friendly while the user provides input.

   KeywordsPCST; Latent SVM; Apriori; Fuzzy Logic

   1. INTRODUCTION

      The fashion industry occupies a significant position in the global economy and involves large industrial chains, including garment design, production, and sales. In fact, in recent years, there has been an expanding demand for clothing all over the world [10]. The apparel market is projected to grow from 1.3 trillion U.S. dollars in 2015 to about 1.5 trillion dollars in 2020 globally, representing that the demand for clothing is going to rise across the entire world [9].

      Dressing in a socially acceptable combination of clothes is considered vital in a modern society, especially where professionalism is synonymous to attire. In everyday life, people need to find appropriate clothes to wear. Wearing clothes that match in color, texture, style, skin tone, etc. is an extremely important aspect of fashion and personality. Outfit selection is a common problem that people face every day, this problem is very broad and it involves a considerable amount of visual and social factors that can be implicit and abstract.

      You stand in front of your wardrobe and see the same clothes, always the same output combinations? What to wear? Need a Change? This is what everyone thinks while getting dressed. In todays fast moving world people dont have enough time to dedicate to fashion and personality as a result they wear same monotonous dress in their routine, most of the people wear the same set of matching dress and repeat them over a week, they are either shy or dont have enough time to try different combinations as a consequence most dresses are not used and remain dumped in the wardrobe forever .While buying they look for match of particular dress but dont take into consideration the ones they have in their closet as a result the clothes get predefined in combination and matching and the ones which do not have any matching pair remains futile

      .Looking at this real life problem, it motivated us to bring a solution which would maximize the utilization of all the dresses.

      In everyday life, people need to find appropriate clothes to wear. Considering the importance of the topic and the need of users, it motivated us to develop a system to provide the best case combinations (Optimized Pairs) of matching clothes as well as provide recommendations. Having looked at the need to provide best case combinations, the rest of the paper is organized as follows. Section 2 throws light on the existing systems, approaches, algorithms and their merits and demerits. The proposed architecture is discussed in Section 3. The paper ends with conclusion in Section 4 and references in Section 5.

   2. LITERATURE SURVEY

1. Literature Related to Existing System

   1. Myntra-Matching Clothes Recommendation:

      On selecting a particular item to buy, Myntra automatically suggests a full set of clothes that are matching to the selected item. For example, on selecting a particular t-shirt, the system automatically generates a combination of watches, shoes, pants, etc. that are matching to the selected t-shirt. This system does not take into consideration private qualities of customers like skin color and existing clothes. It will only suggest clothes that already exist in its database.

   2. Your Closet:

      This is a mobile application that organizes the closet. The user interface is shown in Fig. 1. The application asks customer to input their clothes. It then matches each cloth with other clothes. For example, if there are 4 shirts and 4 pants, the application matches each shirt with each pant and thus provides 16 possibilities. The application does not make matches of clothes depending upon patterns, color and texture of clothes. It also does not have a recommendation system.

      Fig. 1. Your Closet App

   3. Magic Closet:

      This system aims to retrieve clothes from online stores that are matching to the input clothes. These clothes must be fit to a particular occasion. In this system, the user takes a photo of them specifying if they want to use the top or bottom clothes along with the occasion they want to use it for. The system will search for clothing that matches the user query and satisfies the criterion of wearing aesthetically and wearing properly [1].

   4. Which Clothes to wear confidently?

      The basic problem the system addresses is: From the two given images corresponding to a pair of clothes, we have to determine if the pair of clothes matches or not. While there may be several aesthetics espoused by dierent individuals, it takes a simplistic approach in this problem. An example of shirts and ties is used. Various machine learning methods are used to classify if the clothes are matching or not such as Ridge Regression, Standard Neural Network and Siamese Neural Network [2].

   5. Personalized Clothing Recommendation Based on Knowledge Graph:

      This system attempts to exploit the knowledge graph for providing clothing recommendations to the user keeping the user context in mind. The recommendation is done by calculating the similarity in the clothing ontology similar to users collection [8].

   6. Skin and Clothes matching seeded by Color System Selection:

      The main aim of the system is to suggest clothes to user based on skin color. The paper first finds out which color scheme is best suited to represent skin colors and then tries to find a way to recommend if clothes and skin color match. An automated system to determine the highest levels of color suitability between skin and clothing was made [3]. The new

      approach offered in this research was: the connectivity between the fuzzy membership functions of skin color and clothing [3].

   7. Discerning Advisor:

      The system tries to recommend clothes based on skin olor of the customer. Using a neural network, first the skin color is detected. Fuzzy logic is used to map a skin color to the skin color of a fashion model, and clothes suited to that model are recommended [4].

   8. Garment Detectives:

      The garment detection is to detect the presence of clothes in images and somewhat locate their extents, where the localization can be defined from coarse (image) level to fine (pixel) level [5]. A unified system is proposed for detecting and recognizing clothes in customer photos.

   9. Identifying Corners of Clothes by Image Processing: This system aims to find the edges of the clothes for clothes manipulation. This system achieves this by finding pixels that represent the clothes. This system first accepts user image and then performs several image processing operations to improve the efficiency of edge detection. It then uses certain criterion to decide whether a pixel represents an image or not [6].

   10. Real-time Clothing Recognition from Surveillance Videos [7]:

   It is an analysis system of contents of video which is capable of tagging various clothes of different persons is created. First, face detection and tracking is performed and each frame is aligned. The system then proceeds to clothing segmentation using a variant of region growing method. Through this, clothes are detected. The system then proceeds to clothing recognition and indicates the type of clothing – skirt, t-shirt, etc. [7].

2. Literature related to Approaches:

1. Recommendation System:

   1. Occasion Based clothes recommendation using latent SVM:

      The recommendations made by Magic Closet are performed using a Latent SVM model which is learned by using labeled clothes [1]. The model makes recommendations based on visual features, clothing attributes and occasions. For Wearing properly-occasion-attribute potential, feature- occasion potential is taken into account. Wearing aesthetically-attribute-attribute potential is taken into account. Features are extracted and concatenated into a vector to represent a human part. To narrow down the semantic gap between clothing features and occasion concepts, some attributes are manually defined. Category attributes such as jeans, skirts and properties such as pattern, etc. are also defined. The system finds the underlying rules from all the fully annotated clothing photos [1].

      Fig. 2. Latent SVM

   2. Clothes Recommendation based on Knowledge Graph:

      Three kinks of basic ontology are defined for the related entities such as user, cloth and context. The user entity description comprises of user features such as height, weight, skin, etc. The clothes are defined using cloth characteristics like color, texture, pattern, fabric, etc. The context entity consists of information about the current weather, occasion, etc. The Apriori algorithm is used to find correlation between clothing and context attributes and the frequent item set is gathered. The knowledge rules are obtained and applied to the ontology and knowledge graphs. This process is known as knowledge reasoning. After learning about knowledge reasoning, plenty of association rules are obtained. Relevant attributes are connected using edges to construct ontology and the knowledge graph of clothing domain. Personalized recommendation comprises of analysis of knowledge graph, user context and recommendation system. The Top-N algorithm is used to make these recommendations in descending order of ranking. Recommendations are based on similarity of clothing calculated using the clothing ontology [8].

      Fig. 3. Knowledge Graph

   3. Finding Matching Clothes using Machine Learning and Deep Learning:

      For making the system, a dataset is constructed using images of 41 pairs of shirts and matching ties. Only parts of shirt where tie is worn were considered. Non-matching pairs were

      created by pairing each shirt and each tie with itself. Each image is used to create a color histogram. It is then normalized to factor the luminescence. The histogram is created by dividing the RGB color space into equal bins and by determining in which bin each pixel falls. Finally, the sum of pixels at each bin is normalized creating a normalized color histogram, giving a number between 0 and 1 for each feature in the features vector in order to achieve better performance. It is seen unnormalized histograms give lower performance. The color histograms of each pair are concatenated and assigned a corresponding label of +1 and 0 is attached. The feature vector was then passed through various classification algorithms to see which one performs the best. Ridge Regression, Standard NN, Siamese NN were used as the classification algorithms. It was observed that the Standard Neural Network gave the best performance. It is predicted that, artificially creating matching pairs of shirts and ties would make the Siamese Neural Network to perform better [2].

2. Matching skin color and clothes:

   1. Fuzzy Logic and Questionnaire:

      YcbCr color model is fixed and 30 samples were classified into three races. The race classification is used to determine the fuzzy membership function to make the fuzzy rules. Suitability of clothes with color of skin is then to be determined. 30 respondents were given a questionnaire containing images of human races and a wide variety of clothes. Respondent gave ratings of 1 to very mismatch and 10 for very harmonious [3]. The three highest score results of total respondents were ranged as a dominant color of each skin color. To find the relationship between clothes color and skin color, a color model suitable for clothes is selected and fuzzy membership function created. The relationship of fuzzy rule between fuzzy membership function of clothes and skin color were designed to get the best suitability index. The index was required to find suitability of input skin color and clothes color. The suitability index is different for men and women [3].

   2. Fuzzy Logic and Clustering:

      The distribution of skin colors was plotted in an RGB color space and clustered into three clusters- light, medium, dark using fuzzy c-means clustering. Skin color categories for light, medium and dark skin can be defined by three membership functions resulting from clustering. Then we see how much a particular model matches to a particular cluster of skin color. Then, a customer image is taken and his skin color is detected. It is then found out how much does it match with the skin color of a particular model. Colors that are suggested by a specialist for a particular model are suggested to the customer [4].

      Fig. 4. Fuzzy Logic and Clustering

3. Detecting clothes and clothes features from clothes:

   1. Using PCST to find clothes regions [5]:

      Clothing regions are found based on Prize Collecting Steiner Tree (PCST).The clothing detection is done by formulating it as a sub graph problem and recognition is performed using SVM. In clothing detection, the whole picture is divided into super pixels and each super pixel is assigned certain prize depending upon the local features extracted. Each super-pixel forms a node in the graph and each edge is assigned a weight. The PCST solving algorithm is used to find where in the image, clothes are present. The clothing regions are identified as those sub graphs whose sum of scores is maximum. The clothing recognition is done by using the style elements feature vectors of a clothing region and passing it to an SVM classifier [5].

   2. Image processing method to find clothing region [6]: The initial RGB is converted to Grey-scale to remove the background. It is then converted to a binary image to make operations more efficient. Binary erosion and dilation is applied to the image which removes false noise regions and still preserves the topology and shape of the clothes. A corner detection algorithm is applied to the binary image. Edge detection of clothes algorithm is appied to the pixel which is indicated as an object using a threshold method based on the neighboring pixels to change the center of the pixel. The center pixel is now compared with the sum of square of the remaining pixels in the window and using a threshold, it is determined if the center is a clothes edge pixel or not. Corner detection is done by finding the intersection of the edges [6].

   3. Clothes detection using region growing [7]: Clothing categories being high-level semantic concepts, various clues which are relevant to recognize them, such as a persons gender and age, uncovered skin areas, color and

texture features of clothes are used. First, face detection and tracking for each frame is performed and the detected faces are aligned. For clothing segmentation, a light weight region growing based clothing segmentation is performed. After that, multi-class linear SVM classifiers are employed to learn the clothing categories. The candidate rectangular region containing a person is segmented into roughly homogeneous color segments and then knowledge of foreground and background is applied to extract the foreground or clothes. Using Vornoi image, seed points are automatically placed at the center of homogeneous regions. The intensity at each point on a Voronoi image is the distance to the closest edge.

The peaks thus indicate the farthest points from the contours. The canny edge detection on grey-level images is conducted using the local maximums on the Vonoroi image as seeds for region growing. Next, L2 norm of two color pixels in the LAB color space is used to measure their discrepancy and obtain rough homogeneous color segments after region growing. The color pixels in one segment are modeled as a Gaussian distribution, and then two adjacent segments are merged if the Mahalanobis distance of two color distributions is small. Small color segments are merged to the color segments next to them that are most similar. Prior knowledge of foreground and background is applied to extract foreground clothing. A linear SVM is learnt to find recognize these clothing regions [7].

Fig. 5. Real Time Clothing Recommendation

1. Literature related to Algorithms:

   1. Latent SVM:

      It assume that (x, y) pairs is not enough for describing the input-output relationship, but this relationship depends also in unobserved latent variables z. In [1] the model integrates four potentials- visual feature vs attribute, visual feature vs occasion, occasion vs attribute and attribute vs attribute. Embedding these matching rules into the Latent SVM, we ensure that recommended clothes are satisfying the requirement of wearing properly and aesthetically.

   2. Ridge Regression:

      This algorithm tries to find a weights vector for the data- points it encounters to find its score. It attempts to minimize the norm of the weight vector. Although it is not a classification algorithm, it can be utilized to derive results on unseen data which can be compared to an artificial threshold to derive the class. In the given system, a threshold of 0.5 is used [2].

   3. Standard Neural Network:

      Standard neural networks or connectionist systems are computing systems inspired by the neural networks in the human brain. They systems "learn" by considering a large amount of data. It consists of a collection of connected nodes called neurons, which are similar to the neurons in the brain. In the given system, a neural network with two hidden layers is trained using stochastic gradient descent or batch training and using the sigmoid activation functions [2].

   4. Siamese Neural Network:

      A Siamese Neural network is an Artificial Neural Network that works on two inputs simultaneously and computes comparable output vectors. One of the output vectors is

      precompiled, forming a baseline with which the other output vector is compared [2].

   5. Fuzzy Logic:

      Fuzzy logic is a computing approach in which the truthfulness of an object belonging to a set I computed rather than a rigid True or False on which the modern computer is based. Fuzzy rule is used to determine the clothes that are suitable for a given skin color [3] [4].

   6. LVQ Neural Network:

      The system uses LVQ Neural Network for Skin Detection. LVQ is a special case of an artificial neural network. It applies a winner-take-all Hebbian learning-based approach. It is a precursor to self-organizing maps (SOM) and related to neural gas, and to the k-nearest neighbor algorithm (k- NN).The Learning Vector Quantization algorithm (or LVQ for short) is an artificial neural network algorithm that lets you choose how many training instances to hang onto and learns exactly what those instances should look like. LVQ was developed and is best understood as a classification algorithm. Predictions are made using the LVQ code-book vectors in the same way as K-Nearest Neighbors. Based on described situation of the problem, color components are three input features for each pixel to the input of the network. The output layer also has two neurons indicating the classes of skin and non-skin that will be triggered based on class of input sample [12].

   7. Fuzzy C Means Clustering:

      This algorithm assigns a membership to each data point in to a cluster depending upon its distance to the cluster center. Closer the data-point to the cluster center, higher is its membership to that cluster. This clustering is used in the described system to find the fuzzy rules for various skin colors [4].

   8. Profit Collecting Steiner Tree:

   In an undirected graph, where vertices are assigned non- negative weights and with the edges E associated with non- negative costs. The Linear Profit-Collecting Steiner Tree problem (PCST) involves finding a sub-graph that maximizes profit which is the difference between sum of prizes at nodes and sum of costs of edges in the sub graph. In the given system, each super pixel forms a node and the profit associated is found from the local features extracted from each super pixel [5].

2. Merits and Demerits of Systems:

1. Myntra:

   The system does not take into account personal user attributes like skin color and existing user clothes to optimize recommendations.

2. Your Closet App:

   Matches each clothing type with all other clothing types, i.e. each shirt will be matched with each pant. Does not find what clothes are actually matching depending upon color, texture, etc. The system is not automated and requires users to organize their own closets.

3. Magic Closet:

   The system is designed for a Kinect which may not be available to all users. It recommends clothes from online shops than from users existing apparels.

4. Which clothes to wear confidently?

   This approach utilizes Deep Learning and Machine Learning to check if clothes are matching or not and provides satisfactory results. In this system only shirts and ties are matched but this approach can be expanded to a larger variation of clothes.

5. Skin and Clothes Matching Seeded by Color System Selection [3]:

   The system takes support of a questionnaire of common people to develop the fuzzy rules. A more robust method can be adopted. A merit of this system is the usage of the YCbCr color space to eliminate the effect of skin color sensitivity to light.

6. Discerning Advisor:

   In this approach fuzzy c-means clustering is used to map an individual to a skin color and recommend clothes for that skin color recommended by a professional fashion designer. A drawback in this system is the usage of the RGB color space which intensifies the effect of skin color sensitivity to light [3].

7. Garment Detectives:

   The system is satisfactory in the clothing region segmentation but accuracy of clothing recognition can be improved.

8. Identifying Corners of Clothes using Image Processing Method:

   This system requirs user to input two images- one of the background without the clothes and the other of the background with the clothes. This is one disadvantage of this system. The algorithm also does not attain intended results in adverse conditions and thus requires further improvement.

9. Real-Time Clothing Recognition from Surveillance Videos:

   The system gives satisfactory results for real-time applications but we aim to further improve the accuracy of the clothing recognition.

10. Personalized Clothing Recommendation Based on Knowledge Graph:

The system recommends clothes using a knowledge graph and the recommendations are dependent on user attributes which is desirable. This system creates the knowledge graph and clothing ontology artificially instead of extracting automatically.

III. PROPOSED SYSTEM

Fig. 6. Proposed System

Fig. 6 depicts the Proposed System which consists of different modules. User is the one who uses the system. Interface will be provided to the user to access the system. User can give the following inputs. He can upload photos to insert clothes in his digital wardrobe which will be the cache memory of his mobile phone. He needs to provide his personal attributes like age, gender, skin tone etc. He can also give his preference setting like favorite color, brand etc. The interface will have all the features. By using the features as required by the user, he can get the intended results. The system will detect clothes from the image, crop the useful part and do the needful alignment. The system will also do the segmentation, classification and categorization using different modules trained and tested by the dataset. The CNN and RNN models will extract the features from the images and give output in the feature vector format. This output will be used by different algorithms. The system has matching and recommendation models which will provide the user with best case matching combinations as well as useful recommendations. This model will work in unison with the preference model which contains user attributes and give the user specific output. The system will perform different functions in the back-end. These functions are directly related to the features provided to the users. Once the user wants a particular feature it will trigger the system to do a particular function to bring the intended result and then this result will be sent to the interface. The CNN will play a supporting role to the system. All the clothes, attributes and preference setting provided as an input by the user will be used in producing the intended output. Also, the trained models inside the system will use this particular users input to produce the desired results.

CONCLUSION

The report gives an analysis of multifarious systems, their approaches and methodologies, algorithms, technologies that have been used earlier as well as their results. This review paper is demonstrating a comparative study between them, understanding their efficiencies and accuracy of methods being used, their merits and demerits. After having the detailed analysis, the paper also proposes a system that amalgamates the various approaches and provides features of clothes matching and clothes recommendation .This review paper helps us in understanding the basic building blocks of the system that we would be developing in future, using our own algorithm and some of the existing solutions.

REFERENCES

1. Liu, Si; Nguyen, Tam; Feng, Jiashi; Wang, Meng; and Yan, Shuicheng, "H i, Magic Closet, Tell Me What to Wear!"(2012). InComputer Science Fac ulty Publications. Paper 66.

2. Paisios, Nektarios, L. Subramanian, and A. Rubinsteyn. "Choosing which clothes to wear confidently: A tool for pattern matching."( 2012) In Workshop on Frontiers in Accessibility for Pervasive Computing. ACM.

3. Kurnia, Rahmadi & Silvana, Meza & Elfitri, Ikhwana. Skin and Clothes Matching Seeded by Color System Selection. (2015) TELKOMNIKA Indonesian Journal of Electrical Engineering. 14. 10.11591/telkomnika.v14i1.7235.

4. M. A. Akbari, H. Takahashi and M. Nakajima, "Discerning advisor: an intelligent advertising system for clothes considering skin color," (2005) In International Conference on Cyberworlds (CW'05), Singapore, 2005, pp. 8 pp.-209.

5. S. C. Hidayati, K. Hua, Y. Tsao, H. Shuai, J. Liu and W. Cheng, "Garment Detectives: Discovering Clothes and Its Genre in Consumer Photos," (2019) In IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), San Jose, CA, USA, 2019, pp. 471- 474.

6. Yew Cheong Hou and K. S. M. Sahari, "Identifying corners of clothes using image processing method," (2010) In International Conference on Intelligent and Advanced Systems, Manila, 2010, pp. 1-5.

7. M. Yang and K. Yu, "Real-time clothing recognition in surveillance videos," (2011) In 18th IEEE International Conference on Image Processing, Brussels, 2011, pp. 2937-2940.

8. Y. Wen, X. Liu and B. Xu, "Personalized Clothing Recommendation Based on Knowledge Graph,"(2018) 2018 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, pp. 1-5.

9. O'Connell, L. (n.d.). Topic: Apparel Market Worldwide. Retrieved August 30, 2020, from https://www.statista.com/topics/5091/apparel- market-worldwide/

10. Zhang, Yan & Liu, Xiang & Shi, Yunyu & Guo, Yunqi & Xu, Chaoqun & Zhang, Erwen & Tang, Jiaxun & Fang, Zhijun. (2017). Fashion Evaluation Method for Clothing Recommendation Based on Weak Appearance Feature. Scientific Programming. 2017. 1-12. 10.1155/2017/8093057.

11. Abigale J. Stangl, Esha Kothari, Suyog D. Jain, Tom Yeh, Kristen Grauman, Danna Gurari. "BrowseWithMe", Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility – ASSETS '18, 2018.

12. Learning vector quantization. (2020, February 24). Retrieved August 30, 2020, from https://en.wikipedia.org/wiki/Learning_vector_quantization.