Detecting Mask and Unmask People from Crowd using Deep Learning

Detecting Mask and Unmask People from Crowd using Deep Learning

Muhammad Nadeem1, Hamza Ejaz2, Asadullah Ikram3, Muhammad Arsalan Khan4, Khizar Hayat Hashmi5

1Assistant Professor, Department of Computer Science, Sir Syed University of Engineering & Technology, Karachi, Pakistan

2,3,4,5 Department of Computer Science, Sir Syed University of Engineering & Technology, Karachi, Pakistan

Abstract: This study or investigation tends to aim for the detection of face masks and sums or counts on the Picture feed through the utilization of a mainframe vision method known as “Machine Learning and Object Detection”. Open CV and TensorFlow were utilized for creating a CNN model for the detection of face masks. While it was proficient on a dataset that comprised of about three thousand and eight hundred images. Created in a (Corona Safety App) android application, where an operator will be acknowledged and can observe the violations. For accomplishing such a purpose, Postgres was utilized as the backend facility. The operator or user will capture the image from the mobile application along with the existing location, it also sustained the images in the mobile application named gallery. It will upload the image to a c Database with a date, time, and existing location. If the margin of unmasked entities is huge it creates an alert as a notification on the operator device with a low, medium, and high prediction margin of spreading the virus (Covid-19). The application tends to provide several functionalities including taking live location, viewing history, and saving the image to the device. A heat chart can also be regarded which marks the areas with a huge crowd, which can also deliver support to officials recognize the locations and areas that required to be disinfected and sanitized more generally. User can observe their results on web view also.

Keywords: Machine Learning, NumPy, Postgres, TensorFlow, OpenCV

1. INTRODUCTIONThe era of 2020, has fetched us several complications and challenges, specifically in the operating subdivisions. Several developments are converting to operate from home- centered culture. There still exists a few commercials that unlocked or opened after the lockdown due to Covid-19 and enduring its general working including universities and schools, restaurants, sectors of construction, and a small numeral of offices entirely certifying the safety of workplaces like sanitizing hands and wearing masks. The health consultants are operating tough and unremarkable on certifying that these commercials admire all the agreements and protocols for seeking their employees safely. Some health authorities also conduct unvarying inspections and even close these developments, if they observe any sort of violation in protocols for safety. Several among these developments are operating through the computerization and automation of detection like the one breaches or violations, thus reduction in the labor and time spent for similar. The chief objective associated with this investigation is involved in detecting the violations including not trying to wear masks in the company and delivering the notification to officials through utilizing a mobile app.The technology comprises advancements tremendously over the recent era, everything initiating from IoT (Internet of Things) (Raj, 2021) to deep learning and machine learning. CNN is involved in the utilization of several fields, which comprises marine science (Sung, 2017), medical (Duran- Lopez, 2019), and a huge numeral of other applications (Hansen, 2017) and includes a projecting domain associated with machine learning. This plan was imposed through the utilization of TensorFlow and OpenCV where the Convolutional Neural Network (CNN) model was proficient for the detection of facemasks. CoronaSafety App is a mobile application that was made through the assistance of Postgres which is considered as a backend facility imposed for saving images that will be detected through the android app, from where the operator will comprise the capability to observe them on their application.

   The rest of this study is organized in the certified and organized way in which “Section 1” is about providing a transitory and brief introduction associated with the project. While “Section 2” is about the reviews on some previous works associated with this investigation and project. “Section 3” tends to explain the methodology and imposition of the project. “Section 4” of this study acknowledges the backend facility in this project named Postgres with location, time, and date. “Section 5” involves deliberation about the Android Application. “Section 6” involves the discussion about the applications of the project. While is “Section 7” is the conclusion and delivers understandings about the scope of work in the future. The last section is Section 9 which contains a list of references. This segment or section of this project involved reviewing some of the associated works that impose CNN along with the assistance of TensorFlow, OpenCV, and NumPy for the intention, calculation, and betterments associated with object detection.

2. RELATED WORKAkanksha Soni et al. (Soni, 2020) created a model that notices or perceives whether an individual is trying to wear a helmet in an actual instance, thus detecting any defilement or violation. The plan was also imposed through the support of OpenCV, TensorFlow, and NumPy. Their planned model displayed huge betterments, which were associated with a few recent models that delivered falsified predictions whenever an individual garbs clothes throughout the face. In this way, they attained an entire precision of more than ninety percent when evaluated. Chen et al. (Chen, 2020) imposed a model through the assistance of TensorFlow for the identification of ID card numerals. With the support ofOpenCV, the picture comprises an ID card that is pre- managed and the numeral that exists on the ID card is identified and given as consequences with support of a proficient model of CNN.

   When evaluated or tested, it was seen that working out speed is rapid and the precision is huge. Emily Caveness et al. (Caveness, 2020) created TFDV (TensorFlow Data Validation), which tends to provide an ascendable solution for the investigation of data and authentication for ML (Machine Learning). It is organized that in creation which is connected with (TFX) TensorFlow Extended, which is also considered as an end-to-end platform of Machine Learning. Their system has attained several pulls ever since they unlock tracked their plan. Other open-source systems generated for data validation include Apache Spark integrated with built-in components for flowing and also comprises a rapid, comfortable for the utilization of system for processing of huge data. In the era of 2005, Travis Oliphant developed NumPy through the incorporation of several functionalities of competing Numarray into Numeric, with huge betterments and alternations. NumPy is considered open-source software and comprises a huge numeral of contributors. That open-source software operates with objects of python known as the multi-dimensional arrays.

   Arrays are generally considered as the allocation of values and they comprise one or multiple dimensions. NumPy array data structure, which is also known as the ndarray small for the n-dimensional array. Through the utilization of NumPy, logical and mathematical operations associated with the array can be practiced. This discussion group delivers an explanation about the basics of NumPy including environment and arhitecture. It also involves the deliberation of the several functionalities of arras, sorts of indexing, and some additional. As an overview to the Matplotilb is also delivered. NumPy completely delivers assistance to an object-oriented model, initiating, once more with ndarray. For instance, ndarray is a class, having several attributes and techniques. Several of its modes are reflected through functionalities in the most external. NumPy namespace delivers certifications to programmers for coding in whichever paradigm they offered in preference. This elasticity has certified the NumPy array class and NumPy dialect for becoming the de-facto semantic comprises multi- dimensional interchange of data utilized in Python.

3. METHODOLOGY AND IMPLEMENTATIONFigure 1: Project Architecture
   1. Face Mask DetectionThis plan tends to utilize OpenCV and TensorFlow for the training of a CNN model for the detection of facemasks.TensorFlow and OpenCV:

      TensorFlow is considered an open-source stage that is utilized for ML, developed by the team of Google Brain. It is hugely employed for multifaceted numerical calculation and computation that folders collectively a group of algorithms and models comprised of deep learning and machine learning. It can be utilized for several applications including classification of handwritten digits, recognition of image, detection of the object, and general processing of language (Natraj, 2019) through operating and working out neural networks. OpenCV comprises more than forty-seven thousand entities of operator community and evaluated numeral of transfers exceeding18 million. The library comprises more than twenty-five hundred enhanced algorithms, which contains a huge collection of both classic and scale-of-the-art prophecy and visualization for computer and algorithms of machine learning.

      How Does TensorFlow Work?

      With the assistance attained through TensorFlow, producers or developers can develop graphs associated with data flow, which are understood as frameworks that display how statistics or data passes through graphical representations or a sequence of nodes. Seeking of every node as the mathematical node and every edge demonstrating a multi- dimensional array of data or a tensor. This can be preferably imposed in the language of Python where such sensors and nodes are practiced as objects. Yet, the precise mathematical operations are practiced in binaries of C++, which displays an ideal performance; Python involves in looking after of direction of traffic and allocates them for operating collectively as a unit.

      TensorFlow can be operated on several platforms including a local computer or machine, in a cloud, iOS, CPUs or GPUs, and Android devices. TensorFlow can also be operated on custom TPUs (TensorFlow Processing Unit) of Google. The worked-out models can be operated or run on any arrangement for the prediction of consequences. TensorFlow 2.0 which comprises the releasing data in October of 2019, created several necessary alterations from the feedback of operators. With the assistance of another open-source platform of TensorFlow Lite, it is latent to work out or train models on several sorts of devices.

   2. Training the Face Mask Detection ModelThis study or project comprises the utilization of a model known as “CNN (Convolution neural network)” that is especially involved in the detection of facemasks. The CNN model tends to attain in the parameters (input) pictures as an edge through video, practices the picture, and divides that under dual classifications including no mask and mask. This network or model was worked out through the utilization of thirty-eight hundred images. While nineteen hundred each for both categories of no mask and mask.Figure 2: Network Architecture

      The CNN (Convolutional Neural Network) comprises of the two convolution covers or layers each admired through the functionality of relu activation and pooling covering of max. Figure 2 tends to display the architecture of the network. Rectified Linear Unit (Relu Function) is presented for the non-linearity in the CNN network. It is shown as:

      F(x) = max (0, x) (1)

      Pooling layers or coverings will cause a reduction in the numeral of parameters of the pictures are large. Spatial pooling also mentioned as the down-sampling or sub- sampling delivers assistance for reducing the dimensionality of every map creating certified significant understandings that are retained.

      • Sum Pooling
      • Max Pooling
      • Average Pooling

      Among these mentioned types, Max pooling will attain the greatest element from remedied functionality map. The statistics and data are then privileged or flattered through the conversion of it to the 1-dimensional array, which is approved or passed as the parameters to the final covering of output. To support security over appropriate, the network tends to ignore a particular margin of neurons during the workout. In such sort of circumstances, the network falls out about fifty percent of neurons. These components are not understood during some backward or forward permit.

      The final covering of output tends to attain the values and covers them into possibility allocation; this is attained through the assistance of functionality or feature of softmax. This feature is supportive when it arrives in the classification of issues. The last forecast class encompasses the item in the entire list whose self-reliance ranks the highest. It is represented as:

      Figure 3: Softmax Activation Function.

      The last forecast will be centered on the class that comprises the utmost possibility. The model was then obeyed through the utilization of Adam optimizer and precision of more than ninety percent was observed in the collection of validation. Figure 4 plans the precision or accuracy and loss correspondingly (Goodfellow, 2016) (Belciug, 2020).

      Figure 4: Accuracy and Loss

   3. DetectionOpenCV was utilized specifically for capturing the video and the worked-out model was encumbered or loaded through the utilization of TensorFlow. A pre-trained model was utilized particularly for the dictation of faces in an audiovisual. The weights were started from the outline file with the support of OpenCV. After attaining the bounding pacts of the entity’s faces in the edge, that location of interest is collected out from the chief frame, redesigned, and is then approved to the model.Figure 5: Detecting Face masks.
   4. Heat Map

   A heat map is implied for the detection if individuals are thronging in a certain region and hence causing violations in the social distancing standards. Through seeking at the heat map on their application, it can be supportive for the officials in recognizing the regions that required to be clean more generally. For the planning of the heat map, the initial frame is read and in each iteration, the packs are designed on the frame. Red packs or boxes comprises densely occupied and sparely occupied areas are shown by the boxes with blue color. When the program tends to end this picture is saved which is then directed to Firebase storage.

   Figure 6: Heat Map showing densely and sparsely populated areas.

   1. Introduction
4. POSTGRESof his aims. Secondly, an end operator is provided with a particular language of the query in several decision supportTraditional relational assistance of DBMS is a data model comprising of a group of certified associations, each feature of which comprises a certain sort. In existing systems of business, probable modes are floating fact numerals, character strings, dates, and integers. It is generally identified that this sort of data model is inadequate for data practicing of non-business applications. In planning and designing an innovative model and inquiry language, we were directed through the three criteria of design given below:

   Figure 7: Data has been shown in postgresql datbase.

   1. Storage SystemThe database will be moderately stored on a magnetic disk and moderately on an archival medium including an optical disk. The statistics or data that exists on magnetic disk comprises entirely recent database tuples and secondary indexes. The archival medium in the form of an optical disk is reserved on an archival stock comprising historical tuples. There will be an expert that tends to vacuum the tuples from magnetic disk to archival medium as a procedure in the background. Statistics and facts that exist on the magnetic disk will be saved utilizing the general file system of UNIX with an association per file. The optical disk (archival medium) will be arranged as one huge source with tuples from several associations intermixed.All the associations will be saved as the heaps (similar to ASTR76I) with a selection set of secondary indexes. Furthermore, associations can be presented as “nearly ordered” and POSTGRES will try to keep tuples near to arrange sequence on a similar column. In last, the secondary indexes can be represented, which comprise of two distinguished corporal indexes one confined for the tuples of magnetic disk and one for the tuples of the optical disk, each in a distinguished file of UNIX on a magnetic disk. Furthermore, a secondary index will mechanically or automatically be delivered for entire associations in a unique identifier sector, which is explored in the coming subdivision. This sort of index will certify any relation to be sequentially skimmed.
   2. Fast PathThere are three chief reasons associated with choosing the implementation of the functionality of the fast path. Initially, an operator who tends to wish to collaborate with a database through the execution of a sequence of features to steer to expected data can utilize a fast path for the accomplishmentapplications. In those circumstances, it is generally comfortable for the developers of the application to create a parse tree demonstration for a question despite the ASCII one. Yet, it would be considered wanted or desirable for the designer of the application to comprise the capability to straightly call the POSTGRES executors or optimizers. Several DBMSs’ do not deliver certified straight or direct access to the modules of the internal system.

      The third reason for choosing a fast path comprises some complications in it. In the tenacious CLOS covering of Picasso, it becomes important for the run-time processing to give a unique identifier in the form of OID to each created object that is tenacious. It is unwanted for the system to synchronously, put every object straight into the database of POSTGRES and thus gives a POSTGRES recognizer to the object. Such practice would consequences in poor practicing in the execution of a tenacious program of CLOS. Despite this, tenacious CLOS sustains a cache of objects in the space of address in the program and just gives a persistent object into such cache synchronously. There are huge chances, which involve in controlling how the cache is inscribed out to the database at an advanced instance. Unfortunately, a tenacious object must be provided a unique certain identifier at the instance it comes into the cache.

      If tenacious CLOS tends to give certain identifiers, then there will be complicated planning that must be practiced when objects are inscribed out to the database and actual POSTGRES certain identifiers are allocated. Adversely, tenacious or persistent CLOS must sustain its private system for special identifiers, self-governing of the POSTGRES one, strange copying of effort. The resolution selected was to ensure tenacious CLOSS to admittance the routine of POSTGRES that tends to assign special identifiers and permits it to pre-assign N POSTGRES object recognizers which it can then assign to cache those objects. Lastly, these objects can be inscribed to the database of POSTGRES through the utilization of pre-assign special identifiers. When the source of identifiers is tired, tenacious CLOS can appeal to another set.

      In all of the given examples, a software program needs straight admittance to a user-presented or internal functionality of POSTGRES, and hence the query language of POSTGRES has been stretched with param-list (function- name). In the given situation, besides operating inquiries in POSTQUEL, an operator can ask that any function recognized or known to POSTGRES be displayed. This function can be considered one that an operator comprises previously presented as a general, operator or function of POSTQUEL or it can be comprised in the impositions of POSTGRES. Yet, the operator can straightly call the executor, optimizer, parser, or the modes of access, utility routines, or buffer management. Additionally, an operator can present functions, which in turn create calls on internal POSTGRES. Similarly, the user can comprise with valid and considerable control over the low-standard flow of control, much as is attainable through a toolkit of DMS including Exodus [20], but in absence of all efforts associated with the configuration of a tailored DBMS from that toolkit.

      Furthermore, should the operator try to collaborate with his database through creating a grouping of calls of function (technique of invocations), this service certifies the probability. As observed in section 1 (introduction), we do not comprise expectations from this interface to be specifically famous. The above ability is known as a fast path, it is because it delivered straight access to a certain function. As such, it is efficaciously a distant process call service and certifies a program of the user for calling function in another space of address despite its personal space of address.

5. ANDROID APP AND WEB APPThe app was developed in Java with the help of Android Studio, which is built on JetBrains’ IntelliJ IDEA software. The main interface of the app is the login/signup page where the user has to login first if they do not have their account, so the user has to sign up.Figure 8: Login/Signup page.

   When the user login the app the first interface is shown where the user can see their images in the gallery. User can open the camera and click the pictures, where the current location is getting through the mobile, and see the result with a total number of with mask and without a mask and the captured images stored in the database along with the date, time and location.

   Figure 9: Gallery view

   Figure 10: Camera view

   Figure 11: Single user Result.

   Figure 12: Multi-user Result

   Figure 13: triple user Result

   Figure 14: Quad user Result

   Figure 15: Current location

   Figure 16 Alert notification

   Authorities can see the captured images with mask and unmask on the dashboard in the web app along with the date, time, and location.

   Figure 17: dashboard for authorities

   Figure 18: Alert

6. ANDROID APP AND WEB APPOur research shows that multiple people from the mob send the current location along with the date and time to the server. Authorities’ check which location has most unmask people gather.
7. APPLICATIONSHospitals and Clinics: It is considered as the significant aspect that tends to require certain intentions during a particular sort of pandemic. To deliver support to the healthiness and health conditions of nurses and doctors, this framework can be imposed for the detection of whether an individual is trying or wearing a mask or not. In this system, a CCTV camera can be included specifically for the detection of masks and violations associated with regulations of social distancing, which can sequentially deliver support in the reduction of crowding while the individuals (patients) are looking in queues for their approvals of appointment.Airports: Several endpoints and destinations around the whole globe have initiated easing restrictions associated with traveling. This sort of framework can also be imposed in places like airports where indiiduals are looking or waiting at the gates of security clearance, check-in facilities, waiting lobbies, passport control, etc.

   Shops / Workplaces: As several commercials and

   organizations are physically opening their businesses specifically after the lockdown this framework can be imposed for the certifications of employees and consumers who are admiring all the safety agreements associated with Covid-19.

8. CONCLUSION ANDFUTURE SCOPE

The project or investigation has been conducted to explore an effective technique associated with the detection and provide the announcement to the officials, if an individual does not follow the Covid-19 safety agreements or protocols, in any business progression or organization. In the study, a workout model has been created for the detection of face masks specifically in crowds through the utilization of NumPy, TensorFlow, and OpenCV. The expected framework associated with the CNN model includes two convolutional coverings directed through the functionality of relu activation and a mix-pooling covering. Through the assistance of OpenCV, users comprise the capability to capture the video feed by distinguished sources including a video file, IP camera, or webcam. A mobile or android application was made which will attain notification each instance a violation associated with mentioned requirements tends to occur and be detected, and the images attained through detection can also be observed through that application. It was attained through the assistance of the facility and functionality of Postgres. As a study in the coming era, students comprise the capability to operate on evaluating an aspect for detection or prediction of the instance at which it attains packed the huge and the functionality associated with heat map can conspire more precisely.

REFERENCES

1. 1. Soni, A., & Singh, A. P., 2020. Automatic Motorcyclist Helmet Rule Violation Detection using Tensorflow & Keras in OpenCV. In 2020 IEEE International Students Conference on Electrical, Electronics and Computer Science (SCEECS).
   2. Chen, S., Wei, Y., Xu, Z., Sun, P., & Wen, C., 2020. Design and Implementation of Second-generation ID Card Number Identification Model based on TensorFlow. In IEEE International Conference on Information Technology, Big Data and ArtificialIntelligence (ICIBA).
   3. Caveness, E., C., P. S., Peng, Z., Polyzotis, N., Roy, S., & Zinkevich, M., 2020. TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data.
   4. Lu, Y., Zhang, L., & Xie, W., 2020. YOLO-compact: An Efficient YOLO Network for Single Category Real-time Object Detection. In 2020 Chinese Control and Decision Conference (CCDC).
   5. Ullah, M. B., 2020. CPU Based YOLO: A Real-Time Object Detection Algorithm. In 2020 IEEE Region 10 Symposium (TENSYMP).
   6. Raj, A., Maji, K., & Shetty, S. D., 2021. Ethereum for Internet of Things security. In Multimedia Tools and Applications.
   7. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A., 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
   8. Natraj, L., & Shetty, S. D., 2019. A Translation System That Converts English Text to American Sign Language Enhanced with Deep Learning Modules. In International Journal of Innovative Technology and Exploring Engineering Regular Issue, 8(12), 5378- 5383.
   9. Khawas, C., & Shah, P., 2018. Application of Firebase in Android App Development-A Study. In International Journal of Computer Applications, 179(46), 49-53.
   10. Fatima, N. S., Steffy, D., Stella, D., & Devi, S. N., 2020. Enhanced Performance of Android Application Using RecyclerView. In Advances in Intelligent Systems and Computing Advanced Computing and Intelligent Engineering, 189-199.
   11. Nair, L. R., Shetty, S. D., & Shetty, S. D., 2018. Applying spark based machine learning model on streaming big data for health status prediction. In Computers & Electrical Engineering, 65, 393- 399.
   12. Duran-Lopez, L., Dominguez-Morales, J., Amaya- Rodriguez, I., Luna-Perejon, F., Civit-Masot, J., Vicente-Diaz, S., & Linares- Barranco, A., 2019. Breast Cancer Automatic Diagnosis System using Faster Regional Convolutional Neural Networks. In Proceedings of the 11th International Joint Conference on Computational Intelligence.
   13. Sung, M., Yu, S., & Girdhar, Y., 2017. Vision based real- time fish detection using convolutional neural network. In OCEANS 2017 – Aberdeen.
   14. Hansen, D. K., Nasrollahi, K., Rasmusen, C. B., & Moeslund, T.B. (2017). Real-Time Barcode Detection and Classification using Deep Learning. In Proceedings of the 9th International Joint Conference on Computational Intelligence.
   15. Goodfellow, I., Bengio, Y., & Courville, A., 2016. Deep learning, Cambridge (EE. UU.). MIT Press.
   16. Belciug, S., 2020. Artificial intelligence in cancer: Diagnostic to tailored treatment, London, United Kingdom. Academic Press.