Offline Handwritten Text Recognition and Signature Verification

Offline Handwritten Text Recognition and Signature Verification

Rehaan Sajjad Arai1, Skanda Shanubog2, Rithik Jain3, Pushkar Kumar4, Krupashankari S5

1,2,3,4 UG Student, Dept. of Information Science and Engineering Dayananda Sagar College of Engineering

5Assistant Professor, Dept. of Information Science and Engineering, Dayananda Sagar College of Engineering, Bangalore, India

Abstract – Offline Handwritten Text Recognition (HTR) is one of the most interesting challenges in todays date in the field of Image processing. This paper introduces a novel technique to recognise the handwritten text by using Convolutional Recurrent Neural Network along with Connectionist Temporal Classification. This model makes use of the IAM dataset. Offline Signature Verification (SV) is another challenging task in the field of biometrics. This paper demonstrates a novel technique to verify the signature as an original or forged one, and makes use of the Convolutional Siamese network.

Key Words: Convolutional neural network, connectionist temporal classification, IAM database, convolutional siamese network, CEDAR dataset.

1. INTRODUCTION

   Handwritten documents are widely used even today due to various reasons. Some may prefer writing documents down instead of typing it on the computer, because they might not know how to do it on the computer or for some personal reasons. Later, they might find it very difficult and challenging to move the text to digital format. Every person has a different handwriting and it becomes difficult to interpret some peoples handwriting. It is easier to interpret the digital form of the text, compared to handwritten text. A manual approach can be adopted by a person to type each and every word in the handwritten text document, so that we can obtain the digital form of the text. This process is tedious and not feasible, if the document is very huge. Most of the existing systems today, can predict only one character or word at a given time. The novel technique used in this paper, helps us to predict a given line, which reduces the time and the segmentation required. This paper brings our focus on offline handwritten text recognition using deep learning and sequence learning. This paper shows some of the novel techniques to recognise the handwritten text into a machine editable text. Signature Verification is another important task in the field of biometrics and a popular research topic today. Manual verification of signatures is really a very difficult and errorprone task. It is really necessary to automate the task of offline signature verification to categorize the given signature as an original or a forged one. This paper makes use of the Convolutional Siamese Neural Network to classify the signature as a genuine signature or a forged signature based on the euclidean distance calculated.

2. RELATED WORK

   There has been vast research that is going on in the field of text recognition and signature verification. These systems are gaining popularity worldwide and being developed by many. Some of the systems that we have studied and referred to are mentioned. Hong-Phuong Tran et al. [1] made use of a

   Convolutional Recurrent Neural Network which consists of convolutional block for the purpose of image feature extraction, recurrent block for sequence learning and also a labelling block with Connectionist Temporal Classification. The methodology used by them was that the input image is given to the CNN, which generates an image feature vector, which is given as an input to the RNN and generates a probability matrix, which is given as an input to the CTC layers, which then gives the final result which is the predicted sequence. A comparative study was made on different handwriting recognition techniques like character recognition, Support Vector Machine, K-Nearest Neighbour, Multi-layer perceptron neural network, Back Propagation, and Gated Recurrent Unit [2]. The authors in [3] have made use of two different model architectures – LSTM based model and Recurrence free model. The proposed model consisted of GRCL (Gated Recurrent Convolutional Neural Network) block which performed better than the LSTM (Long Short-term memory) layers when feedforward networks are chosen over recurrent networks. The proposed system in [4] used the following methods to recognize characters in Tamil and English languages – pre-processing, segmentation, feature extraction, and classification and recognition. The authors in

   [5] have used Keras for the classification MNIST dataset in which feed-forward neural network and convolutional neural network were used for feature extraction. It was observed that convolutional neural network performed better than the feed- forward neural network. The authors in [6] have made use of OpenCV for image pre-processing, TensorFlow for training Convolutional Neural Network, trained on MNIST dataset, but the system could recognise only letters and digits but not special character. The authors in [7] have made use of feed- forward back propagation neural network. The authors in [8] have made use of convolutional neural network, TensorFlow, and soft-max regression to assign probabilities to handwritten characters between 0 and 1.

   A faster region-based convolutional neural network (R- CNN) was made use of in [9], which is based on object- detection algorithm. The authors in [10] have made use of Convolutional Neural network based on TensorFlow framework on the MNIST dataset to recognise handwritten digit. The authors of [11] made use of convolutional neural network on handwritten text containing struck-out words, trained on the IAM dataset. The authors in [12] have made use of convolutional neural network, bi-LSTM layers, and temporal convolutional network to recognise digits in a three- dimensional space acquired from a Leap Motion Controller. In the papers [13], [20] the authors have developed an offline writer-independent signature verification system using

   convolutional siamese network, by calculating the euclidean

   distance between the pairs. The authors of [14], [18] have made use of the Convolutional Neural Network model, and ReLu activation function to classify signature as original or forged one. The authors of [15] have adopted a novel technique to verify signature based on text recognition, on the GPDS dataset, and it calculates the Mahalanobis distance based on correlations between signatures, instead of the Euclidean distance. The authors of [16] have made use of the Convolutional Neural network for comparing a writer- independent and writer-dependent system making use of various features and algorithms. The authors of [19], have made use of a siamese network on various datasets like CEDAR, GPDS etc. The authors of [17] made use of a template-based and template-matching technique where a two-level verification system is used based on template clustering, which calculates a similarity score matrix. The authors in [21] have made use of neural network technique in classifying the signatures as original or forged.

3. PROPOSED SYSTEM

   1. Handwritten Text Recognition

      The proposed Convolutional Recurrent Neural Network (CRNN) consists of sequential blocks of convolutional layers – for image feature extraction, recurrent layers – for sequence learning, and CTC layer.

      1. Image feature extraction: The Convolutional Neural Network is the first block that is used for the step, image feature extraction. The main goal is to reduce the image to a form which can be computed easily and still retaining the critical predictive features. With this block, we can extract the features of he image. The Convolutional Neural Network block consists of seven layers. The first, second, fourth and sixth layers are followed by a max-pooling layer, which is responsible for down sampling which means it reduces the dimensionality of the matrix. The other layers are followed by batch normalization which is responsible for reducing the internal covariate shift. Every convolutional layer is activated by the Rectified Linear Unit function.

      2. Sequence Learning: After the convolutional layers, the output, that is, the image feature vector, which is one- dimensional and represents a two-dimensional image matrix, is given as an input to the next block that is, Recurrent Neural Network. The proposed system makes use of two bi-directional LSTMs, each consisting of 256 cells, in which one is fed with inputs normally and the other in reverse order. This helps in giving a better prediction because it captures the information both forward and backward.

      3. Labelling: The last block is the Connectionist Temporal Classification, which takes the probability matrix as the input from the Recurrent Neural Network and transforms it into sequence of labels. The output of the CTC classifier, is the predicted sequence of words. The main aim of classifier is to find best path for input sequence and it is called as decoding. The various decoding algorithms that can be made use of by CTC are best path decoding, vanilla beam search, word beam search etc. The proposed system makes use of the best path decoding algorithm.

      4. Overall System Architecture: The given input image of handwritten text is resized to and is given to the Convolutional Neural Network. It takes this input image

         CNN that is the feature vector is sent to the Recurrent Neural Network, which consists of Bi-LSTM layers to obtain a probability matrix. This is fed into the CTC layer, which computes loss and predicts the sequence.

   2. Signature Verification

   1. Pre-processing: The proposed methodology resizes the images to one fixed size 155 X 120 making use of bilinear interpolation. It also inverts the images such that the background pixels have an intensity of 0 and then we normalize each image by dividing each pixel value with maximum of pixel values (255) of the given images. Images are scaled down to 0 or 255 depending on whether the pixel value was below or above 50. Images were grouped in pairs of genuine and forged images, where the label was 1 if both were genuine and of the same writer and 0 otherwise. 13500 image pairs of each label where chosen, 15 percentage of which were used for testing.

   2. Architecture: The proposed methodology makes use of deep Convolutional Siamese Network. Siamese neural networks is nothing but a class of network architectures that contains identical subnetworks. The identical subnetworks are connected by a loss function, that computes similarity metric, which involves Euclidean distance between the feature extraction on both side. Contrastive loss is used to select properties of the images of the same class that will be closer (original or genuine signature) compared to the images of different class (forged). Then, Euclidean distance between two points are calculated. We choose a threshold value on distance, based on which we decide if signatures belong to the similar classes or dissimilar classes. We have made use of the Rectified Linear Units (ReLu) as the activation function. The first convolutional layer filters the input signature image of size 155 X 220 with 48 kernels with a stride of 1 pixel. The second convolutional layer filters the output of the first layer, with 128 kernels of size 5 X 5. The third and fourth convolutional layers have 256 and 96 kernels respectively, without any pooling layers in between, that is, they are connected directly. There are 1024 neurons and 128 neurons in the penultimate and ultimate layers respectively.

   and converts it into an image feature vector. The output of

4. EXPERIMENT AND RESULT

   The proposed system makes use of the HTR IAM dataset which is readily available online. The Train, Validation 1 and Validation 2 sets are both used for training purpose. All the given handwritten input images were one line PNG images with a white background, with their respective XML files and meta data were given. The datasets are divided into four subsets as follows:

   Table -1: IAM Dataset summary

   IAM Input Data
   Sl No.	Name	No. of text lines	No. of writers
   1	Train	6161	283
   2	Validation 1	900	46
   3	Validation 2	940	43
   4	Test	1861	128

   The proposed system for Signature Verification makes use of CEDAR signature dataset, which is readily available. It consists of 24 genuine signatures and 24 forged signatures from 55 different signers. Hence, there are totally 1320 genuine and 1320 forged signatures, and these images are grayscale.

   Table -2: CEDAR Dataset summary

   CEDAR Input Data
   Name	Users	Genuine Signatures	Forged Signatures
   CEDAR	55	24	24

5. CONCLUSION AND FUTURE WORK

   For the handwritten text recognition, convolutional recurrent neural network was used on the IAM dataset, which recognises a line of text with a good accuracy. To extend upon the current system, a system which can detect paragraphs of text at a single time needs to be built for faster results. Also, a line separator can be used to process a paragraph or a page at a time, or having an advanced architecture which does not need a line separator to process an entire paragraph or a page.

   The current system can recognise text only with white background or if there is a high contrast between the text and the background, which is a limitation. Due to the complexity of neural network, time taken to train the model is a constraint. A model can be built which takes lesser training time and faster execution. The Handwritten text recognition can be used where large handwritten documents need to be processed and transformed into machine text, mostly by universities, which makes it easier to store and query the data compared to the handwritten records.

   In the proposed system of Signature Verification, it makes use of Siamese convolutional network for offline signature verification, on the CEDAR dataset, which is a writer- independent system. In the future work, we can use a more enriched model and can make use of lesser number of signatures to verify it. The given signature verification system can be used in applications where signature authentication is required primarily.

6. REFERENCES

1. Hong-Phuong Tran, Andrew Smith, and Eric Dimla (2019). Offline Handwritten Text Recognition using Convolutional Recurrent Neural Network. 51-56. 10.1109/ACOMP.2019.00015.

2. P. C. Vashist, A. Pandey and A. Tripathi, A Comparative Study of Handwriting Recognition Techniques, 2020 International Conference on Computation, Automation and Knowledge Management (ICCAKM), 2020, pp. 456-461, doi: 10.1109/ICCAKM46823.2020.9051464.

3. Ingle, R. and Fujii, Yasuhisa and Deselaers, Thomas and Baccash, Jonathan and Popat, Ashok. (2019). A Scalable Handwritten Text Recognition System. 17-24. 10.1109/ICDAR.2019.00013.

4. N. Murugan, R. Sivakumar, G. Yukesh and J. Vishnupriyan, Recognition of Character from Handwritten, 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), 2020, pp. 417-1419, doi: 10.1109/ICACCS48705.2020.9074424.

5. S. Arora and M. P. S. Bhatia, Handwriting recognition using Deep Learning in Keras, 2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), 2018, pp. 142-145, doi: 10.1109/ICACCCN.2018.8748540.

6. R. Vaidya, D. Trivedi, S. Satra and P. M. Pimpale, Handwritten Character Recognition Using Deep-Learning, 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), 2018, pp. 772-775, doi: 10.1109/ICICCT.2018.8473291.

7. Utkarsh Dwivedi, Pranjal Rajput and Manish Kumar Sharma, Cursive Handwriting Recognition System Using Feature Extraction and Artificial Neural Network, International Research Journal of Engineering and Technology (IRJET), vol. 04, no. 03, Mar 2017, ISSN 2395-0056.

8. Megha Agarwal, Shalika, VinamTomar, Priyanka Gupta, Handwritten Character Recognition using Neural Network and Tensor Flow, International Journal of Innovative Technology and Exploring Engineering (IJITEE) ISSN: 2278-3075, Volume-8, Issue- 6S4, April 2019.

9. J. Yang, P. Ren and X. Kong, Handwriting Text Recognition Based on Faster R-CNN, 2019 Chinese Automation Congress (CAC), 2019, pp. 2450-2454, doi: 10.1109/CAC48633.2019.8997382.

10. P. Ma, Recognition of Handwritten Digit Using Convolutional Neural Network, 2020 International Conference on Computing and Data Science (CDS), 2020, pp. 183-190, doi: 10.1109/CDS49703.2020.00044.

11. H. Nisa, J. A. Thom, V. Ciesielski and R. Tennakoon, A deep learning approach to handwritten text recognition in the presence of struck-out text, 2019 International Conference on Image and Vision Computing New Zealand (IVCNZ), 2019, pp. 1-6, doi: 10.1109- IVCNZ48456.2019.8961024.

12. G. Bastas, K. Kritsis and V. Katsouros, Air-Writing Recognition using Deep Convolutional and Recurrent Neural Network Architectures, 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2020, pp. 7-12, doi: 10.1109/ICFHR2020.2020.00013.

13. Sounak Dey, Anjan Dutta, Toledo J. and Suman Ghosh and Josep Llados, and Umapada Pal(2017). SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification.

14. T. Sultan Rana, H. Muhammad Usman and S. Naseer, Static Handwritten Signature Verification Using Convolution Neural Network, 2019 International Conference on Innovative Computing (ICIC), 2019, pp. 1-6, doi: 10.1109/ICIC48496.2019.8966696.

15. Viriri, S.; Tapamo, J.-R. Signature verification based on handwritten text recognition. Communications in Computer and Information Science 2009, 61, 98105.

16. L. G. Hafemann, R. Sabourin and L. S. Oliveira, Offline handwritten signature verification Literature review, 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), 2017, pp. 1-8, doi: 10.1109/IPTA.2017.8310112.

17. Huacheng Hu, Dezhi Chen, and Jianbin Zheng. 2019. Online Handwriting Signature Verification Based on Template Clustering. In Proceedings of the 2019 3rd International Workshop on Education, Big Data and Information Technology (EBDIT 2019). Association for Computing Machinery, New York, NY, USA, 129135. doi:10.1145/3352740.3352762

18. S. M. A. Navid, S. H. Priya, N. H. Khandakar, Z. Ferdous and A. B. Haque, Signature Verification Using Convolutional Neural Network, 2019 IEEE International Conference on Robotics, Automation, Artificial- intelligence and Internet-of-Things (RAAICON), 2019, pp. 35- 39, doi: 10.1109/RAAICON48939.2019.19.

19. A. Rateria and S. Agarwal, Off-line Signature Verification through Machine Learning, 2018 5th IEEE Uttar Pradesh Section International

    Conference on Electrical, Electronics and Computer Engineering (UPCON), 2018, pp. 1-7, doi: 10.1109/UPCON.2018.8597090.

20. Jagtap, Amruta and Sawat, Dattatray and Hegadi, Rajendra and Hegadi, Ravindra. (2020). Verification of genuine and forged offline signatures using Siamese Neural Network (SNN). Multimedia Tools and Applications. 79. 10.1007/s11042-020-08857-y.

21. Pansare, Ashwini and Bhatia, Shalini. (2012). Handwritten Signature Verification using Neural Network. International Journal of Applied Information Systems. 1. 44-49. 10.5120/ijais12-450114.