An online cursive handwritten medical words recognition system for busy doctors in developing countries for ensuring efficient healthcare service delivery

ABSTRACT Doctors in developing countries are too busy to write digital prescriptions. Ninety-seven percent of Bangladeshi doctors write handwritten prescriptions, the majority of which lack

legibility. Prescriptions are harder to read as they contain multiple languages. This paper proposes a machine learning approach to recognize doctors’ handwriting to create digital

prescriptions. A ‘Handwritten Medical Term Corpus’ dataset is developed containing 17,431 samples of 480 medical terms. In order to improve the recognition efficiency, this paper introduces

a data augmentation technique to widen the variety and increase the sample size. A sequence of line data is extracted from the augmented images of 1,591,100 samples and fed to a

Bidirectional Long Short-Term Memory (LSTM) network. Data augmentation includes pattern Rotating, Shifting, and Stretching (RSS). Eight different combinations are applied to evaluate the

strength of the proposed method. The result shows 93.0% average accuracy (max: 94.5%, min: 92.1%) using Bidirectional LSTM and RSS data augmentation. This accuracy is 19.6% higher than the

recognition result with no data expansion. The proposed handwritten recognition technology can be installed in a smartpen for busy doctors which will recognize the writings and digitize them

in real-time. It is expected that the smartpen will contribute to reduce medical errors, save medical costs and ensure healthy living in developing countries. SIMILAR CONTENT BEING VIEWED

BY OTHERS ENHANCING HANDWRITTEN TEXT RECOGNITION ACCURACY WITH GATED MECHANISMS Article Open access 22 July 2024 A NEW DATASET FOR MONGOLIAN ONLINE HANDWRITTEN RECOGNITION Article Open

access 02 January 2023 NEURAL NETWORK-ASSISTED PERSONALIZED HANDWRITING ANALYSIS FOR PARKINSON’S DISEASE DIAGNOSTICS Article 02 June 2025 INTRODUCTION A global study reports that Bangladeshi

physicians spend less than a minute on each primary consultation whereas physicians of Sweden spend 22.5 min1. This happens due to the massive population and limited number of physicians

available for them in developing countries. The ideal doctor to population ratio recommended by World Health Organization (WHO) is 1:1000, while on the contrary, the ratio in Bangladesh is

only 0.304:10002. The consultation time of doctors includes listening to patients’ problems, inspecting test reports, writing a prescription, explaining the patients’ condition and giving

advises. As they serve a large number of patients in a very short time, they are left with less time for writing a prescription. Thus, the handwriting on the prescription becomes cursive and

indecipherable for the patients and pharmacists. As a result, the pharmacists misread the prescriptions due to the similar appearances or sounds of thousands of medication names3 and end up

providing the patients with wrong medicines. This research conducted an online survey to understand the current state of handwritten prescription usage in the medical practice of

Bangladesh. The survey reports that 97.1% of Bangladeshi doctors still generate handwritten prescriptions. According to Bhuiyan et al.4, incompetency of understanding doctors’ handwritten

prescriptions is an obstacle for getting quality health services. The difficulty of reading these prescriptions often causes adverse medical consequences such as selecting wrong medicine,

improper number of dosage, and even death. National Academies of Science Institute reports that 7000 deaths occur in the US due to the sloppy handwriting of doctors’5. Doctors can be trained

to write legible prescriptions, but they get a very short period of time to serve each patient. This paper proposes a machine learning approach to assist the doctors by recognizing cursive

handwriting of doctors and converting them into readable digital prescriptions. The handwriting recognition system consists of several steps: handwritten data collection, data prepossessing

for simplification, increasing data samples using data augmentation, and building a machine learning model for predicting doctors’ handwriting. Bangladeshi prescriptions are a mixture of

Bangla and English words with Latin abbreviations of medical terms4. Due to the unavailability of enough Bangladeshi prescriptions online, this research started its journey by creating a

‘Handwritten Medical Term Corpus’. Initially, a medical words corpus is created using the 8,324 Bangladeshi prescriptions of the Portable Health Clinic (PHC). PHC is a remote healthcare

system which provides affordable and sustainable health services to the base of the pyramid population of developing countries67 to ensure universal healthcare coverage targeting rurality,

poverty, and disability8. The PHC data has been used for predicting health status of the existing patients910 in terms of healthcare cost reduction11, understanding consumer behavior12.

However, none of these works considered the doctors’ role in the PHC system. The ultimate performance of a remote healthcare system depends on the decision made by the doctor. As mentioned

earlier, 97% of Bangladeshi doctors still write handwriting prescriptions. The trend is changing to digital prescriptions. The analog prescriptions are not archived in digital forms and are

difficult to search for previous medical history. In order to create real-time digital prescriptions, a real-time handwriting recognition system is necessary. At the first step, a ‘Medical

Term Corpus’ is needed. We have created a corpus considering the most used words that appeared in the PHC prescriptions. The corpus contains 480 medical-related words (English: 320 and

Bangla:120). Afterward, the handwriting of 39 healthcare professionals of these 480 words are collected for the recognition purpose. Thus, the ‘Handwritten Medical Term Corpus’ has 17,431

handwritten instances. Recognizing different types of handwriting requires a large dataset collected from various sources which is both costly and time-consuming. One way to deal with this

problem is data augmentation13. This paper proposes a new data augmentation technique - Rotate, Shift, and Stretch (RSS) to generate multitudes of handwriting variations. RSS method takes

each stroke of a handwritten word and creates new data by updating the coordinates. After applying RSS, the extended dataset has 1,591,100 samples. For predicting the handwritten words,

Bidirectional LSTM model is used due to the recent popularity of Recurrent Neural Network (RNN) in the area of handwriting recognition. According to Zhang et al.14, sequence data can contain

rich details of handwriting than static image-like representations. The proposed handwritten technology can be used in a smartpen, specifically designed for doctors. The smartpen will

digitize the handwriting of doctors into readable texts. The database will store each doctors’ writings individually to learn the unique pattern of writing of that particular doctor. This

will lead the tool to recognize the handwriting of each doctor more efficiently. The smartpen will benefit the doctors by saving time and reducing typographical errors of digital

prescriptions. The rest of the article starts with a review on related researches in “Related work”, introduces a handwritten dataset of Bangladeshi doctors in “Handwritten medical term

corpus”, demonstrates the steps of recognition methodology in “Methodology of handwriting recognition”, “Results and discussion” reports the results and findings including the idea of a

smartpen, and finally, the conclusion is given in “Conclusion”. RELATED WORK Over the last few decades, multitudes of deep learning approaches have been proposed for efficient handwriting

recognition using several handwritten datasets of different languages. This section discusses similar research works in the following four sectors: DOCTORS’ HANDWRITING DATASET Few online

datasets are available to design a doctors’ handwriting recognition system. Dibyajyoti et al.15 introduced HP_DocPres dataset with 11,340 samples of handwritten and printed words collected

from various medical prescriptions. This dataset is prepared to differentiate between handwritten and printed texts. However, the words are not labeled so they can’t be used to recognize the

written words by doctors. Another doctors’ handwriting dataset is introduced by Farjado et al.16. This dataset contains 1800 images of 12 medicine names collected from 50 doctors from

clinics and hospitals of Metro Manila, Quezon City, and Taytay, Rizal in the Philippines. However, this dataset is not suitable for recognizing doctors’ handwriting in Bangladeshi

prescriptions due to the limited number of medical terms it contains and the region of data collected being different from our study region. Although doctors’ handwriting dataset is scarce,

there are multitudes of available handwriting datasets both for English and Bangla languages. IAM Dataset by the University of Bern17 is one of the most popular datasets with the largest

handwriting collection in English. This dataset contains 13,353 images of handwritten lines of text created by 657 writers. A similar dataset in Bangla is the Bangla handwriting recognition

dataset by Bappaditya et al.18 that has obtained 79,000 handwritten Bangla word samples written by 77 different writers. BanglaLekha-Isolated19 and ISI20 dataset comes with a vast number of

handwriting samples of individual Bangla characters with numerals. Another popular dataset is CMATERdb121 that has 100 handwritten Bangla pages and 50 handwritten English and Bangla combined

pages with ground-truth annotations. However, these datasets do not contain doctors’ handwriting or any medical terms, hence might perform poorly in recognizing doctors’ handwriting.

OFFLINE HANDWRITTEN CHARACTER RECOGNITION: USING IMAGE DATA AS INPUT Automatic conversion of handwritten texts into images for recognition using Convolutional Neural Network (CNN) is called

Offline Character Recognition22. Shahariar et al.23 proposed a lightweight CNN model for Bangla handwriting recognition. The model has 13 convolutional layers with 2 sub-layers. The

sub-layers are joined together to pass through a max-pooling layer with one 0.25 weighted dropout layer. This model has attained 98%, 96.8% and 96.4% accuracy in BanglaLekha, CMATERdb, and

ISI datasets. A modified LeNet-5 CNN model by Yuan et al.24 obtained an accuracy of 93.7% for uppercase and 90.2% for lowercase for the recognition for English Language characters. Yang et

al.25 presented a path-signature feature method using deep CNN for identifying Chinese character writers. The method was 99.52% accurate with DropStroke data augmentation. ONLINE HANDWRITTEN

CHARACTER RECOGNITION: USING TIME-SERIES DATA AS INPUT Online character recognition considers a sequence of times which is captured by the movements of a specialized pen. The recognition

rate of the online system is more efficient and higher than the offline system22. RNN has recently been widely used in the area of handwriting recognition for showing better recognition

performance. The RNNs work with sequence data of coordinates which contain vast information than static images14. Bappaditya et al.18 used bidirectional LSTM using 65,620 handwritten Bangla

words dataset and has obtained 79% accuracy. Zhang et al.14 proposed a conditional RNN-based generative model combining LSTM and Gated Recurrent Units (GRU). The model is built for

recognizing Chinese handwritten characters and has achieved 98.15% recognition accuracy. Farjado et al.16 used Convolutional RNN (CRNN) for recognizing doctors’ cursive handwriting which

contained 13 convolutional layers followed by 3 bidirectional LSTM layers and has attained 72% accuracy. However, Achkar et al.26 reported obtaining 95% accuracy using the similar CRNN model

with a different dataset for recognizing medical handwritten prescriptions. HANDWRITING RECOGNITION WITH DATA AUGMENTATION In our previous work, SRP (Stroke Rotation and Parallel-shift)

data augmentation technique was applied for expanding the doctors’ cursive handwritten dataset. However, the minimum accuracy of that system was only 68.0%27. For recognizing Bangla

handwriting characters, Shahariar et al.23 applied three data augmentation methods on 10% of the dataset: shifted height and width, rotated images by 10 degrees, and zoomed in the images.

Another data augmentation method named ‘DropStroke’ was used for Chinese character recognition. Chinese characters are very complex as they have many strokes. Thus, the DropStroke method

randomly excludes several strokes and generates new handwritten characters using the combination of the remaining strokes1425. Hayashi et al.13 used a data augmentation technique using

probability distribution for handwriting recognition. This method calculates probability distribution from the features related to the structure of the character. Then, it generates strokes

based on the distribution and forms multitudes of new characters. ETHICS APPROVAL All the authors mentioned in the manuscript have agreed for authorship, read and approved the manuscript,

and given consent for submission and subsequent publication of the manuscript. CONSENT TO PARTICIPATE The written informed consent was obtained from all subjects prior to collecting their

handwritten samples in these studies. CONSENT FOR PUBLICATION The written informed consent was obtained from all subjects prior to collecting their handwritten samples in these studies.

HANDWRITTEN MEDICAL TERM CORPUS In developing countries, Doctors’ handwriting becomes illegible as they have to serve a lot of patients in a short span of time. The writings get more

difficult to read as Bangladeshi prescriptions are a mixture of different languages. A sample of Bangladeshi prescription is given in Fig. 1. Due to the shortage of available Bangladeshi

prescriptions online, this research has introduced a handwriting prescription dataset collected from Bangladeshi doctors. This section describes the data collection and preparation process

for creating a handwritten medical term corpus. CREATING MEDICAL CORPUS FROM DIGITAL PRESCRIPTIONS Medical terms were collected from the remote healthcare prescription database of PHC. PHC

system maintains an electronic journal of patients’ health records. There are major five categories of data in the journal: (1) registration data, (2) survey data, (3) clinical data, iv)

conversation data, and v) prescription data. The foremost section of ‘Handwritten Medical Term Corpus’ is collected from the digital prescriptions of PHC. Figure 2 shows a sample of PHC

prescription data. A total of 8324 digital prescriptions were found in the PHC database. Each prescription contained several columns such as symptoms, medicine names, advises. Initially, a

corpus of medical terms is created using the most frequently appeared words in these columns. The corpus has selected 360 English and 120 Bangla words. These words are sorted according to

the frequency of their appearance in the prescriptions, as shown in Fig. 3. OBTAINING HANDWRITTEN DATA VIA ANDROID APPLICATION A simple android app has been developed to obtain real

handwriting data from doctors and medical professionals. The app displays medical words on the device screen one by one from the ‘Medical Term Corpus’. Then, the data providers write the

corresponding words shown on the screen. Samsung Galaxy Tab S3 was used to capture data as it contains a stylus pen. The data providers write down on the screen using the stylus pen and the

application stores the handwritten words in the database. Besides the writings, the application also receives detailed information such as pen movements (xy coordinates) and the status of

the pen. Status indicates the state of the pen whether it is up or down. Collected information is stored in the database along with the writings as sequential data. The ‘Handwritten Medical

Term Corpus’ also contains the original corpus data to use as truth value while training the machine learning model. The complete data collection process is shown in Fig. 4a. DATASET PROFILE

There are 480 medical words (360 English and 120 Bangla) in the ‘Handwritten Medical Term Corpus’. These words are chosen based on the number of appearances in 8324 Bangladeshi

prescriptions. The handwritings are collected from 39 medical professionals and doctors of Bangladesh. Due to receiving incomplete data from 12 data providers, 1,289 samples are missing in

the dataset. Hence, the dataset has 17,431 handwritten instances of 480 medical-related words. All the data were collected by maintaining authenticity, security, and privacy of the data

providers, and the experiments were performed in accordance with relevant guidelines and regulations. METHODOLOGY OF HANDWRITING RECOGNITION After the data collection phase (Fig. 4a), the

research is administered in three steps, as shown in Fig. 4b–d. First, the collected dataset is analyzed and preprocessed. Then, the proposed RSS (Rotation, Shift, and Stretch) data

augmentation technique is applied to the preprocessed dataset to expand the number of instances. In this step, sequence line data is generated from the extended dataset. Finally, a

bidirectional LSTM model uses the sequence data as input and predicts handwritten medical terms. DATA PREPROCESSING Image-like representations provide very general information about the

data. Whereas raw data is rich with information such as spatial and temporal details. These spatio-temporal information can be constituted in a sequence of variable length14, given in

equation (1). Here, \(x_i\) and \(y_i\) denotes the xy-coordinates of pen movements and \(s_i\) states the stroke number of any point _i_. $$\begin{aligned}

{[}[x_1,y_1,s_1],[x_2,y_2,s_2],\ldots ,[x_n,y_n,s_n]] \end{aligned}$$ (1) As shown in Fig. 4b, the preprocessing is done in three steps. The images are simplified through removing nearby

repetitive points and normalization. Then, a six-dimensional vector is extracted sequentially for each stroke to generate machine learning model input data. The three segments are described

below: REMOVE REDUNDANT POINTS Different styles of handwriting can be found even in a group of people with the same language. Different people follow different ways of writing such as small,

regular, flat, cursive. Each writer creates distinct sampling points even if they are writing the same character. Thus, a general format for each character can be created by removing nearby

redundant points for efficiently estimating the handwritten words. To remove all the redundant points from any handwritten word or character, consider a particular point \((x_i, y_i, s_i)\)

where point _i_ lies in the same stroke with its nearby points, as such \(s_{i-1} = s_i = s_{i+1}\). There are two conditions to determine if point _i_ should be removed: * (i) Distance

between points: If the distance between two points _i_ and \(i-1\) is very small, then point _i_ is removed. In the given Eq. (2), the threshold \(T_{dist}\) = \(0.005 * max(H,W)\), where H

indicates the vertical and W indicates the horizontal widths of the handwriting text input place. Besides, two connecting points lying on a straight line is also be removed.

$$\begin{aligned} \sqrt{(x_i-x_{i-1})^2 + (y_i-y_{i-1})^2} < T_{dist} \end{aligned}$$ (2) * (ii) Cosine similarity: Cosine similarity determines the similarity of an inner product space

between two non-zero vectors. Point _i_ is removed if similarity between two points _i_ and \(i-1\) is greater than the threshold cosine angle value. In the given Eq. (3), the threshold

\(T_{cos}\) is set to 0.99. $$\begin{aligned} \frac{\Delta {x_{i-1}}\Delta {x_i}+\Delta {y_{i-1}}\Delta {y_i}}{{(\Delta {x^2_{i-1}}+\Delta {y^2_{i-1}})}^{0.5} {(\Delta {x^2_i}+\Delta

{y^2_i})}^{0.5}} >T_{cos} \end{aligned}$$ (3) Point _i_ is considered as a redundant point if one of the given conditions is satisfied. After removing the redundant points, the shape of

the handwritten character is well-preserved and each point of the recreated form contains more information14. An example of removing redundant points on a sample handwritten medical term is

shown in Fig. 5. NORMALIZATION After removing the redundant points, the data is normalized for simplification. For _x_ and _y_ coordinates, the maximum \(x_{max}\), \(y_{max}\) and the

minimum \(x_{min}\), \(y_{min}\) is calculated from each data point. Then, _x_ as _X_ and _y_ as _Y_ coordinates is normalized to \(X_{nor}\) and \(Y_{nor}\) using Eq. (4). Thus, the (_x_, 

_y_) coordinates data is scaled between the value of 0 and 1. $$\begin{aligned} X_{nor} = \frac{X - x_{min}}{x_{max}-x_{min}} \end{aligned}$$ (4) GENERATE SEQUENCE DATA After the data

preprocessing steps, this research has connected the normalized points to form straight lines. Then, a six-dimensional vector is generated from the straight line as \(L_i\) with two

connecting points _i_ and \(i+1\), as shown in Eq. (5). $$\begin{aligned} L_i = [x_i, y_i, \Delta {x_i}, \Delta {y_i}, I(s_i=s_{i+1}), I(s_i \ne s_{i+1})] \end{aligned}$$ (5) In the given

equation, \(x_i\) and \(y_i\) are the xy-coordinates which states the starting position of a line. The direction of pen movements in x and y axis is denoted by \(\Delta {x_i}\) and \(\Delta

{y_i}\). The last two expressions determine the status of the pen ([0, 1] indicates pen-up, [1, 0] indicates pen-down). The term \(I(s_i=s_{i+1}) = 1\) indicates that the starting and ending

points of the straight line lies on the same stroke. The last expression \(I(s_i \ne s_{i+1}) = 1\) states that the line has moved to the next stroke. Thus, a new sequence of vectors

\([L_1, L_2, \ldots , L_{n-1}]\) is formed from the (_x_, _y_, _s_) coordinates. This newly generated sequence is denoted as \([x_1,x_2,\ldots ,x_k]\) for simplification, where each \(x_i\)

represents one six-dimensional vector14. DATA AUGMENTATION This research has used data augmentation on the preprocessed data in order to increase the number of instances of ‘Handwritten

Medical Term Corpus’. A new data augmentation approach named RSS (Rotation, Shift, and Stretch) is proposed in this article. RSS method expands data by rotation, shifting, and stretching the

shape of character, as shown in Fig. 4c. This method is specifically designed to expand the variety of handwriting styles. ROTATE (STROKE ROTATION) In stroke rotation, the middle point of a

stroke (_a_, _b_) is determined using the starting point \((x_f, y_f)\) and ending point \((x_l, y_l)\) coordinates, as in Eq. (6). Then, all the points lying on the middle point of that

stroke are rotated. The principle of the rotation process is shown in Fig. 6a. $$\begin{aligned} (a, b) = (\frac{x_f+x_l}{2},\;\frac{y_f+y_l}{2}) \end{aligned}$$ (6) Stroke rotation process

uses Eq. (7) to rotate a point (_x_, _y_) in \(\theta \) angle around the middle point of the stroke (_a_, _b_). The rotated point is (_X_, _Y_). $$\begin{aligned} \left( \begin{array}{r}

X-a \\ Y-b \end{array} \right) = \left( \begin{array}{rr} cos\theta &{} -sin\theta \\ sin\theta &{} cos\theta \end{array} \right) \left( \begin{array}{r} x-a \\ y-b \end{array}

\right) \end{aligned}$$ (7) This equation is applied to all of the points on the stroke to rotate the entire stroke around the midpoint of the stroke. Figure 6b is a sample example where the

blue color is the original instance and the red color is the instance after rotation. This method is applied to the strokes with random values of angles in order to create new augmented

data in different forms. SHIFT (STROKE PARALLEL-SHIFT) In this second method, all of the points on the stroke are added to a constant number (_x_, _y_) in order to shift the strokes in

parallel. The principle of parallel-shifting is given in Fig. 7a. One certain point (_x_, _y_) is shifted to a new point (_X_, _Y_) following Eq. (8). $$\begin{aligned} (X, Y) =

(x+dx,\;y+dy) \end{aligned}$$ (8) The entire stroke is shifted by (_dx_, _dy_) after applying this equation to every point on the stroke. Figure 7b is a sample example where the blue color

is the original instance and the red color is the instance after shifting. This method is applied to the strokes with random values of _dx_ and _dy_ in order to create new augmented data in

different forms. STRETCH (STROKE STRETCHING) When a writer writes very quickly and roughly, the letters may be stretched vertically or horizontally. Hence, the stroke stretching method is

proposed which takes all the strokes of a word and stretches the strokes to change the ratio of the handwritten word. The principle of this method is shown in Fig. 8a. First, the average

value of the y-coordinates is calculated for all the points of a stroke as the reference value _Y_. If a certain point \((x_i, y_i)\) has larger y-coordinate that _Y_, then the value of

\(y_i\) is multiplied by \((1 + r)\), otherwise it’s multiplied by \((1 - r)\), shown in Eq. (9). Here, _r_ is the changing ratio rate and is set to 0.02. $$\begin{aligned} y_i = {\left\{

\begin{array}{ll} y_i \times (1 + r),&{} \text {if } Y < y_i\\ y_i \times (1 - r),&{} \text {otherwise} \end{array}\right. } \end{aligned}$$ (9) However, when the writing is

stretched, the points do not overlap and do not break the shape of the character. Figure 8b is a sample example where the blue color is the original instance and the red color is the

instance after stretching. According to the findings of this research, this is an effective method for identifying rough handwriting. REGENERATING SEQUENCE DATA The original dataset contains

six-dimensional vectors for all the instances. After data augmentation, the vectors need to be regenerated for the new augmented images by updating the parameters. For the rotated images,

the values are updated by adding \(\theta \) with \(\Delta {x_i}\) and \(\Delta {y_i}\), Eq. (10). To update the vector for parallel-shifting, _dx_ and _dy_ is added to \(x_i\) and \(y_i\),

Eq. (11). Finally, the values of \(x_i\) and \(y_i\) are multiplied with _r_ for the stretched images, Eq. (12). However, the values of \(I(s_i=s_{i+1})\) and \(I(s_i \ne s_{i+1})\) remain

unchanged. $$\begin{aligned}{}&\begin{aligned} L_i = [x_i,\;y_i,\;\Delta {x_i} + \theta ,\;\Delta {y_i} + \theta ,\;\\I(s_i=s_{i+1}),\;I(s_i \ne s_{i+1})] \end{aligned} \end{aligned}$$

(10) $$\begin{aligned}{}&\begin{aligned} L_i = [x_i + dx,\;y_i + dy,\;\Delta {x_i},\;\Delta {y_i}\;\\I(s_i=s_{i+1}),\;I(s_i \ne s_{i+1})] \end{aligned} \end{aligned}$$ (11)

$$\begin{aligned}{}&\begin{aligned} L_i = [x_i \times r,\;y_i \times r,\;\Delta {x_i},\;\Delta {y_i},\;\\I(s_i=s_{i+1}),\;I(s_i \ne s_{i+1})] \end{aligned} \end{aligned}$$ (12) The RSS

data augmentation is designed specifically targeting handwriting data as the operations are done by updating the strokes. It can also be used for other datasets if the data are obtained as

time series—a sequence of coordinates. If the time series data is converted into images, it can also be used for the expansion of offline characters. The augmented data is stored in the

‘Augmented Handwritten Medical Term Corpus’ dataset, as shown in Fig. 4c. MACHINE LEARNING MODEL: BIDIRECTIONAL LSTM Handwriting contains multiple strokes with several points. Writing style,

speed, order, shape of the character varies from person to person which information is difficult to achieve from static images. Hence, this research has dealt with raw sequential data

rather than generating image-like representations in order to get rich information about doctors’ handwriting. This research has used Bidirectional LSTM to develop a complete end-to-end

recognition system by operating the sequence data extracted from the line data of the augmented handwritten dataset, as shown in Fig. 4d. Bidirectional LSTM uses both past and future inputs

for prediction, as shown in Fig. 9, whereas the original LSTM considers only past inputs28. In this research, the machine learning model has used both past and future line data to calculate

parameters and predict handwritten medical words. The model architecture for this research is developed using Keras - a neural network library of python. The maximum length of each data is

set to 260 by padding zeroes by the end of the instances. There are 300 hidden LSTM layers with corresponding pooling layers. In order to avoid over-fitting, Dropout is used between pooling

layer and dense layer29. As the model learns the same data many times due to using data augmentation, the Early Stopping method is also used to circumvent overfitting30. The Bidirectional

LSTM model has the following parameters: * Activation function: Softmax31 * Batch size: 51232 * Learning rate: 0.00132 * Number of epochs: 532 * Loss function: Categorical cross-entropy33 *

Optimization function: Adam34 * Dropout: 0.329 RESULTS AND DISCUSSION The ‘Handwritten Medical Term Corpus’ contains 17,431 handwritten samples of 480 medical words. The writings are

obtained from 39 medical professionals. Among the 39 sets, there are 27 complete sets of instances due to receiving incomplete data from 12 writers. Three sets of 480 words are randomly

selected from the complete sets as test data. Thus, the train data has 15,911 and the test data has 1440 handwritten samples. The data augmentation methods are applied only to the train

data. Thus, the Bidirectional LSTM model is trained using the extended sequence data and is evaluated based on its performance on the test set. RESULTS This research has performed eight sets

of experiments by training the model with eight different mixtures of handwritten data. Figure 10 demonstrates the applied eight datasets expanded using different combination of data

augmentation methods. The performance of the model is evaluated based on its accuracy on the same test set. The findings from the model evaluation are as follows, given in Table 1: * 1. NO

DATA EXPANSION: First, the bidirectional LSTM model is trained on the original ‘Handwritten Medical Term Corpus’ with 15,911 instances without applying any data augmentation method. This

case has attained the lowest accuracy. * 2. ROTATE: This case has trained by model by applying only Rotation data augmentation method. Rotation is applied 100 times on each instances and the

data size becomes 1,591,100. This experiment has achieved high maximum accuracy, but the lowest and unacceptable minimum accuracy which is only 3.33%. * 3. SHIFT: Shifting data augmentation

is used 100 times on the train data. It has shown overall good performance in all the above three measurements. * 4. STRETCH: Stretching data augmentation is applied similarly on the train

data and the expanded data size is 1,591,100. This experiment has achieved similar results to Shifting data augmentation. * 5. ROTATE + SHIFT: This experiment has combined the Rotate and

Shift data augmentation methods. First, each instance is rotated ten (10) times and the data size becomes 159,110. Then, Shift method is performed for another ten (10) times on the expanded

instances. Thus, the final data size gets 1,591,100 instances. The maximum accuracy is quite high but the minimum accuracy is comparatively low. * 6. ROTATE + STRETCH: The combined Rotation

and Stretching methods are applied in this experiment in the similar way. This case has attained the highest maximum accuracy. However, the average and minimum accuracy is low comparing to

the other cases. * 7. SHIFT + STRETCH: Shifting and Stretching data augmentation methods are combined in this case. The data size is 1,591,100 by applying the methods ten (10) times each.

This experiment has also accomplished overall considerable accuracy, but better results were observed when these two methods are applied individually on the training data (case 3 and 4). *

8. ROTATE + SHIFT + STRETCH (RSS): Finally, this is the proposed data augmentation technique of this research. It applies all the three methods on the training data one by one. First, the

instances are rotated five (5) times. Then, shifting is performed for another five (5) times on the expanded 79,555 instances. The data size becomes 397,775 where we have applied the stretch

method for four (4) times. Thus, this research introduces the ‘Augmented Handwritten Medical Term Corpus’ which contains 1,591,100 handwritten medical term samples. The Bidirectional LSTM

model with RSS data augmentation has achieved the highest average and minimum accuracy. It could not reach the highest maximum accuracy but the accuracy never plunged under 92.1%. Thus, this

experiment can be considered as the best possible method for recognizing doctors’ cursive handwriting. IOT SMARTPEN FOR DOCTORS: AN APPLICATION EXAMPLE The handwriting recognition tool can

be installed in a doctor’s smartpen, which is an ongoing work at our research institution. As shown in Fig. 11, the smartpen has seven major modules. The same recognition tool can also be

installed in tablet PCs. The HANDWRITING STROKE DETECTOR contains a sensor in the nib of the pen to collect words written by a doctor. The FINGERTIP SENSOR will recognize the authorized user

of the pen so that unauthorized person can not use the pen. The MEMORY will store all the prescriptions so that the doctor can easily find previous health records. It can also store new

patterns of words written by the doctor. In this way, the data set will get larger day by day. The system will also able to capture the unique handwriting habit of that particular doctor.

The HANDWRITING RECOGNITION tool will recognize the words written by the doctor and convert them into text to store it to the memory. A copy of the prescription will also be stored in PHC

ONLINE DATABASE that can be accessed by the patient, authorized pharmacy and family members. The SPEECH RECOGNITION will be used as a sound recorder to generate prescriptions from doctors’

speech through voice recognition. However, that module will use a different technology which is not part of this particular research of handwriting recognition. As mentioned above, the

smartpen is still in the concept level where our handwriting recognition technique can be applied. This smartpen will handle sensitive information such as patients’ identification, medical

history, doctors’ profiles including bio-metric information. The smartpen development process will follow relevant privacy security guidelines such as data privacy for m-health patients35 ,

location based privacy36 , privacy protection of health records from search engines37. CONCLUSION The objective of this research has been to recognize doctors’ handwriting and digitize the

prescriptions in real time. Towards this goal, this paper contributes in three steps- (a) develop a medical term corpus (b) introduce a unique data augmentation technique and (c) use a

machine learning approach for final recognition. It also compares the recognition accuracy in different augmentation stages. The machine learning approach was designed for recognizing

particularly doctors’ cursive handwriting and converting them into digital printed texts. A dataset named ‘Handwritten Medical Term Corpus’ was created from digital prescriptions of PHC that

contains 17,431 handwritten texts of 480 Bangla and English medical-related words. A data augmentation method RSS was proposed for enriching the variety of doctors’ handwriting. RSS method

expanded the data set to 1,591,100 instances which was also introduced in this paper named ‘Augmented Handwritten Medical Term Corpus’. Bidirectional LSTM model was used to create an online

character recognition system for predicting doctors’ handwriting. This research performed eight experiments on the handwritten data set and achieved 93.0% average accuracy (max: 94.5%, min:

92.1%) using Bidirectional LSTM and RSS data augmentation. This accuracy was 19.6% higher than the recognition result with no data expansion. The current accuracy needs to be improved. The

proposed recognition methodology can be implemented in a smartpen for doctors. A brief system architecture of the proposed smartpen is introduced. Doctors will use the smartpen for writing

and the tool will automatically convert the handwriting texts into digital prescriptions. Apart from the proposed method for data augmentation, other representative computational

intelligence algorithms can be used to solve similar problems like Monarch Butterfly Optimization (MBO)38, Earthwarm Optimization Algorithm (EOA)39, Elephant Herding Optimization (EHO)40,

Moth Search (MS) Algorithm41, and Harris Hawk Optimization (HHO)42. More samples will be collected and will be automatically stored in the corpus. Doctors personal handwriting habit will

also be captured. More samples will improve the recognition accuracy. DATA AVAILABILITY The data and materials are available in the Social Technology Lab page at

http://socialtech.gramweb.net/media-archive/codes. CODE AVAILABILITY All codes are available in the Social Technology Lab page at http://socialtech.gramweb.net/media-archive/codes.

REFERENCES * Irving, G. _et al._ International variations in primary care physician consultation time: A systematic review of 67 countries. _BMJ Open_ 7(10), e017902 (2017). Article  Google

Scholar  * Kumar, R. & Pal, R. India achieves WHO recommended doctor population ratio: A call for paradigm shift in public health discourse!. _J. Family Med. Primary Care_ 7(5), 841

(2018). Article  Google Scholar  * Jeetu, G. & Girish, T. Prescription drug labeling medication errors: A big deal for pharmacists. _J. Young Pharm._ 2(1), 107–111 (2010). Article  CAS 

Google Scholar  * Bhuiyan, B. _et al._ Assessing whether medical language is a barrier to receiving healthcare services in Bangladesh: An exploratory study. _BJGP Open_ 3(2), 1–9 (2019).

Article  Google Scholar  * Brits, H. _et al._ Illegible handwriting and other prescription errors on prescriptions at National District Hospital. _Bloemfontein. Prof. Nurs. Today_ 21(2),

53–56 (2017). Google Scholar  * Ahmed, A., Inoue, S., Kai, E., Nakashima, N., & Nohara, Y. Portable Health Clinic: A pervasive way to serve the unreached community for preventive

healthcare. In International Conference on Distributed, Ambient, and Pervasive Interactions (pp. 265–274). Springer (2013). * Ahmed, A., Rebeiro-Hargrave, A., Nohara, Y., Maruf, R., Ghosh,

P., Nakashima, N., & Yasuura, H. Portable health clinic: A telehealthcare system for unreached communities. In Smart Sensors and Systems (pp. 447–467). Springer, Cham (2015). * Podder,

K., Tabassum, S., Khan, L., Salam, K., Maruf, R., & Ahmed, A. Design of a sign language transformer to enable the participation of persons with disabilities in remote healthcare systems

for ensuring universal healthcare coverage. In 2021 IEEE Technology and Engineering Management Conference-Europe (TEMSCON-EUR) (pp. 1–6). IEEE (2021). * Tabassum, S., Sampa, M., Islam, R.,

Yokota, F., Nakashima, N., & Ahmed, A. A data enhancement approach to improve machine learning performance for predicting health status using remote healthcare data. In 2020 2nd

International Conference on Advanced Information and Communication Technology (ICAICT) (pp. 308–312). IEEE (2020). * Sampa, M. _et al._ Blood uric acid prediction with machine learning:

Model development and performance comparison. _JMIR Med. Inform._ 8(10), e18331 (2020). Article  Google Scholar  * Tabassum, S., Sampa, M., Maruf, R., Yokota, F., Nakashima, N., & Ahmed,

A. An analysis on remote healthcare data for future health risk prediction to reduce health management cost. In APAMI 2020: 11th Biennial Conference of the Asia-Pacific Association for

Medical Informatics (pp. 115–119). (2020). * Hossain, M., Okajima, H., Kitaoka, H., Yokota, F., & Ahmed, A. eHealth Consumer Behavior. In Behavior Engineering and Applications (pp.

129–147). Springer, Cham (2018). * Hayashi, T., Gyohten, K., Ohki, H., & Takami, T. A study of data augmentation for handwritten character recognition using deep learning. In 2018 16th

International Conference on Frontiers in Handwriting Recognition (ICFHR) (pp. 552–557). IEEE (2018). * Zhang, X. Y., Yin, F., Zhang, Y. M., Liu, C. L. & Bengio, Y. Drawing and

recognizing Chinese characters with recurrent neural network. _IEEE Trans. Pattern Anal. Mach. Intell._ 40(4), 849–862 (2017). Article  Google Scholar  * Dhar, D., Garain, A., Singh, P.

& Sarkar, R. HP_DocPres: A method for classifying printed and handwritten texts in doctor’s prescription. _Multimed. Tools Appl._ 80(7), 9779–9812 (2020). Article  Google Scholar  *

Fajardo, L., Sorillo, N., Garlit, J., Tomines, C., Abisado, M., Imperial, J., Rodriguez, R., & Fabito, B. Doctor’s cursive handwriting recognition system using deep learning. In 2019

IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM) (pp. 1–6). IEEE (2019). * Bunke, H.

_IAM handwriting database_ (INF, University of Bern, Research Group on Computer Vision and Artificial Intelligence, 2000). * Chakraborty, B., Mukherjee, P., & Bhattacharya, U. Bangla

online handwriting recognition using recurrent neural network architecture. In Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing (pp. 1–8) (2016).

* Biswas, Mithun _et al._ BanglaLekha-Isolated: A multi-purpose comprehensive dataset of Handwritten Bangla Isolated characters. _Data Brief._ 12, 103–107 (2017). Article  Google Scholar  *

Bhattacharya, U. & Chaudhuri, B. B. Handwritten numeral databases of Indian scripts and multistage recognition of mixed numerals. _IEEE Trans. Pattern Anal. Mach. Intell._ 31(3), 444–457

(2009). Article  Google Scholar  * Sarkar, R. _et al._ CMATERdb1: A database of unconstrained handwritten Bangla and Bangla-English mixed script document image. _Int. J. Doc. Anal.

Recognit. (IJDAR)_ 15(1), 71–83 (2012). Article  Google Scholar  * Priya, A., Mishra, S., Raj, S., Mandal, S., & Datta, S. Online and offline character recognition: A survey. In 2016

International Conference on Communication and Signal Processing (ICCSP) (pp. 0967–0970). IEEE (2016). * Rabby, A., Haque, S., Islam, S., Abujar, S. & Hossain, S. Bornonet: Bangla

handwritten characters recognition using convolutional neural network. _Proc. Comput. Sci._ 143, 528–535 (2018). Article  Google Scholar  * Yuan, A., Bai, G., Jiao, L., & Liu, Y. Offline

handwritten English character recognition based on convolutional neural network. In 2012 10th IAPR International Workshop on Document Analysis Systems (pp. 125–129). IEEE. (2012). * Yang,

W., Jin, L., & Liu, M. Character-level Chinese writer identification using path signature feature, dropstroke and deep CNN. arXiv preprint arXiv:1505.04922. (2015). * Achkar, R., Ghayad,

K., Haidar, R., Saleh, S., & Al Hajj, R. Medical Handwritten Prescription Recognition Using CRNN. In 2019 International Conference on Computer, Information and Telecommunication Systems

(CITS) (pp. 1–5). IEEE (2019). * Tabassum, S., Takahashi, R., Rahman, M., Imamura, Y., Sixian, L., Rahman, M., & Ahmed, A. Recognition of doctors’ cursive handwritten medical words by

using bidirectional LSTM and SRP data augmentation. In 2021 IEEE Technology & Engineering Management Conference-Europe (TEMSCON-EUR) (pp. 1–6). IEEE (2021). * Schuster, M. & Paliwal,

K. Bidirectional recurrent neural networks. _IEEE Trans. Signal Process._ 45(11), 2673–2681 (1997). Article  ADS  Google Scholar  * Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I.

& Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. _J. Mach. Learn. Res._ 15(1), 1929–1958 (2014). MathSciNet  MATH  Google Scholar  * Prechelt, L.

Early Stopping—but when?. In Neural Networks: Tricks of the Trade: Second Edition (pp. 53–67). Springer, Berlin, Heidelberg (2012). * Nwankpa, C., Ijomah, W., Gachagan, A., & Marshall,

S. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378 (2018). * Nagaoka, T. Hyperparameter optimization for deep

learning-based automatic melanoma diagnosis system. _Adv. Biomed. Eng._ 9, 225–232 (2020). Article  Google Scholar  * Zhang, Z., & Sabuncu, M. R. Generalized cross entropy loss for

training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS). (2018). * Kingma, D. P., & Ba, J. Adam: A method for stochastic

optimization. arXiv preprint arXiv:1412.6980. (2014). * Regulation, G. D. P. General data protection regulation (GDPR). Intersoft Consulting, Accessed in January 2022. (2018). * Wu, Z. _et

al._ A location privacy-preserving system based on Query range cover-up or location-based services. _IEEE Trans. Veh. Technol._ 69(5), 5244–5254 (2020). Article  Google Scholar  * Wu, Z.,

Shen, S., Lian, X., Su, X. & Chen, E. A dummy-based user privacy protection approach for text information retrieval. _Knowl. Based Syst._ 195, 105679 (2020). Article  Google Scholar  *

Wang, G. G., Deb, S. & Cui, Z. Monarch butterfly optimization. _Neural Comput. Appl._ 31(7), 1995–2014 (2019). Article  Google Scholar  * Wang, G. G., Deb, S. & Coelho, L. D. S.

Earthworm optimisation algorithm: a bio-inspired metaheuristic algorithm for global optimisation problems. _Int. J. Bio-Inspired Comput._ 12(1), 1–22 (2018). Article  Google Scholar  * Wang,

G. G., Deb, S., & Coelho, L. D. S. Elephant herding optimization. In 2015 3rd International Symposium on Computational and Business Intelligence (ISCBI) (pp. 1–5). IEEE (2015). * Wang,

G. G. Moth search algorithm: A bio-inspired metaheuristic algorithm for global optimization problems. _Memetic Comput._ 10(2), 151–164 (2018). Article  Google Scholar  * Heidari, A. A. _et

al._ Harris hawks optimization: Algorithm and applications. _Futur. Gener. Comput. Syst._ 97, 849–872 (2019). Article  Google Scholar  Download references ACKNOWLEDGEMENTS The authors are

grateful to the team in Global Communication Center (GCC) in Grameen Communications, Bangladesh for collecting data samples for creating the ‘Medical Term Corpus’. Kaze Shindo and Ryo

Takahashi of our Social Tech Lab conducted the data augmentation experiments. FUNDING No funding has been received for the research work undertaken in this manuscript. AUTHOR INFORMATION

AUTHORS AND AFFILIATIONS * Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, Japan Shaira Tabassum, Nuren Abedin & Ashir Ahmed * Global Communication

Center, Grameen Communications, Dhaka, Bangladesh Md Mahmudur Rahman, Rafiqul Islam & Ashir Ahmed * Graduate School of Biomedical and Health Sciences, Hiroshima University, Hiroshima,

Japan Md Moshiur Rahman * M A G Osmani Medical College, Sylhet, Bangladesh Mostafa Taufiq Ahmed * Medical Information Center, Kyushu University Hospital, Fukuoka, Japan Rafiqul Islam Authors

* Shaira Tabassum View author publications You can also search for this author inPubMed Google Scholar * Nuren Abedin View author publications You can also search for this author inPubMed 

Google Scholar * Md Mahmudur Rahman View author publications You can also search for this author inPubMed Google Scholar * Md Moshiur Rahman View author publications You can also search for

this author inPubMed Google Scholar * Mostafa Taufiq Ahmed View author publications You can also search for this author inPubMed Google Scholar * Rafiqul Islam View author publications You

can also search for this author inPubMed Google Scholar * Ashir Ahmed View author publications You can also search for this author inPubMed Google Scholar CONTRIBUTIONS S.T., N.A., and A.A.

conducted the experiments, concluded results and findings, and wrote the whole manuscript. M.Ma.R. assisted in data collection. R.I. assisted in data collection and provided guidance in data

privacy. M.Mo.R. and M.T.A provided knowledge on current medical prescription mechanisms in Bangladesh, introduced medical professionals for data collection and commented on the findings.

CORRESPONDING AUTHOR Correspondence to Shaira Tabassum. ETHICS DECLARATIONS COMPETING INTERESTS The authors declare no competing interests. ADDITIONAL INFORMATION PUBLISHER'S NOTE

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. RIGHTS AND PERMISSIONS OPEN ACCESS This article is licensed under a

Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit

to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are

included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons

licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of

this licence, visit http://creativecommons.org/licenses/by/4.0/. Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE Tabassum, S., Abedin, N., Rahman, M.M. _et al._ An online

cursive handwritten medical words recognition system for busy doctors in developing countries for ensuring efficient healthcare service delivery. _Sci Rep_ 12, 3601 (2022).

https://doi.org/10.1038/s41598-022-07571-z Download citation * Received: 19 October 2021 * Accepted: 21 February 2022 * Published: 04 March 2022 * DOI:

https://doi.org/10.1038/s41598-022-07571-z SHARE THIS ARTICLE Anyone you share the following link with will be able to read this content: Get shareable link Sorry, a shareable link is not

currently available for this article. Copy to clipboard Provided by the Springer Nature SharedIt content-sharing initiative