Design and Implementation of A Sign-to-

Speech/Text System for Deaf and Dumb People

DalalAbdulla, Shahrazad Abdulla and Rameesa Manaf Anwar H. Jarndal

Electrical and Computer Engineering Department Electrical and Computer Engineering Department
University of Sharjah University of Sharjah
Sharjah 27272, UAE Sharjah 27272, UAE
[email protected] [email protected]

Abstract— this paper presents an approach for designing paper, a sign to Arabic language speech/text system is
and implementing a smart glove for deaf and dumb people. developed.
There have been several researches done in order to find an According to the best knowledge of the authors, there is no
easier way for non-vocal people to communicate with vocal commercial product for Arabic language and this paper is one
people and express themselves to the hearing world. of the initiative researches in this area. The main advantages
Developments have been made in sign language but mainly in of the developed system, with respect to others, are its
American Sign Language. This research aims to develop a simplicity, low cost, low power and its full mobility. Also, it is
sign to Arabic language translator based on smart glove a hand-glove based and thus it can be used even in dark
interfaced wirelessly with microcontroller and text/voice environment, and there is no restriction on the user movement.
presenting devices. An approach has been developed and Furthermore, only one hand is used to represent all signs,
programmed to display Arabic text. The whole system has which makes it easier and more comfortable. In this system,
been implemented, programmed, cased and tested with very the sign recognition is performed by flex sensors based glove
good results. interfaced to microcontroller wirelessly. The microcontroller
implements an in-house developed program to recognize the
Keywords—sign language, glove, gesture recognition. sign and implement the corresponding Arabic alphabet vocally
and textually. This is easier than sign to word translator, which
I. INTRODUCTION needs complicated system with a lot of hand, facial and body
The serious issue for the deaf and dumb community is gestures to convey the meaning [7]. Thus, an advantage for the
obviously the difficulty in communicating with vocal people. proposed system emphasizes on its capability to translate sign
These people communicate via sign language; however, the to letter, which has low cost and it can be used to represent
main issue is that the majority of people are not familiar with wider range of words. Further, the system is simpler and needs
sign language and they are not willing to learn this language. fewer sensors, and this adds extra reduction for the cost.
This generated an idea to propose this project in which it will
drastically facilitate and improve a communication method
between the non-vocal and vocal people.
Many researches have been conducted in the last ten years
to develop sign to speech/text translators. Most of these are
based on computer vision techniques for sign recognition [1]-
[4]. The efficiency of these techniques depends on the video
quality, which is significantly impacted by the ambient light
Figure 1: Arabic sign language. [9]
and situation of the implemented camera (resolution, focusing,
alignment and mechanical-vibration). Such effects may result II. ARABIC SIGN LANGUAGE
in noisy pictures that reduce the system capability to identify
the sign and recognize the corresponding meaning. Generally, A sign language is movements or actions used by people to
the applicability of these systems is limited in dark areas and communicate with others, without using the vocal codes. It
the user mobility (the user should be properly alignment with combines facial expressions, hand gestures or movement of
the camera). Another system described in [5] presents hand- arms to give the audience an indication of what the person is
gloves based sign to speech/text system. All of these systems trying to convey or say. Arabic sign language is a family of
are computer based, which accordingly increases the cost, size sign language, which is used mainly in the Middle East. As it
and power, and reduces the mobility of these systems. Also, is well known, the Arabic alphabet contains 28 letters and
all of these systems are designed for English language. In this diacritics (see Figure 1), in which each of these letters has its

978-1-5090-5306-3/16/$31.00 ©2016 IEEE

Authorized licensed use limited to: S G BALEKUNDRI INSTITUTE OF TECHNOLOGY - Belagavi. Downloaded on November 16,2024 at 05:19:40 UTC from IEEE Xplore. Restrictions apply.
 own form. In UAE, there is an Emirati Sign Language which A. Hardware
is taught in most of the centers [8]. In the system design, flex sensors are attached to a hand
glove, along each finger length, to work as a gesture
recognition. The output of the flex sensors will vary with the
bending degree of each finger, and the analog output voltages,
which are obtained from these flex sensors, are fed into the
microcontroller. The microcontroller processes these analog
signals and converts them into digital signals. To make it more
flexible for the user, there are three push buttons connected to
the Arduino nano, one of them is to erase one character, the
other one is to clear everything written on LCD and the last
Figure 2: Block diagram of the design.
one is to read from the LCD. Moreover, the program stored in
III. SYSTEM DESIGN the microcontroller will also convert the recognized gesture
The block diagram in Figure 2 shows the entire design of into its equivalent text information. RF trans-receiver has been
the system, which includes the transmitter side and receiver connected to the Arduino to transmit the digital signals
side. In the transmitter side, there are five flex sensors wirelessly. In the receiver side, the digital signals obtained
attached to the glove to detect the bending of each finger. The will be received wirelessly via RF trans-receiver. Finally, the
controller used in the transmitter side is Arduino NANO text information will be displayed on the LCD screen and the
interfaced with RF transmitter. The transmitter is used to text to speech conversion will play out the sound through the
transmit the signals to the receiver. The flex sensors and RF speaker.
transmitter are interfaced with the microcontroller. In the B. Software
receiver side, there is an RF module to receive the signals. The Arduino Software (IDE) was used to program the system.
microcontroller used in the receiver side is Arduino There are two codes, one for the transmitter side and the other
Mega2560. TFT LCD is implemented to display Arabic one is for the receiver side. The flow charts of the codes are
letters, and the Emic-2 module works to produce the sound illustrated in Figures 4 and 5. Figure 4 shows the flow chart of
through 8 Ohms speaker. EMIC 2 TEXT-TO-SPEECH the software design in the transmitting side, while the receiver
MODULE is used to produce the Arabic sound where in side flow chart of the software is shown in Figure 5. Figure 4
EMIC 2 is only defined for English and Spanish letters sounds shows how the code in the transmitter side helps in
according to the specification sheets. So the system did not transmitting the sign which user interprets. Once the user
produce genuine Arabic sounds for letters like “ ‫ع غ ح خ ص ض‬ shows the gesture, the code in the transmitter side checks
‫”ظ ط‬. For example we have used the sound of “Haa” for whether the values are in range for the particular letter and if it
“‫”ح‬,"khaa" for “‫”خ‬, "ain" for “‫”ع‬, "ghain" for “‫”غ‬, "saad" for is in range the transmitter transmits, else the code will wait for
“‫”ص‬, "dhaad" for “‫”ض‬, "dwaa" for “'‫”ط‬, "dhwaa" for “‫”ظ‬. new values and the process keeps repeating.
The components are interfaced to the microcontroller. Figure 3
shows the implemented components to design the system.

IV. SYSTEM IMPLEMENTATION AND PROGRAMING

This part presents the system implementation including the
hardware and software. The hardware of this system
represents the interconnected devices, circuits, modules and
micro-systems. The software denotes the developed algorithm
and the implemented programming language.

Figure 4: A flow chart for the software transmitter side.

Figure 3: The implemented components.

Figure 5: A flow chart for the software of receiver

Authorized licensed use limited to: S G BALEKUNDRI INSTITUTE OF TECHNOLOGY - Belagavi. Downloaded on November 16,2024 at 05:19:40 UTC from IEEE Xplore. Restrictions apply.
 Figure 5 shows how the code in the receiver side works. Once the Hexadecimal values are obtained, it is copied into the
Once the receiver receives a value, the code in the receiver header file of the font to be used in the project. The project
side reads the value received and checks whether the letter uses TFT LCD which is supported by the Adafruit GFX
received meets the requirements to be displayed like it is the libraries and the font selected is FreeMono12pt7b. Thus, the
proper alphabet and then it will be displayed on to the LCD changes are made in this header file.
screen and it can also read from the LCD upon pressing the
push button for sound. If value received is not a proper letter it
will discard and wait for new set of values from the
transmitter. This process will be repeated as long as there are
values being transmitted from the transmitter side
C. Representation of Arabic letter on TFT LCD
Arabic language is unique among other languages; it
contains some letters like "‫ "ض‬which is not present in other
languages therefore it is also known as the "‫ "ض‬language.
There is no readymade similar display product for direct Figure 7: Representing the letter "‫ "غ‬in hexadecimal.
representation of Arabic alphabet such as English ones. This
adds a challenge to be overcome by this project. After too
many researches, it has been realized how English letters are Hexadecimal value obtained from the open office software is
then copied into header file as it can be seen from Figure 8.
represented on the LCD’s, and this method is applied to
After that, it goes to each letter by letter according to the
characterize the Arabic letters. This was achieved by changing Arabic key board format and changes its values by multiples of
one of the English fonts into Arabic by making changes in the 32. Multiples of 32 are used because the matrix is 16x16 and
header file of the English font to denote the Arabic letters. the first value is 0. For example, for letter “‫ ”ا‬the corresponding
Representation of Arabic letters was done in the project by value in the keyboard is “h” so in the header file changes are
defining a 16x16 matrix in the open office software, which made in the matrix of “h” as presented in Figure 9. The same
works similar to an excel file as depicted in Figure 6. The method is continued for the remaining letters like making letter
Arabic letters are drawn in this matrix according to its original “f” to represent Arabic letter “‫”ب‬.
form. Conditional formatting is given in the matrix in a way
that the pixels needed is represented as number 1 which will
be highlighted in black color once the value given is 1,
remaining can be any number and it will stay as white. As an
example representing Arabic letter “‫ ”غ‬is shown in the Figure
6.

Figure 8: Header file FreeMono12pt7b.

Figure 9: Updating header file corresponding to letter h.

Figure 6: Representing the letter "‫"غ‬. V. SYSTEM REALIZATION AND TESTING

Converting the matrix values into binary values (0’s and 1’s) For the system implementation the chosen glove is from the
is done in a new matrix of 16x16 as shown in Figure 7. The north face company which is called ETIP glove. This glove is
values are then converted into its equivalent hexadecimal value made out of cloth and it is very comfortable to wear, in
each row by row and 8 bits by 8 bits using the following addition, it has a really incredible advantage which is that it is
formula: touchscreen connectivity, therefore it can be used while using
"0x"&BIN2HEX(T22&U22&V22&W22&X22&Y22&Z22& touchscreen phone in our daily lives. The case of the system
AA22)&","&"0x"&BIN2HEX(AB22&AC22&AD22&AE22& for the receiver side is made up of plastic which gives it light
AF22&AG22&AH22&AI22)&",". weight to carry along.
The five flex sensors are connected with five 10 kΩ resistors,
This formula is an built-in formula in excel which helps in which are interfaced with the Arduino NANO in the
converting binary data into a hexadecimal string representation. transmitter side. As can be seen in Figure 11, each flex sensor
is connected between +5 V Battery and the resister 10 kΩ

Authorized licensed use limited to: S G BALEKUNDRI INSTITUTE OF TECHNOLOGY - Belagavi. Downloaded on November 16,2024 at 05:19:40 UTC from IEEE Xplore. Restrictions apply.
 resistor. To make the system wireless we have added represents the Arabic sign language letters as a text on LCD
NRF24L01 transmitter receiver module to the transmitter and and outputs an audio through the speaker, which helps in
receiver side which is interfaced directly to the Arduino. limiting the communication barrier between deaf and dumb
Along with the RF module in the receiver side, a graphical with vocal people. The important of the research is related to
TFT LCD has been connected on the Arduino Mega 2560 to its aim to help these class of non-vocal people to communicate
display the transmitting text (see Figure 12). The battery is with others and improve their contributions to growth and
controlled by a switch. Finally, Emic-2 is connected to build their nations. The system has been designed,
Arduino Mega 2560 to display the sound through 0.5-watt programmed, implemented and tested with a very good
speaker of 8 Ω. All these components are interconnected and results. The system could be extended to cover wider range of
mounted inside a black enclosure to finalize the design. Figure sign by using a combination of two gloves instead of one.
13 shows the final realized system.

Figure 14: Demonstration for a sign (see Figure 1) to the corresponding

Figure 11: Connection of flex sensors.
displayed alphabet “‫”ا‬.

ACKNOWLEDGMENT
The authors gratefully acknowledge the support from the
University of Sharjah, Sharjah, UAE.
REFERENCES
[1] Starner, T. and Pentland, A., "Visual recognition of American Sign
Language using hidden Markov models," in Proceedings of the
International Workshop on Automatic Face and Gesture Recognition,
1995.
[2] Starner, T., Weaver, J., and Pentland, A., "Real-time American Sign
Figure 12: Receiving side Language recognition using desk and wearable computer based video,"
IEEE Transactions onPattern Analysis and Machine Intelligence, vol.
The final realized system has been tested and the results 20, no. 12, pp. 1371-1375, 1998.
were very good. Once the person wearing the smart glove dose [3] Bauer, B., Hienz, H., and Kraiss, K., "Video-based continuous sign
language recognition using statistical methods," in ICPR 2000, pp. 463-
an Arabic letter gesture, the LCD displays the letter and the 466, Vol II.
speaker outputs the voice when the sound button is clicked. [4] Bauer, B., Hienz, H., and Kraiss, K., "Video-based continuous sign
Figure 14 show some of the results. language recognition using statistical methods," in ICPR 2000, pp. 463-
466, Vol II.
[5] Brashear, H., Park, K.-H., Lee, S., Henderson, V., Hamilton, H., and
Starner, T., "American Sign Language recognition in game development
for deafchildren," in ASSETS 06, (New York, NY, USA), pp. 79-86,
2006.
[6] ZahoorZafrulla, Helene Brashear, Thad Starner, Harley Hamilton,and
Peter Presti. 2011. American sign language recognition with the kinect.
In Proceedings of the 13th international conference on multimodal
interfaces (ICMI '11). ACM, New York, NY, USA, 279-286.
[7] Olga Katzenelson and Solange Karsenty, “A Sign-to Speech Glove,” in
the Proceedings of the 3nd IUI Workshop on Interacting with Smart
Objects, Haifa,Israel, February 2014, pp.19-23.
[8] ZahoorZafrulla, Automatic Recognition of American Sign Language,
Ph.D. Dissertation, Georgia Institute of Technology, USA, 2014.
[9] M. A. Abdel-Fattah, “Arabic Sign Language: A Perspective,” (Journal),
in Department of Languages and Translation, Palestine,” Birzeit
Figure 13: Final realized system. University, August 24, 2004.
[10] http://www.nawiseh.com/dawrat/esharah/.
[11] http://arduinosensors.com/index.php/interfacing-flex-bend-sensor-with-
VI. CONCLUSION arduino/.
This paper presents glove-based sign-to-text/voice
translating system for deaf and dumb people. The glove

Authorized licensed use limited to: S G BALEKUNDRI INSTITUTE OF TECHNOLOGY - Belagavi. Downloaded on November 16,2024 at 05:19:40 UTC from IEEE Xplore. Restrictions apply.