Sign Languages (SLs) are the main means of communication for deaf people. The access to SL is essential for the fulfillment of basic Human Rights, however there is a shortage of interpreters, which undermines these rights and often endangers the lives of the deaf, especially in cases of emergency or serious health incidents.
HealthSign proposes to develop an application for the automated interpretation of the Greek Sign Language (GSL) over internet with focus on the health services, which are the most common reason to seek for an interpreter. The high demand for interpreters is often not met or requires long waiting. Furthermore, the availability of interpreters facilitates the integration of the deaf community.
Vision is probably the only sensor modality that could be of practical use because (a) only vision can capture manual and non-manual cues, which provide essential information for SLR, (b) camera-equipped hand-held devices with powerful processors are a commodity nowadays and (c) recent advances in computer vision and machine learning render mainstream visual SLR a realistic option.
G1: Develop a database of GSL from native speakers with emphasis on health services
G2: Implement an internet-based platform for synchronous communication and interpretation with health professionals
G3: Develop in parallel a lightweight version which will be able to run on an embedded platform.
G4: Develop algorithms for recognition of SLs, using computer vision and deep learning using the hand and body/facial cues
G5: Implement the algorithms on embedded platforms using FPGAs
The main goal is the viability of the proposed application which will be achieved by (a) simple off-the-shelf equipment for the users (b) efficient implementation of the proposed algorithms (c) simple installation (d) development of a business plan to facilitate the longevity of the proposed product.
E. - V. Pikoulis, Bifis, A., Trigka, M., Constantinopoulos, C., and Kosmopoulos, D., "Context-Aware Automatic Sign Language Video Transcription in Psychiatric Interviews", Sensors, vol. 22, no. 7, p. 2656, 2022 DOI: https://doi.org/10.3390/s22072656
Bifis, A., Trigka, M., Dedegkika, S., Goula, P., Constantinopoulos, C. and Kosmopoulos, D., "A Hierarchical Ontology for Dialogue Acts in Psychiatric Interviews", The 14th PErvasive Technologies Related to Assistive Environments Conference, PETRA 2021, June 29-2 July, Corfu, Greece 2021. DOI: https://doi.org/10.1145/3453892.3461349
E. Sartinas, Psarakis, E., Antzakas, K., and Kosmopoulos, D., "A 2-D Wrist Motion Based Sign Language Video Summarization", British Machine Vision Conference - ORAL PRESENTATION. 2021
D. Kosmopoulos, Oikonomidis, I., Konstantinopoulos, K., Arvanitis, N., Antzakas, K., Bifis, A., Lydakis, G., Roussos, A., and Argyros, A., "Towards a visual Sign Language dataset for home care services", The 15th IEEE International Conference on Face and Gesture Recognition, vol. 1. pp. 622-626, 2020. DOI: https://doi.org/10.1109/FG47880.2020.00099
C. Constantinopoulos, D. Kosmopoulos, A. Argyros, I. Oikonomidis, V. Lampropoulou, K. Antzakas, C. Panagopoulos, A. Menychtas, and C. Theoharatos, "The HealthSign project, current state and future activities", The 10th International Conference on Information, Intelligence, Systems and Applications ”, IISA 2019, Patras, Greece, 15-17 July 2019.
D. Kastaniotis, D. Tsourounis, A. Koureleas, B. Peev, C. Theoharatos, and S. Fotopoulos, “Lip reading in Greek words at unconstrained driving scenario”, The 10th International Conference on Information, Intelligence, Systems and Applications ”, IISA 2019, Patras, Greece, 15-17 July 2019.
N. Arvanitis, C. Constantinopoulos, D. Kosmopoulos, "Translation of Sign Language Glosses to Text Using Sequence-to-Sequence Attention Models", The 15th International Conference on Signal Image Technology & Internet based Systems, SITIS 2019.
V. Nicodemou, I. Oikonomidis, A. Argyros, “Single-shot 3D hand pose estimation using radial basis function networks trained on synthetic data”, Pattern Anal Applic, 1-14, 2019, https://doi.org/10.1007/s10044-019-00801-7
F. Gouidis, P. Panteleris, I. Oikonomidis, A. Argyros, “Accurate Hand Keypoint Localization on Mobile Devices”, Machine Vision Applications (MVA 2019), Tokyo, Japan, May, 2019.
D. Kosmopoulos, "A Prototype Towards Modeling Visual Data Using Decentralized Generative Adversarial Networks," 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, 2018, pp. 4163-4167. doi: 10.1109/ICIP.2018.8451471
D. Kosmopoulos, A. Argyros, C. Theoharatos, V. Lambropoulou, C. Panagopoulos, and I. Maglogiannis. 2018. "The HealthSign Project: Vision and Objectives", In Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference (PETRA '18). ACM, New York, NY, USA, 502-506. DOI: https://doi.org/10.1145/3197768.3201547
The Signal Processing & Communications (SPC) Lab is a Laboratory of the Computer Engineering & Informatics Department (CEID), University of Patras. The SPC Lab presents a research activity in a variety of fields, such as: Signal Processing for Communications, Sensor Networks, System Identification etc.
The Institute of Computer Science has had a relatively long history and recognised tradition, since its establishment in 1983, in conducting basic and applied research, developing applications and products, providing services, and playing a leading role in Greece and internationally, in the fields of Information and Communication Technologies.
IRIDA Labs (Patras, Greece) is bridging the gap between a camera and a human eye by bringing visual perception to any device. They develop software in Computer Vision, using Image Processing and Machine Learning techniques made for any CPU, GPU or DSP/ASP platform or a combination of them using Heterogeneous Programming techniques.
Founded in 2013, BioAssist is the first Greek ICT services company created for seniors and chronic patients. The company's goal is to research, design and study remote monitoring, health support and well-being systems.
The Deaf Studies Unit (DSU) belongs to the Department of Primary Education of the University of Patras, and is a leading center for Deaf Studies, Greek Sign language (ENG), and Deaf Education in Greece. Established in 1994, is the only academic institution in Greece that provides research in the area of deafness, and programs for teachers of the deaf, sign language tutors, and interpreters.
Presentation of the HealthSign app on the O3 TV show...
The third meeting of all partners took place in Heraklion...
Video recordings in Athens have begun...
The second meeting of the HealthSign project...
Κick-off meeting took place in Patras...