This project investigates the effectiveness of using large language models (LLMs) for direct comprehension of document images, aiming to bypass traditional OCR-to-text preprocessing. By allowing the LLM to analyze raw document images, including text, tables, graphs, and visual elements, we explore whether this approach (termed "direct composition") yields better understanding and insights compared to the conventional cascading approach, where OCR is first applied to extract text before LLM processing. We aim to determine whether direct image-to-LLM comprehension improves interpretative accuracy and information retrieval, especially for visually rich documents, and to assess whether this method serves as a complement or alternative to OCR-based processing.

This project aims to develop a pronunciation assessment pipeline for Quranic Arabic, focusing on the accurate evaluation of letters (consonants) and diacritics (vowels). The pipeline includes designing and preparing datasets for training and evaluation, utilizing recordings in Modern Standard Arabic (MSA) style, including Quranic Arabic readings without Tajweed rules. Both orthographic and phonetic transcriptions will be employed to enable character- and phoneme-level evaluation and feedback.
The evaluation dataset will emphasize Quranic Arabic readings and include a subset with deliberate mispronunciations to assess the system's capability for error detection. Preprocessing will involve transcription, vowelization and phonetic annotation.
Model training will focus on fine-tuning existing speech representation models, such as MMS, Hubert, and wav2vec, with considerations for incorporating synthetic and mispronounced data to improve robustness. Evaluation metrics will include Word Error Rate (WER), Character/Phoneme Error Rate (CER/PER), and specific measures for mispronunciation detection, such as False Rejection Rate (FRR), False Acceptance Rate (FAR), and Diagnosis Error Rate (DER).
The initial phase of the project will concentrate on developing a Quranic pronunciation system without Tajweed rules. Subsequent phases will address the challenges of assessing Tajweed recitation pronunciation.

In this project we will extract features relevant for SLT in a gloss-free system and prepare the trainslation system based on the T5 model and / or SignLLaVa. It has been shown that visually pretrained models do not convey the relevant information for the SignLLaVa system and they need to be aligned with the natural (textual) representation of the language. Hence, we want to design a system that will align these SL features with the language features. The systems will be trained on American SL and Arabic SL. Minor effort will be also put into pre-processing of data with multiple signers present.

How do you estimate performance of a multilingual model in novel multilingual scenarios? We are aiming in the long term to develop synthetic code-switching benchmarks as a proxy-evaluation task, ideally capable of both ranking the performance of various multilingual models and estimating their performance in novel unseen domains. Project outcomes include methods for generating synthetic data for codeswitching that could work for arabic dialects as well as benchmarking different ASR approaches on real and synthetic benchmarks.