Paper Title
An End-to-End Deep Learning Framework for Automated Multilingual Video Dubbing

Abstract
The growing demand for producing multimedia content in multiple languages has led to the need to democratise the production of automated video dubbing solutions, with the ability to translate and synthesise speech while maintaining the audiovisual consistency. This paper proposes a kind of end-to-end deep learning approach for automated multilingual video dubbing, which integrates speech recognition, neural machine translation, speech synthesis and lip synchronization in a unified pipeline. Everything audio captured from the input video is transcribed based on Whisper automatic speech recognition model. The transcribed parts are translated with the NLLB - 200 neural machine translation model and rendered to speech using the XTTS multilingual voice - cloning model. Wav2Lip is then used to synchronize the generated speech with the moving lipids of the mouth of the speaker for automated translation of the speech-to-speech video at the same time preserving the audiovisual coherence. Keywords - Multilingual Voice Dubbing, Automatic Speech Recognition, Neural Machine Translation, Text-to-Speech Synthesis, Audio-Video Synchronization