- YouTube plans to launch AI-generated lip sync to automatically dubbed videos.
- It may soon start testing the feature in English, French, German, Portuguese, and Spanish.
- The feature may incur additional fees, which will likely have to be borne by users.
Whether you like it or not, Google isn’t backing down on its attempts to increase engagement by automatically translating audio into your native tongue. Now, following continuous and unwavering stabs at simply translating audio with autodubbing, YouTube may take it a notch higher with realistic-looking audio dubs with accurate lip sync in an attempt to make videos more appealing with AI.
YouTube’s product lead for Auto-dubbing, Buddhika Kottahachchi, recently detailed the technology behind lip-synced autodubs in a conversation with Digital Trends. Kottahachchi informed the publication that the system makes intricate pixel-level changes to modify the speaker’s mouth, ensuring the dub looks realistic. It is powered by a custom AI built to understand the nuances of facial structure, including 3D perception of the lips, teeth, and even different facial expressions. Looking at Google’s success with its Veo3 text-to-video model, we believe it should be able to accomplish the feat with convincing levels of realism.