sync. launches: api for realtime lipsync

sync. recently launched!

‍
^{"Animate anyone in any video to say (or sing) anything you want in any language."}

_{sync.labs is building audio-visual models to generate, modify, and synthesize humans in video.}

Founded by Prady Modukuru, Prajwal K R, and Rudrabha Mukhopadhyay

About

‍They’ve built a state-of-the-art lip-sync model – and they’re building towards real-time face-to-face conversations w/ AI indistinguishable from humans 🦾

Try Sync's playground here: https://app.synclabs.so/playground

How does it work?

Theoretically, their models can support any language — they learn phoneme / viseme mappings (the most basic unit / “token” of how sounds we make map to the shapes our mouths make to create them). It’s simple, but a start towards learning a foundational understanding of humans from video.

Why is this useful?

^{[1] They can dissolve language as a barrier}

Check out how they used it to dub the entire 2-hour Tucker Carlson interview with Putin speaking fluent English.

Imagine millions gaining access to knowledge, entertainment, and connection — regardless of their native tongue.

Realtime at the edge takes us further — live multilingual broadcasts + video calls, even walking around Tokyo w/ a Vision Pro 2 speaking English while everyone else Japanese.

^{‍[2] They can move the human-computer interface beyond text-based-chat}

Keyboard / mice are lossy + low bandwidth. Human communication is rich and goes beyond just the words we say. What if we could compute w/ a face-to-face interaction?

Maybe embedding context around expressions + body language in inputs / outputs would help us interact w/ computers in a more human way. This thread of research is exciting.

^{[3] and more}

Powerful models small enough to run at the edge could unlock a lot:

eg.

Extreme compression for face-to-face video streaming

Enhanced, spatial-aware transcription w/ lip-reading

Detecting deepfakes in the wild

On-device real-time video translation

etc.

‍
‍Learn More

_{🌐 Visit synclabs.so to learn more
‍
‍🧠 Let them know your feedback: Click here
‍
🤝 Play around with the API here: Click here
‍
‍👥 Follow sync.labs on LinkedIn & X}