● Entered Open Source Higgsfield ComfyUI Contest Judging through Nov 6 See it ▸
Humanjava Enterprises · VidBnch

LSDub

Submissions close in
--d--h--m--s

Submissions for Open Source Higgsfield close Nov 3, 9:00 AM PT. LSDub is our entry in the Open Weights, Most Innovative and LTX categories.

Match the lip-sync to the audio, without re-rendering the whole shot.

Built for music videos. Spoken dialogue, film dubbing and other uses are still in development.

Photoreal lip-sync dubbing that warps the face's own mouth (its teeth, gums, lips and tongue) to say new words. No painted mouth. No face swap. Open weights only. Every pixel on screen is the clip's own, moved to fit the audio frame by frame.

Our entry in the Open Source Higgsfield ComfyUI contest.

Watch the 2-min demo ▸ Get it on GitHub How it works

Render the right words in your videos.

The two-minute tour: how LSDub re-syncs a singer without re-rendering the shot, and how it scores every shot. The teeth, the gums, the tongue: all harvested from the face itself, then warped to the new vocal.

Don't paint a mouth. Warp the one that's already there.

Every other lip-sync tool paints a new mouth over the face or swaps the face outright. LSDub does neither. We harvest the clip's own mouth, build it in a canonical dental-arch atlas, and warp that atlas to the audio target for every frame. The result is the person's real teeth and lips — never a model's guess.

Own pixels

Teeth, gums, tongue and lips are the face's own, lifted from its best frames and re-posed. No inpainting, no painted enamel, no borrowed smile.

Open weights

Built on LTX-2.5 and open-weights tooling only. No partner APIs, no closed models — qualifies for Open Weights Only + the LTX bonus.

Any face

One vocal track drives a whole gallery of faces. The same words, the same timing, across every character — a face-chain you play.

One pipeline. Every sung shot.

A music video isn't one face. It's sixty cuts, two singers, wides, inserts and a mic in the way. LSDub runs the whole cut, shot by shot.

  1. castFind the right singer in every shot. Enroll the characters, then track and identify them across the clip and plan the dub from the lyrics.
  2. grammarThe shot-size gate. Every sung word must land on a big enough face, and it has to be its own singer's. Close-up, medium or wide.
  3. singThe standard deep dive on face and lips, run for each shot. Crop the singer to 512 px, build that shot's own mouth kit, warp at 2× fps, snap the p/b/m closures, then gate on measured sync. --auto also renders the native warp and keeps whichever syncs better.
  4. mediumTier 2 for a 120–300 px face: crop, upscale, warp, scale back down. Only the feathered lower face is pasted back, and everything outside it stays bit-exact.
  5. occludersMic and hand masks from open-vocabulary phrase grounding, so the warp goes behind the mic and never over it.
  6. continuityEach painted frame is checked against the last four, carried along the take's own motion. A frame that jumps, smears or shows a ghost lip is flagged before anyone sees it.

Three skill sets, by shot size.

The face's size in the delivered frame decides what LSDub does with it. The lines are parameters.

Close-up · ≥ 240 px

Hooks belong here. The singer is cropped to 512 px anyway, which zooms close-ups about 1.9–2.8×, and gets the full deep dive: shot kit, teeth atlas, 2× fps snap and closure press.

Medium · 120–240 px

For verses. The face is tracked, cropped, upscaled, warped and scaled back down. Only the lower face is pasted, and on track gaps the source passes through untouched.

Wide · 25–120 px

Too small to dub, so this tier is for held notes. The grammar gate flags any sung attack that falls on a wide so you can recut it. LSDub won't fake a mouth it can't read.

High-quality analysis at each stage of the render.

LSDub has a number of checkpoints for ensuring improved results. The QA suite of tools enables video, shot, audio and sync analysis.

Four full-face cuts from the Pleiadians Don't Dance demo: the original take next to the LSDub re-sync, first at regular speed, then the same four cuts slowed to half speed. Each panel shows the vocal waveform, the mouth-opening curves and the live mouth-shape card. Measured sync 0.90 to 0.97.

Before the dub

qa triage new
Auto-detects the shots in a contributed video and marks each one DUB, DUB-MEDIUM or NO, with a reason.
cast
Who is in each shot, and whether their face is safe to dub.
grammar
Checks that every sung word falls on its own singer's face at a big enough size.

Sync

sync-qa
A sync QA video of the mouth opening against the vocal, which also works on a delivered film.
check
Lip-sync checker: viseme agreement, word hits and jaw correlation.
strip
A QA strip that scrolls the vocal waveform under the video, past a fixed playhead.
lag-audit
Is the mouth late? Measured in frames.
snap
Rise time vs vocal onsets, p/b/m closure hit rate and open-in-silence.
annotate
A card on every frame with the sung word, its mouth shape, the openness and the next second of visemes.

The mouth itself

naturalness
Compares the render's mouth motion with the take's own, statistically.
plane-fit
Checks that the mouth sits on the face's 3D plane.
slip-audit
Does the mouth stay pinned? Philtrum and lip slip against the nose, in px.
tooth-scale
Checks the teeth are the face's own size, measured against the IPD.
skull-audit
Midline, axis and teeth-plane alignment.
ghost-check
Fails any frame that shows a second lip edge.
layer-reg
Registration of teeth, cavity and tongue against the lips over time.

Motion

cadence
Motion cadence: finds held and skipped frames.
motion-audit
Motion in the shot: boil, ghost, slide and step.
frame-gate
Checks each frame against its neighbours and the take, giving PASS, SOFT or FAIL plus the causes.

After the dub

qa verify new
How likely each dubbed shot is to have worked, measured on the dubbed video.
scorecard
Scores the film shot by shot, worst first. --compare diffs it against the last cut.
reel
A before/after reel.

A score for every shot.

The scorecard sorts each cut into a class first, so you aren't blamed for editorial choices. Only real performance shots get a sync score.

PERF

A sung shot with the singer at medium or closer and the mouth visible at least 50% of the time. Scored 0–100: PASS from 75, REVIEW from 55, FAIL below 55. A mouth that barely moves is capped at 55.

COVERAGE

Sung, but there's no readable singer: no face, a face that's too small, or a hidden mouth. It shows as grey n/a with the reason. It's an editorial note, not a lip-sync defect.

INSERT

An unsung shot. Only motion, pops and sharpness are checked, and the score is capped at 90 so an insert never outranks a verified sung shot.

Reasons follow a fixed priority: no singer on screen → singer too small → mouth hidden → sync. A shot is never blamed for sync when the real cause is coverage.

Pleiadians · landing cut v3_2 → v3_4
cut 51 · 11C · medium 186 px28 → 85
cut 19 · 05B · Red Nova96
cut 16 · 05A · Red Nova95
cut 58 · Sky Lux close-up88

64 cuts, split into 8 PERF, 45 COVERAGE and 11 INSERT. Fixing continuity took cut 19's measured sync correlation from 0.14 back up to 0.97.

Scorecard contact sheet: v3_4 vs v3_2, shot by shot
Four full-face cuts from the Pleiadians Don't Dance demo: the original take next to the LSDub re-sync, first at regular speed, then the same four cuts slowed to half speed. Each panel shows the vocal waveform, the mouth-opening curves and the live mouth-shape card. Measured sync 0.90 to 0.97.

Open, and clean.

The node pack, the four example workflows and the demo pack are on GitHub (Apache-2.0).

Open weights

Every model in the pipeline is open-weights, and there are no partner APIs or closed endpoints. You can run all of it on your own box.

CPU or GPU

Scan, paint and the QA suite run on any machine. Inside ComfyUI the heavier checks use the GPU automatically when one is free, and hand it back for the render.

ComfyUI nodes

LSDub ships as a ComfyUI node pack, beginning with the vocal stage: split the stem, then isolate the lead vocal that drives the mouth. It runs locally through ComfyUI's own API and the Comfy SDK, on your own ComfyUI and your own weights. No cloud required.

Built with LTX-2.5. Open weights only. Every mouth is the face's own. #OpenSourceHiggsfield