Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Computed Lipsync Improvements #89

Open
drojf opened this issue Sep 24, 2022 · 0 comments
Open

Computed Lipsync Improvements #89

drojf opened this issue Sep 24, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@drojf
Copy link
Contributor

drojf commented Sep 24, 2022

I noticed that the new computed lipsync sometimes has the mouth completely closed near the end of a sentence, if the character is speaking softly.

This is probably due to the current algorithm using fixed thresholds with no hysteresis, so if a character is speaking just below the threshold (currently 30% off max volume), the mouth will always stay closed.

Another related issue is that sometimes a character's mouth will be 'stationary'.

I can think of a couple things to try to fix this:

  1. Try to detect changes in volume + also use fixed thresholds to determine which mouth to use
    a. If the long term average volume is say, less than 50%, always use fully closed or half-open mouth (talking softly)
    b. If the long term average volume is greater than 50%, always use the half-open or fully open mouth (talking loudly)
    c. If the volume suddenly increases, use the more open mouth position, otherwise, use the more closed position
    d. Only hold the mouth in the fully closed position if the volume becomes very low (another threshold? like 10% or 5%)
  2. Use fixed thresholds but:
    a. Add hysteresis only when going from half-closed to fully closed, to prevent character's mouths closing too early.
    b. If a character remains in one mouth position too long, switch to another position even if the average volume hasn't changed
@drojf drojf added the enhancement New feature or request label Sep 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant