Deepspeed .whl - 12.7 (main branch)

[S95Sedan]

Had a look at some of the deepspeed files today and managed to apply the earlier patches to the latest main branch. It still misses some (3) features which had to be disabled (see below) but should be working the same as the old 11.2 version.

Wheel should be here: Github Link

Wheel is compiled with:

Python 3.11
pytorch==2.1.2
torchvision==0.16.2
torchaudio==2.1.2
pytorch-cuda=12.1

And the modification files for those who want to build it themselves here: Github Link

Modules not supported with windows:

DS_BUILD_AIO
DS_BUILD_SPARSE_ATTN
DS_BUILD_EVOFORMER_ATTN

Disabled modules:

DS_BUILD_CUTLASS_OPS
DS_BUILD_RAGGED_DEVICE_OPS
DS_BUILD_INFERENCE_CORE_OPS

[erew123]

@S95Sedan Awesome! Really Awesome! :) Its amazing MS cant get their act together with this.

Thanks for doing this! I will compile up a cuda 11.8 version too (pretty sure I've still got things set up for that!)

[S95Sedan]

@erew123 Yeah its a shame really, i dont understand why they dont have a seperate branch at this point for windows which doesnt have all these issues. Going trough all these hoops to compile it is a bit weird when its a microsoft product.

I assume it will compile the same on cu11.8 opposed to cu12.1. All the instructions should be updated for the latest version. Just make sure 'DS_BUILD_INFERENCE_CORE_OPS' is disabled, that module gave the biggest issue with binding variables. the other two i believe are dskernels modules which needs the compiled module/fixes for that aswell.

Also have some upgraded gui stuff i been using for the ooba frontend if you want to grab that:

Frontend GUI replacement:

def ui():
    with gr.Accordion(params["branding"] + " TTS (XTTSv2)"):
        with gr.Row():
            activate = gr.Checkbox(value=params['activate'], label='Enable TTS')
            autoplay = gr.Checkbox(value=params['autoplay'], label='Autoplay TTS')
            show_text = gr.Checkbox(value=params['show_text'], label='Show Text')
            
        with gr.Row():
            low_vram = gr.Checkbox(value=params['low_vram'], label='Enable Low VRAM Mode')
            low_vram_play = gr.HTML(visible=False)
            deepspeed_checkbox = gr.Checkbox(value=params['deepspeed_activate'], label='Enable DeepSpeed', visible=deepspeed_installed)
            deepspeed_checkbox_play = gr.HTML(visible=False)
            remove_trailing_dots = gr.Checkbox(value=params['remove_trailing_dots'], label='Remove trailing "."')

        with gr.Row():
            tts_radio_buttons = gr.Radio(
            choices=["API TTS", "API Local", "XTTSv2 Local"],
            label="TTS Method (Each sounds slightly different)",
            value=gr_modelchoice  # Set the default value
            )
            tts_radio_buttons_play = gr.HTML(visible=False)
            with gr.Row():
                available_voices = get_available_voices()
                default_voice = params["voice"] # Check if the default voice is in the list of available voices

                if default_voice not in available_voices:
                    default_voice = available_voices[ 0 ] # Choose the first available voice as the default
                # Add allow_custom_value=True to the Dropdown
                voice = gr.Dropdown(available_voices, label="Character Voice", value=default_voice, allow_custom_value=True, )
                create_refresh_button(voice, lambda: None, lambda: {"choices": get_available_voices(), "value": params["voice"],}, "refresh-button",)

        with gr.Row():
            language = gr.Dropdown(languages.keys(), label="Language", value=params["language"])
            with gr.Row():
                narrator_voice_gr = gr.Dropdown(get_available_voices(), label="Narrator Voice", allow_custom_value=True, value=params["narrator_voice"])
                create_refresh_button(narrator_voice_gr, lambda: None, lambda: {"choices": get_available_voices(), "value": params["narrator_voice"],}, "refresh-button")

        with gr.Row():
            local_temperature_gr = gr.Slider(minimum=0, maximum=1, step=0.1, label="Temperature", value=params["local_temperature"])
            local_repetition_penalty_gr = gr.Slider(minimum=0, maximum=20, step=0.5, label="Repetition", value=params["local_repetition_penalty"])

        with gr.Row():
            with gr.Row():
                narrator_enabled_gr = gr.Radio(choices={"Enabled": "true", "Disabled": "false"}, label="Narrator", value="Enabled" if params.get("narrator_enabled") else "Disabled")
                non_quoted_text_is_gr = gr.Radio(choices={"Char": "true", "Narrator": "false"}, label='Text NOT inside of * or " is', value="Char" if non_quoted_text_is else "Narrator")
                explanation_text = gr.HTML(
                f"<p>⚙️ <a href='http://{params['ip_address']}:{params['port_number']}'>Settings and Documentation Page</a><a href='http://{params['ip_address']}:{params['port_number']}'></a>⚙️<br>- Low VRAM Mode and Deepspeed take 15 seconds to be enabled or disabled.<br>- The DeepSpeed checkbox is only visible if DeepSpeed is present.</p>")

        with gr.Row():
            preview_text = gr.Text(
                show_label=False,
                placeholder="Preview text",
                elem_id="silero_preview_text",
            )
            preview_play = gr.Button("Preview")
            preview_audio = gr.HTML(visible=False)

        with gr.Row():
            convert = gr.Button("Permanently replace audios with the message texts")
            convert_cancel = gr.Button("Cancel", visible=False)
            convert_confirm = gr.Button(
                "Confirm (cannot be undone)", variant="stop", visible=False
            )

    # Convert history with confirmation

Needs to be added for temperature and repetition:

    # TTS Settings
    local_temperature_gr.change(lambda x: params.update({"local_temperature": x}), local_temperature_gr, None)
    local_repetition_penalty_gr.change(lambda x: params.update({"local_repetition_penalty": x}), local_repetition_penalty_gr, None)

[erew123]

Oh, also if you pull the recent build, you'll notice I had to add a function for finetuned models too.....

        with gr.Row():
            model_loader_choices=["API TTS", "API Local", "XTTSv2 Local"]
            if finetuned_model:
                model_loader_choices.append("XTTSv2 FT")
            tts_radio_buttons = gr.Radio(
                choices=model_loader_choices,
                label="Select TTS Generation Method (Read NOTE)",
                value=gr_modelchoice,  # Set the default value
            )

basically the

        if finetuned_model:
            model_loader_choices.append("XTTSv2 FT")

[S95Sedan]

Yeah thats fine, should be coming up in a second.

I added the above code in there, also left the TTS temperature and repetition penalty settings (disabled) in the GUI code incase you want to enable them later and parse those values to the tts generation.

As i dont know if it always needs a restart or if it can be done without that by just sending the values. Since its just in the documentations settings.

[erew123]

Something in my head said it DID need a restart... but then I double checked the code and it doesn't look like it does need a restart... I just wasnt applying it dynamically as it was stored in the config file, so only applied on restart. As such, what you've done is a nice addition :) Again thanks!

[S95Sedan]

Yeah figured it was something like that, only had a quick look as i didnt want to break any other code but i think it could just be added to send_generate_request and then parsed forward.

[erew123]

Thats awesome! Im god knows how deep in code, documentation and preparing a few other things... So I will pull it in at some point later (maybe today) when I have things stable. Then I can give it a test, hope it all works (my code I mean) and Ill merge it into main and release it! )

[erew123]

Im just generally tidying around, now that I have time. Im going to mark this closed just for now.

BTW... I will implement the temperature and repetition penalty settings. Its on my feature request list #74

I guess I was releasing so many changes/updates the other day and there will be a small code change to make, I thought id release what I had, see if that stayed stable and then look to add other things like this, which I can do as nice easy updates.