Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flush transcription buffer on mute #557

Closed
wants to merge 11 commits into from
Closed

Flush transcription buffer on mute #557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8146a10
Pick transcriber by id
rpurdel Aug 28, 2024
aa4a42b
Rename transcriberId to transcriberType
rpurdel Aug 28, 2024
98eb5a3
Rebase
rpurdel Sep 30, 2024
50601ca
Add missing Google property
rpurdel Sep 30, 2024
ea614c5
Select transcriber from remote url, use JaaS passcode API to do it on…
rpurdel Sep 30, 2024
4adaab6
Check if tenant is null
rpurdel Sep 30, 2024
01db31e
Keep the old transcriber selection logic
rpurdel Oct 3, 2024
f3b80d3
Remove old remote transcription retrieval url format
rpurdel Oct 8, 2024
4e5bbf4
Adds a check which flushes the buffer if no new audio (participant mu…
rpurdel Oct 2, 2024
0662f57
Rely on the xmpp signaling to flush the buffer
rpurdel Oct 3, 2024
bb6dd3c
Add metrics for the amount of milliseconds sent to each transcriber
rpurdel Oct 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
174 changes: 154 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,27 +103,33 @@ Jitsi Meet will provide subtitles in the video, while plain text
will just be posted in the chat. Jigasi will also provide a link to where the final,
complete transcript will be served when it enters the room if that is configured.

To configure jigasi as a transcriber in a meeting, you will need to have it login with a specific domain that is set as hidden in jitsi-meet config.
To configure jigasi as a transcriber in a meeting, you will need to have it log
in with a specific domain that is set as hidden in jitsi-meet config.
In prosody config (/etc/prosody/conf.d/meet.example.com.cfg.lua) you need to have:

```
VirtualHost "recorder.meet.example.com"
modules_enabled = {
"ping";
}
authentication = "internal_hashed"
```

Restart prosody if you have added the virtual host config and then create the transcriber account:

```
prosodyctl register transcriber recorder.yourdomain.com jigasirecorderexamplepass
```

Edit the /etc/jitsi/meet/meet.example.com-config.js file, add/set the following:
Edit the `/etc/jitsi/meet/meet.example.com-config.js` file, add/set the following:

```
config.hiddenDomain = "recorder.meet.example.com";
config.transcription = { enabled: true };
```

And in jigasi config (/etc/jitsi/jigasi/sip-communicator.properties):
And in jigasi config (`/etc/jitsi/jigasi/sip-communicator.properties`):

```
org.jitsi.jigasi.ENABLE_SIP=false
org.jitsi.jigasi.ENABLE_TRANSCRIPTION=true
Expand All @@ -132,16 +138,21 @@ org.jitsi.jigasi.xmpp.acc.PASS=jigasirecorderexamplepass
org.jitsi.jigasi.xmpp.acc.ANONYMOUS_AUTH=false
org.jitsi.jigasi.xmpp.acc.ALLOW_NON_SECURE=true
```

Configure a transcription provider(Google, Vosk etc.) and restart jigasi.


Google configuration
Jigasi supports multiple transcription services, including Google Cloud speech-to-text
API, Vosk speech recognition server, a custom flavor of Whisper
and Oracle Cloud AI Speech.

Google configuration for transcription
====================

For jigasi to act as a transcriber, it sends the audio of all participants in the
For Jigasi to act as a transcriber, it sends the audio of all participants in the
room to an external speech-to-text service. To use [Google Cloud speech-to-text API](https://cloud.google.com/speech/)
it is required to install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/)
on the machine running Jigasi. To install on a regular [Debian/Ubuntu](https://cloud.google.com/sdk/docs/install#deb) environment:
on the machine running Jigasi. To install on a regular debian/ubuntu environment:

```
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
Expand All @@ -152,19 +163,24 @@ gcloud init
gcloud auth application-default login
```

You will generate a file used for authentication of Google cloud api in jigasi. You will see a result like:
You will generate a file used for authentication of Google cloud api in Jigasi. You will see a result like:
`Credentials saved to file: [/root/.config/gcloud/application_default_credentials.json]`
Move the file to jigasi config and change its permissions:

Move the file to Jigasi config and change its permissions:

```
mv /root/.config/gcloud/application_default_credentials.json /etc/jitsi/jigasi
chown jigasi:jitsi /etc/jitsi/jigasi/application_default_credentials.json
```

In the file `/etc/jitsi/jigasi/config` add at the end:

```
# Credential for Google Cloud Speech API
GOOGLE_APPLICATION_CREDENTIALS=/etc/jitsi/jigasi/application_default_credentials.json
```
Restart jigasi

and restart Jigasi.

Vosk configuration for transcription
==================
Expand All @@ -176,18 +192,18 @@ start the server with a docker:
docker run -d -p 2700:2700 alphacep/kaldi-en:latest
```

Then configure the transcription class with the following properly in `~/jigasi/jigasi-home/sip-communicator.properties`:
Then configure the transcription class with the following property in `/etc/jitsi/jigasi/sip-communicator.properties`:

```
org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.VoskTranscriptionService
```

Finally, configure the websocket URL of the VOSK service in `~/jigasi/jigasi-home/sip-communicator.properties`:
Finally, configure the websocket URL of the VOSK service in `/etc/jitsi/jigasi/sip-communicator.properties`:

If you only have one instance of VOSK server:

```
# org.jitsi.jigasi.transcription.vosk.websocket_url=ws://localhost:2700
org.jitsi.jigasi.transcription.vosk.websocket_url=ws://localhost:2700
```

If you have multiple instances of VOSK for transcribing different languages, configure
Expand All @@ -196,11 +212,42 @@ the URLs of different VOSK instances in JSON format:
# org.jitsi.jigasi.transcription.vosk.websocket_url={"en": "ws://localhost:2700", "fr": "ws://localhost:2710"}
```

Whisper configuration for transcription
==================

If you plan to use our own flavor of Whisper (check [jitsi/skynet](https://github.com/jitsi/skynet)), start by
configuring the following properties in `/etc/jitsi/jigasi/sip-communicator.properties`:

```
org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.WhisperTranscriptionService
org.jitsi.jigasi.transcription.whisper.websocket_url=wss://<YOUR-DOMAIN>:<<PORT>>
```

If you also plan to enable the ASAP authentication, have a look at the
[documentation](https://github.com/jitsi/skynet/blob/master/docs/streaming_whisper_module.md) and at the properties
in the transcription options section of this README.


Oracle Cloud AI Speech configuration for transcription
==================

To use [Oracle Cloud AI Speech](https://docs.oracle.com/en-us/iaas/Content/speech/home.htm), you need to configure the
following properties in `/etc/jitsi/jigasi/sip-communicator.properties`:

```
org.jitsi.jigasi.transcription.customService=org.jitsi.jigasi.transcription.OracleTranscriptionService
org.jitsi.jigasi.transcription.oci.websocketUrl=wss://realtime.aiservice-<<ENV>>.<<REGION>>.oci.oraclecloud.com
```

You also need to place valid OCI credentials under `/usr/share/jigasi/.oci`. Or point to a different location by setting
the `OCI_CONFIG_FILE` environment variable.


LibreTranslate configuration for translation
==================

To use [LibreTranslate](https://github.com/LibreTranslate/LibreTranslate)
for translation, configure the following properties in `~/jigasi/jigasi-home/sip-communicator.properties`:
for translation, configure the following properties in `/etc/jitsi/jigasi/sip-communicator.properties`:

```
org.jitsi.jigasi.transcription.translationService=org.jitsi.jigasi.transcription.LibreTranslateTranslationService
Expand All @@ -220,9 +267,10 @@ Transcription options
=====================

There are several configuration options regarding transcription. These should
be placed in `~/jigasi/jigasi-home/sip-communicator.properties`. The default
be placed in `/etc/jitsi/jigasi/sip-communicator.properties`. The default
value will be used when the property is not set in the property file. A valid
XMPP account must also be set to make Jigasi be able to join a conference room.

<table>
<tr>
<th>Property name</th>
Expand Down Expand Up @@ -252,33 +300,119 @@ XMPP account must also be set to make Jigasi be able to join a conference room.
<tr>
<td>org.jitsi.jigasi.transcription.ADVERTISE_URL</td>
<td>false</td>
<td>Whether or not to advertise the URL which will serve the final
<td>Whether to advertise the URL which will serve the final
transcript when Jigasi joins the room.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.SAVE_JSON</td>
<td>false</td>
<td>Whether or not to save the final transcript in JSON. Note that this
format is not very human readable.</td>
<td>Whether to save the final transcript in JSON. Note that this
format is not very human-readable.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.SAVE_TXT</td>
<td>true</td>
<td>Whether or not to save the final transcript in plain text.</td>
<td>Whether to save the final transcript in plain text.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.SEND_JSON</td>
<td>true</td>
<td>Whether or not to send results, when they come in, to the chatroom
<td>Whether to send results, when they come in, to the chatroom
in JSON. Note that this will result in subtitles being shown.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.SEND_TXT</td>
<td>false</td>
<td>Whether or not to send results, when they come in, to the chatroom
<td>Whether to send results, when they come in, to the chatroom
in plain text. Note that this will result in the chat being somewhat
spammed.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.remoteTranscriptionConfigUrl</td>
<td></td>
<td>
Makes a GET request to https://YOUR-URL/tenant in order to retrieve which transcription service to use.
It expects a JSON response with the <code>transcriberType</code> key set to one of the following values:
<code>GOOGLE</code>, <code>EGHT_WHISPER</code> (see <a href="github.com/jitsi/skynet">jitsi/skynet</a>),
<code>ORACLE_CLOUD_AI_SPEECH</code> or <code>VOSK</code>. If the response is invalid or the request fails,
it will try to use the value of <code>org.jitsi.jigasi.transcription.customService</code>. If no value is
set, it will not make the request.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.remoteTranscriptionConfigUrl.key</td>
<td></td>
<td>Base64 RSA256 private key to sign an ASAP JWT with when issuing the request above.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.remoteTranscriptionConfigUrl.kid</td>
<td></td>
<td>The key's ID.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.remoteTranscriptionConfigUrl.aud</td>
<td></td>
<td>The JWT audience.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.customService</td>
<td></td>
<td>
Which transcription service to use between GoogleCloudTranscriptionService, WhisperTranscriptionService
(see <a href="github.com/jitsi/skynet">jitsi/skynet</a>), OracleTranscriptionService and
VoskTranscriptionService.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.google_model</td>
<td>latest_long</td>
<td>
The model used by the Google speech-to-text API, check the available models
<a href="https://cloud.google.com/speech-to-text/docs/transcription-model">here</a>.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.whisper.private_key</td>
<td></td>
<td>
A base64 RSA256 private key to sign an ASAP JWT with when
<code>org.jitsi.jigasi.transcription.WhisperTranscriptionService</code> is chosen.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.whisper.private_key_name</td>
<td></td>
<td>The key ID for the <code>org.jitsi.jigasi.transcription.WhisperTranscriptionService</code> JWT.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.whisper.jwt_audience</td>
<td></td>
<td>The audience for the <code>org.jitsi.jigasi.transcription.WhisperTranscriptionService</code> JWT.</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.whisper.websocket_url</td>
<td>ws://localhost:8000/ws</td>
<td>
The websocket URL for the <code>org.jitsi.jigasi.transcription.WhisperTranscriptionService</code>
transcription service.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.oci.websocketUrl</td>
<td></td>
<td>
The websocket url for the <code>org.jitsi.jigasi.transcription.OracleTranscriptionService</code>
transcription service.
</td>
</tr>
<tr>
<td>org.jitsi.jigasi.transcription.oci.compartmentId</td>
<td></td>
<td>
The compartment ID for the <code>org.jitsi.jigasi.transcription.OracleTranscriptionService</code>
transcription service.
</td>
</tr>
</table>

Call control MUCs (brewery)
Expand Down
Loading
Loading