Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audio is inconsistently reaching Deepgram (1/50 connections) #9

Open
majumba opened this issue Jul 9, 2024 · 1 comment
Open

Audio is inconsistently reaching Deepgram (1/50 connections) #9

majumba opened this issue Jul 9, 2024 · 1 comment

Comments

@majumba
Copy link

majumba commented Jul 9, 2024

I am getting really inconsistent results trying to adapt this repo for my own use-case (chrome extension). Ever so randomly it does seem to connect successfully and produce transcriptions but I cannot consistently replicate it.

Audio data is being sent from the client (verified through console logs) to the express server which is opening up a connection to Deepgram and then sending to Deepgram servers. I have checked my Deepgram logs and I always get a status 101. But the sane logs also show that the audio duration is always zero so it seems as if the server is not handing off the messages streamed from the browser to Deepgram)

As no audio seems to be received by Deepgram's servers, Deepgram is disconnecting the websocket after 10 seconds or so resulting in a 1006 error.

The audio is being streamed from the mic and seemingly the audio packets are being sent to Deepgram's servers when I check the logs)

The only glimmer of a clue that I have seems to be if the Deepgram instance is made immediately, before messages from the websocket are sent to Deepgram's servers then the transcription works. But this rarely happens and I cannot reproduce it.

However, more often than not, the websocket sends a few messages before the Deepgram instance is created resulting in getReadyState == 2. After the Deepgram connection is made, the audio packets seem to be sent to Deepgram but they never seem to be received.

Has anyone encountered this issue? I have barely modified the express server.

const express = require("express");
const http = require("http");
const WebSocket = require("ws");
const cors = require("cors");
const { createClient, LiveTranscriptionEvents } = require("@deepgram/sdk");

const app = express();
const server = http.createServer(app);
const wss = new WebSocket.Server({ server });

const deepgramClient = createClient(process.env.DG_KEY);
let keepAlive;

const setupDeepgram = (ws) => {
  const dgConnection = deepgramClient.listen.live({
    model: "nova-2",
    language: "id",
    smart_format: true,
  });

  if (keepAlive) clearInterval(keepAlive);
  keepAlive = setInterval(() => {
    console.log("Deepgram: KeepAlive sent");
    if (dgConnection && dgConnection.keepAlive) {
      dgConnection.keepAlive();
    }
  }, 3000);

  dgConnection.on(LiveTranscriptionEvents.Open, () => {
    console.log("Deepgram: Connection opened");

    dgConnection.on(LiveTranscriptionEvents.Transcript, (data) => {
      console.log("Deepgram: Transcript received");
      console.log(data.channel.alternatives[0].transcript);
      ws.send(JSON.stringify(data));
    });

    dgConnection.on(LiveTranscriptionEvents.Close, () => {
      console.log("Deepgram: Connection closed");
      clearInterval(keepAlive);
    });

    dgConnection.on(LiveTranscriptionEvents.Error, (err) => {
      console.error("Deepgram: Error received", err);
    });

    dgConnection.on(LiveTranscriptionEvents.Warning, (warning) => {
      console.warn("Deepgram: Warning received", warning);
    });

    dgConnection.on(LiveTranscriptionEvents.Metadata, (data) => {
      console.log("Deepgram: Metadata received", data);
    });
  });

  return dgConnection;
};

wss.on("connection", (ws) => {
  console.log("ws: client connected");
  let deepgram = setupDeepgram(ws);

  ws.on("message", (message) => {
    if (deepgram.getReadyState() === 1 /* OPEN */) {
      console.log("ws: data sent to deepgram");
      deepgram.send(message);
    } else if (deepgram.getReadyState() >= 2 /* 2 = CLOSING, 3 = CLOSED */
    ) {
      console.log("ws: data couldn't be sent to deepgram");
      console.log("ws: retrying connection to deepgram");
      /* Attempt to reopen the Deepgram connection */
      deepgram.finish();
      deepgram.removeAllListeners();
      deepgram = setupDeepgram(ws);
    } else {
      console.log("ws: data couldn't be sent to deepgram");
    }
  });

  ws.on("close", () => {
    console.log("ws: client disconnected");
    if (deepgram) {
      deepgram.finish();
      deepgram.removeAllListeners();
    }
    clearInterval(keepAlive);
  });

  ws.on("error", (error) => {
    console.error("ws: error occurred", error);
  });
});

app.use(cors());

app.get("/api/status", (req, res) => {
  res.json({ status: "ok" });
});

const PORT = process.env.PORT || 3000;

server.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});





@majumba
Copy link
Author

majumba commented Jul 9, 2024

My logs back from Deepgram. But not super useful other than to tell me no channel was created.

Server running on port 3000
ws: client connected
ws: data couldn't be sent to deepgram
Deepgram: Connection opened
Deepgram: Metadata received
{
  type: 'Metadata',
  transaction_key: 'deprecated',
  request_id: 'b4f36b87-0832-4f62-8bdb-7d5128068f97',
  sha256: '33a3f071d9fc223ff0d386ec54c047977c0ac1ace65bea9c2b91e76aeaef5607',
  created: '2024-07-09T07:09:52.231Z',
  duration: 0,
  channels: 0
}
Deepgram: Connection closed
/home/runner/Deepgram-v1/node_modules/websocket/lib/W3CWebSocket.js:111
        throw new Error('cannot call send() while not connected');
        ^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant