Link to video description
Note: This starter-kit will need you to allocate more memory to your node-RED instance.
It uses the FFmpeg nodes (Conversion, Silence Detect and Segment) and the Watson nodes (Speech to Text and optionally Language Translation) to create live captions for a video.
The UI allows a user to load a video file (eg. Ted Talk) and send the URL to a Node-RED flow.
The Node-RED flow will:
- Download the video
- Convert the video into an audio buffer
- Perform silence detection on the audio
- Split the audio into sentences based on silences
- Convert audio sentence to text using Speech-to-Text
- If translating, it will translate the sentence using Language-Translation
- Save results to global context
The UI will poll the global context to receive new captions and displays them at the correct time under the video.
Load UI:
HTTP-In
(GET) : receives GET request on/tv
HTTP-Request
: sends GET request to Github to get HTML contentChange
: deletemsg.headers
HTTP-Out
: send HTTP response
Node-RED Flow:
HTTP-In
(POST) : receives URLChange
: setmsg.payload
to URLHTTP-Out
: send HTTP responseFFmpeg-Conversion
: downloads video and converts it into a .wav buffer and sets it tomsg.payload
FFmpeg-Silence-Detect
: performs silence detection,msg.silences
contains the start and end times of the silences in seconds, eg.[0, 2]
represents silence beteen 0 and 2 secondsFunction
: get averages for each silence duration and set the result tomsg.times
FFmpeg-Segment
: splits the buffer into a series of messages at each time provided inmsg.times
, each message contains the start and end times onmsg.timesplit
Speech-to-Text
: convert buffer into textChange
: setmsg.payload
tomsg.transcription
Link
: Links two flows for neat looking flowLanguage-Translation
(optional) : converts text into FrenchFunction
: push transcription/translation toglobal.transcription
orglobal.translation
Debug
(optional) : log result
Poll Transcription:
HTTP-In
(GET) : receives GET requestChange
: setmsg.payload
toglobal.transcription
orglobal.translation
HTTP-Out
: send HTTP reponse