Music Mashup and Video Creation
with Node.js, FFMPEG, and Contentful
Website
·
YouTube Channel
·
Buy Me a Coffee
·
Instagram
·
Report Issue
Auto Mashup creates music mashups automatically. It also automatically creates and uploads videos to its YouTube channel and each video's associated thumbnail to its Instagram page (@automaticmashup).
The Auto Mashup project is split into two repositories.
The first repository (https://github.com/amamenko/auto-mashup) handles the scraping of Billboard charts for song details and ranking information, as well as individual song data acquisition (e.g. tempo, key, mode, song structure and timestamps, beat positions, instrumental/vocal stems, etc.) using data from Spotify, Genius, YouTube, and other sources. This data is then stored in Contentful's Content Management System.
The second repository (the one you are visiting right now) is responsible for hosting the front-end code of Auto Mashup's official website, auto-mashup.vercel.app, using the data stored in Contentful to find mixable songs, creating those mashups, selecting non-blacklisted mashups and mixing them altogether, creating an associated video slideshow and uploading it to its YouTube channel, and uploading the associated video thumbnail to the @automaticmashup Instagram page.
The basic functionality of this repository's code logic is:
Front-End Code:
- auto-mashup.vercel.app is a responsive website built with React.
- The burger menu of the site was created with the help of the library react-burger-menu.
- The most recent Auto Mashup YouTube video URL is updated automatically when a new video is posted in Contentful with Contentful's Content Management API. Upon app mount, a request is made via Contentful's Content Delivery API to get the YouTube URL. This URL is then used to embed the YouTube video onto the homepage of the website with the library react-player.
Find Mixable Songs:
- Search the Contentful CMS for songs that have the same mode (either major or minor).
- Narrow down mashup song combinations to those that are within one semitone up or down, have an up to 5% difference in tempo, and in which the instrumental song has at least four continuous song sections that match the sections of the vocal song.
- Remove mashup entries and associated audio assets if either one or both of the instrumental or vocal song entries have been deleted from Contentful due to not being found on any Billboard charts.
Create Mashups:
- Mashups are created within a custom Dockerized AWS Lambda function written in TypeScript. The AWS Lambda Docker image is bundled with Node 18 and FFmpeg/FFprobe binary files.
- If both instrumental and vocal entries exist, both audio assets are downloaded from Contentful and mixed using fluent-ffmpeg. A complex filter is used to trim and delay the matching vocal sections to match the beats of the appropriate associated sections of the instrumental track.
- The pitch and tempo of the vocal audio are manipulated using FFMPEG's rubberband feature to match the instrumental track using a predetermined key-scale and tempo-scale factor. This factor is also used to adjust the beat positions of the vocal track identified with essentia.js.
- Audio inputs are normalized with FFMPEG's loudnorm filter to have similar audio volumes.
- Vocal sections that are below a certain length threshold with respect to their associated instrumental section are looped.
- Fades are added between each vocal section and between each loop.
- The mixed MP3 mashup audio asset and entry containing data about the instrumental and vocal songs is stored in Contentful's CMS using the Content Management API. Additional information about the mix - such as vocal track mix start, mix end, and total duration - is also noted for use when mixing multiple mashups together.
Create Weekly Mix and Video of Selected Mashups:
- Randomly selects either 20 vetted mashups or at least 28 minutes total worth of continuous vetted mashup mixes without any repeats.
- In December, at least 5 holiday songs (instrumental, accompaniment, or both), if available, are included throughout the mashup mix.
- Downloads each mashup's audio MP3 asset and each included song's cover photo from Contentful for use in both thumbnail creation and slideshow inclusion.
- Creates a full audio MP3 mix with FFMPEG of all the selected mashups with a crossfade to transition between each mashup mix.
- Creates a thumbnail JPEG image with dimensions 1280x720 using node-html-to-image. This thumbnail image has to be a JPEG in order to be able to be uploaded to Instagram via instagram-web-api. The thumbnail image contains the Auto Mashup logo and the most recent Billboard ranking week date superimposed onto a collage of eight randomly selected cover photos of songs featured in the current week's mashup mix.
- Creates a slideshow image with dimensions 1280x720 (to ultimately create a 720p video) for each mashup with node-html-to-image. Each slideshow image contains each song's track title, artist, cover photo, latest Billboard chart positions, and key/tempo information.
- Combines all the slideshow images with the full mashup audio mix using FFMPEG to show the appropriate slide image during the accompanying song's duration. Each slide is overlayed over and fades into the next slide.
- Adds a green-screen "Like and Subscribe" overlay banner at the beginning of each video and a green-screen "Thanks for Watching" overlay banner at the end of each video by using FFMPEG's colorkey filter.
- Combines the Auto Mashup intro video (featured in GIF format at the top of this section), hosted on Contentful, with the earlier-created main mix by first re-encoding each video with the x264 library and FFMPEG. The two videos are then concatenated with FFMPEG to create a final merged MP4 video.
Upload to YouTube and Instagram:
- Creates a video title containing the most recent Billboard chart week date and three random artists featured in the current week's mashup video.
- Creates a video description containing details about Auto Mashup as well as its social links. The description also contains the timestamps and song details of all of the mashups featured in the current week's mashup video.
- Uploads the video with a custom thumbnail to Auto Mashup's YouTube channel by means of the youtube-api, which leverages YouTube's Data API - particularly the videos:insert and thumbnails:set endpoints.
- After successful upload to YouTube, records the new YouTube URL to Contentful using the Content Management API.
- Uploads the thumbnail image to Auto Mashup's Instagram account (@automaticmashup) using the instagram-web-api in a similar manner as featured in my other automated Instagram posting project, Daily Pixel Mike. This post features the most recent Billboard week date and three random artists featured in the current week's mashup video. A comment is automatically added to this post that includes a series of hashtags that also include the names of the three random artists featured in the caption.
Server deployed via AWS EC2 instance. Client-side website deployed with Vercel.
Distributed under the MIT License. See LICENSE.txt for more information.
Auto Mashup - [email protected]
Avraham (Avi) Mamenko - [email protected]
Project Link: https://github.com/amamenko/auto-mashup-mix