Rishi Shukla ‐ Logbook
- Looking into separating the highs, mids and lows of a song (Current song being tested is "Gonna Fly Now")
- Completed separation and calculated spectral centroid lows and mids for the first 15 seconds of the song.
- Calculated onset (attack and decay) for the lows as well as the binary attacks. Attack, Decay, Sustain and Release Definitions
Created the H-Bridge to test the peltier and working with it, waiting for ordered extra Peltiers and H bridges to begin devlopment
- Change of scope: Since we can't easily quantify the melodic features via vibrations, and it does not seem to give any meaningful application so far, we have decided to use vibrations for the percussive elements (focusing on rhythm and beat), and the temperature would essentially map the melody's instruments (giving the tone color of the melody).
- Researched on what areas of the arm/body are sensitive to vibrations and sound
Papers to note:
Goal for today: Finish off percussive to beat mapping
- Looked into how to define the spectral envelope properly
Timbre paper Notes:
- To control the temperature pseudocode:
Compute the centroid time series of a piece (base-line of temperature)
Process magnitude envelope of the musical piece (to dictate shape of temperature)
Add both signals together which dictates the shape at a certain base temperature
- Find a way to sync music with the vibration/peltier
- Create a web UI so that we can send the data of peltier mapping over wifi, and we can process on the laptop and sync any film score to the haptic sleeve.
Meeting notes:
- Look into the bark scale as a perceptual scale
Selecting scores for testing with justifications: The following pieces would be selected based on the perceptual emotion they would closely convey. Since we are focusing on four main emotions (based on the four quadrants we're working with)
-
Piece 1: Jaws main theme - This piece is chosen for the notable tension that it provides, which would resemble an excited state (close to anger/fear)
-
Piece 2: Gonna Fly Now - This piece is known to be triumphant and elated, which would measure another excited arousal state (happiness)
-
Piece 3: Anakin's Betrayal - This music piece has more notes of despair and sadness, which veers slowly towards a more excited(angry) state
-
Piece 4: Oogway Ascends - This piece has a more calming nature in the first half, with a phrase of triumph and elation in the middle, which would be interesting to measure the individual's response
-
Piece 5: Day One - A calm beginning that veers into a much more excited states (need to analyse to see what arousal values are received)
-
Need to query on whether it's better to select pieces that an individual doesn't know beforehand or if it's okay if participants do know some of pieces beforehand
Seminar Discussion:
Section 1: Background information on what we're doing
- Background on music experiences
- Existing Haptic devices
- Motivation on choosing temperature
Section 2: Explaining current progress
- AI: Model selection and spectral features, data selection, training
- Software: Visualization UI, wireless communication
- Hardware: Peltier control, Sleeve design
Section 3: Future plans
- Experiment Design
Average arousal plotted graph:
The max value found in the arousal levels is 0.77, and the min is -0.77 so we clip values from -0.8 to 0.8. This is how it looks like:
Spectral features are taken from 15 - 45 seconds as this is the annotations are only available from these regions.