slides/thursday_02.html

<!DOCTYPE html>
<html>
  <head>
    <title>Digital Summer School 2024: THU02</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <link rel="stylesheet" href="../style.css">  </head>
  <body>
    <textarea id="source">


class: center, middle, titlepage
### THU02: *FFmpeg automation and optimisation*
---
class: contentpage
### **Agenda**

1. Building FFmpeg workflows in Python  
  1.1 PyPi packages: ffmpeg-python  
  1.2 Subprocess module  
  1.3 Metadata triggers for FFmpeg commands  
  1.4 Validating your files  
  1.5 Capturing and using filter data  
  
2. FFmpeg encoding optimisation   
  2.1 Hardware configurations    
  2.2 Processing  
  2.3 GNU Parallel  
  2.4 Python parallelisation and multiprocessing  
  2.5 GPU acceleration  
 
---
class: contentpage
### **1. Building FFmpeg workflows in Python**

Combining Python and FFmpeg is merging two of the most powerful tool available to AV digital preservation  

.centre[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/logos_combined.png" width="1000">]

With great power come great responsibility!

---
class: contentpage
### **1.1 PyPi packages: ffmpeg-python**

.center[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/pypi_ffmpeg.png" width="800">]
[pypi.org](https://pypi.org)
---
class: contentpage
### **1.1 PyPi packages: ffmpeg-python**

Complex FFmpeg layout becomes easier to read in a Pythonic way:

```python3
import ffmpeg

in_file = ffmpeg.input('input.mp4')
overlay_file = ffmpeg.input('overlay.png')
(
    ffmpeg
    .concat(
        in_file.trim(start_frame=10, end_frame=20),
        in_file.trim(start_frame=30, end_frame=40),
    )
    .overlay(overlay_file.hflip())
    .drawbox(50, 50, 120, 120, color='red', thickness=5)
    .output('out.mp4')
    .run()
)
```
```sh
ffmpeg -i input.mp4 -i overlay.png -filter_complex "[0]trim=start_frame=10:end_frame=20[v0];\
    [0]trim=start_frame=30:end_frame=40[v1];[v0][v1]concat=n=2[v2];[1]hflip[v3];\
    [v2][v3]overlay=eof_action=repeat[v4];[v4]drawbox=50:50:120:120:red:t=5[v5]"\
    -map [v5] output.mp4
```
---
class: contentpage
### **1.1 PyPi packages: ffmpeg-python**

Set up time! Move your `media` folder into your summer-school-2024 folder if possible. We're going to try using the Python shell, open it by just typing `python3`. Then navigate into your summer-school-2024 folder and we are going to `git pull` updates from the repository:

```sh
cd summer-school-2024/thursday_scripts (Mac/Linux)
cd summer-school-2024\thursday_scripts (Windows)

pip install ffmpeg-python
python3
```
.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-25 at 19.45.09.png" width="500">]

---

class: contentpage
### **1.1 PyPi packages: ffmpeg-python**

<font color="orange">Practise:</font> Use the software to run a horizontal flip of a the film_scan MP4 file you created yesterday. You may need to adjust your path to the video file:
```python3
import ffmpeg
stream = ffmpeg.input('film_scan.mp4')
stream = ffmpeg.hflip(stream)
stream = ffmpeg.output(stream, 'film_scan_flip.mp4')
ffmpeg.run(stream)
```

You can also format your queries like this:
```python3
(
    ffmpeg
    .input('film_scan.mp4')
    .hflip()
    .output('film_scan_flip.mp4')
    .run()
)

(ffmpeg.input('film_scan.mp4').hflip().output('film_scan_flip.mp4').run())
```

---
class: contentpage
### **1.1 PyPi packages: ffmpeg-python**

There's also great reporting using this package to retrieve file metadata:
```python3
result = ffmpeg.probe('film_scan.mp4')
print(result)
```
You are returned with a formatted Python dictionary containing all your metadata. To return just the duration:

```python3
duration = result.get('format').get('duration')
print(duration)
```

<font color="orange">Practise:</font> Use the examples commmands above to query the 'streams' and 'format' dictionary keys, containing stream metadata for the video and audio:
- filename
- format_name
- start_time
- size


---
class: contentpage
### **1.2 Subprocess module**

`subprocess.run()` can also be used to capture results of FFprobe metadata queries
```python3
import subprocess
response = subprocess.run(
  [
    'ffprobe', 'film_scan.mkv'
  ],
  shell=False,
  capture_output=True,
  text=True
)
```
The FFrobe command is supplied as a list of strings, follow by subprocess argument `shell=False`, `capture_output=True` to ensure the response is captured and `text=True` ensures it's formatted as text and not bytes. It returns a 'CompletedProcess' class that allows the data to be viewed with these commands:

```python3
response.args
response.stdout
response.stderr
response.returncode
```
---
class: contentpage
### **1.2 Subprocess module**

Subprocess works for any open-source tool you call from the command line, like MediaInfo:

```python
duration = subprocess.run(
  [
    'mediainfo', '-i', '--Output=Video;%Format%', 'film_scan.mkv'
  ],
  shell=False,
  capture_output=True,
  text=True
)
```
Or MediaConch!
```python
duration = subprocess.run(
  [
    'mediaconch', '-p', 'policy.xml', 'film_scan.mkv'
  ],
  shell=False,
  capture_output=True,
  text=True
)
```
---
class: contentpage
### **1.3 Metadata triggers for FFmpeg commands**

Let's look at the command_builder.py script to see how metadata can be used in code!

.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-18 at 20.18.58.png" width="710">]

---
class: contentpage
### **1.3 Metadata triggers for FFmpeg commands**

From your repository folder `thursday_scripts` import command_builder.py and run functions below. You will need a input file path and output folder or file path to test the encoding command.

```python3
import command_builder as cb

# Experiment with width, height, DAR, PAR possibilities to see the commands that are returned
command = cb.create(
  '../media/film_scan.mp4',
  '../media/',
  1920,
  1080,
  '16x9',
  '16x9')

print(command)
```
If you have a successful `command` variable then copy and paste into the next function:
```python3
success = cb.run(command)
print(success)
```

Hang on to this `success` variable for our next practise

---
class: contentpage
### **1.4 Validating your files**

You can call MediaConch to validate a transcode file straight after encoding to check the file is healthy:
```python3
cmd = ['mediaconch', '-p', 'ffv1_policy.xml', 'input.mkv']
validated = subprocess.run(cmd, shell=False, capture_output=True, text=True)
if validated.stdout.startswith('pass!'):
    # passed!
```

FFrobe will give a very basic true/false response for an AV file's health:
```python3
cmd = ['ffprobe', '-i', fpath, '-loglevel', '-8']
check = subprocess.run(cmd, shell=False)
if check.returncode != 0:
    # Problem reading file
```

Sometimes a file can seem fine but the transcode was interrupted and no duration exists:
```python
cmd = ['mediainfo', '--Output=General;%Duration%', fpath]
check = subprocess.run(cmd, shell=False, capture_output=True, text=True)
if len(check.stderr) == 0:
    # File may be truncated!
```
---
class: contentpage
### **1.4 Validating your files**

The basic MediaConch policy 

.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-19 at 17.21.35.png" width="1000">]

<font color="orange">Practise:</font> Check the transcode from your last encoding against this policy in the `validate_transcode` function of `command_builder.py`. The function returns a success or error string

```python3
success = cb.validate_transcode(outpath)
print(success)
```
---

class: contentpage
### **1.5 Capturing and using filter data**

.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/blackdetect.gif" width="1000">]

The console logs show the blackdetect filter data captured as and when encountered in the encoding:

```sh
[blackdetect @ 0x13a808be0] black_start:196.24 black_end:207.28 black_duration:11.04
[blackdetect @ 0x13a808be0] black_start:369.84 black_end:374.84 black_duration:5 
[blackdetect @ 0x13a808be0] black_start:493.2 black_end:498.08 black_duration:4.88
```

---
class: contentpage
### **1.5 Capturing and using filter data**

This function converts the blackdetect data into a useable list of seconds:  

```python3
def get_blackspace_list(data) -> list:
    '''
    Return a list of start-end timings
    '''
    data_list = data.splitlines()
    time_range = []
    for line in data_list:
        if 'black_start' in line:
            split_line = line.split(":")
            split_start = split_line[1].split('.')[0]
            start = re.sub("[^0-9]", "", split_start)
            split_end = split_line[2].split('.')[0]
            end = re.sub("[^0-9]", "", split_end)
            end = str(int(end) + 1)
            time_range.append(f"{start} - {end}")
    return time_range
```
`[blackdetect @ 0x13a808be0] black_start:196.24 black_end:207.28 black_duration:11.04`  
`['196 - 208', '369 - 375', '493 - 499']`

---
class: contentpage
### **1.5 Capturing and using filter data**

The time_range list can now be checked for clashes with potential thumbnail frames:

```python3
def seconds_clash(time_ranges, second) -> bool:
    '''
    Create range and check for second within
    if clash found return True
    '''
    if not isinstance(second, int):
        second = int(second)

    for item in time_ranges:
        start, end = item.split(" - ")
        st = int(start) - 1
        ed = int(end) + 1
        if second in range(st, ed):
            print(f"Clash {second}: {item}")
            return True
```

The seconds argument needs to be supplied as an integer for use with `range()` but the `isinstance()` can ensure this is fixed if a string is passed. A True bool is returned if a match is made, otherwise the function will return a False.

---
class: contentpage
### **1.5 Capturing and using filter data**

<font color="orange">Practise:</font> Return to your shell and using your `success` variable call `cb.get_black_space` to create your  `time_list`. Then pass `time_list` into `cb.seconds_clash` and see if you receive a True or False
```python
import command_builder as cb

# Experiment with width, height, DAR, PAR possibilities to see the commands that are returned
time_list = cb.get_blackspace_list(success.stdout)
print(time_list)
clash = cb.seconds_clash(time_list, '200')
print(f"Does this clash? {clash}")
```

NOTE: Check your `success` output is in the `stdout` argument returned from `subprocess.CompletedProcess`, sometimes I find it in `stderr` even though the `returncode` is 0
---

class: contentpage
### ** 2. FFmpeg encoding optimisation**

Steps for enhancing FFmpeg encoding:

- Calculating processing speeds  
- Investigating hardware and networks  
- Types of processing approaches  
- Monitoring processes  
- Some open-source tools  
- GPU acceleration  


---
class: contentpage
### ** 2.1 Hardware configuration**

Recommendations from MediaArea for optimising your FFmpeg server:  
- Fast I/O communication speeds between your workstation and your storage (SSD or NAS) 
- Lots of RAM to bridge any bottleneck gaps from I/O interrupts  
- Ample CPU cores to manage processing  

There's a calculation from MediaArea to help you estimate processing speeds:
```
width  x  height  x  components  x  bitdepth  x  framerate
----------------------------------------------------------
1920   x  1080    x  3 (Y,U,V)   x  10        x  25        = ~ 1500 Mbps
4300   x  2920    x  3 (R,G,B)   x  16        x  24        = ~ 1800 Mbps
```

Realtime one pass processing the CPU will need to manage 1500 Mbps  
If for example you have 3 CPUs at 1 GHz each (processing 50Mbps):  
```
1500 Mbps  ÷  50 Mbps  ÷  3 Ghz  =  10 cores
1800 Mpbs  ÷  50 Mbps  ÷  3 Ghz  =  12 cores
```

I/O able to read and write fast enough for 1500/1800 Mbps ~ 200/225 MB/s  
Data converter sites like [unitconverters.net](https://www.unitconverters.net/data-transfer-converter.html) makes these calculations easier!

---
class: contentpage
### ** 2.1 Hardware configuration**

.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-17 at 19.54.09.png" width="900">]

- HTOP interactive system monitor show updated list of active processes bottom, CPU usage in top half
- LSCPU displays information about the CPU architecture of a Linux system, allowing for calculation of CPU:
    Threads per core 2 x Cores per socket 20 x Sockets 2 = 80 Virtual CPU  
- IFTOP is a network analysing tool that reports bandwidth speeds across networks  
- Windows users can use Windows Process Explorer offers similar services, along with NTop and Sniffnet 
---
class: contentpage
### ** 2.2 Processing**

.center[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-20 at 12.32.45.png" width="1000">]
- Serial: Running one task at a time against your CPU
- GNU Parallel: Cues tasks for available CPU and maintains parallel processing
- Python multiprocessing: Each task has sub-processes with own memory allocation spread across CPUs
- FFmpeg Multithreading: Scheduling algorithm shares encoding across threads, not true parallelisation
---
class: contentpage
### ** 2.2 Processing**

.centre[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-20 at 13.00.49.png" width="700">]

HTOP RAWcooked parallel encoding to FFV1 using multithreading. 80 virtual CPUs work against FFmpeg encodings. It shows FFmpeg multithreading combined with GNU parallel processing. This ensures we get the most from our CPU threads at all times.

---
class: contentpage
### **2.3 GNU Parallel**

.left[<img src="https://raw.githubusercontent.com/digitensions/summer-school-2024-local/main/thursday/images/Screenshot 2024-09-17 at 21.23.59.png" width="340">]

GNU parallel separates tasks across CPU cores maintaining maximum CPU and I/O activity at all times. It can be installed with Homebrew for Mac, Unix or Windows Subsystem for Linux can using `apt-get`:
```sh
brew install parallel
sudo apt-get install parallel
```
There are extensive guides in their manual:
```sh
man parallel
```
---
class: contentpage
### **2.3 GNU Parallel**

GNU parallel is highly customisable with a very detailed manual, and online guides to help you optimise your use of it. But, with no arguments supplied at all Parallel will keep the CPUs operating at full capacity. 

This is a command used in RAWcooked encoding workflows at the BFI. As we launch multiple scripts against different storage we set a job maximum 3 to ensure all storage get equal processing:
```sh
grep ^N "${RPATH}list.txt" | parallel --jobs 3 \
  "rawcooked -y --all --no-accept-gaps ${RPATH}{} -o ${MPATH}{}.mkv &>> ${MPATH}{}.mkv.txt"
```
Use a `find` search to only select files modified more than 30 minutes ago and make MP4 encoded copies, and capture logs for the Parallel runs which include execution time:
```sh
find . -name '*.mov' -mmin 30 | parallel --jobs 3 --joblog \
'ffmpeg -i {} -c:v libx264 -c:a aac {}.mp4'
```
---
class: contentpage
### **2.4 Python parellelisation and multiprocessing**

A simple way to parallelise Python scripts is to pipe a list of file paths into GNU parallel then pass one path each to a Python script called by your `$PY3_ENV` environment version of `python3`:
```sh
grep '/mnt/' "${FPATH_LIST}" | parallel --jobs 10 "$PY3_ENV checksum_maker_mediainfo.py {}"
```
This example contains python code to create MD5 checksum and mediainfo reports for each file, but it could easily be used to encode with FFmpeg.  

Alternatively you can use Python's multiprocessing package, which allows code to spawn processes across multiple cores independently. The multiprocessing module has a `Pool()` class making this easily achievable:  
```python3
from multiprocessing import Pool

def main():
  with Pool() as p:
    response = p.map(ffmpeg_encode, file_list)
    print(f"Response code for encoding: {response.returncode}")
    p.close()
```
Iterate the `Pool()` class mapping `file_list` contents to the script's `ffmpeg_encode` function and allocating each to a separate processes, `p`.  

---
class: contentpage
### **2.4 Python parellelisation and multiprocessing**

<font color="orange">Practise:</font> Open the terminal and cd in the repository folder `thursday_scripts`. Identify a folder with several .mov or .mkv media files (duplicate a few examples if needed), then run the script supplying the folder path.

```sh
cd /SUMMER_SCHOOL_2024/thursday_scripts
python3 ffmpeg_multiprocess.py '../media/'
```

If you have means, open `htop` and view the CPU activity as the script processes all your files in rapid succession.
---
class: contentpage
### **2.5 GPU acceleration** 

FFmpeg supports GPU acceleration for some codecs, and can significantly speed up video encoding and transcoding processes. It's particularly effective for real-time encoding and high-resolution video processing.  

GPU acceleration in FFmpeg with hardware-specific APIs:

- NVIDIA GPUs: NVENC (encoding) and NVDEC (decoding)
- AMD GPUs: AMF (Advanced Media Framework)
- Intel GPUs: QSV (Quick Sync Video)

To check for codecs with GPU support for your version of FFmpeg (not included with Package Manager installs):
```sh
ffmpeg -encoders | grep -E "nvenc|amf|qsv"
```

GPU acceleration works best with codecs that have hardware encoding support, like H.264, H.265 and VP9.

You will need to customise your FFmpeg installation so the codecs can be called from any automation, benefitting from the GPU accelerated encoding support.

FFV1 codec does not currently have GPU accelerated encoding support, but it is being planned!

[FFmpeg Intro to Hardware Acceleration](https://trac.ffmpeg.org/wiki/HWAccelIntro)

    </textarea>
    <script src="https://remarkjs.com/downloads/remark-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript">var slideshow = remark.create({ratio: "16:9"});</script>
  </body>
</html>