frames with small deviations

[bgriffen]

Great package, thanks for your work. I was looking at frame to frame differences in a biological image and as a test I grabbed a random video of this beating tissue and wanted to see what it would look like, here is one such frame. Click the link to get the video.

Through pyimof default pyimof.solvers.tvl1(I0, I1). where I1 is frame n and I0 is frame n-1.

Is there a way to reduce the sensitivity to small deviations via some cut-off filter? It seems to be over inflating deviations in regions that don't move at all relative to the primary areas. Basically just need to remove small noisy magnitudes.

As a small note: inside solvers.py, I had to change out nearest with edge as mode='nearest' threw an error.

wI1 = warp(I1, np.array([y+v, x+u]), mode='edge') #nearest

As a side node, I was interested in perhaps building on what you've made to generate maps like this:

https://www.nature.com/articles/s41598-017-10094-7

[rfezzani]

Hello @bgriffen, thank you for your kind words and your interest in pyimof 🙂.

As a small note: inside solvers.py, I had to change out nearest with edge as mode='nearest' threw an error.

I in fact didn't found enough time to give some love to this package, sorry for that 🙏. By the way, I implemented the tvl1 optical flow algorithm in scikit-image, you can find it in skimage.registration.optical_flow_tvl1.

Back to your initial problem, optical flow being based on the hypothesis of pixel intensity preservation during the motion, small displacement may be detected where they are not supposed to be if the image sequence contains small intensity variations 😕. BTW, the most impacting parameter is the attachment parameter lambda_, it controls how smooth the output optical flow is.

Using this script:

from matplotlib import pyplot as plt
import numpy as np
from skimage.color import rgb2gray

import imageio.v3 as iio
import pyimof
from skimage.registration import optical_flow_tvl1 as tvl1

img0 = rgb2gray(iio.imread('/tmp/Video_S1.avi', index=10))
img1 = rgb2gray(iio.imread('/tmp/Video_S1.avi', index=11))

lbda = 5

# u, v = pyimof.solvers.tvl1(img0, img1, lambda_=lbda)
u, v = tvl1(img0, img1, attachment=lbda)
norm = np.sqrt(u*u + v*v)

fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(8, 4))

pyimof.display.plot(u, v, cmap=None, ax=ax0, colorwheel=False)

pyimof.display.quiver(u, v, c=norm, bg=img0, ax=ax1,
                      cmap='jet', bg_cmap='gray')
ax1.set_title(f"Motion range: [{norm.min():.2f}, {norm.max():.2f}]")
fig.suptitle(f"lambda={lbda}")
fig.tight_layout()
plt.show()

you obtain

The application you aiming for is really interesting and could be a very good show case for pyimof or skimage if you are interested in contributing a merge request, you are welcome 😉

[rfezzani]

To filter small intensity variations, you can apply both temporal and spatial filtering. You may also threshold estimated optical flow with small intensity:

from matplotlib import pyplot as plt
import numpy as np
from skimage.color import rgb2gray
from scipy.ndimage import gaussian_filter1d

import imageio.v3 as iio
import pyimof
from skimage.registration import optical_flow_tvl1 as tvl1
from skimage.filters import median

frames = rgb2gray(iio.imread('/tmp/Video_S1.avi', index=None))

# --- Temporal gaussian filtering
frames = gaussian_filter1d(frames, 0.5, axis=0)

# --- Spatial median filtering
radius = 1
size = 2 * radius + 1
ker = np.ones((size, size), bool)
img0 = median(frames[10], ker)
img1 = median(frames[11], ker)

# --- Optical flow estimation
lbda = 10
u, v = tvl1(img0, img1, attachment=lbda)
norm = np.sqrt(u*u + v*v)

# --- Thresholding
mask = norm < 0.25
u[mask] = 0
v[mask] = 0
norm[mask] = 0

# --- Display
fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(8, 4))

pyimof.display.plot(u, v, cmap=None, ax=ax0, colorwheel=False)

pyimof.display.quiver(u, v, c=norm, bg=img0, ax=ax1,
                      cmap='jet', bg_cmap='gray')
ax1.set_title(f"Motion range: [{norm.min():.2f}, {norm.max():.2f}]")
fig.suptitle(f"lambda={lbda}")
fig.tight_layout()
plt.show()