A bit of Optical Flow background

For users of Final Cut Pro X, optical flow is a trademark function to stabilize and interpolate frame rate adjustments of video footage.

In reality optical flow has been a scheme since its inception in 1940 by American psychologist  James J. Gibson. 

Most people have used optical flow without being involved in video post production. The MPEG codec that was the foundation of the DVD renaissance is largely based on optical flow to achieve inter frame compression in contrast to MJPEG that was purely frame based compression.

Optical Flow

Optical flow ​analyzes​ image content and follows the apparent motion of patterns and objects within the frame and even more so ​in the​ following frame. It does so by finding contrast differences from groups of individual pixels from frame to frame.

 

This works very much the same as an optical mouse tracks patterns found on the surface it operates on, just on a much higher resolution. 3200-PB10-R1.png

An array of pixels is sampled, 3×3 and changes in contrast of each pixel is averaged and compared to the next frame. By doing this computation a vector map can be drawn, indicated by but the above image arrows. The vectors are a function of X Y coordinates, or more commonly expressed as U V vs. Time. The smaller constant of time is based on the frame rates. The faster the change, more than one ‘jump’ per frame and the tracking becomes more difficult. In addition the less contrast between moving elements the more difficult the tracking.

Modern high-efficiency computation is further employing pyramid schemes, where the resolution is iterated to various degrees to speed up calculation. pyramid

The image is mastered at lowest resolution first to find out whether there is sufficient information for detecting motion, integration by iteration the image is sampled closer to original resolution. Depending on the algorithm settings, the pyramid raster can be skipped to the next one if sufficient data is recovered. This explains why optical flow data can amass huge amounts of data during computation, since all these maps need to be stored.

Vertex Map

flow-2

The final result is a vertex map, where clusters of pixels are mapped to their apparent movement. Optical flow image stabilization uses this to track the motion of camera or object. Averaging of these vertices, leads to the most dominant movement, which in return is selected for tracking.

Alternatively video codecs use this information to identify which areas of the image, or clusters of pixel need to be stored and which one do not change from frame to frame, leading to the high-compression rates of MPEG and similar codecs. The higher the motion, the less compression can be applied.

Other uses of Optical Flow

CVB-Optical-Flow-App1-I0

Todays machine vision systems, for autonomous cars, 3D face scans in iPhones, apply all their own form of motion detection based on optical flow. In the above image groups of pixels are identified as moving objects, grouped by their apparent pixel displacement and therefore discernible as individual objects.

With additional information of 3D depth, Z-Axis, objects can be isolated even further.

Image processing in video post production lacks the depth information, nevertheless with triangulation a certain level of depth detail can be computed.

Motion Blur Computation

Based on the above, motion blur, depth of field blur can be computed based on computed optical flow vertices. Objects moving outside of a user defined depth of field plane are blurred, depending on motion vertices motion blur can be added.

 

Conclusions

Understanding the basic concept of how optical flow is computed, steps can be taken to facilitate, speed up computation or improve their quality.

Higher contrast in images improves the efficiency in isolating motion. Focus in image is paramount to track individual pixels. The same stands for speed of motion.

When motion of objects or camera is faster than the smallest amount of time from frame to frame, tracking becomes lossy. Filming at higher frame rate, object motion blur due to shutter angle is reduced, inter frame time makes tracking more precise.

Final Cut Pro X, does employ the above approach for stabilizing images, but not by its most basic approach of anchoring the image to the most dominant motion vertex. Depending on the settings and available information in the frames, it warps the image by stretching clusters of vertices to the frame boundaries, which can be observed by the visible deformation of the image in certain cases. Adjusting the settings changes or diminishes this behavior.

 

Leave a comment