Optical flow: reading motion between frames

DIS optical flow on the moving region

The default adaptive AI engine uses OpenCV's DIS (Dense Inverse Search) optical flow with its FAST preset. It runs on the region the engine found moving, or on the box you drew with Set Motion Region… or Live Track, inside a frame shrunk to 320 px wide. A second, half-resolution DIS pass with the ultrafast preset runs alongside it to check whether the flow can be trusted.

Fast

Flow between every 4th frame: 7.5 samples a second on 30 fps video.

Balanced

Every 3rd frame: 10 samples a second on 30 fps video. The default.

High

Every 2nd frame: 15 samples a second on 30 fps video. On flat (non-VR) footage at least 640 px wide, the tracked region's flow is also measured at 640 px. The developer's note records a small gain from this (+0.007 and +0.003 stroke F1 on two films) at about 1.4 times the analysis time.

The interval is a fixed frame count, so on 60 fps video every rate doubles.

You may read elsewhere that AutoScript Sync uses Farneback optical flow. The main engine does not. Farneback appears only in the basic engine, which you get by unticking Use adaptive AI engine for the main Generate button, and in the sampler behind AI Bulk Learn.

From velocity to position

Flow tells you how fast something moved, not where it is. Adding up velocities gives a position, but every small error adds up with it and the trace wanders. So the engine holds the integrated flow to an anchor: a slower, absolute measure of position elected per video from the centre of the moving region, pose keypoints and relative depth. By default the anchor becomes a per-sample blend of every readable channel, if that blend scores at least as well as the single winner.

A Kalman filter with an RTS smoother (a forward pass followed by a backward pass over the whole video) fuses the two into one position trace. The Depth priority setting, 30 to 90 with a default of 85, sets how much of that trace comes from velocity: the higher it is, the lower the anchor's weight. The setting's own tooltip describes the trade: velocity carries stroke size, while the anchor stops drift but flattens depth and cannot follow fast strokes. Changing it needs a regenerate, not just a re-send.

Where tracking drops out, the gap is filled with rhythmic motion continued from the strokes either side, and the share of the timeline filled this way is recorded in the script's metadata.

Cleaning up the rhythm

Two clean-ups run on the finished trace by default. Harmonic folding removes zigzags at double the real stroke rate. Rhythm regularisation moves low-confidence peaks towards the local beat. Points are then placed at each reversal, as described on how AI funscript generation works.

Codec motion vectors and the audio envelope

While the main pass runs, a background thread reads two more signals with PyAV. Both are optional: without PyAV or an audio track, the video-only path runs unchanged.

• Codec motion vectors. Compressed video already stores, for every frame, how blocks of the picture moved. The app reads those vectors for every frame, not just the sampled ones, within a time budget of 25 to 150 seconds that scales with film length. Their speed is used as a signal only when they cover at least 90% of the film.
• A 50 Hz audio loudness envelope, within a budget of 20 to 120 seconds. Audio keeps gap-filled motion in phase with the soundtrack's beat.

Because of the budgets, a long film may be only partly covered. Their main job is as frequency witnesses: they help decide which spans get a second pass that re-reads them at the full frame rate. That pass targets windows where motion looks too fast for the sampling rate or where tracking quality is low. That pass takes at most 4 spans and at most 35% of the video. In the developer's measurement it actually covered 0.2 to 0.6% of each test film, all in the first 40 seconds, because spans are kept in time order. It is not a whole-film full-frame-rate pass. For a span you choose, the timeline editor's Re-analyze does that on demand.

Scene-cut detection

Integrating flow across a cut would treat a change of camera as a huge movement. So the engine checks each pair of sampled frames: a mean pixel difference above 28 marks a candidate cut, and it counts as a cut only if the brightness histograms of the two greyscale frames also correlate below 0.80. The histogram check separates a real cut from fast motion, which changes many pixels but moves the same picture around rather than replacing it. Cuts are detected this way, not from the codec motion vectors.

What flow costs

Flow is not the expensive part. In the developer's flow lab, the CPU flow step cost a median of 4.8 to 5.1 ms per sample at the Balanced setting, and 7.6 to 9.1 ms at High's doubled region resolution. Those figures come from 200 to 400 samples on each of three films, measuring the flow step on its own, not a full generation, so treat them as a sense of scale rather than a benchmark. Since depth moved to the GPU, pose inference has been the largest stage in the developer's measurements.

Questions

Read next

• The whole generation pipelineWhere flow sits between decoding and the saved file.
• Depth estimation for the axis flow missesMotion toward and away from the camera.
• YOLOv8 pose keypointsOne of the anchors that stops the flow trace drifting.
• Fixing a script that feels wrongWhat to try when strokes are missed or too small.