Predictable AI video generation: a proof of concept

What this proof of concept tests

AI video models are good at content and poor at staging. Ask one for three people in a loading bay and you get three people in a loading bay. Ask again and the camera is somewhere else, the people are a different size in frame, and someone you did not ask for walks across the front. For a brand or a series, that means a shot cannot be planned, only re-rolled until one comes out right.

This proof of concept tests whether the parts a director owns can be fixed before anything is generated: where the camera goes, when it moves, where each person stands, and whose face it is. We tested each part on its own with measured runs, then used them together on The Midnight Line, a vertical microdrama set on Singapore's last train.

The same prompt with no staging

We ran one plain prompt four times, with no start image, no reference video and no blocking: three people in a loading bay, a woman walking, a man approaching her, a second man near a crate.

All four runs had roughly three people in them, so the model followed the content. Everything else changed. At the first frame, the lead person filled anywhere from 27% to 91% of the frame height, a 3.4× spread in shot size from an identical prompt. One run changed shot halfway through, unasked, when a figure crossed in front of the camera. Sets, lens and light direction differed in all four.

Blocking the camera move in Blender

Each shot is blocked in Blender first, with simple stand-in figures and a camera. The move is watched on a loop and corrected there, where a change costs nothing. Only an approved blocking goes forward.

The approved blocking reaches the video model in two forms. The first is a rendered clip of the blocking, passed as a reference video. The second is a written description of the timing, read directly from the Blender scene's keyframes rather than written by hand: when the camera moves, how far, and how fast each person is going.

A controlled test showed that both are needed. The blocking was a hold, a quick 90-degree turn between 2.0 and 2.5 seconds, then another hold:

What the model was given	When the turn happened
The Blender blocking itself	1.96–2.46s
Exact timing in the prompt, plus the blocking clip	1.92–2.62s
Exact timing in the prompt only	1.62–3.33s, smeared over 3.4× its length
A vague prompt, plus the blocking clip	No distinct turn at all

With both, the turn landed within about a tenth of a second. On an 8-second shot with several camera beats (trail, pass, overtake, turn, slow down), every beat arrived in order, within about 0.7 seconds of its blocked time. That is close enough to cut to music. It is not frame-accurate, and we do not promise that it is.

Holding each person to a mark

To test position, we blocked a five-second shot with three people, each with a mark to hit. Each person's identity came from a trained face model, and the same start image was used for every version. We then measured where each person actually was in the generated frames against where they were blocked.

What the model was given	Mean distance from the mark, as a share of frame width
Blocking clip, timing prompt and the three faces	0.009
Timing prompt and faces, no blocking clip	0.102
Plain prompt and faces	0.113

Without the blocking clip, people ended up 11 to 12 times further from their marks. With it, all three landed in their blocked third of the frame at every one of the 17 points we checked.

Position also held across re-rolls. When we generated the same blocked shot again, the cast stood where they were blocked each time, so choosing between takes became a call about performance, not a fix for staging.

Holding faces across shots

Each character in a production has a locked written description, a trained identity model, and one approved reference still. Every new shot is made from that still, not from a fresh description, because a fresh description lets the face drift.

We measured identity on two four-second close-ups from The Midnight Line, scoring each face against the character's reference sheet with a published face-matching model. Both faces scored above 0.6 at the first frame and dropped by only 0.03 to 0.04 by the last, against a match threshold of 0.363. The face stayed the same person while the expression changed.

Motion transfer

Motion transfer takes the movement from a reference performance and puts generated characters into it. We tested it on a 10-second fight between two characters, comparing two routes.

A dedicated motion-transfer model kept the reference's body movement, camera and cuts closely, and matched the lead character's face to her reference sheet, but it held on to the reference location: scaffolding and a background figure from the source stayed in the shot. A general video model given the same reference video built a clean new set from our images, but the faces drifted further from their references.

So the choice of route depends on which matters more for the shot: the exact moves, or a clean set and faces. That test used a reference clip we do not own, so its footage is not shown here.

What it does not do yet

Body pose does not transfer from the blocking. A stand-in figure that crouches just gets shorter, which the model reads as nothing. A pose has to be fixed with an end image showing it, which made it reliable.

The timing of an expression cannot be directed. In a test that asked for an emotional beat, the performance landed and the cast held their marks, but the beat arrived about a second early and the camera pushed in closer than blocked.

Size of performance needs checking by eye. One of the Midnight Line close-ups plays open-mouthed fright where the episode needs a mild, deniable chill. The face held; the performance was too big, and that shot needs another take.

The Midnight Line, episode one

The Midnight Line is a near-silent supernatural romance: a woman sleeps past her stop on the last train and brushes past a man no one else can see. Episode one is planned at about 75 seconds in 18 shots, cut first as an animatic before anything was generated.

The episode is the test bed for this proof of concept, not a finished film. So far it has an approved animatic, a blocked dash through the carriage with generated takes, and two generated close-ups whose eyelines oppose, so they cut together as a shot and its reverse.