AI video demos hide the most expensive part of production: correcting the output until it is accurate, credible, and usable in a campaign. A controlled evaluation should measure that correction burden.
Use one real campaign brief
Provide every tool with the same product facts, source images, brand rules, script objective, audience, aspect ratios, and delivery deadline. Do not optimize the brief for one vendor.
Score five dimensions
- Truthfulness: Are product facts, names, UI, and claims preserved?
- Credibility: Do voice, timing, avatar, and visual composition fit the audience?
- Editability: Can errors be corrected without regenerating the whole asset?
- Throughput: How long does it take to deliver several useful variations?
- Economics: What is the effective cost per approved minute or approved ad?
Test variation, not only generation
Marketing teams need different hooks, openings, calls to action, and formats. Ask each tool to produce three materially distinct concepts, not three cosmetic variations.
Record rejection reasons
The most useful dataset is often why an output was rejected: inaccurate product representation, uncanny delivery, generic script, poor pacing, missing editing control, or expensive regeneration. Those reasons should determine the product choice.
The Video Ad Agent lab is testing demand for a system that begins with product truth and produces a set of campaign-ready variations.