Delivered · a video-AI research company
Project Vista
We repaired a 1,549-question video benchmark that other people's models were being scored against, and delivered every question accounted for by name.
- 1,549 questions
- Questions in the repaired benchmark
- 1,191 questions
- Questions given full two-person independent blind review
- 1,549 of 1,549
- Questions accounted for by name at delivery1,530 shipped, 19 delivered as documented exclusions with reasons, 0 withheld, 0 incomplete.
- 250 answer keys
- Answer keys found to be wrong and corrected
ALL 1,549 QUESTIONS
The changes are spread throughout the benchmark.
One point represents one question in delivery order. The map uses anonymous positions and shows which items were repaired, removed, or kept unchanged.
A public video-understanding benchmark had errors in it: wrong answer keys, questions answerable without watching the video, options that were quietly duplicates of each other. Anyone training or evaluating against it inherited those errors silently.
Every question went through independent blind review. A reviewer watched the evidence and wrote their own answer before the existing answer key was ever shown to them - an ordering the database enforces, not a convention the interface asks for. Two reviewers worked each question without seeing one another's work, an adjudicator resolved genuine disagreements, and a senior lead spot-audited pairs that agreed, which is where two people making the same mistake gets caught.
Both reviewers' raw determinations ship alongside the adjudicated outcome, so the client can compute inter-reviewer agreement from the data instead of taking our word for it. Nothing was silently dropped: questions judged unusable ship as documented exclusions carrying their reason, so the original benchmark and the corrected subset can both be reported on.
The work
What the project covers
- The reviewer's own answer is recorded before the existing answer key is revealed, enforced at the database level rather than by interface convention.
- Two independent blind reviewers per question, an adjudicator for disagreements, and senior spot-audits of pairs that agreed.
- A three-model blind panel checks whether a question can be answered from its text alone; if it can, the question leaks its own answer and is flagged.
- An evidence verifier checks that the claimed answer is actually supported by the frames cited for it.
- Both reviewers' raw, unreconciled labels are delivered alongside the adjudicated result.
- Nothing is deleted - questions marked for exclusion ship with their defect code and reason.