Pilot delivered · a maritime analytics company
Project Skerry
Dense annotation of coastal and maritime drone footage, and evaluation sets built from it that frontier vision models still cannot solve.
- 44.1%
- Questions built from our annotations that frontier models answer correctly95% confidence interval 32.2-56.7%, n=59. Per-model, the best score was 70%. A low number is the point - it means the questions are hard.
- 0%
- Frontier-model accuracy on fine-grained vessel typing and countingFrom the earlier proof-of-concept round (n=12). The same models scored 100% on scene-level questions, which is what makes the contrast informative.
- 760 objects
- Objects annotated in a single five-second harbour clipThe dense end of the range. A sparse bridge scene carries around 160.
Four-thousand-pixel drone footage of harbours and coastline, annotated densely enough that a single five-second clip can carry several hundred distinct objects. AI proposes the structured annotation; people are asked only to resolve the calls the system is genuinely uncertain about.
Location and quantity are never left to human phrasing. They are computed from marker geometry, so once a marker sits on the right object the spatial description is correct by construction. Nobody is asked to judge whether a sentence sounds right - they are asked whether a thing exists, where it is, what it is, and how many there are.
The evaluation sets built from these annotations are the real output. Frontier models answer fewer than half of the questions correctly, and score zero on fine-grained vessel typing while scoring perfectly on scene-level questions. That gap is what makes the data worth having.
The work
What the project covers
- AI pre-labels every keyframe; humans resolve only uncertain calls, as single-decision yes/no tasks.
- Spatial descriptions are derived from geometry rather than written by hand, so they are correct by construction.
- Clustering stability is measured by re-running the grouping at wider and narrower thresholds; a knife-edge result triggers a human count check instead of shipping an unreliable number.
- Nine deterministic per-marker quality signals, with failure classes that can never auto-pass regardless of detector confidence.