COMPUTER VISION · SPORTS DATA ENGINEERING · IN PROGRESS
Broadcast video to player and ball positions on the court, in meters
Optical player tracking exists only in arenas that install it, and its data is rarely public. Broadcast video exists for nearly every game. This project asks how much trustworthy spatial data can be recovered from the broadcast alone. The pipeline now runs end to end on real NBA footage. This page is about the decision that has mattered most so far: refusing to trust a camera estimate just because it scored well.
A box around a player is not a court position. The broadcast camera pans and zooms constantly, so turning pixels into meters requires knowing where the camera is pointing in every frame. That camera estimate is the foundation everything else stands on: if it is wrong, every player in the frame is wrong with it.
Real broadcast excerpt, 15 s, camera panning about 28°. Left: tracked players and court lines projected through the estimated camera. Right: the same players on a top-down court in meters; rings show 2σ uncertainty. Portland at Golden State, NBC Sports Bay Area; shown for research demonstration only.
RF-DETR finds players; a basketball-specific heatmap model (WASB) finds the ball.
From a few landmark clicks on one frame, the camera's pan, tilt and zoom are followed through the video using camera motion and the painted court lines.
A multi-object tracker with camera-motion compensation links detections over time.
Each player's ground point becomes court meters with an uncertainty.
My first acceptance rule asked one question: do the projected court lines land on lines in the image? When I tested the camera solver on 17 frames it had never been tuned on, it produced 272 wrong solutions, and that rule accepted 85% of them. An automatic re-initialization after camera cuts failed on all three frames it was tried on; the line-scored version drew the court across the stands and still earned a high alignment score.
Red lines: the court as projected by the estimated camera.
The old rule accepted it. 83% of the template's line points matched an edge, above the 55% threshold.
A crowd is full of edges: railings, signage, rows of seats. A court drawn over the stands can therefore look like an excellent line fit. The rule was checking the solver with the same kind of evidence the solver had just optimized, so it could not catch the solver's mistakes.
The same frame with three camera estimates. These failures are stored as regression tests (tests/unit/test_calibration_gates.py).
The new gate grades every frame on three signals. Line alignment must be stronger: at least 70% of the court's line points must land on an image edge. The projected court must agree with where the wooden floor actually is, which is measured from floor color and is independent of the line fit. And the camera must not jump implausibly from the previous frame. Frames that fail keep their image-space tracking but receive no court coordinates, because a missing position is safer than a wrong one.
| Pose | Line inliers | Floor IoU | Court coverage | Old rule (≥ 0.55) | New gate |
|---|---|---|---|---|---|
| Correct · f6100 | 0.887 | 0.760 | 0.769 | accepts | good |
| Court on stands · f6100 | 0.633 | 0.300 | 0.445 | accepts | failed |
| Court on stands · f2400 | 0.830 | 0.251 | 0.357 | accepts | failed |
| Diverged fit · f6100 | 0.374 | 0.267 | 0.738 | rejects | failed |
Computed by tools/make_gate_figure.py from the regression fixtures.
Three caveats. The 980 perturbed poses all come from real clips; adding 80 from a synthetic arena, the gate accepts 2 of 1,060 (0.2%), both synthetic. One loophole, a tiny projected court sitting entirely inside the floor, was found and closed after I saw the unseen-frame results, which moved acceptance from 14 to 9 of 272, so the 3.3% is partly post-hoc. And stricter is not free: the gate is over-cautious on fast, blurred pans (about 0.6% of frames on the long video). Raising the line threshold to about 0.76 would remove most of the remaining 9 wrong solutions but would also reject about 5% of the correct frames on one wide-angle clip, so I kept the current balance.
A clutter filter discarded ball candidates that projected far from the court. But a ball high in the air always projects far away, so the filter was deleting shots in flight. Removing it added ball frames on four clips and removed none, and all 48 restored frames I sampled showed the real ball.
MEASURED · VISUAL CHECK 48/48Front-row fans whose feet are hidden behind the courtside crowd were being projected onto the court as players. The fix compares each box's height with the expected height of a standing person at that court position. These are proxy numbers from the same 17-second interval used to set the thresholds, not accuracy.
PROXY · DIAGNOSTIC INTERVALRecovering the camera after a cut still needs one manual keyframe. A height-consistency rule helped on the tuning interval but increased track fragmentation on unseen frames, and a longer lost-track window merged players from different teams. Both were left off. Offline track stitching found no link it could make safely.
MEASURED · NEGATIVE RESULTSHuman-reviewed annotations of three 10-second intervals are in progress, so that tracking can be scored with standard metrics (HOTA, IDF1) instead of proxies. After that, the main open problem is identity: telling teammates apart needs an appearance model (re-identification), not jersey color. A companion video index already maps video time to the game clock, which opens a path to aligning public play-by-play data with the footage.
Architecture, data schema, license register, and every measured number with how to reproduce it are in the repository.