01
The idea
Most trackers work in two steps: detect the people in each frame, then link this frame's detections to the tracks from the last one. The linking step is usually the Hungarian algorithm, run on a hand-built cost such as how much two boxes overlap. It works, but you can't pass gradients through it, so the parts that produce the features can't be trained on the thing that actually matters, the matching.
PENMOT replaces it with parts that can be trained. A Set Transformer processes the detections and tracks as unordered sets, with no positional encodings, so shuffling the people doesn't change the answer. We checked that numerically: reordering the inputs changes the output by less than 10⁻⁵. Then the Sinkhorn algorithm, a differentiable form of optimal transport, turns the costs into soft assignments, so the whole pipeline trains end to end. One weight, α, blends the learned costs with plain box overlap.
02
Training it
We trained on five MOT17 sequences and validated on two others, for ten epochs, with the ResNet-50 backbone frozen: 28.3 million trainable parameters, running at about 30 frames per second on a GPU. The matching loss fell from 1.44 to 0.10, a 93% drop, while validation loss levelled off around 1.8.
By the usual signals it learned. The question is what it learned relative to the simple thing it was meant to replace.
03
What the numbers say
Sweeping α from pure learning to mostly geometry tells the story. Every step toward box overlap made the tracker better, in accuracy and in identity switches. The learned costs on their own reached 20.2% MOTA. The best blend, 82.6%, put 90% of the weight on geometry. And plain box overlap, with nothing learned at all, scored 99.05% with 20 identity switches.
| Weight on geometry (α) | MOTA | ID switches |
|---|---|---|
| 0.0 (learned only) | 20.2% | 1,685 |
| 0.3 | 71.7% | 598 |
| 0.5 | 79.7% | 428 |
| 0.9 | 82.6% | 368 |
| Plain box overlap | 99.05% | 20 |
MOT17 validation sequences 11 and 13, from the report.
04
Reading the results again
Coming back to the report for this write-up, three things change how I'd read it.
On the boxes it was given, a plain overlap rule scored 99%. The learned matcher had to beat that, and it didn't.
- Every method in the results table, the overlap baseline included, scores 100% recall and 100% precision. That only happens when the boxes going in are the ground-truth ones. So these MOTA numbers measure linking alone, and the report's ByteTrack figure (80.3%, a real tracker on real detections, on the test set) isn't a fair comparison, even though it sits in the same table.
- MOTA is scored frame by frame. IDF1 asks whether each person keeps one identity across the whole video, and PENMOT's was 1.81%. The trajectory plots show it: one track jumps as much as 500 pixels at a time, between different people.
- The appearance features barely tell people apart. In the report's similarity figure, same-person and different-person similarities almost entirely overlap (means of 0.881 and 0.813). A table elsewhere in the report gives a wider margin, 0.135, and its per-sequence ID-switch counts don't add up to the headline count either. The report disagrees with itself in places, and I'd fix that before quoting it.
05
What I'd do next
The report's own list is the right start: train the appearance features on hard negatives (the similar-looking people they confuse), fine-tune the last layer of the backbone, weight the identity-consistency loss more heavily, and let lost tracks be picked up again. The Sinkhorn-against-Hungarian question deserves a direct test too, since the soft assignments seem to be part of why identities drift.
And I'd change how it's measured: run it on a detector's boxes instead of the ground truth, report IDF1 or HOTA next to MOTA, and leave scores from different setups out of the same table.