MOT Algorithm Comparative Study: 7 Years of Evolution, 22 Experiments, 1 Conclusion

"Does using ReID on every frame improve performance?" Experiment result: No. 32% slower with only 5% improvement.

A comprehensive study comparing 6 Multi-Object Tracking algorithms from 2016 to 2023, where I implemented and evaluated each tracker to answer: "Why do certain trackers work better than others, and when should we use each one?"

Core Discovery: OC-SORT (2022) achieved the highest overall score (43.60), but BoT-SORT might be the better choice for production. Why? Selective ReID application achieves 97% of StrongSORT's accuracy at 2.2x the speed.

This project didn't start with "let me implement some trackers." It started with questions:

• ByteTrack paper: MOTA 80.3 on MOT Challenge
• Our evaluation: 41.40 overall score
• Why the gap? Different datasets, evaluation protocols, and detector quality

This disconnect between published results and deployment reality drives the need for independent, controlled evaluation.

Motion-only trackers like OC-SORT achieve 43.60 overall score, while ReID-based StrongSORT scores only 28.65 despite being 2.2x slower. Why would we accept a 32% speed penalty for minimal gains?

The investigation revealed: BoT-SORT's selective ReID approach solves this dilemma—using appearance features only when motion is ambiguous.

Is it:

• Identity preservation? (BoT-SORT: 0.811 fragmentation)
• Processing speed? (OC-SORT: 30.06 FPS)
• Tracking stability? (ByteTrack: 0.281 stability score)

Or is it the balance of all three? This question shaped my evaluation framework.

Standard MOT Challenge metrics (MOTA/MOTP) measure detection quality more than tracking quality. They require ground truth annotations unavailable in production.

My evaluation framework:

Overall Score = 40% Identity + 30% Stability + 30% Speed

Rationale:

• Identity (40%): User experience depends most on consistent ID assignment. ID switching creates cognitive load and false alerts.
• Stability (30%): Track longevity affects downstream analytics (trajectory analysis, behavior detection).
• Speed (30%): Deployment constraint. Cannot trade speed for accuracy beyond real-time boundary.

Multiple runs establish statistical confidence. Single-run benchmarks can be misleading due to:

• Video-specific characteristics
• Initialization randomness
• System state variance

22 runs across 2 videos (5,025 + 1,311 frames) provide reproducible insights.

Selected to represent algorithmic evolution across 7 years:

Discovery: All modern trackers plateau at 0.81-0.93 fragmentation despite radically different approaches.

ReID Trackers:    BoT-SORT (0.811), StrongSORT (0.815)
Motion Trackers:  OC-SORT (0.845), ByteTrack (0.882)
Baseline:         SORT (0.928)

Why this matters: Fragmentation is fundamentally limited by detection quality and scene complexity, not tracker design. ReID improves fragmentation by ~5-10% but cannot overcome these limits. This explains why motion-only trackers remain competitive.

Implication: Pursuing perfect identity preservation (0.0 fragmentation) is futile without addressing detection and dataset quality first.

Discovery: BoT-SORT achieves ReID-level ID preservation (0.811 fragmentation) at motion-only speeds (29.92 FPS).

Why this matters: The "speed vs. accuracy trade-off" is not a fixed constraint—algorithm design can minimize it. BoT-SORT processes primary matching without ReID (fast Hungarian), invoking ReID only for ambiguous assignments (~10-20% of pairs).

Comparison:

BoT-SORT:    29.92 FPS, 0.811 fragmentation (selective ReID)
StrongSORT:  13.52 FPS, 0.815 fragmentation (exhaustive ReID)
Gain:        2.2x faster for 0.4% worse accuracy

Implication: Future tracking algorithms should focus on selective, adaptive application of expensive operations rather than applying them uniformly.

Discovery: A 2016 algorithm remains competitive at 92% realtime score (27.64 FPS) and 36.34 overall score.

Why this matters: SORT's simplicity is actually an advantage:

• Lower memory footprint (no ReID embeddings)
• Deterministic behavior (pure Hungarian matching)
• Easier to debug and deploy

When to use: Embedded systems, edge devices, scenarios where simplicity matters more than peak accuracy. SORT proves that "good enough" engineering often beats cutting-edge research in production.

Discovery: Best fragmentation does not correlate with best stability.

Why this matters: These metrics measure different quality dimensions:

• Fragmentation: How often IDs switch (count-based)
• Stability: How uniform track durations are (distribution-based)

Implication: Cannot optimize both simultaneously. Must choose based on application needs.

Discovery: StrongSORT (exhaustive ReID) is 2.2x slower than BoT-SORT (selective ReID) for 0.1% accuracy gain.

Per-frame compute (50 tracks, 30 detections):
  StrongSORT:  1,500 ReID evaluations (~50ms)
  BoT-SORT:    150-300 ReID evaluations (~5-10ms)

Why this matters: ReID feature extraction has quadratic complexity (N² pairs to evaluate), while motion matching is linear. Exhaustive evaluation scales poorly in crowded scenes.

Implication: Selective application (BoT-SORT) is asymptotically superior for continuous single-camera tracking. Exhaustive ReID (StrongSORT) should be reserved for cross-camera re-identification.

Core Idea: Linear motion model + Hungarian bipartite matching

# Simplified SORT logic
state = [x, y, area, aspect_ratio, vx, vy]
prediction = kalman_filter.predict(state)
cost_matrix = compute_distances(prediction, detections)
matches = hungarian_matching(cost_matrix)

Performance:

• 27.64 FPS (92% realtime)
• 0.928 fragmentation (worst in class)
• 69 unique IDs detected

Limitation: Cannot handle occlusions or similar-looking objects. Pure motion models fail when tracks are lost for >3 frames.

Why it persists: Minimal dependencies, embeddable, <1ms overhead. Still optimal for resource-constrained environments.

Core Idea: Use detection confidence as primary association signal

# ByteTrack two-stage matching
high_conf_detections = filter(detections, conf > 0.5)
matches_1 = hungarian_match(tracks, high_conf_detections)

low_conf_detections = filter(detections, conf < 0.5)
matches_2 = greedy_match(unmatched_tracks, low_conf_detections)

Impact vs SORT:

• +0.5 FPS (negligible overhead)
• -2.6% fragmentation improvement
• +0.088 stability score (best in class at 0.281)

Why it works: Low-confidence detections maintain track continuity without polluting the main association matrix.

Core Idea: Explicit occlusion handling via confidence decay

# OC-SORT lost track budget
if unmatched_frames < 30 and recent_confidence > threshold:
    keep_track_alive  # May recover from occlusion
else:
    remove_track

Performance:

• 30.06 FPS (100% realtime, highest speed)
• 43.60 overall score (best balanced)
• 0.845 fragmentation

Impact: Removes "track ghost" phenomenon. The -9.9 fewer unique IDs vs ByteTrack is a feature, not a bug—fewer fragmented IDs mean better coherence.

Why it wins: Simplicity + effectiveness. No deep learning required.

Core Idea: Apply ReID only when motion is ambiguous

# BoT-SORT selective ReID
motion_cost = compute_kalman_distance(tracks, detections)
if motion_cost > ambiguity_threshold:
    reid_cost = compute_reid_distance(tracks, detections)
    final_cost = 0.7 * motion_cost + 0.3 * reid_cost
else:
    final_cost = motion_cost  # Fast path

Performance:

• 29.92 FPS (99.7% realtime)
• 0.811 fragmentation (best ID preservation)
• 41.45 overall score (2nd best)

The Secret: Invokes ReID on ~10-20% of associations (ambiguous cases), avoiding the full speed penalty of exhaustive approaches.

Why it's practical: Achieves ReID benefits without GPU bottlenecks in most frames.

Core Idea: Exhaustive deep feature matching on every association

# StrongSORT exhaustive ReID
for all track-detection pairs:
    reid_embedding = resnet50_encoder(detection_crop)
    cost = 0.5 * motion + 0.5 * ||embedding_diff||

Performance:

• 13.52 FPS (StrongSORT, 45% realtime)
• 0.815 fragmentation (nearly identical to BoT-SORT)
• 28.65 overall score (3rd tier)

The Paradox: 2.2x slower than BoT-SORT for 0.4% worse accuracy. Deep ReID network optimizes for different objectives (long-term re-identification) than frame-to-frame tracking requires.

When to use: Cross-camera re-identification or recovery from extended occlusions (30+ frames).

The Problem: Initial implementation with SORT showed persistent ID switching—same person getting multiple IDs within seconds.

Investigation:

• Kalman predictions diverge after 3 unmatched frames
• Hungarian matching reassigns IDs when tracks reappear
• 0.928 fragmentation rate (worst in class)

Solution Attempts:

1. Increased track_buffer (30 → 120 frames) — minimal improvement
2. Adjusted IoU threshold — marginal gains
3. Switched to ByteTrack — 4.6% fragmentation improvement

Learning: Motion-only tracking has a fundamental ceiling. Appearance features (ReID) are necessary for identity preservation beyond ~88% fragmentation.

The Problem: StrongSORT achieved excellent ID preservation (0.815) but ran at 13.52 FPS—unacceptable for 30 FPS deployment.

Investigation:

Profiling breakdown (per frame):
  Detection (YOLO):     15ms
  Kalman prediction:     2ms
  ReID feature extraction: 45ms  ← Bottleneck!
  Hungarian matching:     5ms

Solution: Discovered BoT-SORT's selective ReID approach reduces feature extraction from 1,500 pairs to ~200 pairs per frame.

Learning: Exhaustive application of expensive operations doesn't scale. Selective, conditional execution is key for real-time performance.

The Problem: MOTA/MOTP metrics required ground truth and didn't reflect deployment priorities.

Why standard metrics failed:

• MOTA penalizes missed detections (detector quality, not tracker quality)
• MOTP measures bounding box precision (localization, not identity)
• Both require frame-by-frame ground truth (unavailable in production)

Solution: Designed custom metrics from tracker output:

fragmentation = id_switches / unique_ids
stability = avg_track_length / max_track_length
realtime_score = min(fps / 30.0, 1.0)

Learning: Measurement design determines what you can optimize. Domain-specific metrics reflect real-world priorities better than academic benchmarks.

The Problem: BoT-SORT had best fragmentation (0.811) but worst stability (0.133). ByteTrack showed the opposite pattern.

Investigation:

• BoT-SORT creates very long tracks (max 4,452 frames) but also short ones (avg 590 frames)
• ByteTrack creates uniform-length tracks (max 1,410, avg 396 frames)

Insight: These metrics measure orthogonal qualities:

• Fragmentation: Identity consistency (same person = same ID)
• Stability: Track duration variance (uniform vs. heterogeneous)

Learning: Cannot optimize both simultaneously without understanding the underlying trade-off. Application requirements determine priority.

┌─────────────────────────────────────────────────────────────┐
│                     PersonTracker (Unified Interface)         │
├─────────────────────────────────────────────────────────────┤
│  Video Input → Frame Extraction → Detection → Tracking → Output │
└─────────────────────────────────────────────────────────────┘
         │              │                │            │
    OpenCV        YOLOv11n         6 Trackers   Visualization
    (I/O)        (Detection)       (Pluggable)    (Trails)

Tracker Implementations:
├── SORT: configs/sort.yaml
├── ByteTrack: configs/bytetrack.yaml
├── OC-SORT: configs/ocsort.yaml
├── BoT-SORT: configs/botsort.yaml (with ReID)
├── StrongSORT: (exhaustive ReID)
└── DeepOCSORT: (deep learning + occlusion)

class MultiTrackerEvaluator:
    def run_all(self, video_path) -> Dict[str, float]:
        results = {}
        for tracker_type in TRACKERS:
            metrics = self.evaluate_tracker(tracker_type)
            score = self.compute_overall_score(metrics)
            results[tracker_type] = score
        return sorted(results.items(), key=lambda x: x[1], reverse=True)

Key Design Decisions:

1. Unified Interface: All 6 trackers use identical API, enabling fair comparison
2. Pluggable Architecture: Add new trackers by implementing 3 methods
3. Metric Independence: Each metric computed independently for transparency
4. Reproducibility: Fixed random seeds, identical video inputs

• Detection: YOLOv11n (2.6M parameters, optimized for speed)
• Tracking: Kalman filtering via filterpy library
• Matching: Hungarian algorithm via lapx (fast LAP solver)
• ReID: ResNet-50 embeddings (Motion+ReID trackers only)
• Visualization: OpenCV for real-time trail rendering

# 1. Clone and install dependencies
git clone <repo-url> && cd 3i_Engineer_Assignment
uv venv .venv --python 3.11 && source .venv/bin/activate
uv pip install ultralytics torch opencv-python numpy lapx pyyaml

# 2. Run tracker on sample video
python -m src.main input_video/sample.mp4 --tracker botsort --show

# 3. Reproduce full evaluation
python experiments/evaluate_trackers.py --video input_video/sample.mp4

Single video tracking:

• Processing speed: ~30 FPS (OC-SORT, BoT-SORT)
• Output: output/tracked.mp4 with bounding boxes and IDs

Full evaluation:

• Runtime: ~45 minutes for 6 trackers × 2 videos
• Output: experiments/results/standard_metrics_report.json
• Rankings: OC-SORT (43.60) > BoT-SORT (41.45) > ByteTrack (41.40)

GPU Note: Motion-only trackers (SORT, ByteTrack, OC-SORT) run efficiently on CPU. ReID trackers (BoT-SORT, StrongSORT) benefit from GPU but remain CPU-compatible.

1. Can we predict when ReID is necessary?

   • Current: Fixed threshold on motion distance
   • Future: Learned confidence predictor using track history

2. What if we use multi-scale ReID?

   • Current: Single ResNet-50 embedding
   • Future: Coarse-to-fine cascade (cheap → expensive)

3. How much does detector quality matter?

   • Current: Single detector (YOLOv11n)
   • Future: Evaluate across Faster R-CNN, EfficientDet

4. Can we achieve 40 FPS with deep ReID?

   • Current best: 29.92 FPS (BoT-SORT)
   • Future: Model quantization, distillation, selective encoding

• Evaluate on MOT Challenge benchmark (MOT17, MOT20)
• Cross-camera tracking scenarios (where ReID truly excels)
• Embedded deployment (Raspberry Pi, Jetson Nano)
• Real-time dashboard with tracker comparison

1. SORT (2016): "Simple Online and Realtime Tracking" Bewley et al. - https://arxiv.org/abs/1602.00763

2. ByteTrack (2021): "ByteTrack: Multi-Object Tracking by Associating Every Detection Box" Zhang et al. - https://arxiv.org/abs/2110.06864

3. OC-SORT (2022): "Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking" Cao et al. - https://arxiv.org/abs/2203.14360

4. BoT-SORT (2022): "BoT-SORT: Robust Associations Multi-Pedestrian Tracking" Aharon et al. - https://arxiv.org/abs/2206.14651

5. StrongSORT (2023): "StrongSORT: Make DeepSORT Great Again" Du et al. - https://arxiv.org/abs/2304.14526

• ByteTrack: https://github.com/ifzhang/ByteTrack
• OC-SORT: https://github.com/noahcao/OC_SORT
• BoT-SORT: https://github.com/NirAharon/BoT-SORT
• StrongSORT: https://github.com/dyhBUPT/StrongSORT
• Ultralytics YOLO: https://docs.ultralytics.com/

• MOT Challenge: https://motchallenge.net/ (Standard evaluation dataset)
• KITTI Tracking: http://www.cvlibs.net/datasets/kitti/eval_tracking.php

For deeper technical insights:

• EXPERIMENT_INSIGHTS.md — 800+ line deep dive into algorithm evolution, speed-accuracy trade-offs, and implementation details
• EVALUATION_HIGHLIGHTS.md — TOP 3 tracker comparison and usage recommendations

License: MIT Contact: Open to collaboration on MOT research and production deployment challenges