Object Tracking using OpenCV (C++/Python)

Updated July 30, 2026: The original explanations, worked examples, face-tracking animation, and all three demonstration videos are preserved below. The implementation now uses Python 3 and C++17, keeps MIL as the model-free common path, and adds the DaSiamRPN, NanoTrack v2, and VitTrack model-backed trackers available in current OpenCV 4 and 5. A checksum-pinned downloader, headless operation, capability checks, deterministic validation evidence, and additive four-tracker OpenCV 5 comparisons on the original Chaplin, street, and race-car footage are included.

In this tutorial, we will learn object tracking using OpenCV and the tracking API introduced in OpenCV 3.0. The original article covered eight classic trackers available through OpenCV 4.2 and its contrib modules — BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, and CSRT. Their explanations remain useful, but they no longer share one portable package surface. The refreshed example uses MIL across the exact OpenCV 4.14 and 5.0 builds tested here, with an additional OpenCV 4.13 regression run, and now also runs DaSiamRPN, NanoTrack v2, and VitTrack when their verified ONNX files and OpenCV’s DNN module are present. KCF and CSRT remain capability-gated contrib options: they were present in the tested OpenCV 4 contrib builds and absent from the exact OpenCV 5 builds originally used for the model-free migration, but other contrib-enabled builds and language bindings may expose them.

This problem has been perfectly solved by my friend Boris Babenko as shown in this flawless real-time face tracker below! Jokes aside, the animation demonstrates what we want from an ideal object tracker — speed, accuracy, and robustness to occlusion.

What You Will Learn

• What object tracking means and how it differs from detection and optical flow.
• Why motion and appearance models are both important to a tracker.
• How the classic BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, and CSRT trackers work in practice.
• How to initialize and update a single-object tracker safely in Python 3 and C++17.
• Which tracker APIs remain available across OpenCV 4 and 5.
• How to download, verify, configure, and run DaSiamRPN, NanoTrack v2, and VitTrack.
• What published research says about the strengths and limitations of MIL, DaSiamRPN, NanoTrack, and VitTrack—and which claims actually apply to OpenCV’s implementations.
• How MIL and the three model-backed trackers behave on the original Chaplin clip, and where they have limited success under street-scene occlusion and rapid race-car motion and appearance change.
• How to run the example interactively or headlessly and validate its output.

Table of Contents

• Demo of Object Tracking using OpenCV
• What is Object Tracking?
• Tracking vs Detection
• The OpenCV Tracking API
• Run and Validate the Modern Example
• Run DaSiamRPN, NanoTrack, and VitTrack
• Research-Backed Pros and Cons
• Four-Tracker Comparison and Practical Limitations
• The Tracking Algorithms
• Limitations and Production Guidance
• Frequently Asked Questions
• Conclusion
• Credits and References

Demo of Object tracking using OpenCV

If you do not have the time to read the entire post, just watch this video and learn the usage in this section. But if you really want to learn about object tracking, read on.

What is Object Tracking?

Simply put, locating an object in successive frames of a video is called tracking.

The definition sounds straight forward but in computer vision and machine learning, tracking is a very broad term that encompasses conceptually similar but technically different ideas. For example, all the following different but related ideas are generally studied under Object Tracking

1. Dense Optical flow: These algorithms help estimate the motion vector of every pixel in a video frame.
2. Sparse optical flow: These algorithms, like the Kanade-Lucas-Tomashi (KLT) feature tracker, track the location of a few feature points in an image.
3. Kalman Filtering: A very popular signal processing algorithm used to predict the location of a moving object based on prior motion information. One of the early applications of this algorithm was missile guidance! Also as mentioned here, “the on-board computer that guided the descent of the Apollo 11 lunar module to the moon had a Kalman filter”.
4. Meanshift and Camshift: These are algorithms for locating the maxima of a density function. They are also used for tracking.
5. Single object trackers: In this class of trackers, the first frame is marked using a rectangle to indicate the location of the object we want to track. The object is then tracked in subsequent frames using the tracking algorithm. In most real-life applications, these trackers are used in conjunction with an object detector.
6. Multiple object track finding algorithms: In cases when we have a fast object detector, it makes sense to detect multiple objects in each frame and then run a track finding algorithm that identifies which rectangle in one frame corresponds to a rectangle in the next frame.

Multiple Object Tracking has come a long way. It uses object detection and novel motion prediction algorithms to get accurate tracking information. For example, DeepSORT extends SORT with appearance-based association and re-identification. It is commonly paired with detectors such as YOLO, but it is not tied to one detector architecture.

Tracking vs Detection

If you have ever played with OpenCV face detection, you know that it works in real-time and you can easily detect the face in every frame. So, why do you need tracking in the first place? Let’s explore the different reasons you may want to track objects in a video and not just do repeated detection.

Classic tracking is often cheaper than repeated detection: For many workloads, updating a classic tracker costs less than rerunning a detector, although the result depends on the algorithms, hardware, frame size, and number of targets. When you are tracking an object that was detected in the previous frame, you know a lot about the appearance of the object. You also know the location in the previous frame and the direction and speed of its motion. So in the next frame, you can use all this information to predict the location of the object in the next frame and do a small search around the expected location of the object to accurately locate the object. A good tracking algorithm will use all information it has about the object up to that point while a detection algorithm always starts from scratch. Therefore, while designing an efficient system, an object detector is often run periodically while the tracking algorithm is employed in the frames in between. Why don’t we simply detect the object in the first frame and track it subsequently? It is true that tracking benefits from the extra information it has, but you can also lose track of an object when it goes behind an obstacle for an extended period of time or moves so fast that the tracking algorithm cannot catch up. It is also common for tracking algorithms to accumulate errors and for the bounding box to drift away from the object. To fix these problems, a detection algorithm is run every so often. Detection algorithms are trained on many examples of the object and therefore have more knowledge about the general class. Tracking algorithms know more about the specific instance they are following.

Tracking can help when detection fails: If you are running a face detector on a video and the person’s face gets occluded by an object, the face detector will most likely fail. A good tracking algorithm, on the other hand, will handle some level of occlusion. In the video below, you can see Dr. Boris Babenko, the author of the MIL tracker, demonstrate how the MIL tracker works under occlusion.

Tracking can preserve identity between updates: The output of object detection is an array of rectangles that contain the object. However, there is no identity attached to the object. For example, in the video below, a detector that detects red dots will output rectangles corresponding to all the dots it has detected in a frame. In the next frame, it will output another array of rectangles. In the first frame, a particular dot might be represented by the rectangle at location 10 in the array, and in the second frame, it could be at location 17. While using detection on a frame we have no idea which rectangle corresponds to which object. Tracking provides a way to literally connect the dots, but identity is not guaranteed after a track is lost or through a long occlusion; multi-object systems need explicit association or re-identification.

Re-identification has become an important part of multiple object tracking. FairMOT uses joint detection and re-ID tasks to obtain re-identification and tracking results. Its detection pipeline is an anchor-less approach based on CenterNet. Its speed relative to a classic OpenCV single-object tracker depends on the hardware, detector, video, number of targets, and accuracy requirements.

Object tracking using OpenCV 4 and 5 – the Tracking API

OpenCV provides a tracking API with implementations of several single-object tracking algorithms. OpenCV 4.2 and its contrib modules exposed the eight classic trackers discussed in this article — BOOSTING, MIL, KCF, TLD, MEDIANFLOW, GOTURN, MOSSE, and CSRT. Current package contents differ: MIL was compiled and executed in the exact OpenCV 4.14 and 5.0.0 reference builds used for this refresh and in an additional 4.13.0 regression build, while CSRT and KCF were available in the tested OpenCV 4 contrib Python builds but not in the tested OpenCV 5 builds used for the initial migration. DaSiamRPN, NanoTrack, and VitTrack are current main-module APIs, but they need the DNN module and external ONNX files. Other distributions and language bindings may expose a different surface, so the code checks capabilities at runtime.

Historical note: OpenCV 3.2 had implementations of these 6 trackers — BOOSTING, MIL, TLD, MEDIANFLOW, MOSSE, and GOTURN. OpenCV 3.1 had implementations of these 5 trackers — BOOSTING, MIL, KCF, TLD, MEDIANFLOW. OpenCV 3.0 had implementations of the following 4 trackers — BOOSTING, MIL, TLD, MEDIANFLOW.

Current update: The factory API and module layout changed again after OpenCV 3.3. The refreshed implementation checks the factories and classes actually present in the imported or linked build instead of branching only on the version string.

Before we provide a brief description of the algorithms, let us see the setup and usage. We first choose a tracker that the installed build actually provides. MIL is the model-free cross-version example; KCF and CSRT are optional contrib paths; DaSiamRPN, NanoTrack, and VitTrack are the model-backed paths. We then open a video and grab a frame, define a bounding box containing the object in the first frame, and initialize the tracker. Finally, we read frames from the video and update the tracker in a loop to obtain a new bounding box. The same program can display results interactively or save them during a headless run.

Object tracking using OpenCV – C++17 Code

#include <opencv2/dnn.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/video/tracking.hpp>
#include <opencv2/videoio.hpp>
#include <filesystem>
#include <stdexcept>
#include <string>

#if __has_include(<opencv2/tracking.hpp>)
#include <opencv2/tracking.hpp>
#define LEARNOPENCV_HAS_CONTRIB_TRACKING 1
#else
#define LEARNOPENCV_HAS_CONTRIB_TRACKING 0
#endif

cv::Ptr<cv::Tracker> createTracker(
    const std::string& name,
    const std::filesystem::path& modelsDir) {
#if LEARNOPENCV_HAS_CONTRIB_TRACKING
    if (name == "CSRT") {
        return cv::TrackerCSRT::create();
    }
    if (name == "KCF") {
        return cv::TrackerKCF::create();
    }
#endif
    if (name == "MIL") {
        return cv::TrackerMIL::create();
    }
    if (name == "DASIAMRPN") {
        cv::TrackerDaSiamRPN::Params p;
        p.model = (modelsDir / "dasiamrpn_model.onnx").string();
        p.kernel_cls1 =
            (modelsDir / "dasiamrpn_kernel_cls1.onnx").string();
        p.kernel_r1 =
            (modelsDir / "dasiamrpn_kernel_r1.onnx").string();
        p.backend = cv::dnn::DNN_BACKEND_OPENCV;
        p.target = cv::dnn::DNN_TARGET_CPU;
        return cv::TrackerDaSiamRPN::create(p);
    }
    if (name == "NANO") {
        cv::TrackerNano::Params p;
        p.backbone =
            (modelsDir / "nanotrack_backbone_sim.onnx").string();
        p.neckhead =
            (modelsDir / "nanotrack_head_sim.onnx").string();
        p.backend = cv::dnn::DNN_BACKEND_OPENCV;
        p.target = cv::dnn::DNN_TARGET_CPU;
        return cv::TrackerNano::create(p);
    }
    if (name == "VIT") {
        cv::TrackerVit::Params p;
        p.net =
            (modelsDir / "object_tracking_vittrack_2023sep.onnx").string();
        p.backend = cv::dnn::DNN_BACKEND_OPENCV;
        p.target = cv::dnn::DNN_TARGET_CPU;
        return cv::TrackerVit::create(p);
    }
    throw std::invalid_argument(
        "Requested tracker is unavailable in this OpenCV build.");
}

int main() {
    cv::VideoCapture capture("videos/chaplin.mp4");
    if (!capture.isOpened()) {
        throw std::runtime_error("Could not open input video.");
    }

    cv::Mat frame;
    if (!capture.read(frame) || frame.empty()) {
        throw std::runtime_error("Could not read the first frame.");
    }

    cv::Rect box(287, 23, 86, 320);
    cv::Ptr<cv::Tracker> tracker = createTracker("MIL", "models");
    tracker->init(frame, box);

    while (capture.read(frame) && !frame.empty()) {
        const bool found = tracker->update(frame, box);
        if (found) {
            cv::rectangle(frame, box, cv::Scalar(255, 0, 0), 2);
        } else {
            cv::putText(
                frame, "Tracking failure detected", cv::Point(20, 40),
                cv::FONT_HERSHEY_SIMPLEX, 0.7,
                cv::Scalar(0, 0, 255), 2);
        }
    }

    return 0;
}

The complete C++17 program adds command-line parsing, bounding-box validation, optional ROI selection, output-video checks, snapshots, headless operation, and a machine-readable summary. The focused excerpt above shows the corrected factory and update loop without hiding those safeguards in an obsolete full-file listing.

Object tracking using OpenCV – Python 3 Code

from pathlib import Path
from time import perf_counter
import cv2

MODEL_FILES = {
    "DASIAMRPN": (
        "dasiamrpn_model.onnx",
        "dasiamrpn_kernel_cls1.onnx",
        "dasiamrpn_kernel_r1.onnx",
    ),
    "NANO": (
        "nanotrack_backbone_sim.onnx",
        "nanotrack_head_sim.onnx",
    ),
    "VIT": ("object_tracking_vittrack_2023sep.onnx",),
}
MODEL_CLASSES = {
    "DASIAMRPN": "TrackerDaSiamRPN",
    "NANO": "TrackerNano",
    "VIT": "TrackerVit",
}

def tracker_creator(class_name):
    flat = getattr(cv2, f"{class_name}_create", None)
    if flat is not None:
        return flat
    tracker_class = getattr(cv2, class_name, None)
    return getattr(tracker_class, "create", None)

def create_tracker(name, models_dir=Path("models")):
    name = name.upper()
    if name in MODEL_FILES:
        paths = [models_dir / filename for filename in MODEL_FILES[name]]
        missing = [path.name for path in paths if not path.is_file()]
        if missing:
            raise FileNotFoundError(
                f"Missing {missing}; run download_models.py"
            )

        class_name = MODEL_CLASSES[name]
        params = getattr(cv2, f"{class_name}_Params")()
        if name == "DASIAMRPN":
            params.model = str(paths[0])
            params.kernel_cls1 = str(paths[1])
            params.kernel_r1 = str(paths[2])
        elif name == "NANO":
            params.backbone = str(paths[0])
            params.neckhead = str(paths[1])
        else:
            params.net = str(paths[0])
        params.backend = cv2.dnn.DNN_BACKEND_OPENCV
        params.target = cv2.dnn.DNN_TARGET_CPU
        return tracker_creator(class_name)(params)

    if name in {"MIL", "CSRT", "KCF"}:
        for namespace in (cv2, getattr(cv2, "legacy", None)):
            if namespace is None:
                continue
            creator = getattr(
                namespace, f"Tracker{name}_create", None
            )
            if creator is not None:
                return creator()
            tracker_class = getattr(
                namespace, f"Tracker{name}", None
            )
            if tracker_class is not None:
                return tracker_class.create()

    raise RuntimeError(f"{name} is unavailable in this OpenCV build")

video = cv2.VideoCapture("videos/chaplin.mp4")
if not video.isOpened():
    raise FileNotFoundError("Could not open the input video")

ok, frame = video.read()
if not ok or frame is None:
    raise RuntimeError("Could not read the first frame")

bbox = (287, 23, 86, 320)
tracker = create_tracker("MIL")
init_result = tracker.init(frame, bbox)
if init_result is False:
    raise RuntimeError("Tracker initialization failed")

while True:
    ok, frame = video.read()
    if not ok or frame is None:
        break

    start = perf_counter()
    found, bbox = tracker.update(frame)
    elapsed = perf_counter() - start

    if found:
        x, y, width, height = (int(round(value)) for value in bbox)
        cv2.rectangle(
            frame, (x, y), (x + width, y + height), (255, 0, 0), 2
        )
    else:
        cv2.putText(
            frame, "Tracking failure detected", (20, 40),
            cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2
        )

    update_fps = 1.0 / elapsed if elapsed > 0 else 0.0

The complete Python 3 program also validates that the initial box lies inside the frame, checks output writers and image saves, supports an optional interactive ROI, runs headlessly by default, and prints a JSON summary. The displayed FPS is tracker-update throughput; it is not the source video’s frame rate or an end-to-end pipeline benchmark.

Run and Validate the Modern Example

The original Charlie Chaplin clip remains the input. Run 60 headless updates in Python:

python tracker.py \
  --input videos/chaplin.mp4 \
  --tracker MIL \
  --bbox 287,23,86,320 \
  --max-frames 60 \
  --output tracked.avi \
  --snapshot tracked.png

Build, test, and run the equivalent C++17 program:

cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build --parallel
ctest --test-dir build --output-on-failure

./build/object_tracker \
  --input=videos/chaplin.mp4 \
  --tracker=MIL \
  --bbox=287,23,86,320 \
  --max-frames=60 \
  --output=tracked.avi \
  --snapshot=tracked.png

Add --display to watch either program run. Add --select-roi to choose another target interactively instead of using the default Chaplin box.

Run DaSiamRPN, NanoTrack, and VitTrack

Current OpenCV 4 and 5 builds can expose three additional deep-learning single-object trackers: TrackerDaSiamRPN, TrackerNano, and TrackerVit. They are not detectors and they are not all new to OpenCV 5; each still needs a first-frame bounding box, and VitTrack’s API is available from OpenCV 4.9 onward. Unlike MIL, these trackers also require OpenCV’s dnn module and external ONNX files.

The ONNX binaries are intentionally not stored in the companion source tree. The included downloader uses immutable upstream revisions, limits every response to the expected byte count, verifies the full SHA-256 digest, and only then atomically places the file in models/. Download all three sets:

python download_models.py --tracker all

DaSiamRPN accounts for nearly all of the download. If you only want one of the smaller trackers, fetch a single set instead:

python download_models.py --tracker nano
python download_models.py --tracker vit
python download_models.py --tracker dasiamrpn

Use --models-dir /path/to/models with the downloader and the tracking program if you do not want to use the default directory beside the scripts. Re-running the downloader skips every file that already has the pinned size and checksum.

Run the model-backed trackers in Python

python tracker.py \
  --tracker DASIAMRPN --models-dir models \
  --bbox 287,23,86,320 --max-frames 60 \
  --output dasiamrpn.avi --snapshot dasiamrpn.png

python tracker.py \
  --tracker NANO --models-dir models \
  --bbox 287,23,86,320 --max-frames 60 \
  --output nano.avi --snapshot nano.png

python tracker.py \
  --tracker VIT --models-dir models \
  --bbox 250,15,180,340 --max-frames 60 \
  --output vit.avi --snapshot vit.png

Build and run the model-backed trackers in C++17

cmake -S . -B build -DBUILD_TESTING=ON \
  -DTRACKING_MODELS_DIR="$PWD/models"
cmake --build build --parallel
ctest --test-dir build --output-on-failure

./build/object_tracker \
  --tracker=DASIAMRPN --models-dir=models \
  --bbox=287,23,86,320 --max-frames=60

./build/object_tracker \
  --tracker=NANO --models-dir=models \
  --bbox=287,23,86,320 --max-frames=60

./build/object_tracker \
  --tracker=VIT --models-dir=models \
  --bbox=250,15,180,340 --max-frames=60

The original narrow Chaplin box is retained for the classic example and works as a model-loading smoke test for DaSiamRPN and NanoTrack. VitTrack changed that tall, narrow box sharply during sequence testing, so its documented command initializes a wider full-body box. This is a practical reminder that initialization geometry is part of a tracker’s input contract; use --select-roi --display to choose the target on a new video.

The companion implementation explicitly selects OpenCV’s DNN backend and CPU target for a reproducible cross-version path. CUDA, OpenVINO, and other backend/target combinations are build-dependent; query the targets available in the installed build and measure them on the intended hardware before changing those parameters.

Research-Backed Pros and Cons of the Four Trackers

The papers behind these trackers used different datasets, hardware, evaluation protocols, and model variants. Their published numbers are useful evidence about each design, but they are not a head-to-head ranking of the four OpenCV 5 implementations used here. In particular, the OpenCV DaSiamRPN wrapper omits parts of the full research system, NanoTrack v2 has no peer-reviewed method paper of its own, and OpenCV’s compact VitTrack model is influenced by—but is not the same model as—the full OSTrack system. We therefore use the papers to explain design tradeoffs and use the videos below only as sequence-specific observations.

MIL—online adaptation without a downloaded model. Babenko, Yang, and Belongie’s CVPR 2009 paper introduced the positive-bag update to cope with uncertainty about the target’s exact location. Their implementation ran at about twenty-five frames per second on a Core 2 Quad desktop, a historical result rather than an OpenCV 5 speed promise. The authors tracked location with a simple greedy motion model and later reported that the method handled partial occlusion and changing appearance on average across their small test set, while still failing under long complete occlusion and benefiting less on articulated targets. The extended TPAMI study also contains counterexamples where part-based or semi-supervised methods performed better. Practically, MIL’s online adaptation is useful, but it does not provide global re-detection: if the target moves outside the local search region or a wrong patch is used repeatedly for updates, drift can become self-reinforcing.

DaSiamRPN—learned matching and box regression, but not the full long-term paper in OpenCV. Zhu et al.’s ECCV 2018 paper used semantic negative pairs and online distractor templates to make Siamese features less likely to jump to similar objects. Its full system also expanded from local to global search after a confidence-defined failure. The authors reported 160 FPS for the short-term version and 110 FPS for the long-term version on a Titan X, plus relative gains of 9.6% on VOT2016 and 35.9% on UAV20L. Those results belong to the paper’s implementation and protocols; VOT also reinitialized a failed tracker after five frames. OpenCV’s TrackerDaSiamRPN source computes the template kernels once, uses a fixed 271-pixel local search, and implements neither online distractor-template reranking nor the paper’s global recovery. This boundary explains why fast displacement, full occlusion, and leaving the frame remain hard in our comparison.

NanoTrack v2—an engineering implementation with published research ancestry. The NanoTrack source states that its design mainly refers to SiamBAN and LightTrack. SiamBAN provides the published basis for anchor-free parallel classification and box regression; LightTrack provides the research case for jointly designing a compact backbone and head for constrained hardware. NanoTrack v2 itself is not the exact model evaluated in either paper and does not appear to have a peer-reviewed method paper, so their benchmark values must not be transferred to it. The safe claim is narrower: OpenCV’s implementation is exceptionally compact, but it freezes the initial template, searches around the previous estimate, and returns a box even when that box has visually drifted. A source audit also found that OpenCV 5.0.0 computes NanoTrack’s scale-change penalty using the target’s center position, while the reference Python implementation uses the previous target size. That makes the OpenCV penalty appear position-dependent; until upstream confirms or corrects it, this should be treated as a likely porting discrepancy and the results below as evidence for the stock OpenCV implementation, not for the NanoTrack design in general.

VitTrack—compact one-stream interaction with a usable confidence score. OpenCV’s model takes its high-level direction from the OSTrack paper, which jointly processes template and search tokens so that feature extraction and relation modeling inform each other. The paper reported 73.7% average overlap on GOT-10k for its full OSTrack model. That number does not describe OpenCV VitTrack: the OpenCV contributor documented that the compact model was trained separately, used the same training datasets rather than OSTrack weights, and changed the patch embedding for a smaller network. The implementation provenance therefore matters as much as the research citation. In this tutorial, VitTrack’s practical advantage is its 0.68 MiB model and explicit tracking score. Its limitation is that a threshold only rejects a low-scoring estimate; a high score can still belong to the wrong person, car, or background patch.

Four-Tracker Comparison and Practical Limitations

These comparisons use the clean source footage behind the original demonstration rather than replacing it with a new example. Each video shows MIL, DaSiamRPN, NanoTrack, and VitTrack simultaneously in one four-panel view, using the same source frames and initial box in every panel. There are no title cards or sequential replays.

The Chaplin comparison uses frames 0 through 149 from the original videos/chaplin.mp4 bundled with the companion example. For this shared four-way comparison, every tracker starts from the same wider full-body box, (250, 15, 180, 340); the original narrow Chaplin box remains unchanged in the classic example commands above. Chaplin turns away to a rear view, returns to face the camera, and finally moves toward the left edge. DaSiamRPN stays relatively tight around him, MIL reports a broader region, and NanoTrack and VitTrack follow the motion with boundary clipping at points in the sequence, including near the end. These are visible observations from this sequence, not a general accuracy ranking.

In the 330-frame street sequence, the same woman in black carrying the lime-green purse becomes partially and then substantially occluded by passing cyclists. Similar nearby people create additional target ambiguity. In the frame-exact 65-frame race-car section used by the original article, the same intentionally narrow target around the blue car’s front grille and bumper undergoes fast motion plus rapid appearance, viewpoint, and scale change. These conditions expose the trackers’ limited success; the drift and failures are intentional evidence of practical limitations, not a tracker ranking.

Each colored rectangle shows the visible portion of the box reported by that tracker. When an estimate extends beyond the source image, the panel labels it BOX OFF FRAME and the companion audit JSON retains the full, unclipped coordinates. No detector resets, manual corrections, or hidden smoothing were applied. A tracker can continue returning a box after it has visually drifted, so the videos deliberately retain both good estimates and visible failures.

The Chaplin clip is already included in the companion project as videos/chaplin.mp4. Download the street clip from tiburi’s street-video page and the race-car clip from DistillVideos’ race-car page. Both external source pages currently provide the videos under the Pixabay Content License. After downloading the verified tracker models as shown above, reproduce the three comparison-only timelines with:

python render_tracker_comparison.py \
  --input videos/chaplin.mp4 \
  --trackers MIL,DASIAMRPN,NANO,VIT \
  --models-dir models \
  --bbox 250,15,180,340 \
  --start-frame 0 --end-frame 149 \
  --scene-title "Charlie Chaplin Comparison" \
  --together-only \
  --output charlie-chaplin-opencv5-four-trackers.mp4

python render_tracker_comparison.py \
  --input street-5025.mp4 \
  --trackers MIL,DASIAMRPN,NANO,VIT \
  --models-dir models \
  --bbox 351,321,57,249 \
  --start-frame 0 --end-frame 329 \
  --scene-title "Street Occlusion Limitation" \
  --together-only \
  --output street-scene-opencv5-four-trackers.mp4

python render_tracker_comparison.py \
  --input car-74.mp4 \
  --trackers MIL,DASIAMRPN,NANO,VIT \
  --models-dir models \
  --bbox 734,420,300,100 \
  --start-frame 256 --end-frame 320 \
  --scene-title "Race-Car Motion Limitation" \
  --together-only \
  --output race-car-opencv5-four-trackers.mp4

The Python suite verifies the six pinned model records, atomic download behavior, missing-model diagnostics, MIL output files, five real updates with each of DaSiamRPN, NanoTrack, and VitTrack, the comparison-only four-panel timeline, and rollback of the renderer’s paired MP4/JSON output after an injected installation failure. All 17 tests passed with exact OpenCV 4.13.0, 4.14.0, and 5.0.0 Python environments. The native C++ CTest builds then ran MIL and all three model-backed trackers for five updates apiece; all four tests passed against exact OpenCV 4.13.0, 4.14.0, and 5.0.0 builds containing the required DNN module. Longer MIL evidence runs returned a box for 30 of 30 updates in the 4.13.0 and 5.0.0 evidence environments. The OpenCV 4.13 MIL run shown above moved the box from (287, 23, 86, 320) to (318, 30, 86, 320); final estimates differ by tracker, initialization, and version. These checks prove that every documented construction and execution path runs; they are smoke tests, not an accuracy or speed ranking.

Object tracking using OpenCV – the Algorithms

In this section, we will dig a bit into different tracking algorithms. The goal is not to have a deep theoretical understanding of every tracker, but to understand them from a practical standpoint.

Availability note: The descriptions below preserve the article’s original practical guide. Some rankings are explicitly labeled as historical observations, and not every tracker is exposed by current OpenCV packages. Check the factory available in the build you actually use.

Let me begin by first explaining some general principles behind tracking. In tracking, our goal is to find an object in the current frame given we have tracked the object successfully in all ( or nearly all ) previous frames.

Since we have tracked the object up until the current frame, we know how it has been moving. In other words, we know the parameters of the motion model. The motion model is just a fancy way of saying that you know the location and the velocity ( speed + direction of motion ) of the object in previous frames. If you knew nothing else about the object, you could predict the new location based on the current motion model, and you would be pretty close to where the new location of the object is.

But we have more information than just the motion of the object. We know how the object looks in each of the previous frames. In other words, we can build an appearance model that encodes what the object looks like. This appearance model can be used to search in a small neighborhood of the location predicted by the motion model to more accurately predict the location of the object.

The motion model predicts the approximate location of the object. The appearance model fine tunes this estimate to provide a more accurate estimate based on appearance.

If the object was very simple and did not change it’s appearance much, we could use a simple template as an appearance model and look for that template. However, real life is not that simple. The appearance of an object can change dramatically. To tackle this problem, in many modern trackers, this appearance model is a classifier that is trained in an online manner. Don’t panic! Let me explain in simpler terms.

The job of the classifier is to classify a rectangular region of an image as either an object or background. The classifier takes in an image patch as input and returns a score between 0 and 1 to indicate the probability that the image patch contains the object. The score is 0 when it is absolutely sure the image patch is the background and 1 when it is absolutely sure the patch is the object.

In machine learning, we use the word “online” to refer to algorithms that are trained on the fly at run time. An offline classifier may need thousands of examples to train a classifier, but an online classifier is typically trained using very few examples at run time.

A classifier is trained by feeding it positive ( object ) and negative ( background ) examples. If you want to build a classifier for detecting cats, you train it with thousands of images containing cats and thousands of images that do not contain cats. This way the classifier learns to differentiate what is a cat and what is not. While building an online classifier, we do not have the luxury of having thousands of examples of the positive and negative classes.

Let’s look at how different tracking algorithms approach this problem of online training.

BOOSTING Tracker

This tracker is based on an online version of AdaBoost — the algorithm that the HAAR cascade based face detector uses internally. This classifier needs to be trained at runtime with positive and negative examples of the object. The initial bounding box supplied by the user ( or by another object detection algorithm ) is taken as a positive example for the object, and many image patches outside the bounding box are treated as the background.

Given a new frame, the classifier is run on every pixel in the neighborhood of the previous location and the score of the classifier is recorded. The new location of the object is the one where the score is maximum. So now we have one more positive example for the classifier. As more frames come in, the classifier is updated with this additional data.

Historical author observation: In the original comparison, I found no compelling reason to choose BOOSTING over the newer MIL and KCF alternatives that were then available.

Historical author observation: Tracking performance was mediocre on the original examples, and failure was not reported reliably. Treat that as an observation from those tests, not a universal benchmark.

MIL Tracker

This tracker is similar in idea to the BOOSTING tracker described above. The big difference is that instead of considering only the current location of the object as a positive example, it looks in a small neighborhood around the current location to generate several potential positive examples. You may be thinking that it is a bad idea because in most of these “positive” examples the object is not centered.

This is where Multiple Instance Learning ( MIL ) comes to rescue. In MIL, you do not specify positive and negative examples, but positive and negative “bags”. The collection of images in the positive bag are not all positive examples. Instead, only one image in the positive bag needs to be a positive example!

In our example, a positive bag contains the patch centered on the current location of the object and also patches in a small neighborhood around it. Even if the current location of the tracked object is not accurate, when samples from the neighborhood of the current location are put in the positive bag, there is a good chance that this bag contains at least one image in which the object is nicely centered. The MIL publication page has the paper and citation for readers who want to dig deeper into the tracker.

Historical author observation: MIL performed well on the original examples, drifted less than BOOSTING, and handled some partial occlusion. In the refreshed example, MIL is selected because it is the tested model-free path shared by the OpenCV 4 and 5 builds—not because it is universally the most accurate tracker.

Limitations: A successful update is not a guarantee that the box is correct, and MIL should not be expected to recover reliably from full occlusion.

KCF Tracker

KCF stands for Kernelized Correlation Filters. This tracker builds on the ideas presented in the previous two trackers. This tracker utilizes the fact that the multiple positive samples used in the MIL tracker have large overlapping regions. This overlapping data leads to some nice mathematical properties that are exploited by this tracker to make tracking efficient.

Historical author observation: KCF was faster and more accurate than MIL on the original examples and reported failure more usefully than BOOSTING or MIL. Current performance depends on the video and build; the refreshed program exposes KCF only when the installed OpenCV package provides it.

Limitation: KCF should not be expected to recover reliably from full occlusion. 

TLD Tracker

TLD stands for Tracking, learning, and detection. As the name suggests, this tracker decomposes the long term tracking task into three components — (short term) tracking, learning, and detection. From the author’s paper, “The tracker follows the object from frame to frame. The detector localizes all appearances that have been observed so far and corrects the tracker if necessary.

The learning estimates detector’s errors and updates it to avoid these errors in the future.” This output of this tracker tends to jump around a bit. For example, if you are tracking a pedestrian and there are other pedestrians in the scene, this tracker can sometimes temporarily track a different pedestrian than the one you intended to track. On the positive side, this track appears to track an object over a larger scale, motion, and occlusion. If you have a video sequence where the object is hidden behind another object, this tracker may be a good choice.

Historical author observation: TLD handled multi-frame occlusion and scale changes well on the original examples.

Historical author observation: The original tests also produced many false positives. Results vary with the sequence, and current package availability varies.

MEDIANFLOW Tracker

Internally, this tracker tracks the object in both forward and backward directions in time and measures the discrepancies between these two trajectories. Minimizing this ForwardBackward error enables them to reliably detect tracking failures and select reliable trajectories in video sequences.

In my original tests, I found this tracker worked best when the motion was predictable and small. Unlike other trackers that kept going even when tracking had clearly failed, MEDIANFLOW reported failure more usefully on those examples.

Historical author observation: It reported failure well and worked very well when motion was predictable and there was no occlusion.

Historical author observation: It failed under large motion. Current package availability varies.

GOTURN tracker

Of the eight trackers in the original OpenCV 4.2 list, GOTURN was the only one based on a Convolutional Neural Network (CNN). It was designed to handle viewpoint, lighting, and deformation changes, but it does not handle occlusion well. OpenCV now has other model-backed tracking options, so this is a historical distinction rather than a description of the whole modern library.

Notice: GOTURN uses a Caffe model and prototxt file. Those external model files and their preprocessing contract must be supplied separately, so GOTURN is outside the tested portable path in this tutorial. Consult the official OpenCV GOTURN documentation for the current interface and model requirements.

Update: GOTURN object tracking algorithm has been ported to OpenCV.

MOSSE tracker

Minimum Output Sum of Squared Error (MOSSE) uses adaptive correlation filters initialized from a single frame. It was designed for high update speed and can tolerate some lighting, scale, pose, and non-rigid changes. Its peak-to-sidelobe ratio can help identify occlusion. Actual speed, accuracy, and recovery behavior depend on the hardware, implementation, initialization, and video, so the original fixed-FPS comparison should not be treated as a benchmark.

CSRT tracker

In the Discriminative Correlation Filter with Channel and Spatial Reliability (DCF-CSR), a spatial reliability map adjusts the filter support to the useful part of the selected region. This can improve localization for non-rectangular targets. The OpenCV implementation uses HOG and Color Names features. CSRT is commonly chosen when tighter localization matters more than update speed, but its speed and accuracy must be measured on the target workload rather than assumed from a fixed FPS value.

Limitations and Production Guidance

Single-object tracking still has several hard failure modes:

• Bad initialization: Background inside the first box becomes part of the appearance model.
• Occlusion: The tracker may drift to a similar nearby region.
• Scale and pose change: A rigid box may not represent a deforming target.
• Leaving the frame: A success flag can lag behind actual target loss.
• Camera cuts: Appearance continuity disappears immediately.
• Model and backend mismatch: A model-backed tracker can fail to construct when the ONNX file, preprocessing contract, DNN module, backend, or target does not match the installed build.

For production, validate the first box, monitor a quality signal instead of trusting only the Boolean result, rerun a detector periodically or when quality drops, and measure end-to-end latency including decoding and output. Evaluate on representative occlusions, lighting changes, camera motion, and target motion.

The example now runs NanoTrack, VitTrack, and DaSiamRPN, but that support does not make them drop-in quality replacements for MIL. DaSiamRPN and NanoTrack can continue returning a successful update after visual drift, while VitTrack applies an internal score threshold. A high score still does not prove identity. Keep the checksum-pinned model files with the deployment, record the backend and target, and evaluate accuracy and latency on labeled examples from the real workload.

Frequently Asked Questions

Which modern OpenCV tracker should I choose?

There is no universal winner. Use MIL when you want the tested model-free baseline. Among the model-backed choices in this tutorial, DaSiamRPN provides learned local matching with a 154.35 MiB download, NanoTrack v2 prioritizes compact deployment at 1.70 MiB, and VitTrack uses a 0.68 MiB model with a configurable score threshold in OpenCV 4.11 and later. Measure accuracy and end-to-end latency on labeled clips from your own workload.

Are DaSiamRPN, NanoTrack, and VitTrack object detectors?

No. They are trackers, not detectors. Each needs a bounding box on the first frame. A detector can initialize that box, periodically correct drift, or reacquire the target after it is lost.

Are the model-backed trackers exclusive to OpenCV 5?

No. The exact companion paths were tested with OpenCV 4.14.0 and 5.0.0, with an additional OpenCV 4.13.0 regression run. VitTrack’s API arrived in OpenCV 4.9, while its score-threshold rejection behavior appears in OpenCV 4.11 and later. Packaging and language bindings still vary, so inspect the factories in the build you actually import or link. The OpenCV 4 tracking API and OpenCV 5 tracking API document the available classes.

Why does the initial bounding box matter so much?

The model-backed trackers used here form a first-frame target template and then search near the previous estimate. Too much background, clipped target parts, or the wrong aspect ratio in the initial box can therefore affect every later match. Start with a tight, representative ROI and test it on the real sequence.

Can these trackers recover after full occlusion or very fast motion?

Not reliably. No tracker shown here provides reliable global re-detection. Local search makes long occlusion, large frame-to-frame displacement, and leaving the frame difficult, and OpenCV’s DaSiamRPN wrapper omits the research system’s global recovery stage. Production systems should monitor quality and reinitialize from a detector when needed.

Does a successful update or high tracking score prove target identity?

No. update() returning true means a box was returned, not that it still surrounds the intended target. getTrackingScore() is tracker-specific: MIL does not provide a meaningful learned confidence, while the model-backed scores are not calibrated for comparison with one another. In OpenCV 4.11 and later, including the tested 4.13.0, 4.14.0, and 5.0.0 builds, VitTrack’s threshold can reject a weak estimate; a high score can still belong to the wrong object or background patch.

How do I download and verify the tracker models?

Run python download_models.py --tracker all to fetch all six ONNX files with pinned size and SHA-256 checks. DaSiamRPN needs three files, NanoTrack v2 needs a backbone and head, and VitTrack needs one model. Use dasiamrpn, nano, or vit instead of all when you need only one tracker.

Do model-backed trackers use a GPU automatically?

No. The reproducible path in this tutorial explicitly uses DNN_BACKEND_OPENCV and DNN_TARGET_CPU. The tracker parameter objects can request other DNN backend and target combinations, but only when the installed OpenCV build and the selected model support them. Query the available targets and benchmark the complete pipeline before changing the defaults.

Can I compare the published tracker benchmarks directly?

No. The cited papers use different datasets, hardware, protocols, and model variants. OpenCV’s DaSiamRPN omits parts of the published system, NanoTrack v2 is not the exact model evaluated in its related papers, and compact VitTrack is not the full OSTrack model. Treat published numbers as design evidence, not a ranking of these OpenCV implementations.

When should I use multi-object tracking instead?

Each tracker instance follows one initialized box. If targets enter and leave, cross paths, or need persistent identities, use a pipeline with detection, data association, lifecycle management, and—when the application requires it—appearance-based re-identification. Running one single-object tracker per target does not provide those system-level guarantees by itself.

Conclusion

The core lesson of the original article still holds: a tracker combines motion and appearance information to follow a specific target between detections. The refreshed implementation preserves the original Chaplin, race-car, street-scene, occlusion, identity, and face-tracking examples while replacing obsolete factory code with tested Python 3 and C++17 paths. MIL provides the common model-free path; DaSiamRPN, NanoTrack v2, and VitTrack now provide three tested model-backed paths with reproducible downloads; runtime capability checks keep optional trackers honest.

From idea to working model to real-time deployment

Big Vision takes computer vision projects through the full journey, not just the easy parts.

Satya Mallick Founder - Big Vision & LearnOpenCV

Talk to our Team

Credits and References

Video Credits: All videos used in this post are in the public domain — Charlie Chaplin, Race Car, and Street Scene. Dr. Boris Babenko generously gave permission to use his animation in this post.

1. Bolme, David S.; Beveridge, J. Ross; Draper, Bruce A.; Lui, Yui Man. Visual Object Tracking using Adaptive Correlation Filters. In CVPR, 2010.
2. OpenCV 4 Tracking API.
3. OpenCV 5 Object Tracking API.
4. OpenCV 5 model-backed object tracker sample.
5. Zhu, Zheng; Wang, Qiang; Li, Bo; Wu, Wei; Yan, Junjie; Hu, Weiming. Distractor-aware Siamese Networks for Visual Object Tracking. In ECCV, 2018.
6. NanoTrack implementation and v2 model source.
7. OpenCV Model Zoo VitTrack model and demo.
8. Babenko, Boris; Yang, Ming-Hsuan; Belongie, Serge. Visual Tracking with Online Multiple Instance Learning. In CVPR, 2009.
9. Babenko, Boris; Yang, Ming-Hsuan; Belongie, Serge. Robust Object Tracking with Online Multiple Instance Learning. In IEEE TPAMI, 2011.
10. Chen, Zedu; Zhong, Bineng; Li, Guorong; Zhang, Shengping; Ji, Rongrong. Siamese Box Adaptive Network for Visual Tracking. In CVPR, 2020.
11. Yan, Bin; Peng, Houwen; Wu, Kan; Wang, Dong; Fu, Jianlong; Lu, Huchuan. LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search. In CVPR, 2021.
12. Ye, Botao; Chang, Hong; Ma, Bingpeng; Shan, Shiguang; Chen, Xilin. Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework. In ECCV, 2022.
13. Henriques, João F.; Caseiro, Rui; Martins, Pedro; Batista, Jorge. High-Speed Tracking with Kernelized Correlation Filters. In IEEE TPAMI, 2015.