MAEG 5720 Computer Vision in Practice – Project 1
Aim: To familiarize with template tracking and detecting moving object in a video stream
In this project, you will implement a simple ‘translation-only’ Lucas-Kanade tracker to track a local 2D
template in a sequence of image. The Lucas-Kanade works on two consecutive frames each time. We
assume the image intensity doesn’t change significantly between two frames. The tracking algorithm
estimate the deformations between two image frames.
The template tracking algorithm estimate the constant motion (at any image position (π₯π, π¦π) of the
consecutive image frames πΌπ π‘π πΌπ+1) inside a widow W by inverting the Harris matrix. The motion is
expressed as the offset (π’, π£) that is to be added to the image position (π₯π, π¦π) and this will be the
newly tracked position in image frame πΌπ+1
The Algorithm:
Given:
(1) The current image frame πΌπ
(2) The template T in the previous image frame πΌπ−1, centered at image position (π₯π, π¦π)
Goal:
(1) Compute the displacement (π’, π£) using Lucas-Kanade flow
where ππ₯, ππ¦ is the gradient of the template π in the x and y direction respectively in the
window π and πΌπ‘
is the pixel difference between π(πΌ1) and π
(2) Translate π by (π’, π£) to π′
(3) Compare π′(πΌπ) and ππ(π’, π£)π(πΌ1)π′(πΌ1)
Several Enhancements of the tracking algorithm
(1) As the algorithm assumes very small motion between every two adjacent image frames, any
displacement (π’, π£)>1 pixel cannot be computed. A solution to iterate the same algorithm
several times is needed. At each iteration step, the window π is shifted by the recovered
motion of π′. Then template π and π′
(πΌ) is compared (e.g. SSD, or correlation). If the
difference is larger than the predefined threshold, the iteration will continue until it converges.
(2) We are interested in the centre movement of the window πand therefore you may consider
giving higher weight to those value of πΌπ₯(π₯, π¦) and πΌπ¦(π₯, π¦) near the centre. (e.g. Gaussian)
(3) For large movement, it will be advantages to reduce the resolution of the image (coarse-to-fine
approach).
What to do?
You will track a specific object in an image sequence by template tracking. You will apply Lucas-Kanade
tracking technique to track a template in the image sequence. You will be given a CarSequence.rar file.
You will write a MATLAB function to implement the template tracking method of translation-only
model.
In the package, there are a .rar file and two .m files:
(1) CarSequence.rar contains the image sequences and the template image.
(2) The script test_motion.m is provided to you for testing your function. This script simply (1)
repeatedly calls the function trackTemplate (that you will code) on every consecutive pair of
images in the file CarSequence.rar. (2) make an AVI movie out of the moving_image returned for
every image processed, (3) save it as a file motion.avi for your offline viewing, and (4) at last
save your tracking windows to track_coordinates.mat. Please set the path_to_images and
numimages prior to using this function. Please do not submit your AVI file.
(3) The trackTemplate.m is the code you will write to track the template in every consecutive pair of
images. The script firstly (1) finds the template position in the first image. (2) tracks the
template over the rest of the images and (3) at last return the template window coordinates for
each image in the image sequence. All image and template will be transformed to gray-scale
image for convenience. Four parameters in the trackTemplate function
a. Img1 is the template image,
b. Img2 is the image on which you shall track the template
c. Windows 1x4 matrix is the coordinate of top-left and bottom right corners of W. You’ll
probably notice that the tracker will lose the template before the end of the sequence.
To fix this, you may smooth the template and the windows using Gaussian function
d. Sigma is the S.D. of the Gaussian function. Please try the range from 1 to 10.
Summary of codes to compose
Compose the following modules of codes in MATLAB:
(1) Test_motion.m: After determining the template position in each of the images, display each
color image in the sequence with a bounding box. If your tracker loses the object before the end
of the sequence, display a message and exit
(2) trackTemplate.m: For the first image, find the initial template window position; for each of the
image sequence, iteratively update the window position until the motion (π’, π£) is small. Then
compare the two windows in the current image and the previous image to ensure the error is
small enough.
Detail and Hints
(1) Please make a .MAT file which has the top-left and bottom-right corners of your tracked object
for evaluating the tracking result. The .MAT file should contain a variable called TrackedObject
which is an nx4 matrix. Ni is the number of image frames in the input sequence CarSequence
and each row in the matrix contains 4 numbers: x1, x2 y1 y2 representing the coordinates of the
top-left and bottom-right corners of the object you have tracked. Please name the .MAT file as
track_coorinates.mat.
(2) Since determining the motion vector (u,v) for each frame is an iterative procedure. Please be
reminded to send the maximum number of iterations in case your template is lost.
(3) In each step, the template will not be moving by integer amount of pixel, you may need to use
interp2 to interpolate the value of gradient.
(4) The meshgrid function can be useful for creating array of (π₯, π¦) coordinate for all pixels within a
car’s image windows.
(5) Please vectorize your operations to enhance the running efficiency.
What to submit?
(1) One MATLAB function: trackTemplate
(2) A MAT file: track_coordinates.mat file which contains the track coordinate of car_template.jpg
by the trackTemplate function.
(3) A PDF file: miniProject.pdf to describe your template tracking algorithm including the
assumptions and parameter choices. (Gaussian sigma value and termination criteria). Please also
include a short discussion about the effect of different smoothing sigma values and possible
causes