Phase 2: Tracking Loss

Why we needed it?

When the IR LEDS are tracked, some reasons might cause its loss, like:

1- Getting out of the projection area.

2- Some LEDs turn off.

3- Barriers get into the way of tracking the sensors.

And so the projected image will get distorted, so we needed to estimate the coordinates of the lost IR LEDS.

Actually after implementing this part in our project, the tracking was improved a lot :D.

Scenarios:

  We've handled the first 3 scenarios, and ignored when only 1 sensor is available as it's uncommon in most cases.

Algorithm:

  

 We mainly Store the last successful tracking for the 4 LEDs in order to be able to get the coordinates of the lost LEDs.

Of course after reading the algorithm details, you'll find a really important problem that appears "Point Identification" (the new tracked sensors map to which old sensors). We'll discuss this problem and the suggested solution to it in a later article.

1- 1 LED lost,  3 Tracked   (Affine)

1- We simply calculate the affine matrix (2*3) that maps "3 Tracked sensors" to "their corresponding old positions".

RotationMatrix2D map = CameraCalibration.GetAffineTransform(src, dst);

src: their corresponding old positions.

dst:   3 Tracked sensors

2- Then take this matrix and multiply it with the corresponding old position of the lost sensor. Which will output the correct estimated position of the sensor.

2- 2 LEDs lost, 2 Tracked  (Simplified Affine)

Since there's no ready made fn that makes the simplified affine, so we implemented it ourselves.

We need to calculate 2 dimensions of translation, one degree of rotation, & one degree of scaling.

1- We simply construct 2 vectors, Vector1 (new tracked 2 points), Vector2  (their corresponding old points).

2- Translate scene origin, Translate scene (-x0,-y0) 

hint: (x0,y0) id the start point of Vector1

3- Calculate scaling factor, S= |vector2| / |vector1|

hint: |vector2| is its magnitude

4- Calculate rotation angle, Theta= Theta2- Theta1

hint: Theta2= Vector1.y / Vector.x

5- Calculate the matrix that encapsulates the rotation & scaling, and multiply it with the scene

CvInvoke.cv2DRotationMatrix(center, rotation_angle, scale_factor, map.Ptr);

hint: center is the origin of course

6- Translate the scene to (x1,y1)

hint: (x1,y1) is the start point of Vector2

Phase 2: Calibration

This part of the project really confused us for a long time, we were so convinced that we won't need manual calibration and we spent the whole time trying to implement automatic calibration. But, then we faced some problems that I'll discuss in details that lead to the necessity of using manual calibration.

First, What's Calibration?
It is a preprocessing stage before warping, in which we align the coordinates of the display to the wii-mote coordinates.

Manual Calibration : suggests that the user at the beginning of the application, marks the corners of the projector, and save the corresponding wii-mote coordinates. Then maps the corners of the projector to the corners of the wii.

Automatic Calibration: doesn't need any user interference to map the display to wii-mote coordinates.


Second, Why use calibration?

Well, if we took the coordinates directly from the wii-mote without any preprocessing and projected on those coordinates, we'll find that the image is not positioned directly on the display, since the wii-mote coordinates system [1024*768] is different than the projector/ display coordinate system [Screen resolution]. And if we positioned the wii-mote away from the projector, that might lead to distorted image. So we do this process to calculate origin (0,0) of the screen maps to what coordinate in the wii-mote.


Third, we're going to discuss in details each approach.

Manual Calibration in details,

How,
1- At the beginning of the application, a white screen is projected and small crosshairs at the corners of the screen are drawn.
2- The user user selects those crosshairs using the IR pen.
3- Then we save the corresponding wiimote coordinates.
4- After that we compute a homography matrix, where the source points are the screen corners, & the destination points are are the saved wiimote coordinates.

5- After that whenever we save coordinates of the IR LEDs, we multiply it by the above homography matrix.

Advantages,

1- A very big advantage is that it doesn't place any constraints on the place of the wii-mote relative to the projector.
2- It's accurate, the position of the image is placed on the display without any errors.

Disadvantages,

1- A major disadvantage is that it needs user interference to calibrate.

Automatic Calibration in details,

How,
1- without any user interference, we take the coordinates of the IR LEDs and do some calculations on it to output the mapped display coordinates.
2- Those calculations are illustrated below: (similar to normalization equations)

pt.x = (int)((newpt.x / 1023) * this.Width);

pt.y = this.Height - (int)((newpt.y / 767) * this.Height – 225);

Hint:
1- in the Y equation, we subtract [Height – newpt], because the origin of the screen is top left while the origin of the wii-mote is bottom left.

2- in the Y equation also, we subtract with a constant value 225. It is due to the difference between the projector lense and the wii-mote just placed above. You won't believe how we got this number, but we actually took a ruler measured the distance between them it was about 2 cm, then took this ruler and measured the screen. With some cross multiplication :D, we calculated (2 cm) corresponds to how many pixels (225).

Advantages,

1- It's obvious that its main advantage, is that it won't need user interaction.

Disadvantages,

1- It places a constraint that the wii-mote must be placed above the projector.
2- The big problem appeared, when we found that the size of the image wasn't perfectly matching to the size of the display.


Last,  We decided to go for manual calibration as it's more accurate.

Second Phase

During the second phase we mainly focused on improving the display and solving the problems that popped out during phase 1.

what We mainly did in second phase:
1- Calibration
2- Tracking Loss
3- Point Identification
4- Constructing Frame's HW

We'll discuss each one of those modules in other articles.

After we finished this phase the building block of the wii display was finished,
we had a movable projected display that now can be used in many applications.

it was only left to implement some of the applications, and add extra features to the display, which we'll see in the upcoming phases.

Phase 1: Image Warping

1- Background Information: 

-Why we need Image Warping?

To project the image properly on the display while moving, we need to calculate the transformation matrix that's being applied to the image before projecting it in order to fit the movement of the display.

- How image warping is implemented?

As the display moves new coordinates are given to the IRLEDS , so using these new coordinates (destination) and the corners of the image (source), we can estimate the homography matrix.

The homography matrix (transformation) is applied to the image and we get the warped image to be projected. This is all done by openCV functions :
-“cvFindHomography”: which computes the homography matrix using source & destination points (IRLEDS coordinates).
-“cvWarpPerspective”: This applies the transformation calculated above to the image to be projected.

 

2- Into the details:


- Transformations Taxonomy:
There are  4 types of transformations:

 the following picture shows the effect of applying different transformation types on an image

and since that our display moves in 8 degrees of  freedom, so we used the Homography matrix.

dof: (forward/backward, up/down, left/right (translation) combined with rotation about three perpendicular axes (pitch, yaw, roll))

for more details abt the projective transformation see this ppt.


- Forward Warping VS Inverse Warping
This ppt explains the concept of forward & inverse warping,
remeber if u're using the inverse warping you should use the inverse of the homography matrix not the homography matrix itself 
 

3- C# Code: 

This code uses openCV 1.1,  this book explains the openCv fns used below

also we use Emgu .Net Wrapper and this is its Wiki for documentation abt it

This's just a simple test code, to warp an image from srcp to dstp

private void warping()
        {
           
            imageBox2.Image=src;

            float[,] srcp = { { 0, 0 }, { src.Width-1, 0 }, { 0, src.Height-1 }, { src.Width-1, src.Height-1} };
            float[,] dstp = { { src.Width * 0.05f, src.Height * 0.33f }, { src.Width * 0.9f, src.Height * 0.25f }, { src.Width * 0.2f, src.Height * 0.7f }, { src.Width * 0.8f, src.Height * 0.9f } };
            float[,] homog = new float[3, 3];

            Matrix srcpm = new Matrix(srcp);
            Matrix dstpm = new Matrix(dstp);
            Matrix homogm = new Matrix(homog);
            IntPtr n = new IntPtr();

            CvInvoke.cvFindHomography(srcpm.Ptr, dstpm.Ptr, homogm.Ptr, Emgu.CV.CvEnum.HOMOGRAPHY_METHOD.DEFAULT, 0, n);
            src= src.WarpPerspective(homogm, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC, Emgu.CV.CvEnum.WARP.CV_WARP_INVERSE_MAP, new Bgr(Color.White));
            imageBox1.Image = src;
        }