Phase 2: Tracking Loss

Why we needed it?

When the IR LEDS are tracked, some reasons might cause its loss, like:

1- Getting out of the projection area.

2- Some LEDs turn off.

3- Barriers get into the way of tracking the sensors.

And so the projected image will get distorted, so we needed to estimate the coordinates of the lost IR LEDS.

Actually after implementing this part in our project, the tracking was improved a lot :D.

Scenarios:

  We've handled the first 3 scenarios, and ignored when only 1 sensor is available as it's uncommon in most cases.

Algorithm:

  

 We mainly Store the last successful tracking for the 4 LEDs in order to be able to get the coordinates of the lost LEDs.

Of course after reading the algorithm details, you'll find a really important problem that appears "Point Identification" (the new tracked sensors map to which old sensors). We'll discuss this problem and the suggested solution to it in a later article.

1- 1 LED lost,  3 Tracked   (Affine)

1- We simply calculate the affine matrix (2*3) that maps "3 Tracked sensors" to "their corresponding old positions".

RotationMatrix2D map = CameraCalibration.GetAffineTransform(src, dst);

src: their corresponding old positions.

dst:   3 Tracked sensors

2- Then take this matrix and multiply it with the corresponding old position of the lost sensor. Which will output the correct estimated position of the sensor.

2- 2 LEDs lost, 2 Tracked  (Simplified Affine)

Since there's no ready made fn that makes the simplified affine, so we implemented it ourselves.

We need to calculate 2 dimensions of translation, one degree of rotation, & one degree of scaling.

1- We simply construct 2 vectors, Vector1 (new tracked 2 points), Vector2  (their corresponding old points).

2- Translate scene origin, Translate scene (-x0,-y0) 

hint: (x0,y0) id the start point of Vector1

3- Calculate scaling factor, S= |vector2| / |vector1|

hint: |vector2| is its magnitude

4- Calculate rotation angle, Theta= Theta2- Theta1

hint: Theta2= Vector1.y / Vector.x

5- Calculate the matrix that encapsulates the rotation & scaling, and multiply it with the scene

CvInvoke.cv2DRotationMatrix(center, rotation_angle, scale_factor, map.Ptr);

hint: center is the origin of course

6- Translate the scene to (x1,y1)

hint: (x1,y1) is the start point of Vector2

Phase 2: Calibration

This part of the project really confused us for a long time, we were so convinced that we won't need manual calibration and we spent the whole time trying to implement automatic calibration. But, then we faced some problems that I'll discuss in details that lead to the necessity of using manual calibration.

First, What's Calibration?
It is a preprocessing stage before warping, in which we align the coordinates of the display to the wii-mote coordinates.

Manual Calibration : suggests that the user at the beginning of the application, marks the corners of the projector, and save the corresponding wii-mote coordinates. Then maps the corners of the projector to the corners of the wii.

Automatic Calibration: doesn't need any user interference to map the display to wii-mote coordinates.


Second, Why use calibration?

Well, if we took the coordinates directly from the wii-mote without any preprocessing and projected on those coordinates, we'll find that the image is not positioned directly on the display, since the wii-mote coordinates system [1024*768] is different than the projector/ display coordinate system [Screen resolution]. And if we positioned the wii-mote away from the projector, that might lead to distorted image. So we do this process to calculate origin (0,0) of the screen maps to what coordinate in the wii-mote.


Third, we're going to discuss in details each approach.

Manual Calibration in details,

How,
1- At the beginning of the application, a white screen is projected and small crosshairs at the corners of the screen are drawn.
2- The user user selects those crosshairs using the IR pen.
3- Then we save the corresponding wiimote coordinates.
4- After that we compute a homography matrix, where the source points are the screen corners, & the destination points are are the saved wiimote coordinates.

5- After that whenever we save coordinates of the IR LEDs, we multiply it by the above homography matrix.

Advantages,

1- A very big advantage is that it doesn't place any constraints on the place of the wii-mote relative to the projector.
2- It's accurate, the position of the image is placed on the display without any errors.

Disadvantages,

1- A major disadvantage is that it needs user interference to calibrate.

Automatic Calibration in details,

How,
1- without any user interference, we take the coordinates of the IR LEDs and do some calculations on it to output the mapped display coordinates.
2- Those calculations are illustrated below: (similar to normalization equations)

pt.x = (int)((newpt.x / 1023) * this.Width);

pt.y = this.Height - (int)((newpt.y / 767) * this.Height – 225);

Hint:
1- in the Y equation, we subtract [Height – newpt], because the origin of the screen is top left while the origin of the wii-mote is bottom left.

2- in the Y equation also, we subtract with a constant value 225. It is due to the difference between the projector lense and the wii-mote just placed above. You won't believe how we got this number, but we actually took a ruler measured the distance between them it was about 2 cm, then took this ruler and measured the screen. With some cross multiplication :D, we calculated (2 cm) corresponds to how many pixels (225).

Advantages,

1- It's obvious that its main advantage, is that it won't need user interaction.

Disadvantages,

1- It places a constraint that the wii-mote must be placed above the projector.
2- The big problem appeared, when we found that the size of the image wasn't perfectly matching to the size of the display.


Last,  We decided to go for manual calibration as it's more accurate.

Second Phase

During the second phase we mainly focused on improving the display and solving the problems that popped out during phase 1.

what We mainly did in second phase:
1- Calibration
2- Tracking Loss
3- Point Identification
4- Constructing Frame's HW

We'll discuss each one of those modules in other articles.

After we finished this phase the building block of the wii display was finished,
we had a movable projected display that now can be used in many applications.

it was only left to implement some of the applications, and add extra features to the display, which we'll see in the upcoming phases.

Phase 1: Image Warping

1- Background Information: 

-Why we need Image Warping?

To project the image properly on the display while moving, we need to calculate the transformation matrix that's being applied to the image before projecting it in order to fit the movement of the display.

- How image warping is implemented?

As the display moves new coordinates are given to the IRLEDS , so using these new coordinates (destination) and the corners of the image (source), we can estimate the homography matrix.

The homography matrix (transformation) is applied to the image and we get the warped image to be projected. This is all done by openCV functions :
-“cvFindHomography”: which computes the homography matrix using source & destination points (IRLEDS coordinates).
-“cvWarpPerspective”: This applies the transformation calculated above to the image to be projected.

 

2- Into the details:


- Transformations Taxonomy:
There are  4 types of transformations:

 the following picture shows the effect of applying different transformation types on an image

and since that our display moves in 8 degrees of  freedom, so we used the Homography matrix.

dof: (forward/backward, up/down, left/right (translation) combined with rotation about three perpendicular axes (pitch, yaw, roll))

for more details abt the projective transformation see this ppt.


- Forward Warping VS Inverse Warping
This ppt explains the concept of forward & inverse warping,
remeber if u're using the inverse warping you should use the inverse of the homography matrix not the homography matrix itself 
 

3- C# Code: 

This code uses openCV 1.1,  this book explains the openCv fns used below

also we use Emgu .Net Wrapper and this is its Wiki for documentation abt it

This's just a simple test code, to warp an image from srcp to dstp

private void warping()
        {
           
            imageBox2.Image=src;

            float[,] srcp = { { 0, 0 }, { src.Width-1, 0 }, { 0, src.Height-1 }, { src.Width-1, src.Height-1} };
            float[,] dstp = { { src.Width * 0.05f, src.Height * 0.33f }, { src.Width * 0.9f, src.Height * 0.25f }, { src.Width * 0.2f, src.Height * 0.7f }, { src.Width * 0.8f, src.Height * 0.9f } };
            float[,] homog = new float[3, 3];

            Matrix srcpm = new Matrix(srcp);
            Matrix dstpm = new Matrix(dstp);
            Matrix homogm = new Matrix(homog);
            IntPtr n = new IntPtr();

            CvInvoke.cvFindHomography(srcpm.Ptr, dstpm.Ptr, homogm.Ptr, Emgu.CV.CvEnum.HOMOGRAPHY_METHOD.DEFAULT, 0, n);
            src= src.WarpPerspective(homogm, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC, Emgu.CV.CvEnum.WARP.CV_WARP_INVERSE_MAP, new Bgr(Color.White));
            imageBox1.Image = src;
        }

Phase I:: Basic IR Tracking

At first we had to collect all the information about the wii-mote device and its capabilities, before implementing the IR Tracking module. After that we began to search for ready made libraries that deals with the wii-mote device.

1- Background Information:

Wiibrew is wonderful source of information on this device, to know its details.






But for this project we only needed certain information which are,


- Connecting to wii:
In order to interface with the wiimote, it must first be placed in discoverable mode by either pressing the 1 and 2 buttons at the same time, or by pressing the red sync button under the battery cover. Once in this mode, the Wii Remote can be queried by the Bluetooth HID driver on the host.

- Wii Reports:
This block includes an enumeration of reports that the device understands. A report can be thought of similar to a network port assigned to a particular service. An "Input" report is sent by the Wii Remote to the host. An "Output" report is sent by the host to the Wii Remote. The Wii Remote has a number of different data reporting modes. Each of these modes combines certain Core data features with data from external peripherals, and sends it to the host through one of the report IDs.

- Wii IR Camera:
We use the IR Camera inside, which has an object tracking processor (IR Blob Tracking).
It's capable of tracking up to 4 moving objects, and these data are the only data available to the host. The IR camera has an effective field of view is about 33 degrees horizontally and 23 degrees vertically (as measured on one unit).

2- Implementation Details:

We used Brian Peek's WiimoteLib ,
and its wrapper  BobCat Library

and inside the Bobcat wrapper we used the first layer ( Code Layers )
because it had 2 main features that we needed
a) it performs smoothing
b) it keeps track of points to keep point indexes consistent as they move.

First Phase

as we started to work, we made a plan to which we shall stick for the the rest of our work...
we divided our project into modules which is much easier for implementation and maintenance.
And we focused in first phase on:
1- Basic IR Tracking
2- Image Warping
3- Automatic Calibration

We'll discuss each one of those modules in other articles, 

but for now we'll discuss some of the problems that we faced during implementation

 
Problems faced:

The projected image wasn't correctly aligned with size of the of the display ...so figured that we should manually calibrate the wii to projector before projecting.

another problem is that sometimes the image freezes when moving it , this's because wii loses track of some of the Ir Leds ....so we need to track the lost Leds.

Also the image sometimes turned upside down this was because of point identification, which means identifying the LED as the top left, top right, ...
and the solution was either use motion modeling ,data grid, or optical flow...
we thought we shall start with the data grid solution and see the results.

Related Work

We were mainly inspired by Johnny Lee's Thesis 2008, projector based Tracking
http://johnnylee.net/academic/p104-lee.pdf

Those are some videos for his work





And this is Johnny Lee's Blog, for more information about him
http://procrastineering.blogspot.com/2008_06_01_archive.html


And of course we were really inspired by Sixth Sense next Gen by Pranav Mistry that was posted on Nov 2009



We actually had those ideas on mind even before watching the video and after watching it we were really inspired to move on with our project, as he had everything that we wanted to do except one thing which was to support multiple projected display, instead of working on only one display

Also another project that we felt is similar to our work especially in the part of projecting dual image,
is Microsoft Second Light, designated to launch in 2011

This is a video about it