Phase 2: Tracking Loss

Why we needed it?

When the IR LEDS are tracked, some reasons might cause its loss, like:

1- Getting out of the projection area.

2- Some LEDs turn off.

3- Barriers get into the way of tracking the sensors.

And so the projected image will get distorted, so we needed to estimate the coordinates of the lost IR LEDS.

Actually after implementing this part in our project, the tracking was improved a lot :D.

Scenarios:

  We've handled the first 3 scenarios, and ignored when only 1 sensor is available as it's uncommon in most cases.

Algorithm:

  

 We mainly Store the last successful tracking for the 4 LEDs in order to be able to get the coordinates of the lost LEDs.

Of course after reading the algorithm details, you'll find a really important problem that appears "Point Identification" (the new tracked sensors map to which old sensors). We'll discuss this problem and the suggested solution to it in a later article.

1- 1 LED lost,  3 Tracked   (Affine)

1- We simply calculate the affine matrix (2*3) that maps "3 Tracked sensors" to "their corresponding old positions".

RotationMatrix2D map = CameraCalibration.GetAffineTransform(src, dst);

src: their corresponding old positions.

dst:   3 Tracked sensors

2- Then take this matrix and multiply it with the corresponding old position of the lost sensor. Which will output the correct estimated position of the sensor.

2- 2 LEDs lost, 2 Tracked  (Simplified Affine)

Since there's no ready made fn that makes the simplified affine, so we implemented it ourselves.

We need to calculate 2 dimensions of translation, one degree of rotation, & one degree of scaling.

1- We simply construct 2 vectors, Vector1 (new tracked 2 points), Vector2  (their corresponding old points).

2- Translate scene origin, Translate scene (-x0,-y0) 

hint: (x0,y0) id the start point of Vector1

3- Calculate scaling factor, S= |vector2| / |vector1|

hint: |vector2| is its magnitude

4- Calculate rotation angle, Theta= Theta2- Theta1

hint: Theta2= Vector1.y / Vector.x

5- Calculate the matrix that encapsulates the rotation & scaling, and multiply it with the scene

CvInvoke.cv2DRotationMatrix(center, rotation_angle, scale_factor, map.Ptr);

hint: center is the origin of course

6- Translate the scene to (x1,y1)

hint: (x1,y1) is the start point of Vector2

Phase 2: Calibration

This part of the project really confused us for a long time, we were so convinced that we won't need manual calibration and we spent the whole time trying to implement automatic calibration. But, then we faced some problems that I'll discuss in details that lead to the necessity of using manual calibration.

First, What's Calibration?
It is a preprocessing stage before warping, in which we align the coordinates of the display to the wii-mote coordinates.

Manual Calibration : suggests that the user at the beginning of the application, marks the corners of the projector, and save the corresponding wii-mote coordinates. Then maps the corners of the projector to the corners of the wii.

Automatic Calibration: doesn't need any user interference to map the display to wii-mote coordinates.


Second, Why use calibration?

Well, if we took the coordinates directly from the wii-mote without any preprocessing and projected on those coordinates, we'll find that the image is not positioned directly on the display, since the wii-mote coordinates system [1024*768] is different than the projector/ display coordinate system [Screen resolution]. And if we positioned the wii-mote away from the projector, that might lead to distorted image. So we do this process to calculate origin (0,0) of the screen maps to what coordinate in the wii-mote.


Third, we're going to discuss in details each approach.

Manual Calibration in details,

How,
1- At the beginning of the application, a white screen is projected and small crosshairs at the corners of the screen are drawn.
2- The user user selects those crosshairs using the IR pen.
3- Then we save the corresponding wiimote coordinates.
4- After that we compute a homography matrix, where the source points are the screen corners, & the destination points are are the saved wiimote coordinates.

5- After that whenever we save coordinates of the IR LEDs, we multiply it by the above homography matrix.

Advantages,

1- A very big advantage is that it doesn't place any constraints on the place of the wii-mote relative to the projector.
2- It's accurate, the position of the image is placed on the display without any errors.

Disadvantages,

1- A major disadvantage is that it needs user interference to calibrate.

Automatic Calibration in details,

How,
1- without any user interference, we take the coordinates of the IR LEDs and do some calculations on it to output the mapped display coordinates.
2- Those calculations are illustrated below: (similar to normalization equations)

pt.x = (int)((newpt.x / 1023) * this.Width);

pt.y = this.Height - (int)((newpt.y / 767) * this.Height – 225);

Hint:
1- in the Y equation, we subtract [Height – newpt], because the origin of the screen is top left while the origin of the wii-mote is bottom left.

2- in the Y equation also, we subtract with a constant value 225. It is due to the difference between the projector lense and the wii-mote just placed above. You won't believe how we got this number, but we actually took a ruler measured the distance between them it was about 2 cm, then took this ruler and measured the screen. With some cross multiplication :D, we calculated (2 cm) corresponds to how many pixels (225).

Advantages,

1- It's obvious that its main advantage, is that it won't need user interaction.

Disadvantages,

1- It places a constraint that the wii-mote must be placed above the projector.
2- The big problem appeared, when we found that the size of the image wasn't perfectly matching to the size of the display.


Last,  We decided to go for manual calibration as it's more accurate.

Second Phase

During the second phase we mainly focused on improving the display and solving the problems that popped out during phase 1.

what We mainly did in second phase:
1- Calibration
2- Tracking Loss
3- Point Identification
4- Constructing Frame's HW

We'll discuss each one of those modules in other articles.

After we finished this phase the building block of the wii display was finished,
we had a movable projected display that now can be used in many applications.

it was only left to implement some of the applications, and add extra features to the display, which we'll see in the upcoming phases.

Phase 1: Image Warping

1- Background Information: 

-Why we need Image Warping?

To project the image properly on the display while moving, we need to calculate the transformation matrix that's being applied to the image before projecting it in order to fit the movement of the display.

- How image warping is implemented?

As the display moves new coordinates are given to the IRLEDS , so using these new coordinates (destination) and the corners of the image (source), we can estimate the homography matrix.

The homography matrix (transformation) is applied to the image and we get the warped image to be projected. This is all done by openCV functions :
-“cvFindHomography”: which computes the homography matrix using source & destination points (IRLEDS coordinates).
-“cvWarpPerspective”: This applies the transformation calculated above to the image to be projected.

 

2- Into the details:


- Transformations Taxonomy:
There are  4 types of transformations:

 the following picture shows the effect of applying different transformation types on an image

and since that our display moves in 8 degrees of  freedom, so we used the Homography matrix.

dof: (forward/backward, up/down, left/right (translation) combined with rotation about three perpendicular axes (pitch, yaw, roll))

for more details abt the projective transformation see this ppt.


- Forward Warping VS Inverse Warping
This ppt explains the concept of forward & inverse warping,
remeber if u're using the inverse warping you should use the inverse of the homography matrix not the homography matrix itself 
 

3- C# Code: 

This code uses openCV 1.1,  this book explains the openCv fns used below

also we use Emgu .Net Wrapper and this is its Wiki for documentation abt it

This's just a simple test code, to warp an image from srcp to dstp

private void warping()
        {
           
            imageBox2.Image=src;

            float[,] srcp = { { 0, 0 }, { src.Width-1, 0 }, { 0, src.Height-1 }, { src.Width-1, src.Height-1} };
            float[,] dstp = { { src.Width * 0.05f, src.Height * 0.33f }, { src.Width * 0.9f, src.Height * 0.25f }, { src.Width * 0.2f, src.Height * 0.7f }, { src.Width * 0.8f, src.Height * 0.9f } };
            float[,] homog = new float[3, 3];

            Matrix srcpm = new Matrix(srcp);
            Matrix dstpm = new Matrix(dstp);
            Matrix homogm = new Matrix(homog);
            IntPtr n = new IntPtr();

            CvInvoke.cvFindHomography(srcpm.Ptr, dstpm.Ptr, homogm.Ptr, Emgu.CV.CvEnum.HOMOGRAPHY_METHOD.DEFAULT, 0, n);
            src= src.WarpPerspective(homogm, Emgu.CV.CvEnum.INTER.CV_INTER_CUBIC, Emgu.CV.CvEnum.WARP.CV_WARP_INVERSE_MAP, new Bgr(Color.White));
            imageBox1.Image = src;
        }

Phase I:: Basic IR Tracking

At first we had to collect all the information about the wii-mote device and its capabilities, before implementing the IR Tracking module. After that we began to search for ready made libraries that deals with the wii-mote device.

1- Background Information:

Wiibrew is wonderful source of information on this device, to know its details.






But for this project we only needed certain information which are,


- Connecting to wii:
In order to interface with the wiimote, it must first be placed in discoverable mode by either pressing the 1 and 2 buttons at the same time, or by pressing the red sync button under the battery cover. Once in this mode, the Wii Remote can be queried by the Bluetooth HID driver on the host.

- Wii Reports:
This block includes an enumeration of reports that the device understands. A report can be thought of similar to a network port assigned to a particular service. An "Input" report is sent by the Wii Remote to the host. An "Output" report is sent by the host to the Wii Remote. The Wii Remote has a number of different data reporting modes. Each of these modes combines certain Core data features with data from external peripherals, and sends it to the host through one of the report IDs.

- Wii IR Camera:
We use the IR Camera inside, which has an object tracking processor (IR Blob Tracking).
It's capable of tracking up to 4 moving objects, and these data are the only data available to the host. The IR camera has an effective field of view is about 33 degrees horizontally and 23 degrees vertically (as measured on one unit).

2- Implementation Details:

We used Brian Peek's WiimoteLib ,
and its wrapper  BobCat Library

and inside the Bobcat wrapper we used the first layer ( Code Layers )
because it had 2 main features that we needed
a) it performs smoothing
b) it keeps track of points to keep point indexes consistent as they move.

First Phase

as we started to work, we made a plan to which we shall stick for the the rest of our work...
we divided our project into modules which is much easier for implementation and maintenance.
And we focused in first phase on:
1- Basic IR Tracking
2- Image Warping
3- Automatic Calibration

We'll discuss each one of those modules in other articles, 

but for now we'll discuss some of the problems that we faced during implementation

 
Problems faced:

The projected image wasn't correctly aligned with size of the of the display ...so figured that we should manually calibrate the wii to projector before projecting.

another problem is that sometimes the image freezes when moving it , this's because wii loses track of some of the Ir Leds ....so we need to track the lost Leds.

Also the image sometimes turned upside down this was because of point identification, which means identifying the LED as the top left, top right, ...
and the solution was either use motion modeling ,data grid, or optical flow...
we thought we shall start with the data grid solution and see the results.

Related Work

We were mainly inspired by Johnny Lee's Thesis 2008, projector based Tracking
http://johnnylee.net/academic/p104-lee.pdf

Those are some videos for his work





And this is Johnny Lee's Blog, for more information about him
http://procrastineering.blogspot.com/2008_06_01_archive.html


And of course we were really inspired by Sixth Sense next Gen by Pranav Mistry that was posted on Nov 2009



We actually had those ideas on mind even before watching the video and after watching it we were really inspired to move on with our project, as he had everything that we wanted to do except one thing which was to support multiple projected display, instead of working on only one display

Also another project that we felt is similar to our work especially in the part of projecting dual image,
is Microsoft Second Light, designated to launch in 2011

This is a video about it

Wii-Display, Idea "Technical Point of View"

Tools:

1- Projector of any kind




2- Wii-mote (mounted over projector)

3- IR LEDs (they will be mounted on the frame)

 







4- IR pen

Software: 

- Our System Architecture is divided to 4 main modules: 

1- IR Tracking: (Based on WiimoteLib)

Operation:

- Calibration (Automatic/ Manual)

- Tracking Loss Strategies

- Point Identification

Output: 2D co-ordinates of 4 IR LEDs

2- Image Warping: (Based on OpenCV, Emgu wrapper)

Input: Co-ordinates of IR LEDs

Operation:

- Computing Homography Matrix

- Dual Image projection

Output: Warped image to be projected

3- Orientation Sensitivity:

Input: Co-ordinates of 4 IR LEDs

Operation:

- Computing Orientation direction (angle)

- Handling Orientation Change event

Output: Desired action according to designed purpose.

4- Controlling mouse: (Based on Win API functions)

Input: Co-ordinate of IR pen.

Output: mouse actions

Wii - Display "the big idea"

as we were searching for an idea for our GP , we wanted to do something new and useful .
when we (actually "menna siam" did) hit johnny lee's idea "projector based location discovery and tracking " we were amazed by it, so we decided to go for it.

The main idea of the wii - Display:
there are some obstacles concerning the usual display:
1- The display on a typical device whether a mobile or a monitor remains fixed in size and rigid thus forcing constraints on the interaction with it.
2- those devices are composed of highly sophisticated hardware.

so, the solution is by combining Motion Tracking  +  Image Projection,

                                   










We'll construct a simple frame with IR LEDs attached to its corners.
This frame will act as a movable projected display.

Our Project aims to Creating a flexible display using any kind of surfaces,Creating a low cost display with the simplest hardware.

as we were thinking about applications for this project , we thought that it can be used as :
1- Wii-Book: where it can be used inside classrooms as an interactive book
2- Map Viewer: Use the display to project different information about the map behind the display, like projecting the zooming of the part of the map behind the display.

3- eBook reader : inside libraries.
4- Tablet PC : The display can be used as Low cost tablet PC by using interactivity feature.
and so many others....
so , it's the time for work and trials.

Our GP Journey

Our journey to this GP was really weird. At first, people were saying this is a very difficult project you'll never be able to do this stuff. Then, ironically we found it's not impossible at all, to the extent that some people said that it's so easy. Actually there were others who told us that it's not useful.

But, then we decided we're all gonna pray este5ara and see what happens. We entered to Dr Haytham and told him about our project, and he was so enthusiastic and he liked it and told us to go on with it. And yeah that was a relief, but yet we had another problem which application to use for the project.

Which application to use that would really make the project worth the effort. We've searched & thought a lot. “Application”, we all really felt that this word was haunting us. Whenever we go people say wow what a great project, but what's its application anyway? Yeah those words really irritated us, but they were right. And for some time we felt skeptic about the project whether to continue with it or change it totally. But, we didn't because it was the one idea that we all loved, and we all wanted to participate in it for different reasons. And we took the decision to continue with it and squeeze our minds for any application. What we've settled upon at last, was 2 applications actually, one was the GIS viewer and the other was the wii-Book, they'll be explained later in another post.

At the end, we selected the project each with her own reasons. Mohga loved the idea of learning new stuff. Menna Hisham was fascinated with the wii-mote device and she liked that the idea had some hardware that we had to built with our own hands. As for Menna siam really wanted to do sth that's new and useful especially for the teaching field, she always had that dream of changing the teaching system to the better cause it's the one thing that can change the whole nation

Our Team

We're 3 senior students at the faculty of computer and information sciences Ainshams university, computer science department. The 3 of us share the passion & love of programming and learning new stuff.

Who we are:

Menna Hisham:is the top speed coder we've ever seen msa.

Very very committed to her work and very punctual if you run late on her you're dead meat for sure :D.

Also she's a tech maniac she loves any hardware device that's related even remotely to technology, of course she has the number one device that all people should get an ipod touch.

Mohga Nabil: is a real pro, we've always pictured her as CEO of Microsoft Redmond:D. she Loves to learn new stuff each day, the most hardworking girl we've seen. And she's an artist, she doesn't just write some chunk of code and that's it, she always put a solid design before coding, honestly when we read her code we feel that it's a master piece :D msa.

Menna Siam:is our GEEk :D. all our friends and her family know that about her.

she first fell in love with coding when she was at school, and her love for it grows year by year. she wants to be a computer scientist not just coder. she loves to feel that she's a mathematician, understanding all this hard math & probability stuff and actually using them in programming just fascinates her.

Our Technical Skills:

Menna Hisham: she's a very talented .Net developer. And She's skilled in searching about any kind of topic and has a relatively good knowledge about wii-mote which will help us in the project.

Mohga Nabil: She is maniac to algorithms and coding. has good knowledge about web application development, especially using PHP and MySql. She had an internship program at Specialized Solutions (2009).

Menna siam : She is a very talented reseracher/scientist . Has a good knowledge about computer graphics .She had an internship program at Nile University as a research assistant (2009).

Common Programming Languages: C++, C#, Prolog, Assembly

Common Courses: Algorithms, Data Structures, Object-Oriented Programming, Design Patterns, Computer Graphics, Computer Architecture, System Analysis and Design, Windows Programming, Artificial Intelligence, and Simulation and Modeling.