5th OpenCV PSMove Example (Multi-Controller Tracking, World Distance Estimation, Build-Toolchain)

This is the fifth blogpost about the implementation of a tracker for the colored sphere of the PSMove controller. This time it is less about the multiple controller tracking, real world distance estimation, performance issues, and the build-toolchain.

1. Multi Controller Tracking

A lot has happened in the meantime. Most exciting regarding the tracker are its ability of multi controller tracking, increased robustness and the calulation for the distance to the camera.

Have a look at a short video featuring tracking of two controllers.

There are some things that are noticeable in that example. First of all and of course the obvious ... two controllers are tracked at the same time at reasonable speed of 350-900 FPS depending of its distance to the camera. The farer away the controller is from the camera the higher the FPS and vice versa. This is caused by the different levels of regions of interest i use to search for the colored blob in the image. But i already told you that in one of the previous posts [3rd OpenCV Example].

2. Real world distance estimation

The Example now displays also a estimate of the distance between the controller and the camera in [mm]. This might get very handy vor applications that like to make use of the 3D position of the controller. You might ask why to use the distance in [mm] rather than the spheres radius. Using the realworld unit has two advantages. One is, that [mm] is easier to understand as it relates directly to the real world in which the interaction takes place. And the second ist, that the distance in [mm] has 1:1 relation with the user on the Z-Axis, while the readius has a logarithmic relation. To put it simply ... it makes app development easier :P.

3. Increased worst case performance 

One big problem of the old examples was, that if the controller was not visible in the camera image at all, the framerate dropped drastically (around 60FPS on a 2.5Ghz dual core) which was a no go on slower systems e.g. laptops that are in power-saving mode. To increase the FPS, if the controller is not tracked, the tracker simply only scans a quater of the image, and if it does not find the controller, it immediately returns NOT_FOUND. In the next iteration a different quater of the image is evaluated an so on. This is best recogizeable in the video at second 00:17 when i hide the controllers behind my back. 

4. Increased robustness to occlusions

 Another nice feature, is its increased robustness to occlusions. Have a look of the screenshots taken from the video above.

In all of the screenshots you can see, that the circle of the controller is estimated very good, although it is partly occluded. The estimation for the center of the sphere is quite simple and can therefore only correct occlusions smaller than one half of the spheres size. The estimation utilizes the idea, that the two most distant points of a detected contour are equal to the diameter of the sphere and the center of these two points is the the center of the very same. This assumption is true, if the occlusion occurs from one side or from two conterparting sides. For others situations it may fail, but the approach tourned out to be convenient and fast.

5. The new buildprocess and PSMoveAPI integration

Thanks to Thomas Perl the optical tracking of the controller has now been integrated into the branch "tracker" of the official psmoveapi repository hosted on github. It'll soon be moved the master branch but for now just stick with the "tracker" branch. It also contains a detailed description how to build the whole system and the different examples (also the one from the video) that are part of the psmoveapi. As the build-instructions for windows are quite long, they are included at the end of this post :).

6. Remaining problems of the optical tracking

Unfortunately the optical tracking has still some problems.

1. Only the Magenta, Cyan and Blue can be tracked robustly. Others colors suffer a lot of motion blur which resuls in a low tracking performance. Especially regarding the z-position of the controller.
2. Strong daylight coming trough windows might be big problem for tracking performance, too. Avoid direct sunlight  by closing curtains if the controllers cannot be tracked.
3. There are still too much false detections (sphere estimated to small / wrong position) when the controller is partly occluded. 
4. Artifical light, especial the one emitted of fluorescent light, causes unwanted jitter in the estimation of the 3D position.

enjoy!
   cherio benjamin


System requirements:

• psmove-api + tracker [branch tracker] -> https://github.com/thp/psmoveapi.git
• CL-Driver for the PS-Eye
• CL-SDK for the PS-Eye
• Windows operating system
• Linux operating system (tested on Ubuntu 12.04)
• OpenCV 2.4 headers  --> http://opencv.willowgarage.com/wiki/
• OpenCV 2.4 library build --> OpenCV 2.4 win binary 

---- Build Instructions For Windows -----------

Get and install
- MinGW       : http://sourceforge.net/projects/mingw/files/latest/download?source=files
- CMake       : http://www.cmake.org/cmake/resources/software.html
- OpenCV      : http://sourceforge.net/projects/opencvlibrary/files/opencv-win/
- GIT         : e.g. http://code.google.com/p/msysgit/
- PSEyeDriver : http://codelaboratories.com/get/cl-eye-driver/
[optional]
- CLEyeSDK    : http://codelaboratories.com/get/cl-eye-sdk/

1. build and configure OpenCV with cmake
    :: you may skip to build OpenCV on your own, however i had no luck
    :: the binary distribution did not work on my system
    cd <where you extracted opencv>
    mkdir build
    cd build
    cmake .. -G "MinGW Makefiles"
    mingw32-make
    :: now go for a coffe-break

2. Get you clone of the psmoveapi
    git clone https://github.com/thp/psmoveapi.git
   
3. Check out the "tracker" branch
    cd psmoveapi
    git checkout tracker

4. Init and update the submodules
    git submodule init
    git submodule update
   
4. Copy blue-tooth headers and library to your MinGW installation
    :: e.g. MinGW installed at C:\MinGW\
    :: e.g. your cloned repository is at D:\dev\psmoveapi
   
    copy D:\dev\psmoveapi\external\mingw-w64-headers\*.h  C:\MinGW\include
    copy D:\dev\psmoveapi\external\mingw-w64-headers\*.a  C:\MinGW\lib

5.  make OpenCV known to your system and the cmake toolchain
    set OpenCV_DIR=<the path where you extracted opencv>
    set PATH=%PATH%;%OpenCV_DIR%\build\bin

7. prepare a new build with cmake for the psmoveapi
    ::
    mkdir build
    cd build
    :: only with OpenCV Camera access
    cmake .. -G "MinGW Makefiles"
    :: additionally with Code Laboratories PS Eye SDK
    cmake .. -G "MinGW Makefiles" -DPSMOVE_USE_CL_EYE_SDK=ON
   
8. finally build
    mingw32-make
   
9. start one of the desired test applications

------------------------------------------------