Multi-threaded LED writing

Today I've experimented a bit with using multi-threading in Linux to write LED status updates. On Linux, setting LEDs actually blocks the process - depending on the controller - for 30 ms up to 500 ms in some cases. I'm not yet sure if there is a way to make this faster (in Mac OS X on the same hardware, updating LEDs is no problem, but maybe OS X does not wait for an acknowledgement when sending out the set LEDs packet).

You can find the multi-threading branch in the multithreaded branch on Github. Now that we have the nice test_read_performance tool, we can compare the results with the previous results from the OS X installation (same hardware):

 -- PS Move API Sensor Reading Performance Test --

Testing STATIC READ performance (non-changing LED setting)
1000 reads in 13043 ms = 76.669478 reads/sec (144x seq jump = 14.40 %)
1000 reads in 12563 ms = 79.598822 reads/sec (100x seq jump = 10.00 %)
1000 reads in 12451 ms = 80.314834 reads/sec (88x seq jump = 8.80 %)
=====
Mean over 3 rounds: 78.861043 reads/sec

Testing SMART READ performance (rate-limited LED setting)
1000 reads in 20073 ms = 49.818164 reads/sec (177x seq jump = 17.70 %)
1000 reads in 20113 ms = 49.719087 reads/sec (196x seq jump = 19.60 %)
1000 reads in 19858 ms = 50.357539 reads/sec (190x seq jump = 19.00 %)
=====
Mean over 3 rounds: 49.964930 reads/sec

Testing BAD READ performance (continous LED setting)
1000 reads in 26187 ms = 38.186887 reads/sec (310x seq jump = 31.00 %)
1000 reads in 25985 ms = 38.483741 reads/sec (306x seq jump = 30.60 %)
1000 reads in 26784 ms = 37.335723 reads/sec (343x seq jump = 34.30 %)
=====
Mean over 3 rounds: 38.002116 reads/sec

Testing RAW READ performance (no LED setting)
1000 reads in 12612 ms = 79.289565 reads/sec (101x seq jump = 10.10 %)
1000 reads in 12639 ms = 79.120184 reads/sec (109x seq jump = 10.90 %)
1000 reads in 12576 ms = 79.516539 reads/sec (107x seq jump = 10.70 %)
=====
Mean over 3 rounds: 79.308762 reads/sec
Still not as good as on OS X, but a good starting point. Without multi-threading on Linux, the results are much worse when the LEDs are updated often:

 -- PS Move API Sensor Reading Performance Test --

Testing STATIC READ performance (non-changing LED setting)
1000 reads in 12890 ms = 77.579519 reads/sec (125x seq jump = 12.50 %)
1000 reads in 12660 ms = 78.988942 reads/sec (104x seq jump = 10.40 %)
1000 reads in 12650 ms = 79.051383 reads/sec (108x seq jump = 10.80 %)
=====
Mean over 3 rounds: 78.539948 reads/sec

Testing SMART READ performance (rate-limited LED setting)
1000 reads in 14596 ms = 68.511921 reads/sec (161x seq jump = 16.10 %)
1000 reads in 14315 ms = 69.856794 reads/sec (143x seq jump = 14.30 %)
1000 reads in 14224 ms = 70.303712 reads/sec (139x seq jump = 13.90 %)
=====
Mean over 3 rounds: 69.557475 reads/sec

Testing BAD READ performance (continous LED setting)
1000 reads in 41132 ms = 24.311971 reads/sec (69x seq jump = 6.90 %)
1000 reads in 41014 ms = 24.381918 reads/sec (70x seq jump = 7.00 %)
1000 reads in 41153 ms = 24.299565 reads/sec (76x seq jump = 7.60 %)
=====
Mean over 3 rounds: 24.331151 reads/sec

Testing RAW READ performance (no LED setting)
1000 reads in 12683 ms = 78.845699 reads/sec (114x seq jump = 11.40 %)
1000 reads in 12723 ms = 78.597815 reads/sec (114x seq jump = 11.40 %)
1000 reads in 12640 ms = 79.113924 reads/sec (107x seq jump = 10.70 %)
=====
Mean over 3 rounds: 78.852478 reads/sec

Interestingly, the multi-threaded variant is worse in the SMART READ test (the rate-limited LED update variant). Not sure why this is the case - maybe it's some threading overhead. For some of my older PS Move controllers that I have here, it's even worse - seems like some controllers are faster to respond than others. I wonder if there's a difference in the firmware or if it's different hardware (the faster controllers is the one that I bought more recently).

Profiling sensor reading performance

Right now, the PS Move API is single-threaded. This makes debugging easier, and simplifies the code and usage of it. What this also means is that setting the LED colors will slow down sensor reading (not taking into account any Bluetooth communication slowdown that might happen even in multi-threaded scenarios).

In order to get high-quality sensor readings, the read rate should be as high as possible. On my MacBook Pro in OS X 10.7.4, I currently get an average of 83.49 sensor readings per second if I don't send any LED updates. If I continuously send LED updates (one update after every sensor reading), the reading rate drops to 51.02 sensor readings per second (thats ~ 61% of the best performance).

In order to improve the situation while still being able to set the LEDs (after all, we need the LEDs on for tracking the controller position with the camera) there are two options: First, if the color is static (i.e. it does not change or does not change often), we only have to send an update every 5 seconds (the LEDs keep glowing in the set color for 5 seconds after an update). And even if we want to modify the color during tracking, we can be smart about the updates and ignore some requests if the update rate is too high.

I have implemented both variants in the latest PS Move API code (you can find it in the Git repository) now, and also wrote a small test application (which you can also find in the Git repository as "test_read_performance" when you build the source) to compare the different methods. Here are the results (Git revision 64a3214 on a 2.53 GHz Intel Core 2 Duo, static read = 4000ms between updates, smart read = 120ms between updates):


 -- PS Move API Sensor Reading Performance Test --


Testing STATIC READ performance (non-changing LED setting)
1000 reads in 12373 ms = 80.821143 reads/sec (56x seq jump = 5.60 %)
1000 reads in 11954 ms = 83.654007 reads/sec (29x seq jump = 2.90 %)
1000 reads in 12090 ms = 82.712986 reads/sec (36x seq jump = 3.60 %)
=====
Mean over 3 rounds: 82.396047 reads/sec


Testing SMART READ performance (rate-limited LED setting)
1000 reads in 16386 ms = 61.027707 reads/sec (283x seq jump = 28.30 %)
1000 reads in 16673 ms = 59.977209 reads/sec (304x seq jump = 30.40 %)
1000 reads in 16736 ms = 59.751434 reads/sec (306x seq jump = 30.60 %)
=====
Mean over 3 rounds: 60.252116 reads/sec


Testing BAD READ performance (continous LED setting)
1000 reads in 19593 ms = 51.038636 reads/sec (479x seq jump = 47.90 %)
1000 reads in 19732 ms = 50.679100 reads/sec (486x seq jump = 48.60 %)
1000 reads in 19471 ms = 51.358430 reads/sec (469x seq jump = 46.90 %)
=====
Mean over 3 rounds: 51.025391 reads/sec


Testing RAW READ performance (no LED setting)
1000 reads in 12105 ms = 82.610492 reads/sec (40x seq jump = 4.00 %)
1000 reads in 11987 ms = 83.423709 reads/sec (31x seq jump = 3.10 %)
1000 reads in 11843 ms = 84.438065 reads/sec (21x seq jump = 2.10 %)
=====
Mean over 3 rounds: 83.490753 reads/sec

The ideal situation is RAW READ (not updating the LEDs at all). STATIC READ is when we set the LEDs to a fixed color and only send an update every 4 seconds - the performance impact is not really noticeable in practice. When we try to update the LED colors all the time, but have the new rate-limiting feature enabled (SMART READ), performance drops to 60.25 reads per second (that's 72% of the best performance), but we get an LED update every 120 ms (that's the current threshold for rate limiting and might change in the future). And as discussed above, if we don't do any rate-limiting (BAD READ), the performance is even worse.

The "seq jump" counts are the non-continuous readings of the sequence number that the controller sends with every report. A "seq jump" means that we missed one (or more) sensor readings from the controller. As the sequence number is something internal to the controller, I'm not sure if we can get any better than the 2-4% "seq jumps" in the RAW READ case, but it's good to know that we can nearly read as fast as the controller itself is able to process the data internally.

In practice, even if we do update the LEDs while tracking, we might not update the color every 120 ms and therefore the practical read performance should be between 60.25 and 83.49 reads per second.