Controlling the Kokokaka Interactive Piano

Here's some fun I had with an afternoon hack session.

The start

To start, look at this video. Play with it. It's an interactive piano. Wait for the whole video to load, and then click the piano keys with your mouse. You'll see that clicking a key moves the video's position to the spot where that note is played. In other words, you get to play the piano on the YouTube video by clicking keys.



That was fun; I wanted to automate playing the piano with a script. Could I do it? Well, yes. For the most part.

Controlling the Mouse

This turned out to be rather difficult, and in the end I could not get it quite right. But it's good enough.

Since this was a Mac workstation, I assumed it would be an easy matter to get AppleScript to do this, but I gave up after about 45 minutes of trial and error. I couldn't even find an API, let alone a reliable one. (This also included a short foray into experimenting with the Automator. Nope, it wasn't going to work.)

I finally found an answer on Mac OSX Hints with a full example of how to build a command-line binary that positions and clicks the mouse.

The secret sauce of this example was the Quartz Events method CGPostMouseEvent.

Did it work? Yes. No. Not really. Calling ./click -x x -y y certainly sent the mouse to the correct position, but it didn't seem to actually generate the correct click event. Do you know what worked instead? Calling it twice. Yep.

./click -x x -y y ; ./click -x x -y y works.

Most of the time.

I tried all sorts of things, like creating delays in between the clicks, or delays between the mouse-down and mouse-up events. No luck. Anyway, this was the simplest correct solution. That's what they call elegance, you know: the simplest, correct solution. Doesn't feel like it.

I noticed CGPostMouseEvent was deprecated and replaced with the pair of methods CGEventCreateMouseEvent and CGEventPost. I replaced the code with the newer methods, which resulted in no reasonable change to the application. But I didn't switch back.

The major problems with this solution were not only that there could be a risk of skipping, it also, the browser seemed to get overwhelmed by these fake mouse events, enough so that the video would stop playing. The mouse click events would continue to be delivered, and the video would be repositioned in time to the correct spot, but the video was paused; so no music came out. The faster the music played, the more likely the video would halt. Too bad.

A final comment about mouse control. I found that this only worked with Safari. It didn't work well (or at all) with Firefox or Chrome.

// File: 
// click.m
//
// Compile with: 
// gcc -o click click.m -framework ApplicationServices -framework Foundation
//
// Usage:
// ./click -x pixels -y pixels 
//
// additional data found at
// http://stackoverflow.com/questions/2369806/simulating-mouse-clicks-on-mac-os-x-does-not-work-for-some-applications

#import <Foundation/Foundation.h>
#import <ApplicationServices/ApplicationServices.h>

// Read more: http://www.faqs.org/faqs/unix-faq/faq/part4/section-6.html#ixzz0k416qomt

int main(int argc, char *argv[]) {
  NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
  NSUserDefaults *args = [NSUserDefaults standardUserDefaults];

  int x = [args integerForKey:@"x"];
  int y = [args integerForKey:@"y"];

  CGPoint pt;
  pt.x = x;
  pt.y = y;

  CGEventRef evt = CGEventCreateMouseEvent(
      NULL, kCGEventLeftMouseDown, pt, kCGMouseButtonLeft);
  CGEventPost(kCGHIDEventTap, evt);

  evt = CGEventCreateMouseEvent(
      NULL, kCGEventLeftMouseUp, pt, kCGMouseButtonLeft);
  CGEventPost(kCGHIDEventTap, evt);

  [pool release];
  return 0;
}

Finding positions

Now I had a relatively reliable procedure for generating mouse click events. Where should I click the mouse?

This was easy to solve. I put the browser in the upper left corner and used Grab to get a shot of the full screen. Then I loaded the image in Gimp and used the cursor position indicator. Given a full screen snapshot, the cursor position on the loaded screenshot image would correspond with a mouse position click event.

From there I could generate positions on the display that, when clicked, would generate the correct results.

my %points =  (
    toggle => [18, 580],
    c => [20, 500],
    cs => [40, 300],
    d => [80, 500],
    ds => [120, 300],
    e => [140, 500],
    f => [220, 500],
    fs => [240, 300],
    g => [280, 500],
    gs => [320,  300],
    a => [360, 500],
    as => [400,  300],
    b => [430, 500],
    C => [500, 500],
    Cs => [530, 300],
    D => [560, 500],
    Ds => [610, 300],
    E => [620, 500]
  );

Scripting the result

Coding up the controller wasn't very hard. I went through several versions, and settled on this:

#!/opt/local/bin/perl

use strict;

# system("gcc -o click click.m -framework ApplicationServices -framework Foundation");

my $tempo = 1;

sub _nap ($) {
 select undef, undef, undef, $_[0];
}

sub nap ($) {
 _nap($_[0] * $tempo);
}

sub note {
  my ($x, $y) = @_;
  system("./click -x $x -y $y");
  _nap .05;
  system("./click -x $x -y $y");
}

my %points =  (
    toggle => [18, 580],
    c => [20, 500],
    cs => [40, 300],
    d => [80, 500],
    ds => [120, 300],
    e => [140, 500],
    f => [220, 500],
    fs => [240, 300],
    g => [280, 500],
    gs => [320,  300],
    a => [360, 500],
    as => [400,  300],
    b => [430, 500],
    C => [500, 500],
    Cs => [530, 300],
    D => [560, 500],
    Ds => [610, 300],
    E => [620, 500]
  );

sub play($) {
  my @notes = split(/;/, $_[0]);
  foreach my $note (@notes) {
      my ($key, $duration) = split(",", $note);
      $duration = 1 unless defined $duration;
      print "$key-$duration- ";
      my $rpoint = $points{$key};
      print "$rpoint ";
      my @point = @$rpoint;
      print @point . $point[0] . $point[1] . "\n";
      note @point;
    nap $duration;
  }
}

sub scale {
 $tempo = .3;
 play ("E;Ds;D;Cs;C;b;as;a;gs;g;fs;f;e;ds;d;cs;c");
}

sub test() {
 $tempo = .2; 
 play ("c;d;e");
}

sub test2() {
 $tempo = .1; 
 play ("E;c;Ds;cs;D;d;Cs;ds;C;e;b;f;as;fs;a;g;gs");
}

sub twinkle {
  $tempo = 1;
  play("c;c;g;g;a;a;g,2;f;f;e;e;d;d;c,2");
}

sub fur_elise {
  $tempo = .5;
  play(
  "E;Ds;E;Ds;E;b;D;C;a,3;c;e;a;b,3;e;gs;b;C,3;e;" .
  "E;Ds;E;Ds;E;b;D;C;a,3;c;e;a;b,3;e;C;b;a,4;" .
  "b;C;D;E,3;a;f;e;d,3;g;E;D;C,3;f;D;C;b,3;e;" .
  "E;Ds;E;Ds;E;b;D;C;a,3;c;e;a;b,3;e;C;b;a,4;");
}

my %songs = (
  scale => \&scale,
  twinkle => \&twinkle,
  test => \&test,
  test2 => \&test2,
  fur_elise => \&fur_elise,
);

my $song = $ARGV[0];
print $song;
my $rsong = $songs{$song};
&$rsong();

Notice that _nap is actually a call to the Perl builtin method 'select'? It's a bit of a hack I read about in a couple of places; this is one.

The final result

Did it work? You tell me!



Success!

Parting thoughts

It's too bad there was no easy way to get a reliable mouse click on the Safari browser. That would make such a difference.

Collecting this data from a script feels a little cheap. I'd much rather have a keyboard send events that control the video. Anyone who has a spare electronic keyboard and a couple of hours of spare time, let me know.