Dygraphs Bugs and the Start of Automated Javascript Testing

Short version:

Someone contributed wrote a bunch of code to Dygraphs. It broke an existing behavior, but also inadvertently (or advertently)   fixed another one that was broken before his patch was submitted. The new broken behavior was more of a problem than the old broken behavior, so Nealie provided a fix which restored the old broken behavior. And then I went ahead and did a bunch of tests. And as always, patches are welcome!

Long version:

Nealie submitted a rather comprehensive patch to Dygraphs that changed the way zooming events behaved. Here's the use case that he inadvertently broke:

• Open http://dygraphs.com/tests/zoom.html
• Use the mouse to zoom in on a graph along both the X and Y axes.
• Perform an unzoom by clicking the “Unzoom” button. This effectively calls g.updateOptions({dateRange: null, valueWindow: null})
• Expected behavior: both the original x and y ranges are restored.
• Actual behavior: Only the x-axis range is restored.

Now, here’s the use case that was broken before his patch, and subsequently fixed:

• Open http://dygraphs.com/tests/zoom.html
• Use the mouse to zoom in on a graph along both the X and Y axes.
• Make the following call from a javascript console: g.updateOptions({})
• Expected behavior: this should effectively be a no-op.
• Actual behavior: The y-axis restores to the original range.

In other words, Nealie’s patch made g.updateOptions({}) work just fine, while breaking g.updateOptions({dateRange: null, valueWindow: null});

I figured this out by basically looking at a series of commits and checkout then out at different times along the main code path. You can see my work here. If you read that document, you’ll notice that when performing the same zoom operations via g.updateOptions({…}), neither bug appears. It’s only when a zoom is performed via mouse operations.

Much of this could have been avoided if we had automated tests. Fortunately, I’m working on that, and in fact an initial attempt at them is now in a pull request. It uses the js-test-driver framework. If we had these automated tests in place beforehand, then it might be safe to say that neither bug would have made it in to the main branch.

For instance, here’s a simple test that verifies that verifies calls to updateOptions should do what I expect:

RangeTestCase.prototype.testRangeSetOperations = function() {
  var graph = document.getElementById("graph");
  var g = new Dygraph(graph, ZERO_TO_FIFTY, { });
  assertEquals([10, 20], g.xAxisRange());
  assertEquals([0, 55], g.yAxisRange(0));

  g.updateOptions({ dateWindow : [ 12, 18 ] });
  assertEquals([12, 18], g.xAxisRange());
  assertEquals([0, 55], g.yAxisRange(0));

  g.updateOptions({ valueRange : [ 10, 40 ] });
  assertEquals([12, 18], g.xAxisRange());
  assertEquals([10, 40], g.yAxisRange(0));

  g.updateOptions({  });
  assertEquals([12, 18], g.xAxisRange());
  assertEquals([10, 40], g.yAxisRange(0));

  g.updateOptions({ dateWindow : null, valueRange : null });
  assertEquals([10, 20], g.xAxisRange());
  assertEquals([0, 55], g.yAxisRange(0));
};

To be fair, that test doesn’t really test the use cases above, where the mouse operations are the cause of broken behavior. I haven’t written those tests yet, but here’s one I did write, which verifies the behavior of a mouse double-click:

RangeTestCase.prototype.testDoubleClick = function() {
  var graph = document.getElementById("graph");
  var g = new Dygraph(graph, ZERO_TO_FIFTY, { });

  assertEquals([10, 20], g.xAxisRange());
  assertEquals([0, 55], g.yAxisRange(0));

  g.updateOptions({ dateWindow : [ 12, 18 ] });
  g.updateOptions({ valueRange : [ 10, 40 ] });
  assertEquals([12, 18], g.xAxisRange());
  assertEquals([10, 40], g.yAxisRange(0));

  var evt = document.createEvent('MouseEvents');
  evt.initMouseEvent(
     'dblclick', true, true, document.defaultView,
     2, 0, 0, 0, 0
     0, false, false, false, false, 0, null);
  g.canvas_.dispatchEvent(evt);
  assertEquals([10, 20], g.xAxisRange());
  assertEquals([0, 55], g.yAxisRange(0));
}


You get the idea.

If you read those tests carefully, you would probably have four criticisms:

1. Accessing to the private attribute canvas_ should be discouraged.
2. All those magic numbers used for initializing the event.
3. Having to actually dispatch the event via the DOM.
4. Questioning browser compatibility - what browsers would that work on?

Here are my answers:

1. Yes, you’re right. That should be fixed.
2. Yes, you’re right. That should read like DygraphTestUtils.dispatchDoubleClick(g);
3. You’re right about that one also. Right now the API doesn’t quite make that possible, but it could be worked on if necessary. (Psst … patches welcome!)
4. Yeah, I have no idea if that would work on anything other than Chrome. One thing at a time.

Finally, just what will we do about this behavior? Here's what I think:

First, I think we should accept Nealie's bug fix. Second, figure out why and how the both cases can be fixed. Third, get some comprehensive automated tests in place, make the tests easy to write and the framework easy to run. Fourth, and probably most important, change the submission guidelines to require a) automated tests to be run as part of the patch submission process and b) require new tests to be part of patches of a certain level of complexity.

Thoughts about Proxying the HTML5 canvas

This post is a blackboard for my thoughts around how to write tests that verify what's drawn on an HTML5 canvas. I'm still relatively new to Javascript, DOM and Canvas, so if there's an established solution to mocking canvas operations, let me know. My research hasn't yielded anything.

As part of my recent involvement with Dygraphs, I've become interested in automated Javascript testing. I'm currently writing some very basic automated sanity tests for Dygraphs using jstestdriver, and those tests should be committed to the Dygraphs repository fairly soon.

Many of the tests I want to write have to do with the graph's internal state, and not its visual aspects, but at some point I will need to test what's drawn. Because Dygraphs has an HTML5 canvas, I have limited options:

1. Visual inspection. To some degree this is what we already do. Dygraphs has over 80 pages that represent visual tests. Most tests require a quick glance, but some require playing with values on the page to see how the rendering changes. (This last problem is something I've contributed to.) And finally we just can't look at every single test with every release. Besides, we're looking to create automated tests, so let's move on to the other choices.
2. Perform pixel-perfect tests against golden images. This works well when setting up tests, but the lack of a consistent canvas implementation across all browsers means that I should expect to have to create a golden image per test per browser version. This type of test can also be a problem when making small changes (such as minor tweaks in color and so on) require generating new golden images across the board, which comes with its own problems.
3. Perform fuzzy-match tests against golden images. This addresses many issues, but I still need to generate golden images. Not ideal.
4. Pixel count. This isn't my idea; someone else suggested it. This could theoretically work, but it seems complicated. "Assert that the canvas has 50 blue pixels" reads less like an effect and more like a side-effect.
5. Implement a fake canvas of some kind. This is the type of solution I want to implement. Replace the canvas with some object so I can verify that a line is drawn from a to b by writing something like assertSegment(point(0, 0), point(50, 50));.

What I really want is a proxy that not only receives calls from a client and logs them for post-operation validation, but also actually draws on a real canvas. That way in the event of a regression I can view the backing canvas, which will serve as a clue to what changed. It also helps me understand just what kind of test is being written in the first place.

Today I set about to prove out the idea of a canvas proxy, and the truth is, it's just not going to happen the way I want. Here are the two showstopping problems I see so far:

1. You can't fake properties. (You really can't do that with Java and EasyMock either, but in Java you don't often see statements like canvas.style.top = top;.
2. The proxy has to be a DOM element. Otherwise calls like document.appendChild(canvas) will fail. HTML canvas elements, in the end, are just DOM elements. The Proxy is not. I could start by creating a DOM element, and add methods to it (such as getContext()) but while my experience with faking the DOM is limited, my intuition suggests that's a dead end.

The DOM element problem can be mitigated by faking not the canvas, but the rendering context that comes from calls to getContext('2d'). I always expected to need to fake out the context. This might work, but we will still have trouble with properties.

context.strokeStyle = "#000000";
context.fillStyle = "#FFFF00";
context.beginPath();
context.arc(100,100,50,0,Math.PI*2,true);

It may be possible to mitigate the properties problem with a hack that keeps properties in sync between the proxy and proxied object. For example:

ProxyContext = function(proxied) {
  this.__proxied = proxied;
  this.__copyIn();
}

ProxyContext.prototype.__copyIn = function() {
  this.strokeStyle = proxied.strokeStyle;
  this.fillStyle = proxied.fillStyle;
  ...
}

ProxyContext.prototype.__copyOut = function() {
  proxied.strokeStyle = this.strokeStyle;
  proxied.fillStyle = this.fillStyle;
  ...
}

ProxyContext.prototype.beginPath = function() {
  this.__copyOut();
  proxied.beginPath();
  this.logCall("beginPath");
  this.__copyIn();
}

This could work. I'm pretty sure it could work. Hey Internet, why won't this work?

I fear that a decent solution requires an abstraction layer that turns all mutations on the context into functions (e.g. context.strokeStyle becomes context.setStrokeStyle().) Is that good Javascript practice, or is that just not the Javascript way?

You Should Try Dygraphs

Over the last six months I've been working with an open source Javascript library called Dygraphs. Dygraphs, run by Dan Vanderkam, provides an interactive graphing library using the HTML5 canvas.

Four things have made me appreciate Dygraphs:

• It's got an wide array of configurable options.
• It has an active community on the mailing list.
• The documentation continues to improve. The options listing above was a recent improvement that is now my regular go-to reference.
• The tests. Really I should call them examples. Because I can just often analyze them and get 80% of the way to the solution of the problem I'm working on.

I've recently made a few contributions to the project, including 2-dimensional pan and zoom and displaying graphs in a log scale.

If you need a graphing library I recommend you give Dygraphs a try.