Google Testing Blog

Google @ ICST 2010

Saturday, March 06, 2010

pointer to the programlink to the abstract
Posted by Patrick Copeland

5 comments

Google

Labels: Patrick Copeland

Flashback. It's 1990. Chances are you do not own a cell phone. And if you do it weighs more than a full sized laptop does now. You certainly have no iPod. The music in your car comes from the one or two local radio stations that play songs you can tolerate and a glove box full of CDs and cassettes. Yes, I said cassettes...you know the ones next to those paper road maps. Music on the go? We carried our boom boxes on our shoulder back then.
If you are a news junkie, you get your fix from the newspaper or you wait until 6 ... or 11. Sports? Same. Oh and I hope you don't like soccer or hockey because you can't watch that stuff in this country more often than every four years. Go find a phone book if you want to call someone and complain.
I could go on, and on, and on, but you get the point. Oh wait, one more: how many of you had an email address in 1990? Be honest. And the people reading this blog are among the most likely to answer that affirmatively.
The world is different. The last 20 years has changed the human condition in ways that no other 20 year period can match. Imagine taking a 16 year old from 1990 and transplanting him or her to a 2010 high school. Culture shock indeed. Imagine transporting a soccer mom, a politician, a university professor... Pick just about any profession and the contrast would be so stark that those 1990 skills would be a debilitating liability.
Except one: that of a software tester. A circa 1990 tester would come from a mainframe/terminal world. Or if they were on the real cutting edge, a locally networked PC. They'd fit into the data center/slim client world with nary a hiccup. They'd know all about testing techniques because input partitioning, boundary cases, load and stress, etc, are still what we do today. Scripting? Yep, he'd be good there too. Syntax may have changed a bit, but that wouldn't take our time traveler long to pick up. That GEICO caveman may look funny at the disco, but he has the goods to get the job done.
Don't get me wrong, software testing has been full of innovation. We've minted patents and PhD theses. We built tools and automated the crud out of certain types of interfaces. But those interfaces change and that automation, we find to our distress, is rarely reuseable. How much real innovation have we had in this discipline that has actually stood the test of time? I argue that we've thrown most of it away. A disposable two decades. It was too tied to the application, the domain, the technology. Each project we start out basically anew, reinventing the testing wheel over and over. Each year's innovation looks much the same as the year before. 1990 quickly turns into 2010 and we remain stuck in the same old rut.
The challenge for the next twenty years will be to make a 2010 tester feel like a complete noob when transported to 2030. Indeed, I think this may be accomplished in far less than 20 years if we all work together. Imagine, for example, testing infrastructure built into the platform. Not enough for you? Imagine writing a single simple script that exercises your app, the browser and the OS at the same time and using the same language. Not enough for you? Imagine building an app and having it automatically download all applicable test suites and execute them on itself. Anyway, what are you working on?
Interested? Progress reports will be given at the following locations this year:
Swiss Testing Day, Zurich, March 17 2010
STAR East, Orlando, May 2010
GTAC, TBD, Fall 2010
Here's to an interesting future.

10 comments

Google

Labels: James Whittaker

Interview with Copeland

Tuesday, February 23, 2010

Part 1, Part 2, Part 3
Posted by Patrick Copeland

2 comments

Google

Labels: Patrick Copeland

Testing in the Data Center (Manufacturing No More)

Tuesday, February 09, 2010

was wasbuilt grownshippedmade available

19 comments

Google

Labels: James Whittaker

Interviewing Insights and Test Frameworks

Tuesday, January 05, 2010

Google is hiring. We have openings for security testers, test tool developers, automation experts and manual testers. That's right, I said manual testers.

As a result of all this interviewing I've been reading a lot of interview feedback and wanted to pass along some insights about how these applicants approach solving the testing problems we ask in our interviews. I think the patterns I note in this post are interesting insights into the mind of the software tester, at least the ones who want to work for Google.

One of the things our interviewers like to ask is 'how would you test product xyz?' The answers help us judge a tester's instincts, but after reading many hundreds of these interviews I have noticed marked patterns in how testers approach solving such problems. It's as though testers have a default testing framework built into their thinking that guides them in choosing test cases and defines the way they approach test design.

In fact, these built-in frameworks seem to drive a tester's thinking to the extent that when I manage to identify the framework a tester is using, I can predict with a high degree of accuracy how they will answer the interviewers' questions. The framework defines what kind of tester they are. I find this intriguing and wonder if others have similar or counter examples to cite.

Here are the frameworks I have seen just in the last two weeks:

The Input Domain Framework treats software as an input-output mechanism. Subscribers of this framework think in terms of sets of inputs, rules about which inputs are more important and relationships between inputs, input sequences and outputs. This is a common model in random testing, model-based testing and the testing of protocols and APIs. An applicant who uses this framework will talk about which inputs they would use to test a specific application and try to justify why those inputs are important.

The Divide and Conquer Framework treats software as a set of features. Subscribers begin by decomposing an app into its features, prioritizing them and then working through that list in order. Often the decomposition is multi-layered creating a bunch of small testing problems out of one very large one. You don't test the feature so much as you test its constituent parts. An applicant who uses this framework is less concerned with actual test cases and more concerned with reducing the size of the problem to something manageable.

The Fishbowl Framework is a big picture approach to testing in which we manipulate the application while watching and comparing the results. Put the app in a fishbowl, swirl it around in the water and watch what happens. The emphasis is more on the watching and analyzing than it is on exactly how we manipulate the features. An applicant who uses this framework chooses tests that cause visible output and large state changes.

The Storybook Framework consists of developing specific scenarios and making sure the software does what is is supposed to do when presented with those scenarios. Stories start with the expected path and work outward. They don't always get beyond the expected. This framework tests coherence of behavior more than subtle errors. Applicants who employ this framework often take a user's point of view and talk about using the application to get real work done.

The Pessimists Framework starts with edge cases. Subscribers test erroneous input, bad data, misconfigured environments and so on. This is a common strategy on mature products where the main paths are well trodden. Applicants who use this framework like to assume that the main paths will get tested naturally as part of normal dev use and dog-fooding and that the testing challenge is concentrated on lower probability scenarios. They are quick to take credit for prior testing, assume its rationality and pound on problematic scenarios.

There are more and I am taking furious notes to try and make sense of them all. As I get to know the testers who work in my organization, it doesn't take long to see which frameworks they employ and in what order (many are driven by multiple frameworks). Indeed, after studying an applicant's first interview, I can almost always identify the framework they use to answer testing questions and can often predict how they are going to answer the questions other interviewers ask even before I read that far.

Now some interesting questions come out of this that I am still looking into. Which of these frameworks is best? Which is best suited to certain types of functionality? Which is better for getting a job at Google? Already patterns are emerging.

One thing is for sure, we're interviewing at a rate that will provide me with lots of data on this subject. Contact me if you'd like to participate in this little study!