future - proofing innovation sprints: a guide to

experimentation & iteration

introduction

1
“
There is no such thing as
a failed experiment, only
experiments with
unexpected outcomes.
- Futurist Richard Buckminster Fuller
2
moving from It’s not easy to "retrofit" your company
with that "big experimental engine"

curiosity to later. It’s a cultural shift. You have to be in

the mindset of constantly testing new

value-add ideas, new business models,

products, and new processes. It helps
new

only hire people familiar with the

As Peter Diamandis is fond of saying, experimentation/ data-driven mindset
"The only constant is change, and the and sets the stage for experimentation
rate of change is increasing." in the beginning. During the early days at
Amazon, the company created a
In business, as in life, standing still standard experimental platform
means death. The only way to succeed in available to almost everyone. If people
business is to experiment and innovate wanted to test a new button or feature
constantly. Think of it as Darwinian on the website, they could. The problem
evolution at hyperspeed. Hyper-growth was that many of these experiments
and experimentation are very closely came from curiosity rather than a
linked. Case in point, Jeff Bezos likes to clearly-defined value proposition. After
say, "Our success at Amazon is a much trial and error, Amazon’s solution
function of how many experiments we was to create an "Experiments Group"
do per year, per month, per week, per that became the company’s home for
day...." experimentation, now referred to as
Amazon Experimentation & Optim-
Jeff Holden, who has built experimental ization.
engines at Amazon, Groupon, and Uber,
agrees that innovation is the key to
success. "The philosophy is you have to
build your company to be a big
experimental engine, and it has to start
right at the beginning."

4
 Main takeaway: Build a team inside your
organization that has an experimental
the experiments group
ethos, and make sure that each
always asked two key experiment, value proposition, and
questions: hypothesis are really thought through
• The first question the group would before you invest the time and energy to
ask was: What’s our hypothesis? actually proceed.

• The second question was: What’s

the value proposition to our
company?

Holden explained, "If you couldn’t

articulate your hypothesis crisply or
your hypothesis didn’t matter for
Amazon, Uber or Groupon, then they did
not run that experiment. Oftentimes
you'd send folks back to the drawing
board or ask them to recast the
experiment. The company learned, and
we got much better. You have to be able
to interpret the experimental results
well. It’s statistics. Know the difference
between statistically significant and
insignificant results." Uber, for example,
runs thousands of experiments per
month to test different features. They
A/B test key core features of the
business and choose the one that
performs best.

5
forming your did, you'd change how you operate. And
to get good questions and experiments,
experiment you must create a culture that
incentivizes asking good questions and
questions designing good experiments. At
(Google) X, Teller set up a Get Weirder
Every good experiment starts with a Award. The whole point of the Get
curious inquiry. Weirder Award was to focus the team on
experiments and to drive home the point
that they needed to think in terms of
experiments. Teams would be
teller recommends 3 challenged to ask "weird" questions to
principles for creating put forth crazy ideas around framing
experiments: problems differently and to design
experiments that really push the limits.
• Principle 1: Any experiment
where you already know the
Critically, Teller only gives out the Get
outcome is a bad experiment.
Weirder Award after the experiments are

• Principle 2: Any experiment in run. He shared, "If you give out the award

which the outcome will not after they've run the experiment,

change what you are doing is a independent of the results, then people

bad experiment. start to feel that you don’t actually care

about the outcome. You care about the
• Principle 3: Everything else
quality of the question. So every two
(especially where the input and
weeks, we would give out an award for
output are quantifiable) is a
the best experiment." Doing so
good experiment.
constantly (and viscerally) reinforced
the behavior of asking good questions.

You must ask the questions you don’t An innovation sprint is a fantastic tool for
currently know the answer to, but if you rapid experimentation in your company.

6
Participating in a sprint orients the can’t feed your team with two pizzas, it’s
entire team and aims efforts at hitting too large." An ideal team involves
clearly-defined goals. Sprints are useful representation from the engineering,
starting points when kicking off a new sales, customer support, and marketing
feature, workflow, product, or business departments. Additionally, "the decision
or solving problems with an existing maker" is critical.
product. There are five sprint phases,
typically done sequentially over five
days following forming your sprint team.
Try them with your team. each person brings
an essential
understanding to
the process:
pre-innovation
• Engineering understands
sprint: forming product capabilities.

your team • Sales understand customer

preferences.

To kick off an innovation sprint, • Customer support sees

assemble a diverse team from within customer challenges.

your company of five to eight experts in • Marketing holds key insights

different areas of the business. Your that tie it all together.
sweet spot is seven people—any more
will only create drag. Why seven people?
There’s no "CEO" of a team of seven
people. There’s no need for an org chart Over the next five days, the team will
or a communication plan. Information develop and test a number of different
flows freely, and there’s reasonable idea things.
diversity. Similar to Jeff Bezos'
"two-pizza rule," which states: "If you

7
01
You can run sprints on key features ("What should my landing page look like?" or
"How should we price it?"), product variations ("Do people prefer this type of
product or something else?"), or entirely new businesses ("Does this business
solve a key problem for a customer?" or "Are they willing to pay for this
product?").

8
01
day

01
such as what the company or product
should achieve in, say, eight months.

Ensure everyone is aligned and excited

about this goal, and write it down and
post it where everyone can see it.

Then, map out a series of difficult

questions that represent potential
barriers to achieving that goal, such as:
map & define
the problem • What are the technical and

Day 1 is about defining the problem you market risks?

want to solve. Develop a shared • What if people don’t care about it?
understanding of the working context,
including the problem, the business, the • What if they won’t use it?

customer, the value proposition, and • What might they not trust?
how you will determine success. By the
end of this phase, you should also aim to
identify some of your most significant
risks and make plans to mitigate them. After you've mapped out the questions,
Common understanding will empower spend time trying to answer the most
everyone’s decision-making and important of them based on the
contributions to the project. Under- information you know. Next, brainstorm
standing your risks also enables you to and list as many different types of
stay risk-averse and avoid investing time customers (demographics) that you
and money in things that rely on believe have the problem you are trying
unknowns or assumptions. The decision to solve. Now pick one, either the one
maker in the group should facilitate the you understand best or the largest
agreement around a long-term goal, cohort. This is the customer for whom

9
you will be running the innovation sprint. Based on this understanding, assign
one team member to recruit five real potential customers who fit these
characteristics from outside the company. These five people will be asked to
physically show up on Day 5 to test the product you’re designing.

why five customers?

In the '90s, computer scientist and web usability consultant Jakob Nielsen did a
study to answer this question, "How many interviews does it take to spot the
most important patterns?" As it turns out, 85 percent of the problems were
observed after just five people reviewed a product

1
day
2
day
3
day
4
day
5
day

map sketch decide prototype test

Additionally, the deadline of having actual customers onsite in five days creates
an added incentive to have a workable product finished by then.

The team’s last task for Day 1 is to "ask the experts." Someone in the world knows
the most about your customers. Somebody also knows the most about
technology, marketing channels, the business, etc. Your goal is to find them and
pick their brains. This is usually where people find the most exciting ideas.

10
01
day

02
combined and recombined in new ways
to create something better. Today, the
team will first sketch competitors'
solutions and put them on the wall for
everyone to see. Next, each team
member should sketch new solutions
that combine these LEGO bricks to
address the problem identified on Day 1.
Team members should do their
sketching independently and submit

create solutions their sketches anonymously.

member can submit one, two, or even
Each

Day 2 of your innovation sprint is about three solutions. This individual approach
generating insights and potential prevents the "groupthink" that occurs
solutions to your customers' problems. with traditional brainstorming.
Have your team explore as many ways of
solving the problems as possible, The goal is for each individual to create
regardless of how realistic, feasible, or fully fleshed-out ideas, empowering
viable they may or may not be. The introverted team members to contribute
opportunity this phase generates equally. Note: You need to be quite
enables your team to evaluate and thorough with these sketches—they
rationally eliminate options and identify should illustrate how the solution would
potentially viable solutions to move work for the target customer, what the
forward with. This phase is also crucial to customer sees going through the app or
innovation and marketplace differ- website, and so on. The sketches can be
entiation. Think of ideas in product rough, but they should be very detailed
design as LEGO bricks. They can be and precise.

9
day

03
Interestingly, the person who sketched
an idea isn’t allowed to speak until the
end of the group’s critique. This rule
minimizes bias and ensures the process
is truly meritocratic, allowing people to
be as honest as possible with their
critiques. In a group of seven sprint
participants, there may be anywhere
from seven to a dozen or more proposed

narrow down storyboard solutions.

your ideas
Day 3 is about converging. Take all the
possibilities exposed during Days 1 and
2, eliminate the wild and currently
unfeasible ideas, and hone in on the
ideas you feel are best. These ideas will
guide the implementation of a prototype
on Day 4 that will be tested with existing
or potential customers. Only some ideas
are actionable or feasible, and only some
will fit the situation and problem
context. Exploring many alternative
solutions helps provide confidence that
you are heading in the right direction. By
Day 3, the suspense will be high. Team
members should tape their sketches on Your team members should vote on the
a wall or whiteboard and then have solutions they believe in and are most
people mark their favorite ideas with a excited about testing. The clincher,
dot. The team should engage in a timed however, is that the decision-maker
critique of each concept. casts super-votes, which supersede all

12
other votes, to choose up to three solutions to test thoroughly on Day 5. Ideally,
this person will pick the single best prototype to answer the question your team
defined at the beginning of the week. Next, have the team create a storyboard.
Team members should flesh out the top idea(s), detailing everything from
discovery such as how a user comes across the product for the first time—for
example, on a store shelf, via a web search, or news article) to every other part of
the user experience. In other words, this storyboard needs to be able to stand
alone without a verbal explanation. That way, it may be evaluated unbiasedly on
its merits alone and not based on who created the storyboard. The idea behind
the storyboard will then be transformed into a high-resolution mockup that
looks and feels like an actual product.

13
01
day

04
to be. Once you know what works and
what doesn’t, you can confidently invest
time and money in more permanent
implementation. A good rule of thumb is
to build just enough to learn, but not
more. Once the team has an idea of how
everything will look and feel, the group
will assign tasks to different team
members to build a prototype. These
prototypes don’t have to be fully

build the mockup functional but should look like they are.
The team needs to ensure everything
Day 4 is all about bringing the ideas to appears realistic and cohesive. Buttons
life. Build a prototype that you can test might not fully work, but when pressed
with existing or potential customers. should drive the customer down a
Design the prototype for learning about different path.
specific unknowns and assumptions.
Determine its medium by time Jake Knapp, a partner at Google
constraints and learning goals. Paper, Ventures, calls this "Goldilocks
Keynote, and simple HTML/CSS are all Quality"—the prototype should have just
excellent prototyping tools for software enough quality to evoke honest
products, and 3D printing works well for customer reactions. After team
hardware. The prototype storyboard and members build their mockup, they will
the first three phases of the sprint regroup to flesh out the full experience
should make prototype-building fairly they want to put their five potential
straightforward. customers through on the following day.

There should be little uncertainty

around what must be done. A prototype
is a very low-cost way of gaining valuable
insights about what the product needs

14
01
day

05
person (videoconferencing can suffice,
in some cases). Identify "the
interviewer" on your team. This person
will interact one-on-one with your
customers who are testing the product.
The rest of your design team will observe
customer reactions via live video in a
separate room. The interviewer will ask
the customers different questions (you
should write a script):
observe & learn
from customer
• What do you think about what
reactions just happened?

On Day 5, the real fun begins. The five • How would you compare the

people you recruited on Day 1 will arrive different options?

to test your product/prototype, and • What worked?

you’ll get detailed and authentic
feedback from real customers. It’s • What didn’t?

important to test with existing or • Would you buy this?

potential customers because they're
the ones for whom you want your
product to work and be valuable. Their
experiences with the problem and Critically, the team should observe
knowledge of the context influence their customer reactions rather than simply
interaction with your product which listening to feedback to ensure the
non-customers won’t have. Your insights they glean are as honest as
customers will show you the product possible. By the end of this observation,
they need. It’s best to bring them you should be able to see patterns. You'll
face-to-face to test the product in know what’s working well and

15
Successful innovation isn’t just about coming up with great ideas— it’s also
about celebrating productive failure. Empowering employees to explore
unknown territories and try out bold new concepts enables organizations to
future-proof themselves, creating a "big experimental engine" that will drive
sustainable growth. With the right questions and meaningful experimentation,
you are able to unlock your most valuable resource - time.

Ready to take your leadership skills and vision for the future to the next level?

Our five-day Singularity Executive Program focuses on how exponential

technologies will shape the future and inspire you to develop a future-proof
mindset geared toward innovation while building relationships with other
hand-selected leaders from around the world, excited about what lies ahead.

Looking to future-proof your organization?

For over ten years, we’ve helped companies of all sizes understand the changing
landscapes of technology and shift their culture from fearing disruption
to driving it with our Singularity Enterprise Solutions.

17
keep exploring
additional future-proofing resources

6 key principles to help

leaders shift to an
innovation mindset

creating an innovation
culture with massive
transformative purpose