skip to Main Content

Experimentation

Strategy execution insight through testing

The methods and mindsets of experimentation are hugely valuable for organisations seeking agility in fast-changing environments.

Experimentation can be applied both as a mindset and method – ideally developed and learned through designing and conducting a series of experiments that generate insight and inform action.

Experimentation expert Dr Andrew MacLennan, introduces the topic and explains its value.

Strategy Execution Ltd MD

Why experiment?

Experiments are an incredibly useful way for organisations to learn and catalyse effective change. As the business environment becomes more volatile and uncertain, the need to foster agility and ‘free up’ thinking has grown. Whilst other forms of research, analysis, planning and execution remain useful, for some organisational challenges they cannot provide the answers. Rather than calculate, predict, assume and estimate, it can be better to explore, try, test, learn and refine. 

You’ve got to experiment to figure out what works.

– Andrew Wiel

Experiments are usually conducted on a small scale at first, making them easier and faster to implement than typical organisational changes. Designed well, they can produce insights very quickly – and often these insights are more robust than those produced from conventional methods. For example, focus groups might suggest that lots of customers would buy a new product – an experiment can show that they actually do! Experiments are also often less expensive than alternative approaches to gathering insights – and because they typically help to reduce risks, can offer significant financial returns if these insights lead to wide-scale implementation of changes.

Experiments have led to some of humankind’s greatest achievements, and have the potential to stimulate extraordinary innovation and change within organisations too – by challenging assumptions and fostering action learning.

What makes a experimenter?

Curiosity

Curiosity is the most important characteristic in an experimenter. The hunger to better understand drives great experiments.

bias for action

A bias for action drives fast, small experiments which produce insights to steer subsequent phases of experimentation and organisational change.

Boldness

Experiments are good for exploring 'edgy' topics and for challenging convention. Courage can be required to overcome organisational inertia.

Teamwork

Different perspectives, skills, learning styles, experience, stakeholder connections mean that diverse teams run great experiments.

What is an experiment?

An experiment is a systematic procedure designed to discover facts through altering conditions and observing effects.

Crucially, experimentation is not just ‘trying out new things’ – which most organisations do all the time. Experiments are systematic procedures. They use the fundamental building blocks of the scientific approach – relying on clear hypotheses, careful sampling and management of bias, systematic observation, robust measurement and so on.

Experimentation is not just 'trying out new things'

Andrew MacLennan

Also important in this definition is the point that experimentation involves changing conditions. Most other forms of research commonly used in organisations are observational – for example conducting survey questionnaires or focus groups. Although these can be useful, by themselves, they are not experiments: they don’t change the conditions of respondents in the way that experiments change the conditions for their subjects.

This distinction is often important. For example, surveyed customers might say what they think they would do under particular circumstances; whereas an experiment can show what they actually do. 

Classic experimentation

Management experiments are not new. One of the most famous early management studies, the Hawthorne experiments, saw lighting levels in a Western Electric factory adjusted, to observe effects on worker productivity.

In another illuminating case from 1930, Kellogg’s experimented by changing the working schedule at its Battle Creek cereal plant from three eight-hour shifts to four six-hour shifts, hypothesising that shorter working hours would increase productivity. 300 extra workers were hired for the fourth shift, and hourly pay was increased to reduce the impact on weekly wages. The hypothesis was validated. In spite of these extra costs, productivity rose so much – and mistakes, waste, absenteeism, staff turnover, accidents and insurance premiums fell so much – that profitability increased.

Employees strongly favoured the new arrangements, which created additional family and leisure time and reduced unemployment in the community.

Experimentation on the high street

One food & beverage company recently decided to co-create products with consumers via experiments.

It set up a pop-up store in a major city and invited paying consumers to experiment with different ingredients for one of its big brand products. The store was mobbed as hundreds of people tried out combinations the company had never considered would be appealing. A huge amount of positive social media activity was generated by customers, with many posting photos of their creations. A second pop-up store was opened in another city, this time charging considerably more, and the result was the same. 

The company reckoned that much more useful insights were being generated several times faster than using conventional focus groups. For example, these led to one product being priced 600% higher than had been planned – successfully. Furthermore, the revenue generated through the experimentation more than covered all the costs!  

Experimentation on the web

Advertisers on the web constantly experiment with different adverts, to test their effectiveness. Rather than rely on advertising executives or customer focus groups selecting ‘winners’ from a range of possible adverts, multiple options are delivered to a small but representative sample of web users for a short time. The ads that generate the most clicks are then rolled-out to the entire target audience. 

A coffee manufacturer recently ran some online adverts for instant coffee, on a major Asian online shopping platform. Three different adverts were automatically served to random samples of users – each suggesting a different benefit of drinking the coffee. Otherwise, the adverts were identical – each showing the same image, using the same colours and so on. One advert, which hinted at coffee’s potential to help overcome boredom at work, generated twice as many click-throughs and a much higher rate of purchase than the alternatives tested. This translated in to much higher sales volumes and revenues than for consumers who viewed the other adverts or, indeed, no adverts at all.

Even more sophisticated forms of such experiments are possible where characteristics of web users are known and used to target adverts. Amazon, for example, can determine which adverts are likely to be most effective for customers depending upon their address, age, gender, purchase history, and so on.

Netflix uses a similar approach to recommend titles to view. Not only are individuals shown titles likely to be of interest to them, but different algorithms that determine which titles are of interest are matched to individuals through experiments conducted on each user.

Experiments are not...

Experiments are not like typical change initiatives. They don’t by themselves necessarily deliver specific outputs or have direct business performance outcomes. They are explorative, focused on discovery, insight and learning. However, most experiments will inform or develop into change initiatives – making them highly impactful.

Those who experiment in conventional organisations have to be careful not to fall back on a ‘big project’ mindset – planning to deliver ‘perfect’ and complex solutions over long time horizons.

What to experiment on

Experiments can create insight around a very broad range of topics and issues. For example, they can be:

  • problems or opportunities
  • strategic- or operationally-focused
  • externally- or internally-oriented
  • short-term or long-term
  • process- or people-oriented

Similarly, experiments can support innovation around new ideas or optimisation of existing activities – and often a combination of the two.

To get started, it’s useful to think of several phases:

  1. identifying important challenges*
  2. developing ideas to address the challenge
  3. finally, developing experiments that can test out these ideas
*  ‘challenge’ is used in a neutral sense – i.e. it could be an opportunity or a problem

FAQs

Experimentation can usefully address a wide range of challenges. They could relate to customers, competitors, suppliers, staff, regulators, products, services, processes, systems, resources, and so on. 

Much can be gained by asking questions about potential challenges. Staying in a ‘questioning mode’ is wise – rather than leaping to solutions too early. It’s useful to undertake some initial:

  • secondary research – finding out what is known and has already been explored
  • stakeholder engagement – finding out who is knowledgeable about and interested in the challenge

Initial research can lead challenges being ruled out, before potential solutions/responses are considered.

Initial ideas about how to address challenges, through potential solutions or responses can be evaluated using two key criteria:

  • scalability potential of the solution/response
  • feasibility of developing the solution/response – usually it’s best for individuals or teams to develop ideas that are reasonably within their spheres of influence, so they can gain traction with experimentation

It’s always a good idea to hold your initial ideas lightly. Exploring ideas often leads to new information and perspectives being developed – particularly when ideas are shared with others. Those who hold tightly on to fixed ideas gain less from this exploration.

Teamwork is vital for good experimentation.  Effective teams are diverse – bringing varied perspectives, relationships, skills and strengths to the group.  However, teams need to remain small enough to be agile – around four to six members is usually ideal. Amazon calls these ‘two pizza teams’ – if a a team cannot be fed on two pizzas, it’s too big.

Other stakeholders can of course be engaged by each team to provide support as required.

The other magic ingredient is passion – teams should form around issues about which members are passionate!  This will heighten chances of success.

This is a difficult question to answer as it very much depends on the experimentation and the context. But generally, any one experiment should not be too demanding, for two reasons.

First, experimentation is best adopted as part of the way people tackle their ‘day jobs’. It’s less effective as a ‘bolt on’ activity. Those who successfully adopt an experimentation mindset typically start to see good opportunities for experimentation all the time.

Second, although experiments can address major strategic challenges, and experimenters can ‘think big’, they should always aim to start small, with tightly-scoped experiments that seek insight around highly specific (but important) elements of potential solutions/responses. If an experiment seems ‘too big’ and too too time-consuming, it is probably the wrong experiment.

Experimentation works best when initial experiments are undertaken speedily on a very small scale, and insights are quickly created and recycled into an ongoing experimentation process. So it’s almost always best to undertake a series of experiments, but starting with very small and less formal tests and trials.

Absolutely! Experimentation can become a mindset for individuals and a policy for organisations. Experiments can focus on business challenges of all sorts. Experimentation is a very flexible friend!

Absolutely. It makes great sense to engage stakeholders and draw upon contributions from diverse teams.

Experiments should of course be designed realistically – reflecting what is possible in the businesses and, for example, requiring only data that can be measured feasibly. The best way to ensure that experiments are feasible is to limit their scope.

The scope of an experiment refers to its breadth and depth – for example, which products are launched, which processes are redesigned, which regions are involved or which customers are targeted. In most cases, it is best (and usually necessary) to severely restrict the scope of initial experiments. It is usually better to keep them small – so that they can feasibly be undertaken quickly, inexpensively and without unnecessary risk. Then they can be refined and repeated or expanded, once insight has been generated. 

Certainly. The focus of experimentation can change at every stage:

  • When exploring business challenges, some issues will be discarded in favour of others where there is greater potential to create value
  • When considering ideas for solutions or responses to business challenges, it is wise to consider a range of possibilities – and pursue the ones with the greatest potential
  • When shaping experiments to test out these solutions or responses, many options will emerge, some of which will prove to be more valuable and feasible than others.

Often, it is after connecting with stakeholders and undertaking secondary research, that refinement to experimentation topics becomes feasible. Also, each phase of experimentation is likely to create insights that influence later decisions about what to experiment upon.

This iterative and agile approach is a healthy and positive feature of an experimental approach.

There are no rules about this – it very much depends on the context. It’s usually best if initial experiments are undertaken very quickly, to gain early feedback and then perhaps further developed and expanded. However, sometimes the context demands greater patience, or means that there are limited opportunities to keep experimenting.

Experiments should create insight – and conversely reduce uncertainty.  If insights focus on important issues, experiment teams should be able to identify lots of ways to convert them into value. 

It is ideal if experimenters can directly apply the insights within their own roles – or in some cases change the shape of their roles to incorporate this responsibility. Many experiments lead to new solutions being developed and mainstreamed or scaled-up. Ideally, experimenters should lead this change.

It may also be valuable to systematically disseminate insights created, to inform wider stakeholders and influence and inspire them to act, or behave differently. It is usually the responsibility of the experiment team to do this to extract the greatest possible value from the insights they have created.

Back To Top