Leading Product Teams to Experiment Better

Introduction

As a leader of product managers, one of the key roles you have is helping your teams be able to find the right opportunities and solutions that generate value for the customer and the business. Most of our ideas for what these right opportunities and solutions are will miss the mark. We never know less than we do at the beginning, and as such we are most likely to not know or consider something. I know of no better way to tease out what we don’t know, uncover risks and test assumptions than through running experiments. As a product leader, you need to be equipped to coach product managers to be able to successfully experiment.

As I have worked with product managers and tried to help them experiment more often and more successfully, I have found some common areas that product leaders need to be prepared to handle with their product managers. I have found 4 key areas that highlight the struggles PMs have with experimenting:

1. Permission — Does the product team’s leadership, the company and key stakeholders allow the product team to experiment, knowing that this experimenting will result in uncovering what works but also what does not work?
2. Space — Does the product team have sufficient time and resources needed to experiment?
3. Ability — Does the product team have the necessary skills to be able to design, execute and measure experiments that lead to meaningful product outcomes?
4. Purpose — Does the product team understand what they are trying to learn and why it is important relative to their customers and the business’s strategy?

Permission

Permission has to do with whether or not the product team is allowed to experiment. Do the managers, stakeholders, and company culture allow the team to test their way to achieving an outcome.

Product teams struggling with this will express concerns that when they’ve tried to experiment they’ve been shot down. First, make sure that this is actually the real problem, permission is generally not given unless there is first trust. I don’t let my son drive our car, not because I am a tyrant but because I don’t trust him to do so without crashing (he’s 10).

For product teams trust comes as they demonstrate the ability to and understanding of why they are experimenting. Product teams that know how to effectively experiment and understand the companies goals so they know the reasons for experimenting will get way more trust and therefore permission than those that don’t. Because of that make sure first that the product team has the ability to and understands the reasons for experimenting before looking at issues of permission. Until you feel like the team has these areas of ability and purpose up to par you are wasting your time focusing on permission.

That said, to establish permission the product team is going to have to start with small relatively inconsequential experiments. Find a place where you can test things and when you win you can parade it all over and when your experiment doesn’t play out it doesn’t sink the ship.

Evangelism will also be critical to helping product teams get permission. If their small experiments are only known to the product team trust won’t grow and permission won’t expand. Help them find ways to share what they are doing and the impact it is having. Never lie, you’ll be found out and you’ll be in a worse spot than you were before, call your failures for what they are and explain the impact of the successes.

I need to acknowledge that some companies will not ever permit product teams to experiment. Frankly, these companies might be using terms like product managers and product teams, but what they want are what have been called Feature Teams by Marty Cagan. They just need someone to help ensure what they want built gets built. If you find yourself in a company like this and want to do modern product management you may have to leave if you are unable to influence change.

Space

Space is a question of does the product team have sufficient time and resources needed to conduct experiments.

Helping with space will generally lead to helping with prioritization, the product team has decided they will work on other activities that are not experimenting. There are several reasons why this might be the case, but I would not recommend tackling these at first. Issues of space are usually masking other issues the product team is likely dealing with. First, ensure that the team has the ability to experiment and understands the purpose of experimenting. If they are lacking in either of those two areas they will busy themselves with things that others tell them are important or that seem to matter. They will be too busy to do any meaningful experimenting, but the painful reality is that they are embracing that busyness because they are sure what to do otherwise.

If the team has the ability and reason and space really is the issue; the product team needs to take a hard, and probably painful look, at how they are choosing to spend their time. Then they need to cull some things to make room for experimenting. Ideally, the product team can remove things themselves and communicate the impact and reasoning to stakeholders. There will be scenarios where the team’s leaders may need to get involved to help make space. However, the product team should be given every opportunity to do this themselves, with managers coaching, doing so results in greater confidence in the product team from those they work with.

Ability

Ability has to do with whether or not the product team is able to find, design, execute and measure experiments. The skills involved with this are one of the primary reasons for having a dedicated product team. Can they quickly, inexpensively remove ambiguity and find optimal solutions through experimenting. Knowing the hows of experimenting is where the product team and their leaders should focus first when trying to experiment more and more effectively.

Helping a product team learn to experiment will involve coaching them in these key areas:

1. Determining what needs to be learned.
2. Designing an experiment that will provide learnings.
3. Quickly getting the experiment in front of real users.
4. Gathering and interrupting data to be able to learn more in ways that progress the team.

Each of these parts is crucial to successfully experimenting. Missing any part will cause the product team to be unable to find the right solution and they will become frustrated with the experiment and revert to whatever is comfortable.

1. Determining what needs to be learned

The product team needs to be able to know what they are striving to accomplish (more on that in the Purpose section) and be able to call out what they need to know to accomplish that. This seems obvious but it has to be addressed and don’t assume it is a given. If the team is trying to increase sign-ups they need to learn what drives sign-ups, not what drives views.

A helpful tool for determining what needs to be learned is the Riskiest Assumption Test or RAT. Whenever we assess an opportunity or come up with a solution we have assumptions baked in. We don’t always realize we have them but we do. This is okay, but you must make sure the assumptions get called out. The assumptions that have the most potential to cause the team to fail if they prove untrue are the Riskiest Assumptions and running experiments to vet those assumptions is critical to being successful.

2. Designing an experiment that will provide learnings

Once the team knows what they need to learn they need to design an experiment that will help them learn that thing. This includes finding the right segment of your customers, planning for what will be different for that segment of customers, and ensuring that data will be available on the results.

Getting the customer segment right is critical and is a point that product teams get hung up on. Say your team needs to learn if new site visitors will be more inclined to do business with you if you offer a way for them to trial your software without providing any info. If the team tried this experiment on previous customers they would be missing the mark, they must try it on a new customer. It seems obvious but make sure the segment you are testing lines up with the real customers. Experiment as close to the intended customer segment as possible. This often means experiment with the intended customer segment.

The team also needs to plan what the experience will be for that segment. What will they see that will be different, who will work with them, what will the flow or process they go through be like? Make sure that this experience once run will result in learning what you actually need to learn.

3. Quickly getting the experiment in front of real users

Because of the messy and uncertain nature of product development the team needs to be able to move fast. If they don’t someone else will and you are at a real risk of losing. The experiments do not need to be things that are scalable and completely polished. Product teams need to be able to craft tests that don’t take long to spin up, are as real as possible, and are done with people that actually use your product.

Running your prototype by some friends or people in the coffee shop is no substitute for getting people that actually use and are deciding if your product is worth their money.

4. Gathering and interrupting data

Once the experiment is going and especially after it is done the team will need to look at the data and decide what happened. They will need to be able to access and analyze the data. They need to be unbiased and should not be overly vested in one experiment succeeding. They are trying to accomplish an outcome for the company, not make sure one feature becomes a reality.

It is very important before the experiment to define the goal and hold yourself to it after the experiment. This is easiest when the team already has an objective they are trying to influence. I’ve seen many experiments that were called a success simply because no one said what success would look like and so they defined it based on what went well in the experiment. Don’t fall for this trap. Pick your target before you shoot.

As a leader of product teams when you are wanting to help your team experiment more you will probably spend 90% of your time helping them with ability. They will learn this as they practice and you provide feedback on how they could have experimented better. Think of yourself as a coach, they are the players and you have the opportunity to watch and coach them to awesome things.

Purpose

The final area is purpose. This simply means ensuring the product team knows the reason why they are experimenting, what are they trying to accomplish for the organization. Without a clear purpose, the team may have the ability, space, and permission but will risk learning things that don’t actually help the organization and the product.

Addressing this is a matter of helping the product teams have context into the overall strategy of the organization and then providing them with objectives for how their team will push that strategy forward. This does not take a lot of time but will need to be done regularly. The team will need to be reminded from time to time and that is your job.

Conclusion

A product team that experiments successfully will be able to uncover more value for a business and their customers than one that does not. Being able to experiment requires a mix of ability, purpose, space, and permission. As a product leader, a large part of your job is to help your product teams experiment successfully. Coach them to awesomeness and before long your teams will surprise at the results they will be delivering.