Choosing an idea
How do you decide what to do (in a rigorous defendable way?)
There are many ways to choose an idea or what to do next. Some of them involve a strict process, some of them just require relying on your ‘gut’. Whilst there’s no ‘right’ way to do it, it’s helpful to have some logic to the decision in case you ever needed to defend or justify that to someone else.
Decision Speed vs Decision Quality
The first thing I would say is to consider is: “is this a decision I could easily reverse if it turns out to be the wrong one?” If it is, the requirement for a rigorous defendable process is much weaker and the speed of the decision making may offer a number of advantages. Just picking something and getting on with it with the minimum of wasted time and has some advantages. Especially if the correct decision will only be discovered through experimentation rather than careful planning.
So I find it helpful to first consider if this idea can be reversed and whether making a choice (any choice) and trying it will help me know which is the correct choice to make.
As an example, imagine you’re in a foreign country and are presented with a selection of different dishes to try, which plate of food do you eat? Assuming you have no food allergies, there is almost no negative impact of making the wrong decision. You can choose something else immediately if you don’t like your first choice.
Now contrast the decision making process needed when trying different foods to the decision making process of whether to jump off a cliff into the sea or not… you’ve got to have done your homework and made very sure that there is a safe place to land. This is not a decision that can be easily reversed if it goes wrong!
Warning…
Sometimes you can make a quick decision because you thought it was easy to course correct if it turned out to be wrong, and then once you start, the process can run away with you and you discover you’ve actually invested a lot in this quick selection.
For quick decisions, always keep an eye on the clock and your wallet and regularly check in to make sure this is still making sense.
A flat surface with no cost to changing direction can become a gentle slope that can before you know it, quickly become steeper and steeper… before you know it, you might be slipping off the ‘decision cliff’ that you didn’t think existed when you started.
"If you're good at course correcting, being wrong may be less costly than you think"
Jeff Bezos
Idea Scorecards
Delivered value over effort to deliver (from Lombardo et al. Product Roadmaps)
The best way I know of making a decision is to construct some form of ‘scorecard’ for all you options. It consists of a number of elements and can be adapted to whatever situation you require. The basic building blocks are:
Value - What are the benefits of this idea, in the case of a product idea, it might be which customer needs (CN) or which (of your) business objectives (BO) does it satisfy and to what degree?
Effort - Every option requires some level of effort to be expended in order to make it a reality, is the level of effort low or high.
Feasible? - How confident are you that you can deliver what is needed for this idea to work?
Translated into common English, the equation terms might read something like this: What are the needs of your potential customers, what are your objectives and targets as a business, how much effort is all this likely to take and do we think we can actually do it with the resources and time we have?
The result of the formula is a priority score. The higher the number, the more better the choice. Use a simple number scoring system (a 5 point one would be fine). Since you are most likely to only be reviewing the better ideas, rather than everything, I’d suggest going from -1 to 4.
High numbers are good for Value but low numbers are good for Effort
In this example, Feature A satisfies the customer need and both business objectives reasonably well, but it doesn’t require too much effort and we’re 75% confident we can pull it off. We’re certain we can deliver Feature B although it’s not a desirable. Whereas Feature C has great scores for 2 of the 3 requirements but will need a huge effort and is unlikely to work.
As a result we should try Feature A first.
I believe lots of decision makers, even if they appear not to have a process, use some form of this formula sub-consciously. Delivered value over effort to deliver equals a good starting point. It has a nice advantage, when written down, that the decisions behind each number can be challenged. It’s nice and transparent.
Why I like it…
Of all the assessment methods I’ve come across, this one forces the user to consider a number of important options and is very flexible to different scenarios but also it does it in a transparent way. It’s easy to interrogate the different rankings, the different scores because they aren’t hidden behind layers of analysis.
At the end of the day, it’s just a tool that does 2 things: 1. help you decide which idea is best and 2. communicate that decision to others. It’s not making the decision for you, so if you find number rankings confusing, then replace them with words or even descriptive sentences. The goal is to have all the facts in front of you in one place. Not to hide the reasoning or shirk the responsibility.
“Learning to choose is hard. Learning to choose well is harder. And learning to choose well in a world of unlimited possibilities is harder still, perhaps too hard.”
Barry Schwartz, The Paradox of Choice: Why More Is Less
A few alternative selection methods:
Comparison Matrix
In formal engineering settings, a commonly taught approach is the Multiple-Criteria Decision Analysis (or Pugh Matrix). It’s similar in approach to comparing the pros and cons of different ideas but instead a list of determining factors is drawn up and weighted in terms of significance to the final outcome. Each design is then compared to a baseline solution as either better, same or worse than the baseline for each of these factors.
You could also give a score for each factor if you wanted more levels of granularity to your grading rather than just better, same or worse.
Whilst this is a popular analytical method. I have some strongly held criticisms of it:
The result is easy to manipulate, in years of teaching and literally thousands of Pugh Matrix’s looked at, I don’t think I’ve ever seen a result that was not what the student wanted it to be. But maybe just years of poor practise and incentives skewed towards just getting the job done rather than doing it properly has given me a biased opinion.
It’s less transparent than the scorecard method, the several stages of scoring, weightings and hide a lot of assumptions that are not easy to unpick. Choosing the datum can have a drastic impact on the outcome.
Dot Sticking (or other voting methods)
Another popular approach and all based on the premise that a democratic system is a fair one that will produce a good outcome. Whilst often a useful approach to force progress on a situation, it again is not very transparent to the real reasons behind a decision. Tom Kelley (of IDEO fame) uses dot sticking after a lengthy ideation session, not to pick a winner but to help narrow the field and focus the teams attention onto a cluster of related ideas.
I worry that sometimes people voting just follow the crowd, popular ideas can attract votes unjustifiably, whilst good ideas slip through unnoticed. Notice in the above video, Tom Kelley does not use the whole team to vote on ideas but a smaller group of people more experienced or closer to the problem being tackled. This is because they are better informed and more knowledgeable than the group used to generate the ideas in the first place.
One person who has taken this idea of informed voting to an extreme level is Ray Dalio and his investment firm Bridgewater Capital where he aims for an “Idea Meritocracy”.
A few things to try and avoid.
Shiny Object Syndrome:
Everything you start has a long tail of work associated with it. Prototyping, testing, documenting, training etc… before you know if it makes any sense. So try not to jump about from one idea to the next too quickly without good reason, you’ll swamp the development team with half started / half finished projects.
Your Gut (alone):
There is no problem with basing decisions based on you ‘gut reaction’, but prioritising solely on gut feeling can kill team morale. The lack of rigour to the process means that the decision maker is highly likely to change their mind. Look to test and review your gut reaction with data otherwise you might quickly lose your team.
Support Requests & Me-Too Features:
The customer support team is a vital source of data and insight for product improvements. However, making something easier to use can never be more important than making something more desirable.
Customer input and requests can often lead to a closing of the gap between you and the competition. Once you have easily comparable products, you have essentially created a commodity and it’s a race to the bottom. Choose instead to differentiate yourself, distancing yourself from the competition, rather than racing towards them by copying.
The Siren’s Call of Big Markets
There’s a natural draw to preferring ideas that impact the most people, that appeal to everyone, that have the largest possible market potential. But be warned! It’s like a Siren’s call, calling you onto the rocks!
The more attractive the idea may seem because of it’s huge market potential, the more likely it’ll fail. You can’t appeal to everyone, you can’t make something that everyone will want (or at least not right away in day one).
Find a niche, a tribe, a keen dedicated group of people that has the problem you are solving and is reachable, addressable, desirable. If you win here, you can grow from there. But trying to ‘boil the ocean’ won’t work.
Ideas should be hard to kill. Dismissing any idea, no matter how silly it seems, at first sight prevents you from uncovering deeper insights, challenging assumptions and coming up with better ideas.
Be quick to judge, but slow to dismiss.
Being critical of an idea can often force you to make the idea better, but only if you’re forced to sit with it, to live with it, to have to make it work. If you can dismiss ideas, you’ll never develop any of them and it’s in the development where ideas realise their full potential.
Selecting ideas is just another step in idea generation.
There seems to be a perception that selecting an idea means you’ve made it to the end. After the hundreds of ideas we created, using our idea generation methods, all we need to do now is pick the best one.
Linus makes it sound easy when he said “I had lots of ideas, then threw the bad ones away” and many people seem to take it literally… Have some ideas, reduce the number down to a few feasible ones and then pick the best, using one of the methods described above.
The too simplistic way to think about it:
The common perception, but a flawed, idea selection process.
This is a gross simplification of the process and results in potentially great or ground breaking ideas being rejected with casual disregard. I often see a quick comparison between the pros and cons of an idea cut great sway through ideas that are being rejected at a first glance.
All the ideas at this stage should not be considered more than weak hunches. They deserve more attention before being thrown in the trash.
The reality of the idea selection process is not a quick judgement but a multi-stage idea generation process where ideas are continuously improved, built upon and reviewed. Only once the idea generation phase has been repeated several times, on a more and more focused idea space can any rigorous selection process be made.
A better way to think about it:
The idea selection process more closely resembles an iterative series of design review, generation and improvement.
The initial ideas will have many issues and problems, and after a quick cull to the most promising few, a thorough review should be undertaken. But this review is the fodder for another generation phase. Every weakness and problem is an opportunity for creative endeavour.
Repeat the process until you have a small selection of credible and well thought through ideas. Only then can a credible selection process take place.
Ideas worth pursuing are hard to find
In a competitive environment the obvious ideas are almost certainly worthless. Not because they are not good, but because every competitor would have already thought of them. Go deeper and think harder, uncover something new.
Do not be put off by the flaws in any idea, they are an opportunity to be creative and discover something new and innovative.
Quitting is not the same as failure. Successful people quit more often.
The following is abridged from the above podcast, I strongly encourage you to listen, but a summary of some important notes are here:
“What distinguishes great poker players from everybody else is mainly quitting. They quit a lot more. So they're just very good at cutting their losses. So they fold more hands to start. Once they've committed money to a pot, they fold a lot more. They change tactics or strategies, like, in the middle of things. Um, and you have to be willing to do that.”
Annie Duke ($4m poker prize money winner and author of books on decision making)
Regret asymmetry and the bias behind It:
“There are two really important biases to think about because they collide here when it comes to quitting. The first cognitive bias is called status quo bias. Status quo bias is that we have a preference for the path that we're already on. So we don't like to change. We have a preference to keep going the way that we're going.
That collides with another bias, which is called omission commission bias, uh, and what that is, is that failing to act does not feel as much like a decision as acting does. Right? So let's say that I'm in a career or I'm in a relationship, and I just stay the path. It doesn't feel like I've made a decision. If I move, if I quit my job and change, if I break up in the relationship, now it feels like I've actually made a decision.
Now, the reason why this is an error is because the decision to stay in the job is also an active decision, and you should treat it the same way.”
The danger of goal-setting and how to do it properly:
“Another strategy we can use to quit closer to when we should is to increase the flexibility in how we set goals… So there's this amazing work by Maurice Schweitzer, who's at Wharton at University of Pennsylvania, we have this idea that goals are just generally good but he's saying there's a real downside to goals which is when you have a goal and it does two things to you:
It privileges certain values that you might have and deprivileges other values that you might have. So we need to be very thoughtful about, when I think about this goal, first of all, what am I not seeing? And what am I giving up that I'm following this goal?
Goals are also pass/fail. So, in a lot of ways, it's better to have never tried to go up Everest at all than to have gotten within 300 feet of the summit and turned around.
So how do we solve for this? There's kind of two ways that we solve for it. The first way that we can do is to remember the word “unless”. And this is really important, think about these things in advance. It's totally fine to say, "This is my goal unless," right? So my goal is to reach the summit unless there's really bad weather. So this idea of, "This is my goal unless," allows you to say, "I am setting this goal given what my information is right now."
The second thing, I think... So I was speaking to Ken Kamler, who is really amazing. He had been a doctor on Everest, actually, six different times. And he said something, I think, that was really profound. He said, "People forget when they're climbing Everest that the goal of Everest is not to get to the summit. The goal is to get back down to the base of the mountain." And why I think that that's so powerful is that he's talking about a time horizon problem, is that we get really wrapped up in the short term a lot when we ought to be thinking about the long term.”
Longer term thinking allows for many more changes of course and changes of direction without being seen as a quitter. Focus on the longer term bigger picture issues, family, happiness, wealth, health etc… and then the decisions along the way are all to those goals.