The Practical Advantage Of The ICE Score As A Test Prioritization Framework
Think of the ICE Score as a minimum viable prioritization framework. It’s not objectively perfect but it’s good enough to get the job done.
The GrowthHackers team has been using the ICE score to prioritize tests from the start. I’ve used the ICE score for 2+ years and found it to work extremely well. There’s been some criticism, however, of the ICE score that is worth addressing now that I have some experience with this framework.
As a quick refresher, ICE stands for:
- Impact: How impactful do I expect this test to be?
- Confidence: How sure am I that this test will prove my hypothesis?
- Ease: How easily can I get launch this test?
Each of these criteria is graded from 1–10 and the average presented as the ICE score.
ICE is sometimes criticized as being too subjective. The arguments are:
- if multiple people score an idea to test, they would all score it differently.
- the same idea might be scored differently by the same person at different times. This difference would then affect the final prioritization list.
- team members who wanted their tests prioritized could simply manipulate scores to get tests approved (though let’s face it, if you’re worried about this, you have bigger problems than the wrong tests being run).
I have nothing against less subjective approaches but I think the issue with these criticisms is that they:
- misunderstand the purpose of the ICE score and
- minimize or neglect the role of the growth process & the growth team
The Purpose of the ICE Score
Let’s get this out of the way first: The ICE score is not intended to be the perfect system for prioritizing individual ideas. It is actually a system of relative prioritization. The goal is to prevent you from being bogged down in trying to fine-tune the score too much. Think of the ICE Score as a minimum viable prioritization framework. It’s not objectively perfect but it’s good enough to get the job done.
But just because it’s not perfect doesn’t mean it’s not data-informed. Data analysis and information from past tests should inform Impact and Confidence scores. The Ease score should be a function of conversations with appropriate members of your team if you don’t have specific expertise.
Your worst case scenario is that you assign some combination of Impact, Confidence and Ease a few points higher or lower than if you had perfect data. I can say this with some authority having generated over 550 ideas for the team.
So why is this non-perfect scoring of ideas not a big deal?
As you participate in the growth process you’ll find your sense of scoring become finer tuned. This is because after each test you will have real information on the Impact of that test. Comparing this with the pre-test Impact score will help you understand why those differences existed. Similarly, test results will inform you of whether the pre-test data you relied on for Confidence was adequate or not. Lastly, with each growth meeting, feedback on Ease scores (or developing specific expertise) will tell you more about how to assign this score moving forward.
The point is that the “good enough” characteristic of the ICE score works well BECAUSE it is paired with the discipline of a growth process.
The Growth Process: The Role Of Objectives & The Growth Team
Before testing any idea at GrowthHackers, we establish objectives. Objectives determine our current area of focus. This is a signal to the team that most of the new ideas they come up with should relate to this objective. It also means that the only ideas that are nominated and hence tested are ones that map to this objective. Read more about this objective driven process here.
At GrowthHackers, we currently have 1500+ ideas in our backlog. Assigning an objective eliminates approximately 95% to 98% of existing ideas from contention. That generally creates a pool of somewhere between 30–75 ideas that we can pick from to nominate every week for testing. If no or very few ideas map to this objective the team comes up with more ideas.
Each member of our team can only nominate up to 2 ideas that map to our current objective. Our growth meeting is attended by 4–5 team members representing marketing, sales, product and engineering. Ideally, everyone nominates ideas that someone else came up with but it’s ok to nominate one of your own if you feel strongly about it. This helps cuts down on personal bias overrunning nominations.
So now we’re down to a maximum of 10 ideas that can even be considered for testing any given week. Some weeks it’s less than 10 ideas because some of them get nominated by multiple people. Of these, we green light 3–5 ideas to test that week.
When any one of us is making the case for an idea to be tested, its the responsibility of the team with their varying expertise to ask questions and provide a reality check to the scoring applied. For example, if an idea had a really high ease score, feedback from product or engineering team members might lower that score, potentially taking it out of contention that week. Similarly, lack of supporting data for high Impact scores will lead to a lowering of that score.
Wrapping Up
It was important for me to show how the ICE score for a testable idea works in the context of a real growth process versus a theoretical scenario. This should give you a sense of how small the possibility of “non-optimal” tests being run is in any given week if your team follows a systematic growth process.
If you haven’t given the ICE score and a growth process a genuine shot, I encourage you to check out Growth Hackers Projects.
If you found this post useful click or tap the clap icon below to help others discover it on Medium.
