Story points, and velocity that is derived from them (story points completed for an iteration), are forms of vanity metrics. Vanity metrics have been given a number of definitions including:
- Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies 1
- Vanity metrics are a type of metric that may appear good for the company or suggest success, but in reality, they lack actionable data and have very little connection to a company’s progress or the impact of their marketing strategy 2
- Measurements that give “the rosiest picture possible” but do not accurately reflect the key drivers of a business.3
When I think about vanity metrics I also ask myself – does the customer care about this?
Many people may argue that story points can help inform future strategies, but in the vast majority of cases because they are not necessarily linked to things that a customer cares about or things that are truly actionable. Do story points help us understand things such as:
- Whether a customer is going to base a purchase decision on this?
- Will this enable customer retention?
- Does this improve the customer experience?
- Is this enabling a customer need / solving their problem?
Story points really don’t help us action anything with regards to the above.
This is usually where someone comes up with the argument that velocity will help us determine how much time something takes. Really? Are you sure about that? First of all, story points are “estimates” and are relative so they’re not connected with anything concrete like time. Thus velocity is a sum of our “estimates” of a collection of stories – they have nothing to do with the reality of how long something takes or how difficult/complex something actually was.
Furthermore, we often see teams that need to stabilise or “understand their velocity” which could take about 6 sprints (with 2 week sprints that’s a quarter’s worth of work). That is, we need to take a long time to get a decent sample to understand velocity. But then something changes – team members join / leave, we make changes to our process. Oftentimes, scrum masters say they need time to rebaseline their velocity – this may take another 3-6 sprints in which case we really don’t have any good data to infer how long things can take.
In many cases, customers don’t necessarily care about user stories – they actually care about the larger feature / capability that they are tied to. In which case, we’re really tracking the wrong thing with story points. Generally speaking, customers are concerned with things such as time, cost and whether a product or service is fit for their purpose.
I also often see many scrum teams at the end of the sprint splitting large items so they can “claim the points”. Often this is why scrum masters tell me they behave in this way. In that sense, the method is feeding the metric not the other way round. There is no improvement, insight or benefit from using story points other than to report “status quo” to management (which is also not the purpose of story points).
A better practice would be to start measuring something that the customer cares about and that you can action. For example, think about those key items that relate to customer requests and track metrics that will allow you to improve the delivery of the product / services in relation to those. Oftentimes, a key metric is lead time. There can be other types of metrics that you can use to help create concrete actions to improve your business and answer the questions I described above, such as conversions/conversion rate, customer satisfaction, customer churn rate etc. Looking at all aspects of end to end delivery of metrics such as lead time can lead to improvements that have real impact and that you never would have considered had you stuck with story points.
Vanity metrics can get in the way of real business goals and improvements. They prevent us from focusing on the right things because we spend our time focusing on the vanity metric instead. Stop using story points and velocity as these things are just vanity metrics. Measure and focus on data that is actionable and has an impact on your business.
References:
- https://www.tableau.com/learn/articles/vanity-metrics
- https://www.masterclass.com/articles/vanity-metrics
- https://en.wikipedia.org/wiki/Lean_startup