How we improved our sprints when we stopped estimating stories

Written by Andre Schweighofer, Technical Product Owner

Despite our best intentions, software projects often get delayed. We, the members of the Premium team, try our best to make realistic predictions, yet we often fail. For some, the solution is to take every estimate with a grain of salt or to massively inflate our timelines. Others joke that in order to find realistic estimates, the best method is to take the estimates of two senior engineers and multiply them to get an accurate due date. If you have worked in an agile or Scrum team before, you might have used story points to estimate your user stories. We used the same strategy in the Premium team.

In this article, we outline the issues we encountered while trying to create realistic estimates and how we ultimately changed our entire process.

What our workflow looked like

After the arrival of our new Scrum master, we reset our workflow to reflect a pretty standard Scrum process. We used the backlog refinement to discuss our user stories and assigned story points to each. If there were significant differences between team members’ estimations, we knew that the story was not clear enough and needed more thorough review.

During our sprint planning, we would closely monitor our velocity. Velocity is an average of story points achieved over several iterations. We could then use this number as a guideline for how much work we should plan for the upcoming sprint. We also used this number to estimate how much we would be likely to achieve over a longer timeframe or when we would reach a certain milestone.

Our estimation process and use of velocity was never really questioned. It was a procedure ingrained in the team. It was our ritual. In theory, this ritual had a few advantages:

• It allowed us to follow a simple procedure which lead to consistent results
• It helped us point out unclarities in stories and foster a discussion
• Because story points can abstract factors like time, complexity and effort, it helped make sure we didn’t get lost in those details

Ultimately it helped us plan our work better. 

“Our estimation process and usage of velocity was never really questioned. It was a procedure ingrained in the team. It was a ritual.“

Inspecting a faulty ritual

If we consider the estimation process as a ritual of the entire agile community, it appears to be a rather inconsistent one. The steps and procedures aren’t exactly straightforward. No team follows the exact same steps or treats their estimates and velocity the same way. If you ask the community for best practices, you won’t get clear answers. Some of the questions often raised are: 

• Do you estimate effort or complexity? 
• What about bugs?
• Is one story point equal to one day of work or some other time-based unit? 
• If not, why do we use it to predict the scope of an iteration which has a fixed duration of fourteen days?

While many thought leaders in the agile community have their own answers to those questions, there is no single correct answer. You will have to find your own answers and what works for your team. At best, your team will develop a clear understanding of your process, and you only have to face those hard questions again when your team changes. At worst, you and your colleagues might already have different perceptions of what you are doing and why without even realising it.

The ritual is also pretty time consuming. If you try it and do not see the expected results, you’ll often be counseled to simply try harder or invest more time. It takes considerable effort to keep the ritual running. You need some anchor or some reference story to make sure your estimates continue to mean the same thing.

One of the traps in this process we stumbled into was related to proxy discussions. Instead of spending time clarifying our stories, we repeatedly discussed our estimations. While this might have highlighted an issue in the story, it did not always lead us to the desired outcome–a clear, well understood user story.

To make matters worse, while trying to improve our estimates we faced a typical challenge–our team members changed. Even though we looked at our velocity to limit our sprint scope and create realistic forecasts, our gut was still the main driver of our predictions.

So we did not experience the benefits of using story points in our team, but we did suffer from the amount of time and energy we dedicated to the process. We soon decided to take a different approach.

“We did not experience the benefits of using story points in our team, but we did suffer from the amount of time and energy we dedicated to the process.”

Let’s not and say we did

Some of our team members are followers of Allen Holub. He is a strong advocate of a #noEstimates approach. During a retrospective, we decided to give it a try. We wanted to test what would happen if we skipped the estimations and, instead, simply relied on our gut feeling entirely. We also wanted to see if we could find a way to predict our sprints and release forecasts without creating a separate process.

We immediately felt the benefits of not having to deal with story points anymore. There was simply no opportunity to escape talking about the complicated details of the work in front of us. With the help of the INVEST principle (don’t worry about the E, a story can be estimable without actually having an estimate), we managed to reduce the time spent in meetings which left us with more time to actually work. At the same time, we encountered fewer surprises during our sprints. Using our new approach, we were able to decrease our average cycle time by 30% in three months. This new process felt much more natural to the team. It was also easier to sustain. We did not rely on any ritual master to keep us going anymore.

“We managed to reduce the time spent in meetings, and we also encountered fewer surprises during sprints.”

Measures of progress

In the Premium team, we break our stories down into tasks. Our goal with tasks is not to slice our stories into same-sized units like “takes 1 hour of work”. Instead, we use our tasks to break stories down into their natural, smallest units. That means that one task may only take 30 minutes, and another one might take a whole day. A task is a logical unit of the story, can be worked on independently and is actionable.

It turned out that while our number of completed stories varies from sprint to sprint, our number of tasks does not. With a simple calculation of “tasks per person per day” we found a metric which we could use to more accurately predict our velocity. This metric beats story points on many levels: 

• There is no additional process required to maintain this number
• It emerges from the work we are doing already
• It saves us time and allows us to focus on delivering value

We now use this metric to check our sprint commitment, so we can be sure we are not overloading ourselves with work or undercutting our process. 

Extending our experiment 

We liked our #noEstimates approach so much that we tried to apply it to other processes as well. At Runtastic, we use quarterly OKRs to align our team’s goals with the company and other teams. In the past, we spent a lot of our OKR planning time trying to come up with magic estimations to limit our workload for the upcoming quarter. We compared the progress we made toward our OKRs when we used estimations vs. when we did not, and the results were almost exactly the same. So, out with the magic estimations and in with the #noEstimates metrics! We calculated the average amount of key results achieved over the last few quarters and used this number as our guide to set goals for the upcoming quarter. Since our key results are mostly deliverables, we now have a much better idea of how many features we will be able to release. Looking at the data and hard facts helped us to make quick decisions and reduce our overly optimistic commitments.

What we learned from this

With this simple experiment we proved to ourselves that we should always question our processes. Story points are such a widely accepted concept that we automatically accepted its relevance for us. While we cannot not say that story points have no merit, we learned that, for our team, in our current situation, it was not the right tool.

“Never leave a process unquestioned”

Another takeaway we learned is to worry less about processes and focus more on agile values. I think what the team did is a perfect example of the Scrum theory: Inspect and adapt. The agile values and principles continue to be a great source for helping us reflect on what we are doing and how we can get even better at delivering value. We learned how we can reduce the time we spend on mundane tasks and, instead, have more joyful interactions within the team. By making this change we are not just embracing agile values, but also our Runtastic values. 

***