Agile Metrics for Teams and Projects

A few weeks ago I was on a video call with someone about managing large-scale, multidisciplinary, distributed teams. A few years back I coined the term “agility for adults” as the title for a talk. That term captures the fact that, beyond managing small teams and models for motivating and inspiring people, working at scale also requires more advanced models of agile metrics.

Beyond the little sticky notes (known by the popular 3M brand name post-it) and the aspirational phrases that sound nice for motivating teams, the real challenge of agility, when it’s aimed at achieving things at scale, lies in mobilizing a much larger organizational structure.

Big Goals, Big Challenges

Completing initiatives or reaching goals of a much bigger magnitude isn’t a simple task. And whenever we think about this problem, the same questions come up: how do we build agile indicators and metrics to monitor the operation? Agile metrics that really let us see how good we are, or how well our projects are going? What agile metrics let us make better-informed decisions?

That said, and for the purposes of this article, I want to be clear that I’m a lover of coding, of software development. My professional life took me down the path of managing teams and projects, but I’ve never stopped “cranking out” code. Never for more than a year at a time. Sometimes I do it as a hobby. I regularly take courses on some online platform to stay a bit current, or support development projects by writing a few lines of code myself. From the underappreciated PHP to the powerful Go .

Code is something I’m passionate about. Thanks to this mix of managing people and writing colorful little letters[ref]The most popular source code editors usually have a feature called syntax highlighting that highlights words, key instructions, and symbols, making it easier to tell code and its structure apart.[/ref], I’ve been lucky enough to take part in, coordinate, and lead projects with a handful, many, or hundreds of people, across design, development, testing, and operations. From that vantage point, I want to share the agile metrics I’ve used, and still use, in my projects and with my teams.

This article is a summary of the metrics, indicators, and dashboards I’ve worked with successfully over the years, and that today, in my role as a consultant, I like to implement with the teams I serve.

Measures, Metrics, and Indicators

For a long time, I thought measure, metric, and indicator were essentially the same thing. Over time I found subtle differences between them, and with experience I understood just how far apart they really are.

So, what’s the difference between measures, metrics, and indicators? Let’s go through some definitions and a solid example.

A Comparative Example: Measure, Metric, and Indicator

To understand the difference through an example that uses measures, metrics, and indicators, I’d like to walk through the following.

Joanna took a 100-question test and got 50 correct.

Subjective Assessment of the Data

In your opinion, how do you think Joanna did? Let’s take a look.

• Joanna took a 100-question test (measurement)
• According to the test’s creators, Joanna answered 50 questions correctly (measurement). This measurement comes from comparing each question and answer against the expected question and answer (standard).
• Joanna got 50% of the questions right (metric). This metric is the ratio between total questions and correct questions.

Experience and Value Judgments

At this point, you’ll probably say: “Joanna failed.” The truth is, you’re using a metric as if it were an indicator. That happens when you assume the goal of the test was to get 100% of the answers right. That might be true in school or college, but in a business context, that’s just a value judgment - a very subjective opinion.

To define or identify an indicator, let’s add a bit more information to the example.

After evaluating more than 15,000 tests taken by the same number of people, we discovered that Joanna got one of the highest scores, beating more than 99% of the people evaluated.

Wait, I’m slow! Now it got interesting. Even though she only answered 50% of the questions correctly, Joanna is in the top 1% of scorers. What do you think now? Do you have any sense of Joanna and her performance? Does that 99% tell you something? Could you offer a more objective opinion of Joanna and her test?

Of course you could. That 99% tells us Joanna is one of the top 150 out of 15,000. And we can definitely say Joanna did well - or at least “less badly,” if you want to be strict about the fact she only answered 50% correctly. In both cases, that number (99%) is an indicator. If 50% was a cold metric that, under some bias or paradigm, got assumed to be an indicator, 99% actually tells us something, and reduces the subjectivity of our judgment.

How Do You Correctly Define Agile Indicators and Metrics?

Now that the difference between measurement, metric, and indicator is clear, let’s get into examples of agile indicators and metrics. That said, it’s not possible to define metrics and indicators that apply to every context. To keep this article light and simple, I’m going to set two different goals:

1. Measuring an agile team’s performance with indicators and metrics
2. Measuring a project’s performance and its progress to completion

While a lot of the measurements are the same, the metrics and indicators aim to establish different relationships and signals. Don’t forget the temporary nature of projects - by definition, they have a beginning and an end.

Let’s use an example: if you’re in charge of an organization’s IT department, under normal conditions, you wouldn’t be thinking about arbitrarily disbanding the team. But when we talk about projects, we inevitably think of a team that, with absolute certainty, will eventually disband - what those in the know call adjourning.

What Are Story Points?

This article is about agile metrics. One concept I need to clarify before moving on is Story Points, or SP. That doesn’t mean I’m talking exclusively about Scrum , though. To understand how agile team metrics work, we need to define the context and determine:

1. A measurement period, like Sprints in Scrum, iterations - a generic term across every agile and hybrid model - or Program Increments in SAFe .
2. A unit of measure for effort - one that, ideally, somehow captures complexity and risk too. Imagine having to consolidate a dashboard across multiple teams using person-hours, function points, and SP - what a nightmare!

The first part is simple: define a fixed time period - a week, an iteration, a month, a quarter, or a year. The second part requires a deeper discussion. The most common units of measure we use for effort include:

1. Person-hours - how many hours a task or deliverable will take to complete.
2. Ideal person-days - how many ideal days, meaning days with no interruptions or task switching, it would take someone to complete a task or deliverable. Ideal days almost always come with a productivity factor. That is, 3 ideal days with a 50% factor means it’ll take 6 days to complete.
3. Function points - hard to explain if you’re not in programming, but here’s a link for the curious.
4. Story Points

But what are Story Points, really? Let me walk you through a simple example.

Defining Story Points

Story points, or SP, are a unit of measure for expressing a relative estimate of effort. This estimate captures, in a single value:

• Effort
• Complexity
• Risk

Here’s an example to help you understand the concept.

Someone needs to install 20 windows in a 10-story building. All identical, same dimensions, anchored with the same mechanisms. The building has no structural quirks that would change the installation process.

Someone in charge of scheduling the work might assume that, if installing a window takes between 20 and 30 minutes, the entire process of installing 20 windows takes between 400 and 600 minutes.

But if we consider that not all the windows are on the same floor, and that there’s additional risk tied to height, we might instead assume that[ref]The window example is, of course, drastically oversimplified, to highlight the fact that an activity’s duration isn’t the only variable considered in calculating SP.[/ref]:

• Windows on the first 3 floors get a relative score of 1SP
• Windows on floors 4 to 8 get a relative score of 2SP - higher risk and complexity
• Windows on floors 9 and 10 get a relative score of 3SP - much higher risk and complexity

As you can see, SP account for the fact that greater height means greater complexity or risk. We’re not just evaluating how long activities take - we’re also weighing other dimensions, like risk or complexity.

Story Points are a relative unit of measure that only makes sense once you have more than one request or requirement to analyze. Saying, in isolation, that requirement ABC weighs or equals 3SP, 5SP, or 20SP means nothing at all. But saying that request ABC is 3SP and XYZ is 5SP tells us XYZ involves almost twice the effort, complexity, and risk of ABC.

Metrics for an Agile Team

Beyond projects, there’s a whole universe of management where we can think of work as continuous and ongoing. I know, nobody lives forever, and these days, a lot of people don’t even last two years in the same job. Still, the way we understand a team’s performance is a continuous one.

Which metrics are useful for a team? Here are the most common ones.

Capacity: How Much Do We Think We Can Complete?

My favorite definition of capacity is: the amount of work we believe, or project, we can complete in a defined period of time.

Capacity is measured (measurement) in person-hours, function points, or story points. We don’t always have the information needed to pin the measurement to one particular unit, and part of a Scrum Master’s or facilitator’s job - in roles like RTE or DASSM - is to validate that the chosen metric and measurement actually favor the organization’s context, instead of becoming a “headache” instead of a “relief”[ref]In the agile world, relative estimation is favored. Among the most popular relative estimation practices is Planning Poker. Likewise, the most commonly used unit is story points, but that doesn’t make it the only option.[/ref]

Velocity: What’s Our Capacity, and How Much Have We Completed?

Velocity is a concept tied to the amount of work a team can carry out or complete in a period. It was originally designed to measure “individual productivity” on eXtreme Programming (XP) teams - which didn’t make some people very happy. Velocity was originally used to determine a team’s “load factor .” Load factor was a messy concept that ended up being replaced by Story Points.

Today, Velocity is a quantity of points - of relative effort, like SP - within a given period. We can classify it as:

• Planned Velocity: The number of points a team expects to complete in a period.
• Actual Velocity: The number of points a team actually completed in a period - worth noting that, in Scrum, “completed” includes both the Development Team’s work and the PO’s approval - that is, done and approved.

Completion Rate: What Have We Successfully Completed?

We can define completion rate as the amount (measurement) of work actually delivered or resolved within a defined time period (another measurement). But in highly variable contexts, I like to split this measurement into two:

1. Completion against the plan: how much of what we planned to do did we manage to complete?
2. Completion against effort: how much of what we believed we could complete actually got completed?

At first glance, you might think these are the same thing. And surely some Scrum-maniac will insist that a Sprint shouldn’t include any additional work. But that’s not always the case[ref]The term Scrum-maniac refers to a blind love that ignores the realities of an academic management model. I’m surprised at how politically correct that definition of mine turned out to be.[/ref].

If the time period allows for it, or the scope of work is extremely variable, we’re very likely to run into new items, or even swap some out - ones we consider to be of similar magnitude. In that context, the tasks planned at the start of the period aren’t necessarily a good metric for completion.

Variability: How Stable Is Our Plan?

As a result, if completion requires understanding how much the original plan - the one defined at the start of the period - has changed, a metric that scores how much the original plan has drifted can be very useful and revealing for the team.

Calculating variability can be challenging, and it requires a lot of discipline. What we want to avoid is working on things that end up not contributing to our metrics in any way. So every new activity or task within a time period should be marked or classified as either part of, or a consequence of, the original plan, or as new and unexpected.

Quality: How Well Are We Doing the Work?

Well, we couldn’t leave quality out. In this case, I like to think of quality as a ratio relative to the amount of work delivered. A lot of teams limit themselves to “counting” defects or incidents. As you probably already know, that’s a mistake. A measurement isn’t a metric, and it’s definitely not an indicator.

The dictionary defines quality as:

A property or set of properties inherent to something, that allow us to judge its value.

Diccionario de la lengua española

That’s why one quality metric I like to use is what some authors call the defect rate. What percentage of the delivery is, in some way, compromised or affected by quality-related problems?

A similar metric - and my favorite - is defect density. This metric is the ratio between the number of defects and the size of the delivery.

Metrics for Agile Projects

Of course, projects can benefit from team metrics too. But unlike a team, projects move toward a specific completion or closing point. Similarly, project metrics can be used to track a release.

Progress: How Far Have We Come?

When you define a goal to reach, you can establish a progress metric. To do that, you need at least one of the following - though personally, I try to have both:

1. A date - a milestone
2. A specific objective - tied to the product or service in development

With a limit or goal in place, we can define a progress metric based on the amount of work completed and its impact on the objective - what people often call value - compared against the total capacity planned to reach the milestone.

Danger! Progress to Completion Against a Variable Scope

The biggest difficulty in measuring progress within an “adaptive” context is the constant push to lock down scope. It’s hard, and for some people even annoying, to track progress against a goal that keeps moving. Within an agile context, the concept of a goal or value is used often, and preferred over scope. That way, scope can vary depending on the objective you’re trying to reach.

Example: A Fixed Goal with Variable Scope

Let’s go with another example, but this time, let’s leave Joanna alone.

It’s Friday night, and Pedro, a single guy about to turn 30, wants to go out and have fun. But he doesn’t have much money, and tomorrow, Saturday, he has lunch planned with his parents that he wants to attend.

So Pedro has defined:

1. A clear objective
2. An estimated budget - a limit on resources
3. A deadline - let’s say he doesn’t want to show up hungover to lunch with his parents, or at least wants to be able to hide his hangover.

But Pedro doesn’t have a plan, and he isn’t sure how he’s going to “have fun.” So let’s see how scope can vary in light of the objective.

First Scenario

Pedro decides to call his friends and meet up at one of their homes. If Pedro is anything like me, they’ll probably drink a few beers, reminisce about old times, maybe play a video game tournament, and have some pizza at some point in the night. By 3 or 4 a.m., Pedro will be home, happy to have gotten together with his friends and had a night full of “friendship.”

Second Scenario

Pedro decides to go to his favorite bar. There, he runs into someone he really likes, and there’s a clear chance of having a great night. It’s a moment of “flirting,” of romance, a moment to make a move. Pedro decides to approach and give it a shot. The next day, without going into much detail, Pedro is happy and feels like he had a night full of “romance.”

Third Scenario

Pedro decides to go to his favorite bar, but he doesn’t recognize anyone there. He feels a bit lonely, but sees the opportunity to talk to new people. Between a few drinks and some good music, he makes new friends. By the end of the night he’s home, having met new people, feeling like he unexpectedly enjoyed his favorite bar.

Each of these scenarios lays out different activities and behaviors within a shared objective and the same constraints - known in project jargon as constraints. Do you now see how scope can vary based on the objective? I haven’t said it’s simple, or fast, but it’s possible in certain contexts.

Debt: How Much Have We Added or Traded, Inside or Outside the Original Plan?

Well, if we have a progress metric, and we know scope is variable, it’s a very good idea to measure and track the project’s level of debt against the plan. In other words, what percentage of the originally planned or projected work is now outside the team’s capacity for the established date and objective.

What happens if Pedro is at a table with new friends, and right then, the person he likes shows up? It’s inevitable - for that same night, and with the same budget, he has to make decisions. Every decision carries a cost, and certain specific activities will end up outside his capacity - for that period.

Examples of Agile Metrics

Now let me walk you through some examples, along with explanations I hope will help tie the concepts together. Let’s take a look.

Agile Metrics for a Dedicated Team

[wpdatatable id=1]

Even though the team never hits 100% completion of the plan - which, in my experience, is actually the best-case scenario - we can still tell it has a stable actual velocity hovering around 30SP. You might think a team that plans should hit 100%, but the truth is, that practice undermines honest, transparent estimates.

Notice I use the word period instead of Sprint, because using Scrum isn’t a requirement for these metrics - though they do work well with it.

Hitting 100% of a Sprint or Period’s Plan

This is a really bad idea, and I’ll explain why.

Imagine you’re on an agile team, and during iteration planning, you agreed to complete requirements totaling 100SP. During the iteration, one of your teammates gets sick and can’t come in, and some of the requirements get blocked by dependencies on a vendor. What does that mean?

1. It’s clear the team isn’t going to hit its 100SP goal.
2. If it’s not the team’s fault, we can’t immediately call it “bad planning” - a common mistake made by people who believe humans are machines and that plans must be followed to the letter.
3. Worse yet, out of that 100SP, the team no longer has any additional items to pick up and reassign work to, to avoid wasting time.

What should we do? Should we suspend the iteration? Run an emergency planning session? What happens to our commitments and our agile metrics?

The team isn’t just blocked - it also has no other items available to pick up and “recompose” its 100SP goal. For those who enjoy the math: estimates should be ranges, not absolute points. If a team’s projected capacity is 100SP, the team should plan for something closer to 110SP or 115SP, understanding they might have some high-performance days. Likewise, if something with a negative impact happens, the PO should have “a plan B” to reorganize the team around what they actually can work on.

Agile Metrics “in Trouble” - What’s Going On?

[wpdatatable id=2]

At first glance, something happened starting in period 4. But what happened? Why is “actual velocity” dropping?

The simple solution, of course, is to make the most of reflection spaces with the team, identify the causes, and propose solutions. But at a leadership level, expressing what’s happening through data can be useful. Leadership isn’t always in constant contact with agile teams, and often relies on reports and metrics instead. So let’s look at a bit more information on this case.

[wpdatatable id=3]

Well, this expanded table tells us a lot. For some reason, the team started getting assigned unplanned work within the same period - for example, adjustments to existing features, changes, or enhancements. Never defects, though.

How Many Points Should I Assign to Defects?

This is a point of debate. Should you quantify the effort behind defects or incidents you need to fix, or not? Yes and no. In my experience, agile metrics can be gamed if incidents or defects carry weight in the team’s metrics. But:

1. It’s good to quantify them to determine relative effort.
2. It’s bad to quantify them when determining the team’s velocity.

In my opinion, if a team has to invest a large chunk of its time and capacity fixing errors - things that don’t work the way we agreed they should - the best way to reflect that in the metrics is as a “loss of capacity.” Otherwise, you could end up with teams that show high relative scores but deliver nothing but fixed defects.

In short, a team that spends its time on fixes should see its velocity drop as a result.

Are There Agile Metrics for the Quality of Delivered Work?

Let’s look at a bit more information from our agile team.

[wpdatatable id=4]

Simply counting defects is a bad idea. Not all defects are equal. Some defects take more work than others, so it’s a good idea to use relative values to quantify defects - remember, I wrote earlier that “it’s good to quantify them to determine relative effort.”

So classifying them by size, or using something similar to SP, is a good idea.

That said, the best approach is to establish a ratio between the SP delivered and the defects injected during a period - yes, “injected” sounds strange, I know.

[wpdatatable id=5]

In this second example, we can see how defect density - the number of defects injected per SP delivered - is climbing. It’s possible that, by introducing new requirements into the period, the plan or the impact of those requirements isn’t being properly assessed. That, of course, affects delivery capacity (lower actual velocity) and delivery quality (higher defect density).

Scaled Models and Program Increments

Well, so far we’ve only discussed team-level metrics. Every single metric presented here can be used across multiple teams - if we normalize the data - and across different periods, like iterations, months, and Program Increments (PI).

If “variability” still bothers you, consider the following:

1. You might decide not to accept anything new within the same iteration.
2. Anything additional or new goes into the next iteration - 0% variability for that iteration.
3. That said, that iteration is part of a PI, so it’s smart to measure the iteration’s variability to improve your measurements and projections for the next PI Planning.

Considerations

As you can see, agile metrics apply to agile projects, agile teams, and large-scale initiatives. If you only keep metrics at the team level, you lose the chance to scale. In large organizations, with hundreds or thousands of employees, it’s not possible to run retrospectives and easily communicate the reasons and actions behind everything. That’s why setting up metrics, and supporting people in reading and understanding them well, can help you earn the organizational support you need. We all need “political” support in a large organization - it’s key - and giving visibility to what’s actually happening is the first step toward getting it.