Tag: Operational Management

When Data Lies

Apparently, your astrological sign can determine whether aspirin saves your life.

At least, that’s what one analysis of a major heart attack study seemed to suggest.

In 1988, researchers published results from ISIS-2, a large clinical trial examining treatments for patients experiencing a suspected heart attack. The study found that aspirin reduced the risk of death.

The results attracted considerable interest, including requests to break the data down into increasingly specific subgroups.

And that’s where the story gets interesting.

When the results were examined by astrological sign, aspirin appeared to provide no benefit to patients born under Gemini or Libra.

Wait. What?

Did aspirin somehow work differently depending on when you were born?

Of course not.

The finding became a memorable illustration of a fundamental problem in data analysis: if you slice data enough different ways, eventually you’re going to find a pattern that looks meaningful but isn’t.

The problem wasn’t necessarily the data. It was the conclusion someone might draw from it.

This is something I think about a lot as an operations leader.

We break productivity down by shift, department, supervisor, tenure, product, location, and day of the week.

We find a correlation and immediately start looking for a cause.

We build a dashboard, identify an outlier, and launch an initiative to fix it.

But before we act, we should ask a few questions.

  • Is the pattern truly meaningful? Or did we find it because we looked at enough combinations?
  • Is there a plausible explanation? Can we explain why the result occurred, or are we just observing that it did?
  • Can we reproduce it? Does the pattern persist when we collect more data or test it under different conditions?
  • Would this change our decision? If the metric moves, do we actually know what action to take?

I’m a big believer in data. You can’t manage what you don’t understand, and data can expose problems that intuition alone will miss.

But being data-driven doesn’t mean believing every number we produce.

It means understanding how the data was collected, recognizing the limitations of the analysis, testing our assumptions, and being willing to challenge a result that doesn’t make sense.

Data should inform our judgment, not replace it.

The objective isn’t to find a pattern.

It’s to understand the system well enough to make a better decision.

How do you distinguish a meaningful signal from statistical noise in your own operation?

Measuring Everything, Managing Nothing

Up is good.

Down is good.

I can’t count how many times I saw those words on Air Force dashboards.

I sat through countless meetings reviewing performance metrics. Break Rates. Repeats. Recurs. FMC rates.

Many of the charts had graph with an arrow in the corner accompanied by one of those simple reminders.

That guidance always struck me as odd.

If someone reviewing the dashboard needs a reminder about whether a metric should go up or down, perhaps the metric isn’t telling the story as clearly as we think it is. If a decision maker needs guidance on whether the metric should be moving one way or another, maybe it’s not even an important thing to be tracking.

Which raises some larger questions:

What does a good dashboard actually look like?

How many KPIs should an organization be tracking?

There is no shortage of things to measure. In fact, many organizations today are moving toward hyper-tracking, collecting data on nearly every action an employee takes throughout the day.

The technology exists to measure almost everything.

The challenge is deciding what actually matters.

A dashboard should not be a collection of every metric available. It should be a decision-making tool.

The best dashboards answer three simple questions:

Are we winning?
Where are we struggling?
What action should we take next?

I’ve always believed every KPI should pass a simple test:

If the metric moves significantly tomorrow, what would I do differently?

If the answer is “nothing,” it probably doesn’t belong on the dashboard.

Too many organizations confuse measuring with managing.

More data isn’t always better.

Sometimes it’s just more data.

The goal isn’t to track everything.

The goal is to understand enough to make better decisions.