Comparing Periods and Sites
The most requested analysis and the most frequently wrong. What to normalise for before concluding anything changed.
Procedure
Comparisons between periods, sites and teams are asked for constantly and are almost always unfair as first computed.
What differs before performance does
Case mix, which is usually the largest factor.
Volume, which affects cycle time in both directions.
Staffing and experience.
System version, where a release changed the process.
Seasonality, including holidays and month ends.
The activity mapping version, which if it changed makes the comparison invalid outright.
Normalise or the comparison measures the differences rather than the performance.
Doing it properly
Same query, same boundary, same exclusion rules, same mapping version.
Same period length, and the same season where seasonality exists.
Filter to comparable case types, or report by type.
Report per unit of work, not per period.
Report the spread, since a shift in the tail matters more than a shift in the median for most processes.
Check the difference against ordinary variation before treating it as a change.
Comparing a unit against itself
The analysis that avoids the fairness problem entirely and is usually more informative.
Plot each site or team's own series over time, with its normal range.
Look for changes within the series, which are attributable to something that happened.
Compare trends rather than levels across units.
A unit improving faster than another is a finding about practice, and the practice can be copied.
Site comparisons specifically
Rarely worth attempting as a ranking.
Layout, mix, systems and volume differ enough that a normalised comparison is close to impossible.
Compare practices instead: what does the better-performing site do differently, described concretely.
Ranking sites produces defensive reporting and hidden problems, which costs more than the comparison is worth.
The mapping version trap
Worth separating because it is invisible.
A refinement to activity naming changes variant counts, path frequencies and sometimes durations.
A comparison across mapping versions is not a comparison.
Version the mapping and report the version with every figure.
When a number moves unexpectedly, check the mapping version first. It is the explanation more often than the process.
Presenting a comparison
State what was normalised for and what was not.
Show both series, not two numbers.
Say whether the difference exceeds ordinary variation.
Lead with the explanation, not the ranking.
Decline the request where no fair comparison is possible, with the reason and an alternative. A number produced under pressure will be quoted without its caveats.
Before-and-after that survives challenge
Most improvement claims in this field rest on two numbers and nothing else.
Same log definition, same boundary, same exclusion rules. A change in any of them invalidates the comparison silently.
Comparable period length and season.
Check the case mix, which shifts and can produce the entire apparent change.
Report the spread, not only the median.
Show the whole series, so a reader can see whether the change coincided with the intervention or preceded it.
Use a control: another region, another product line, or the historical trend, chosen before the intervention rather than after.