Profiling the Log First
Every log has the same defects. Finding them before drawing conclusions, because bad event data produces confident wrong maps.
Procedure
The most common failure in this field is confident analysis of a log that does not mean what it appears to.
The recurring defects
Incomplete cases at both ends of the window: started before it, or not yet finished.
Missing events for steps the system does not record.
Batch timestamps.
Duplicate events from a retry or a double extraction.
Cases with one event, which are either noise or a real finding about abandonment.
Test and training data mixed into production.
Migration artefacts, where a system cutover created thousands of cases at one instant.
Finding them
Case length distribution. Cases with one event and cases with hundreds both need explaining.
Duration distribution per activity pair. Negative and impossible values appear immediately.
Events per second. Spikes indicate batching or bulk loading.
Start and end activity frequency. A case ending on an unexpected activity is either incomplete or a variant worth knowing about.
Case volume over time. Gaps indicate extraction failure; spikes indicate migration.
Activity frequency, where a code appearing once may be a defect and may be the interesting case.
The window problem
Cases running across the boundary distort every duration.
Two options: filter to cases entirely within the window, or report incomplete cases separately.
Filtering biases toward fast cases, which is the more dangerous error since it flatters the cycle time.
Report both, and state which the headline figure uses.
Handling what you find
Exclude, do not correct. Removing implausible records is defensible; adjusting timestamps is not.
Document every rule and the volume it removes.
Report the exclusion rate alongside every figure, trended.
Never silently drop data, because the finding will be questioned and the answer must be immediate.
The reality check
Trace twenty real cases by hand through the source systems.
Compare against what the log shows.
Show the traces to the operation and ask whether they look right.
When the data and the operation disagree, the data is wrong more often. That assumption has saved more projects than any technique.
The standing report
Published with every refresh, on the same page as the findings.
Cases in scope, and excluded, with reasons.
Incomplete case proportion.
Events with defective timestamps.
Unmapped activity codes.
Match rate, where cases were joined across systems.
A finding presented without these is asking to be believed rather than checked, and the first time it turns out to rest on a defect, everything the project produces is discounted.
The exclusion register
Every analysis excludes records, and undocumented exclusions make findings unreproducible.
One row per rule: what it removes, why, and the volume.
Stable between baseline and follow-up, or the comparison is invalid silently.
Versioned with the mapping.
Reported as a proportion alongside every figure.
Never silently applied. The finding will be questioned and the answer must be immediate rather than reconstructed.
Test and training data
A defect that produces spectacular findings and no reality.
Test cases in production systems are common, particularly after a release.
They frequently have implausible durations, zero values, or names like "test".
Identify them by user, by value, by naming pattern, or by asking.
Exclude and count them, because the count itself is occasionally a finding about environment discipline.
Check after every system release, which is when a new batch appears.
The checks to run every time
Six checks, run before any finding leaves the analyst's screen.
Cases with no start or no end event, which distort every duration.
Events out of chronological order, which indicate a clock or extraction fault.
Duplicate events.
Identical timestamps in large clusters, which indicate batch writes.
Cases with implausible durations, at both extremes.
Activities appearing in only a handful of cases, which are frequently mapping errors rather than rare behaviour.
Report the proportion each check removed, because the exclusions will be questioned and the answer must be immediate.