Getting the Data Out
Where the events actually live, how to extract them without disrupting production, and the fields worth arguing for.
Procedure
Extraction is the bulk of the effort in most projects and the part nobody budgets for.
Where events live
Status history tables, which record each state change. The ideal source and not always present.
Audit and change logs, which record who changed what and when. Frequently the richest source.
Document tables with created and modified timestamps, which give two events and nothing between.
Workflow and approval engines, which are purpose-built logs.
Message and interface logs for events crossing systems.
Ask for the status history first. Where it exists the project is weeks shorter.
Where they do not
Tables holding only the current state, which record no history at all. A field showing "approved" with no timestamp of when tells you nothing.
Overwritten fields, where a modification destroys the previous value.
Archived data, where history is purged after a period, which caps how far back the analysis can reach.
Establish this early, because a process whose history is not retained cannot be mined retrospectively and needs logging turned on before anything else.
Extracting without disruption
A read replica or a nightly export, not a query against production.
Incremental by timestamp, not full reloads.
Agreed with whoever owns the system, in advance, with a stated volume.
Batch windows outside peak, which is a condition most system owners will impose anyway.
The fields worth arguing for
Start and end timestamps, not one. This separates working from waiting and it is the single most valuable addition.
Lifecycle state, which reveals suspension.
Resource, with the identity decision made deliberately.
Reason codes on rejections and cancellations.
Case attributes for segmentation.
Ask for all of them in the first request, because a second extraction request is much harder to obtain than a broad first one.
What slows extraction down
Access approval, which is frequently the longest single delay and should be started first.
Finding who owns the table, in older systems.
Understanding the state model, which is undocumented and lives in one person's head.
Data volume, where a year of events is larger than anyone estimated.
Personal data review, which is required and is faster if the resource field is excluded or pseudonymised at source.
Practical sequencing
Start the access request on day one, in parallel with everything else.
Extract a week first, to validate the structure before requesting a year.
Profile that week for defects before committing to the full extract.
Then extract the full period, with the fields and the rules already settled.
Starting the access request first
The longest single delay in most projects, and it runs in parallel with everything else.
Submit it on day one, before you know exactly what you need.
Ask for read access to the history tables for the process, for a named period.
Name a business sponsor, which shortens most approval routes considerably.
State the volume and the schedule, which is what system owners actually care about.
Expect a personal data review if the resource field is included, and consider excluding it in the first request to avoid the delay.
When no history exists
Some processes record only the current state, and they cannot be mined retrospectively.
Check first, because discovering it in week four is expensive.
Options, in order: find an audit log that captures changes; find an interface log; enable history logging and wait.
Waiting is a real option. Turning on logging and mining in three months is better than a year spent reconstructing.
Meanwhile, mine a different process that does have history, which keeps the programme moving and builds the credibility for the wait.
Extraction that survives repetition
The first extraction is a project; the tenth should be a scheduled job.
Read from a replica or an export, never from the live transactional system at analysis volume.
Incremental by timestamp, not full reloads.
Retain the raw source events alongside the mapped log, because the activity vocabulary will change and remapping from raw is possible while reconstructing raw is not.
Version the mapping so a change in variant counts can be attributed to a mapping change rather than to the process.
Document the source query, which is what makes a finding reproducible by someone else.