Lesson 35 — Measuring Analyst Efficiency
Security Copilot can make analysts faster, but speed alone does not tell us whether security operations have improved.
A mature SOC measures whether analysts reach accurate conclusions sooner, investigate more consistently, explore evidence more deeply and spend less time on repetitive work.
This lesson builds a practical model for measuring Security Copilot without confusing AI activity with genuine productivity.
What you will learn
Measure the operational value of Security Copilot using meaningful SOC outcomes.
The measurement cycle
↓
Select repeatable SOC activities
↓
Measure time, quality and effort
↓
Introduce Security Copilot
↓
Measure the same activities again
↓
Compare speed AND quality
↓
Review consistency, depth and workload
↓
Check security outcomes
↓
Refine workflows and measure again
What should a SOC measure?
| Area | Question | Example indicator |
|---|---|---|
| Time | Is repetitive work taking less time? | Median triage or investigation time. |
| Quality | Are conclusions accurate and evidence-based? | Peer-review corrections. |
| Consistency | Are required investigation steps followed? | Evidence-check completion. |
| Depth | Are analysts exploring relevant evidence? | Validated pivots and data sources. |
| Workload | Has repetitive effort reduced? | Time spent summarising and documenting. |
| Response | Are meaningful decisions happening sooner? | Time to escalation or containment. |
| Learning | Are analysts becoming more capable? | Independent investigation performance. |
Learning objectives
Establish useful baselines, measure time and quality, evaluate consistency and depth, assess workload, connect AI use to operational outcomes and avoid misleading vanity metrics.
Efficiency is not just speed
An analyst who closes an incident five minutes faster but misses a compromised account has not become more efficient. Useful efficiency combines time, quality and outcome.
Establish a baseline first
Record how selected SOC activities perform before a new Copilot workflow is introduced.
Choose comparable work
Compare similar incident types and complexity. A simple phishing alert should not be compared directly with a multi-stage identity compromise.
Measure median time
Median values can be more useful than averages when a few unusually long investigations would distort the result.
Measure triage time
Track how long analysts take to understand an alert, identify affected entities and decide whether deeper investigation is required.
Measure investigation time
Track time from investigation start to a defensible conclusion, escalation or response decision.
Measure reporting time
Copilot may reduce the time required to turn validated technical evidence into readable reports and handovers.
Measure repetitive effort
Look for reductions in manual summarisation, formatting, query explanation and evidence organisation.
Measure investigation quality
Review whether conclusions are supported by evidence, important telemetry was checked and uncertainty was documented.
Track corrections
Record how often peer review identifies inaccurate statements, missed evidence or unsupported conclusions.
Track missed evidence
A faster investigation is not an improvement if analysts overlook relevant identity, endpoint, email or cloud evidence.
Measure validation behaviour
Check whether analysts continue opening source telemetry rather than accepting generated summaries as evidence.
Measure false confidence
Review cases where generated language sounded more certain than the underlying evidence justified.
Measure escalation quality
Determine whether escalations contain enough validated evidence for responders to act without repeating basic investigation work.
Measure consistency
Verify whether required investigation steps are being followed across analysts and shifts.
Use investigation checklists
Measure completion of required evidence checks for repeatable scenarios such as phishing and compromised accounts.
Compare analyst variation
Look for excessive differences in investigation quality between analysts handling similar incidents.
Measure handover quality
Review whether handovers preserve status, evidence, uncertainty, owners and next actions.
Measure investigation depth
Efficiency should create more time for useful reasoning, not simply faster closure.
Count meaningful pivots
Review whether investigations explore relevant users, devices, IP addresses, applications and resources.
Measure telemetry coverage
Check whether analysts use appropriate identity, endpoint, email, cloud and SIEM evidence.
Measure hypothesis testing
Look for evidence that analysts tested alternative explanations rather than only confirming the first theory.
Do not reward unnecessary depth
More queries are not automatically better. Investigation depth must remain relevant to risk and evidence.
Measure analyst workload
Efficiency gains should reduce repetitive effort and create capacity for investigation, hunting, engineering and learning.
Measure queue pressure
Check whether priority incidents are processed more effectively without rushed closures or reduced quality.
Measure time recovered
Track where saved time goes. Ideally it becomes capacity for higher-value security work.
Measure analyst experience
Combine operational data with structured analyst feedback about usefulness, friction, trust and workflow fit.
Measure analyst development
Copilot should support capability growth rather than dependency on generated answers.
Test independent skills
Periodically assess whether analysts can interpret telemetry, construct queries and explain reasoning without Copilot.
Measure query improvement
Track whether analysts improve at understanding and refining KQL rather than merely copying generated queries.
Measure reasoning quality
Review whether analysts can explain why evidence matters and what alternative interpretations remain possible.
Operational outcomes matter
Ultimately, Copilot should contribute to better security decisions and more effective SOC operations.
Time to escalation
Measure whether serious incidents reach the correct responder sooner with enough evidence to justify escalation.
Time to containment
Compare how quickly validated incidents move from detection to approved containment where appropriate.
Reopened incidents
An increase in reopened incidents can reveal that apparent efficiency is actually premature closure.
Repeat investigation work
Measure how often senior analysts must repeat basic evidence gathering because the original investigation was incomplete.
Detection improvement
Recovered capacity may enable better hunting, tuning and detection engineering.
Do not measure prompt volume
Ten prompts do not represent ten units of productivity. One precise prompt may be more useful than twenty vague ones.
Do not measure words generated
Longer summaries can increase review time and hide the evidence that actually matters.
Do not reward blind adoption
High Copilot usage is not automatically desirable if analysts use it where traditional tools are faster or more reliable.
Do not measure speed alone
Time savings must be reviewed alongside accuracy, evidence quality and security outcomes.
Beware selection bias
If Copilot is only used on easy incidents, measured improvements may not represent the broader SOC workload.
Beware novelty effects
Early changes may reflect enthusiasm, training effort or unfamiliarity. Measure over a meaningful period.
Create a balanced scorecard
Combine speed, quality, consistency, depth, workload and operational outcomes rather than relying on one KPI.
Use trends, not snapshots
Track measurements over time to distinguish sustainable improvement from temporary variation.
Segment the results
Review metrics by incident type, analyst experience and complexity where comparisons are meaningful.
Review unintended consequences
Look for over-trust, skill erosion, unnecessary prompting, sensitive-data issues and premature closure.
Turn measurement into improvement
Use findings to refine promptbooks, training, review gates, workflows and governance.
Final measurement principle
If Copilot saves time while maintaining or improving investigation quality and security outcomes, the efficiency gain is meaningful.
Example SOC efficiency scorecard
| Metric | Baseline | With Copilot | Interpretation |
|---|---|---|---|
| Median initial triage time | 12 min | 7 min | Potential efficiency gain. |
| Incident summary preparation | 18 min | 6 min | Strong reduction in repetitive work. |
| Peer-review corrections | 8% | 7% | Quality maintained or improved. |
| Evidence checks completed | 82% | 95% | Improved consistency. |
| Incidents reopened | 3% | 3% | No sign of premature closure. |
| Proactive hunting | 4 hrs/week | 7 hrs/week | Recovered capacity redirected. |
Important: These figures are illustrative training examples, not Microsoft benchmarks. Each organisation should establish its own baseline.
Example measurement prompt
Agent Foskett investigation: “The analyst who won the prompt competition”
↓
One analyst had used Security Copilot 684 times
↓
Someone declared him the SOC's “AI productivity champion”
↓
Agent Foskett checked the incident data
↓
His median investigation time had barely changed
↓
Several prompts repeatedly requested the same summaries
↓
Peer review found unsupported conclusions
↓
Another analyst had used Copilot far less
↓
Her handover time had dropped by half
↓
Her evidence-check completion had improved
↓
She was spending more time threat hunting
↓
The dashboard was redesigned
↓
Prompt count disappeared
↓
Time, quality, consistency and outcomes took its place
Key takeaways
- Efficiency combines speed, quality and security outcome.
- Establish a baseline before claiming improvement.
- Compare similar investigation types and complexity.
- Track evidence quality and peer-review corrections.
- Measure consistency and meaningful investigation depth.
- Check whether recovered time moves to higher-value work.
- Monitor analyst development and dependency.
- Prompt count is a poor productivity measure.
- Use a balanced scorecard and trends over time.
- Measure whether Copilot helps the SOC make better decisions with less unnecessary effort.
Related Agent Foskett resources
Continue Module 4 — Operational Security Copilot
How do you measure Security Copilot analyst efficiency?
Measure Security Copilot with a balanced combination of time saved, investigation quality, consistency, depth, workload and operational security outcomes rather than prompt volume alone.
