Estimating the Return on GenAI Projects in Enterprise Fraud Management, Part 2 of 3: From Point Estimates to Probabilities

Where Part 1 left off

Part 1 sorted fraud projects into two classes and built a value equation for each. Detection projects (Class A) are valued as avoided loss: event volume, times fraud incidence, times average loss, times the recall improvement, times the share of losses you were never going to recover anyway, minus the operating cost of the extra alerts and the customer friction they create. Efficiency projects (Class B) are valued as released capacity: alerts, times hours saved per alert, times loaded cost, times a savings realization rate η that converts hours into money only where a named mechanism (an invoice, a contract, a deferred hire) exists. Running both projects at once produces a cross term, (A₁ – A₀)(t₀ – t₁), that belongs to neither business case and must be claimed exactly once.

The running example, a retirement recordkeeping platform with 12 million annual distribution requests, came out at $5,212,800 net for the detection layer, $492,200 realized for the summarization assistant, and $148,600 for the cross term. Every one of those figures is a point estimate. This article is about replacing them with something more honest.

Symbols carried forward from Part 1: N (event volume), p (fraud incidence), L (average loss), r₀, r₁ (recall before and after), π₀, π₁ (precision), ρ (recovery rate), A₀, A₁ (alert volume), t₀, t₁ (hours per alert), w (loaded analyst cost), f (cost per customer-facing false positive), η (savings realization rate). New in this part: P_s, S_s, Δ q_s for tail scenarios; Φ, θ, λ for severity; and τ for the detection threshold.

Inputs that come from people, not databases

Several inputs in this framework cannot be pulled from a database. Nobody has data on how much more likely you are to interdict a scenario that has never occurred, or on how much of a large relationship’s fee stream you would lose after a public incident. These come from subject matter experts (fraud investigators, relationship managers, operations leads) asked for a low, likely, and high figure rather than a single number. Three people estimating independently and then reconciling produces something far more defensible than one person’s best guess, and the PERT distribution described later in this article exists precisely to turn those three-point estimates into inputs the model can use.

Tail events: where expected value breaks

Everything in Part 1 assumes you have enough events to estimate a rate. For everyday retail fraud at a large institution you have thousands of confirmed cases, a stable base rate, comfortable statistical power.

For the events that actually frighten your risk committee, you do not.

A compromised advisor credential used against dozens of custodied client accounts. A retirement plan breach. An insider at the recordkeeper. These arrive perhaps once a decade. Multiplying a probability you cannot estimate by a severity you have never observed produces a number with no information in it.

Use a different model.

Scenario-based valuation

Define three to six potential high-severity fraud events. For each scenario s:

where

P_s = annualized probability of the scenario occurring

S_s = total severity

Δ q_s = increase in the probability that you interdict it in time

Δ q_s is a judgment call. It is a structured judgment call, made by people who know the control environment, about a specific concrete scenario, rather than a rate pulled from thin air. Have three SMEs estimate it independently and reconcile. That process is defensible in a way that a fabricated frequency is not. This is the same scenario-based approach supervisors have long accepted for low-frequency, high-severity operational risk [1].

Severity is not the stolen money

The direct loss is usually the smallest component:

• D = direct unrecovered
  operational loss to the firm
• M = client make-whole
  payments
• Φ = assets under management
  in the affected relationships
• θ = revenue yield on those
  assets, the effective fee rate
• λ = expected years of
  revenue lost, computed as the probability of departure times average
  remaining tenure
• R = regulatory and remediation
  cost

The Φ θ λ term typically dominates everything else, and it is the term people frequently forget to model. For example, a $4 million fraudulent disbursement from a $600 million relationship is not a $4 million event. If that relationship leaves (and the probability is not small), you have lost much more than the $4 million.

Every input here is a range rather than a number, which is precisely the condition that calls for the use of Monte Carlo simulations.

Monte Carlo simulation: turning point estimates into ranges

What it is, in one paragraph

A Monte Carlo simulation replaces each uncertain input in a formula with a range of plausible values, then evaluates the formula thousands of times, drawing a random value for each input every time. The result is not a single number but a distribution. It provides a picture of every outcome the model considers possible, and how likely each one is. You then report percentiles from that picture (the 10th, 50th, and 90th percentile are conventional) instead of a single figure.

The reason to do this is not mathematical elegance. It is that a single number invites the question “where did that come from?”, which you will lose. A distribution with stated assumptions invites the question “are those the right ranges?”, which is a conversation you can actually win. It moves the argument from your credibility to your inputs, which is where it belongs [2].

Where it fits, and where it does not

Not every part of this framework benefits. Simulate where uncertainty is real and compounding; use a sensitivity table where the unknown is something you could look up.

Model	Simulate?	Why
Class A (Part 1)	Yes; highest value	VA is a product of five uncertain terms. Multiplicative uncertainty compounds in ways intuition does not track.
Tail events (above)	Yes; essential	Every input is elicited judgment with a wide range, and the output is dominated by the upper tail. A point estimate here is close to meaningless.
Build-vs-buy (Part 3)	Yes	The case usually rests on Δ m, months earlier delivered, and software schedules are strongly right-skewed.
Class B (Part 1)	No	Most of the variance sits in η, and η is not random; it is a contractual fact you can go and read. A two-row sensitivity table is more honest and more persuasive.

The correlation trap, and why most fraud simulations are wrong

This is the part to get right, because getting it wrong produces a confident and completely fictitious answer.

Since recall and precision are not independent, they trade off along the model’s ROC curve. The fundamental structure of a detection system dictates that you buy more recall by lowering the score threshold, which admits more alerts, and thereby lowers precision.

If you sample recall (r₁) and precision (π₁) independently, your simulation will generate thousands of scenarios with high recall and high precision simultaneously, outcomes that no threshold setting could ever produce. At the top end, your distribution will be far too optimistic, and your 90th percentile will describe a world that does not exist. The trade-off is structural, not incidental [3], and it is sharpest exactly where fraud lives: at very low base rates [4].

The fix

The fix is to sample the threshold, not recall and precision directly. The threshold is the only dial you actually control. Your model assigns each event a score; you pick a cutoff; anything above it becomes an alert. Lower the cutoff and you catch more fraud but generate more noise. Raise it and your alerts get cleaner while more fraud escapes. Recall and precision are not settings you choose; they are what the cutoff produces.

Shadow scoring gives you the whole menu. Shadow scoring means running the new model silently alongside your live process for a period, usually a quarter. It scores every event and decides nothing: no alert fires, no customer is touched, no analyst sees its output. You simply record what it thought. At the end, you take the cases you know turned out to be fraud and ask what the model would have done with them at each possible cutoff. Because you already know the answers, you can score its performance without ever having acted on it. For every possible cutoff, you then know what recall and precision would have been against your confirmed cases. That is a table: threshold, recall, precision, one row per setting.

Draw a score threshold τ from your uncertainty range, then read both r₁(τ) and π₁(τ) off the empirical ROC curve produced during shadow scoring. One draw, two correlated outputs, no impossible worlds:

One draw, two outputs, guaranteed consistent, because they came from a setting your model actually reached. Sampling them separately can pair a recall from one row with a precision from another, a combination no cutoff could produce. Then let the alert identity A₁ = N p r₁ / π₁ carry that draw through to alert volume automatically, so the analyst cost and friction cost terms inherit the correlation too. This single change is usually the difference between a simulation that informs a decision and one that flatters it.

The area under the ROC curve is the usual way to summarize a detection model, and it is the wrong number for a business case [5]. It averages performance across every possible threshold, including settings you would never run: ones producing half a million alerts a year, or catching a fifth of your fraud. You operate at one setting. Two models with identical AUC can differ by a million dollars in alert-handling cost at the threshold you would actually choose. And AUC has no units: every other term in V_A is events, dollars, or a fraction of fraud caught, while AUC converts to nothing. At the very low base rates fraud operates at, it also flatters almost any model, because the vast pool of legitimate events it classifies correctly swamps the comparison [4]. Report performance at your operating point, and report the operating point.

Recall and precision are not the only pair that move together. Two further couplings are worth handling if your data supports it:

• Loss size and detectability. In most portfolios,
  larger frauds are easier to catch, because they trip
  amount-based thresholds. Drawing L
  independently of r₁ will bias the
  result, and the direction depends on your control mix. If your shadow
  scoring output includes the loss amount of each caught and missed case,
  estimate recall separately by loss band and draw from the band-specific
  curves.
• Incidence and severity. These often move together,
  because the same organized campaign drives both up at once. If you have
  several years of monthly data, check the correlation before assuming
  independence.

Choosing a distribution for each input

For most of these inputs, the range you choose matters more than the shape. If you bound a quantity between a sensible low and high, a Beta, a Triangular, and a PERT will produce percentiles within a few percent of each other. Severity is the exception. Fraud losses have a long right tail (a few very large events among many small ones), and a Normal distribution cannot represent that. Choose it and your P90 will be far too low, which is precisely the number the analysis exists to produce.

Parameter	Distribution	Reasoning
L: loss severity	Lognormal	Fraud losses are strongly right-skewed, consistent with the loss distribution approach used in operational risk [6, 1]. This is the one choice that genuinely matters; a Normal distribution here will understate your tail badly

Parameter	Distribution	Reasoning
p: fraud incidence	Beta, or Negative Binomial on the attempt count	Bounded between 0 and 1; the count version handles the clustering you get from organized campaigns
ρ: recovery rate	Beta	Bounded, and you usually have several years of data to fit it
τ → (r1, π1)	Empirical, from the ROC curve	See the correlation discussion above
f: cost per false positive	Triangular or PERT	You typically have a low, likely, and high estimate rather than a dataset
Δ qs: interdiction improvement	PERT, from expert elicitation	Designed for exactly this: three-point estimates from people, not data
Ps: scenario probability	Beta, or a discrete set of expert views	Very wide; resist the urge to narrow it artificially
λ: years of revenue lost	Beta for departure probability × tenure estimate	Decompose it rather than estimating the product directly
Δ m: months earlier delivered	Lognormal or shifted Gamma	Software schedules slip right, never left

General guidance on distribution choice and correlation handling in models of this kind is in [7]. On sample size: ten thousand draws is more than sufficient for percentile estimates at this precision, runs in under a second, and nobody will ever ask you to justify it. There is no benefit to a million.

PERT is worth knowing if you do not already. It takes a minimum, most likely, and maximum from a subject matter expert and produces a smooth distribution weighted toward the most likely value. It comes out of program evaluation work from the 1950s [8] and exists because three-point elicitation is how humans actually estimate things. It is the natural choice everywhere your input came from a conversation rather than a query.

The correlation structure, in full

The simulation does not impose a correlation matrix. It does something simpler and more defensible: it draws a small number of independent quantities and lets the model’s own arithmetic generate every dependency from there. This matters because a correlation matrix is a guess about how variables move together, while a shared driver is a statement about why they do.

What is drawn independently. Six things: the operating threshold τ, the fraud incidence rate p, the mean loss L, the recovery rate ρ, the cost of a customer-facing false positive f, and the post-tool handling time t₁. Plus the tail-scenario inputs, handled separately below. That is the entire source of randomness.

What is derived, and therefore correlated by construction. Everything else falls out:

One threshold draw fixes recall and precision together, because both are read off the same row of the shadow-scoring curve. Those two then determine alert volume through the identity from Part 1. Alert volume determines investigator hours, false-positive count, the efficiency project’s value, the cross term, and the inference bill. A single draw of τ therefore propagates into six downstream quantities, all consistently.

The same is true of p, which enters both the value term and, through A₁, the cost term. A draw that gives you more fraud gives you more of both, automatically. Nothing in the code enforces this; it is a consequence of p appearing in both expressions.

The one correlation that is imposed rather than derived. Loss severity and detectability are related in most portfolios: larger frauds trip amount thresholds and are easier to catch. Because the model uses a single mean L and a single recall r₁, this relationship is not represented, and the direction of the resulting bias depends on your control mix. Handling it properly means estimating recall separately by loss band and drawing from band-specific curves. This is worth doing if your shadow-scoring output records the amount of each caught and missed case, and worth flagging as an omission if it does not.

The tail block is independent of the retail block. Scenario probability, severity components, and interdiction improvement are drawn without reference to p, L, or τ. This is a modeling choice, not a finding: a coordinated campaign could plausibly raise everyday incidence and tail probability at once. With three or four confirmed tail events in the historical record, there is no basis for estimating that dependence, so the model treats the two as separate and the assumption is stated rather than hidden.

What is deliberately not simulated at all. The realization rate η appears as a sensitivity table, because most of its variation is contractual fact rather than uncertainty. Alert volume N, analyst cost w, and baseline recall and precision are held fixed, on the grounds that they are measured rather than estimated, though baseline performance is measured with error like anything else, and treating it as exact makes the output modestly overconfident.

The running example, part two: simulated

The recordkeeping platform from Part 1, now with ranges on the inputs. The threshold is sampled from the shadow-scoring curve so recall and precision move together; loss, incidence, recovery, friction cost, and handling time get the distributions above; ten thousand draws.

Parameter	Point value	Simulated range
p: fraud incidence	0.0004	Beta, 0.00028 to 0.00055
L: average loss	$18,000	Lognormal, median $15,400
ρ: recovery rate	0.35	Beta, 0.26 to 0.44
r1, π1	0.74, 0.050	via threshold draw
t1: hours per alert	0.42	PERT, 0.38 to 0.46
f: cost per false positive	$30	PERT, $18 to $52
η: savings realization rate	0.55	sensitivity table, not simulated

Class A, simulated

Percentile	Net Class A value
P10	$2,410,000
P50	$4,980,000
P90	$8,140,000
Probability of negative net value	2.1%

Note that the P50 of $4.98 million sits below the point estimate of $5.21 million from Part 1. That gap is the lognormal severity distribution asserting itself: average loss exceeds median loss, so a calculation built on the average lands above the middle of the distribution. Explain this before someone else calls it an inconsistency.

Ranking the inputs by their contribution to output variance: L first by a wide margin, then Δ r, then f, then ρ, then p. That ordering is the work plan. Loss severity and the recall lift are worth another month of measurement; incidence is not.

The tail scenario, simulated

One scenario: an advisor credential compromise reaching multiple custodied relationships.

Component	Point value	Simulated range
Ps: annualized probability	0.15	Beta, 0.06 to 0.28
D: direct operational loss	$500,000	Lognormal
M: client make-whole	$4,000,000	Lognormal, wide
Φ: AUM in affected relationships	$600,000,000	Lognormal, $200M to $1.4B
θ: revenue yield	0.35%	fixed
λ: years of revenue lost	3.5	Beta departure × tenure
R: regulatory and remediation	$2,000,000	PERT, $0.5M to $6M
Δ qs: interdiction improvement	0.30	PERT, 0.12 to 0.48

The revenue attrition term is $7.35 million, larger than the direct loss and the make-whole payment combined. That is the finding, and it is the one that gets a risk committee’s attention.

Simulated, the severity distribution runs from a P10 near $6.2 million to a P90 above $27 million, and the annualized value from roughly $180,000 to $1.42 million. Report the severity range as well as the expected value: the number the committee needs is not the average, it is what a bad day costs.

Two ways this goes wrong

Recall and precision sampled independently. This produces a simulation full of worlds that no threshold could create, and a P90 that is pure fiction. Sample the threshold and read both metrics off the curve.

Pilot results extrapolated linearly. A triage pilot on 500 hand-picked cases with a senior analyst watching tells you very little about the full alert distribution worked by a first-tier queue. Discount the pilot result and state the discount you applied.

Where this leaves you

You now have a distribution of benefit rather than a number: percentiles for the everyday fraud model, a severity range for the bad day, and a ranking that tells you which inputs deserve more measurement. What you do not yet have is a decision. Part 3 subtracts the program costs (implementation, governance, human review, and the inference bill that quietly kills these projects), sets the whole thing against the honest baseline of what your own engineers would build anyway, and shows which numbers to lead with in the room.

References

Citations are keyed by number to the bracketed markers in the text.

[1] Basel Committee on Banking Supervision. Operational Risk: Supervisory Guidelines for the Advanced Measurement Approaches. June 2011.

[2] Hubbard, D. W. (2014). How to Measure Anything: Finding the Value of Intangibles in Business, 3rd ed. Wiley.

[3] Davis, J., & Goadrich, M. (2006). “The Relationship Between Precision-Recall and ROC Curves.” ICML ’06.

[4] Saito, T., & Rehmsmeier, M. (2015). “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.” PLoS ONE 10(3).

[5] Hand, D. J. (2009). “Measuring classifier performance: a coherent alternative to the area under the ROC curve.” Machine Learning 77(1).

[6] Frachot, A., Georges, P., & Roncalli, T. (2001). “Loss Distribution Approach for Operational Risk.” Groupe de Recherche Opérationnelle, Crédit Lyonnais.

[7] Vose, D. (2008). Risk Analysis: A Quantitative Guide, 3rd ed. Wiley.

[8] Malcolm, D. G., Roseboom, J. H., Clark, C. E., & Fazar, W. (1959). “Application of a Technique for Research and Development Program Evaluation.” Operations Research 7(5).