Ask a fraud executive at a financial institution what a generative AI project is worth and you will usually get one of two answers: a very confident number that will more than likely fall apart under scrutiny, or an honest “I can’t quantify it.”
Both responses come from the same place. If you are reading this post, I don’t have to tell you that these days there is a lot of energy around understanding the return on investment on Gen AI projects. Fraud is an even more specialized case.
Return on investment is a simple ratio, (V – C)/C, which asks a simple question: is what we get out of this worth what we put in?
You calculate it by subtracting the cost from the value and dividing by the cost. Fraud projects make that question a little harder to answer, because most projects earn their keep by bringing money in, while a fraud project earns its keep by stopping money from going out. So nothing arrives. A theft that was prevented leaves no record anywhere (no invoice, no deposit, no line in the accounting ledgers) so the entire value of the work is something you have to estimate rather than count. Everything that follows is about how to make that estimate defensible.
The benefit of a fraud project is difficult to measure, for three reasons that compound each other.
The thing you are trying to measure is an absence. A fraud you prevented leaves no trace. You are always estimating a counterfactual (what would have happened in a world where you did nothing), and counterfactuals are, by definition, unobservable. This is the central difficulty in valuing any preventive control [1, 2].
The baseline will not hold still. Fraud losses move on their own. This is the same problem we see with building fraud prevention and detection models. A ring gets arrested. Senior management decides to shift your customer mix. Or some unrelated control gets tightened upstream and quietly changes your numbers. Any measured change in loss is the sum of your project and everything else happening in the world, and the everything-else term is often larger.
Two completely different loss populations are usually averaged together. A retirement platform experiences hundreds or thousands of small account takeovers a year and, if it is unlucky, one catastrophic event a decade. These obey different statistics and require different math. Blending them produces a number that probably describes neither.
This series lays out a framework that handles all three. It is written for the person who has to build the business case and defend it in front of Finance, and assumes no statistical background.
How the series is organized
Both halves of the ROI ratio are built up across the series rather than in one place. Costs come in two kinds. Operating costs scale with what the system does: set the detection threshold lower and you get more alerts, more investigator hours, and more friction on legitimate customers. They are inseparable from the recall decision that created them, so they are modeled alongside the value they buy, in this article. Program costs are what you pay to have the capability at all: building it, governing it, running inference, keeping the evaluation set current. These do not move when you adjust the threshold, and they arrive in Part 3.
| Installment | What it builds |
|---|---|
| Part 1 (this article) | The value model: detection projects, efficiency projects, and the term that appears when you run both, with operating costs attached |
| Part 2 | What to do when the inputs are ranges: tail events, expert elicitation, and Monte Carlo simulation |
| Part 3 | The decision: program costs, the build-versus-buy baseline, and presenting the result |
A single worked example, a large retirement recordkeeping platform evaluating two projects, runs through all three parts. This article computes its point estimates; Part 2 turns them into ranges; Part 3 subtracts the costs and turns the result into a decision.
The first move: sort your projects into two classes
The first thing to understand here is that not all projects in an enterprise fraud world should be viewed under the same lens. So we must explicitly account for this. Specifically, in fraud there are two main classes, and they need different value models, different measurement approaches, and different skepticism.
Class A: Detection projects. These are projects which hopefully will change what you catch. Examples of these projects are: a better scoring model, a new signal source, a graph-based linkage layer, an LLM that reads unstructured case notes and surfaces a connection nobody saw. The value is therefore avoided loss. The hard part is proving causation against a moving baseline.
Class B: Efficiency projects. These are projects which will hopefully change what it costs to operate. Examples of these projects are: an assistant that drafts case summaries, a tool that pre-populates regulatory filings, a copilot that answers investigator questions about an entity. The value is time saved (released capacity). Causation is much easier to establish, since you can measure the same analysts before and after, but converting released hours into actual money is the hard part.
These classes of projects do not compete for credit, because they are measured on different numbers. A detection project claims loss reduction; an efficiency project claims cost per case. Nobody has to arbitrate between them.
They are not completely independent, because they collide in the alert queue, which sits between them. This will be addressed later in this article.
Notation
Everything below reuses the same symbols. Define them once and the rest is arithmetic.
| Symbol | Meaning | Typical unit |
|---|---|---|
| N | Number of events in the measurement period that are candidates for fraud | events per year |
| p | Fraud incidence rate: the fraction of those events that are fraud attempts | decimal fraction |
| L | Average loss the firm bears per fraud attempt that gets through | dollars |
| r0, r1 | Recall before and after the project: the fraction of fraud attempts you catch | decimal fraction |
| ρ | Recovery rate: the fraction of a loss you eventually get back through clawback, chargeback, or insurance | decimal fraction |
| A0, A1 | Alert volume before and after | alerts per year |
| π0, π1 | Precision before and after: the fraction of alerts that turn out to be real fraud | decimal fraction |
| t0, t1 | Average analyst time to work one alert, before and after | hours |
| w | Fully loaded analyst cost | dollars per hour |
| f | Cost of one false positive that reaches the customer | dollars |
| η | Savings realization rate: the fraction of saved hours that actually becomes money | decimal fraction |
Two terms deserve a plain-English gloss before we go further, because they are the two most misunderstood quantities in fraud analytics.
Recall answers: of all the fraud that was attempted against us, what share did we catch? If 1,000 fraud attempts were made and you stopped 620 of them, your recall is 0.62.
Precision answers: of all the alerts we generated, what share were actually fraud? If you produced 50,000 alerts and 3,000 were real, your precision is 0.06, six percent. That number looks alarming to outsiders and is entirely normal in fraud operations.
These two are linked by an identity that will do a lot of work for us:
In words: your alert volume equals the number of frauds you catch, divided by the share of alerts that are fraud. This identity is a restatement of the formal relationship between precision-recall and ROC space [3]. Note it, because it means you cannot change recall without changing alert volume, and alert volume is what your operations team lives and dies by. Any detection project that promises more recall is implicitly promising your investigators more work, whether or not the business case says so.
Class A: Detection projects
The value side
Ideally, a successfully implemented detection project turns losses you would have absorbed into losses you avoid. The gross value of losses in a period is:
N = total events
p = the share of total events that are fraud attempts
L = what each fraud attempt costs you
(r1 – r0) = number of additional frauds caught
(1 – ρ) = portion that was never coming back anyway
That last factor, (1 – ρ), is tricky, and omitting it can inflate the benefit by half. Cost-benefit analysis calls this the incrementality rule: compare what happens with the project against what would have happened without it. The recovery team was getting that money back either way.
Let’s suppose you already recover 35% of fraud losses after the fact. Catching a fraud before the money leaves does not save you the full loss. It saves you the 65% you were never going to recover. The other 35% you were getting back regardless; the new system just saved you the work of chasing it.
If your recovery process is genuinely improved by earlier detection (and it usually is, since speed matters enormously for ACH returns), you can model ρ as changing too:
This refinement is worth making only if you have actual recovery data by time-to-detection. If you do not, use a single ρ and say so.
The cost side
Detection is not free, and its cost is not the software license. It is the alerts.
where c = t · w is the fully loaded cost of working one alert.
The first term is intuitive: more alerts, more analyst hours. We use the alert identity above to compute A1.
The second term (which is usually missing) is what decides most projects
ROI cases routinely omit the costs their controls impose on other departments, because those costs never appear in the fraud budget. For example, there are costs associated with (a) a step-up authentication challenge, which can lead to a customer leaving, and (b) a frozen retirement distribution, which generates a complaint to the plan sponsor.
At scale this term can be enormous. Consider a large institution with forty million accounts. A change that adds two percentage points of step-up authentication generates 800,000 additional friction events, and authentication strength is itself a supervised expectation, not a free dial to turn [4]. At even $8 apiece in downstream contact and abandonment, that is $6.4 million; potentially more than the entire loss reduction the project was built to deliver. When we think of false positives in terms of dollars and not rates it hits a little differently.
The measurement problem, and what to do about it
You now have a formula. The difficulty is that r1, the recall you will actually achieve in production, is the one number you cannot look up. And you cannot infer it by watching your loss numbers fall, because the baseline drifts.
Holdout. The foolproof method is to have a holdout: route a random slice of events, typically 5%, through the old process, and compare. Randomization means the two groups differ only by the treatment, so any difference in outcome is caused by the treatment. This approach gives you clean causal attribution, and the practical design pitfalls are well catalogued [5]. Unfortunately, for a fraud project (especially a detection-related project) to knowingly under-protect a group of clients may be unethical, so this option may be off the table for most organizations.
Shadow scoring. Run the new model silently alongside the old one for a predetermined time. It scores everything; it decides nothing. Then compare what each would have caught on the confirmed cases. This gives you a defensible estimate of Δ r with no customer impact whatsoever. It also produces the ROC curve, which Part 2 will put to heavy use.
Staggered rollout. Deploy channel by channel or segment by segment, and compare the change in each group against the groups that have not received it yet. This is the difference-in-differences approach [6], and it controls for the drifting baseline as long as the drift affects all segments similarly.
Interleaving, for ranking changes. If your GenAI component reorders an investigator queue rather than making decisions, merge the old and new rankings into one list, present it, and track which source contributed each case an investigator confirmed as fraud. This is dramatically more sensitive than a split test at the same sample size, because every investigator sees both systems. The technique and its validation come from search ranking [7, 8], but the argument transfers directly to a scored queue.
Class B: Efficiency projects
The value side
An efficiency project releases capacity. Its gross value is:
A = alert volume
t0 – t1 = hours saved per alert
w = loaded hourly cost
η = savings realization rate
The first three factors are straightforward and everyone includes them. The fourth is where honest business cases separate from optimistic ones.
The savings realization rate is the whole argument
The savings realization rate, written η, is the fraction of released hours that becomes hard savings rather than soft.
An efficiency tool produces hours. A business case has to produce dollars. Hours only cross that gap through some specific mechanism: an invoice that shrinks, a contract renegotiated at renewal, a budgeted hire that does not happen. Where no such mechanism exists, the hours return to the pool as slack, and η is near zero however real the time saving was.
Setting η to one, which most business cases do by omission rather than by argument, asserts that every minute saved is a minute billed. Name your mechanism instead:
| Mechanism | Typical η | Why |
|---|---|---|
| Outsourced contact center billed per contact or per minute | ~1.0 | Fewer minutes, smaller invoice, immediately |
| Contract or BPO investigator capacity | 0.6 to 0.9 | Converts at contract renewal, subject to minimums |
| Avoided hire against an approved headcount plan | ~1.0 for the deferred heads | Real, but capped at the number of heads actually in the plan |
| Existing salaried staff, no headcount change | 0.0 to 0.3 | Hours return to the pool as slack unless something absorbs them |
| Backlog reduction or improved SLA | Not a cost saving | Real value, but it belongs in a different column (see below) |
That bottom row causes more arguments than any other. If your team is running a persistent backlog, an efficiency tool will not reduce cost; it will increase throughput at constant cost. That is genuinely valuable (faster case closure means shorter fraud windows and better recovery rates), but it is detection value, not cost savings, and it should be reported in the Class A column where it can be measured properly. Claiming it as headcount reduction while the backlog persists is the single most common way a fraud business case loses credibility.
The practical rule: express the benefit in units Finance can book. “We defer 14 of the 22 investigator hires in the FY27 plan” is a line item. “We save 12,800 analyst hours” is a talking point.
Building t1 honestly: the review tax
When modeling the efficiency gains through Gen AI, the temptation is to just use the generation time as the new handling time. For example, a model drafts a case summary in twelve seconds, so the summary now costs twelve seconds. This may be undercounting the time, and that matters.
It is typical that for anything with regulatory or customer consequence, a human in the loop is a requirement, and therefore a human reviews the output. To model that explicitly we can use the following equation:
Where:
tgen = machine time (negligible)
treview = human verification time
q = acceptance rate, the fraction of outputs a reviewer accepts without material change
trework = time to fix the rest
This raises two interesting points.
First, the acceptance rate matters a lot. In the example of something like case generation, if reviewers stop trusting the output and begin deriving it from scratch, at that point t1 (the average analyst time to work one alert after the tool is in place) can exceed t0 (the time an analyst currently spends working one alert end to end, with no assistance), thereby suggesting your tool has made things worse. This is a real and frequently observed failure mode. Measure q during the pilot and treat it as a go/no-go gate. Field experiments show quality can fall on tasks outside a model’s competence even while speed rises [9], which is exactly the condition a low acceptance rate signals.
Second, for regulatory filings, assume review never goes away. Model the reviewed throughput, not the unreviewed one, indefinitely. Any business case that assumes review tapers off after year one is assuming a regulatory posture no institution has actually adopted.
Measurement
Efficiency projects are much easier to measure than detection projects, and there is no ethical obstacle at all. The comparison group is analysts or cases, not customers.
If your case management system timestamps case open and close, you have t0 without running anything. This may turn out to be a better measure, since it covers the full queue over a long period rather than a few weeks of observed behavior. On the other hand, there are some pitfalls to consider: (a) timestamps measure elapsed time, not effort, so cases left open over a weekend or worked in parallel with others will inflate the figure; in these cases, you must check the distribution for the implausible tail before trusting the mean; and (b) because an average across a mixed queue is not the baseline for a tool that only touches one part of it, you should segment by case type.
Historical data gives you the baseline; it cannot give you t1, the acceptance rate q, or any evidence that the tool caused a change rather than coinciding with one. These measures must be obtained from a pilot (or User Acceptance Test).
Whatever the pilot design, pair every time metric with a quality gate. Speed without one is not a measurement, it is a hazard. Score an independent QA sample with reviewers who do not know which cases received the assistant. A tool that cuts handling time 30% while dropping QA pass rate from 96% to 88% has not saved you anything; it has moved cost from the investigation team to the compliance team, where it will show up later and larger. Measured productivity gains from LLM assistants also concentrate among less experienced staff [10], so a pilot staffed with senior investigators will not tell you what a first-tier queue will do.
The coupling term: where A and B collide
Detection projects and efficiency projects claim different kinds of value since they serve different purposes, so neither can take credit that belongs to the other. But they share the alert queue, and that creates a genuine accounting problem.
Interaction effects are well understood in factorial experimental design, and the same concern appears in cost-benefit analysis as the problem of double-counting overlapping benefits. The cross-term, (A1 – A0)(t0 – t1), which we will derive below, is exactly a two-factor interaction written out algebraically.
To better understand this, let’s look at an example: say a detection project raises alert volume from A0 to A1, and an efficiency project lowers handling time from t0 to t1. Each business case was built independently, using today’s numbers as its baseline.
Total handling hours before either project: A0 t0.
After both projects have been implemented, total handling hours
is: A1 t1.So the true change is:
The problem arises because each case computed its effect holding the other constant:
Add them and subtract from the truth:
where (Δ HA + Δ HB) = the naive sum, what you get by adding two independently written business cases.
The cross term (A1 – A0)(t0 – t1) belongs to neither project. It is the efficiency gain applied to the additional alerts the detection project created: savings that exist only because both projects happened.
The term is always positive. Whether your business cases have missed it or counted it twice depends on how they were written. If both cases were built on today’s baseline, adding them together understates the benefit. This is the default situation, and neither counted the cross term, so the real saving exceeds what the two cases claim between them. Add it as an explicit line rather than leaving it out and hoping someone notices; an understated case can lose a funding decision just as easily as an overstated one loses credibility.
The other case is a little more subtle. It can lead to double counting. The detection team knows its project will add alerts, but argues it can absorb them without new headcount, because the triage assistant is coming and will make each alert cheaper to work. Meanwhile the efficiency team, writing its own case, books that same reduction in cost per alert as a headcount saving. Both are relying on the same hours; as a result, the cross term is exactly how much you have counted twice. Double counting is common and it is exactly what a sharp reviewer looks for. The defense is simple: build both cases on the same explicitly stated baseline, and report the cross term as its own line. One sentence in a footnote prevents an argument in a review meeting.
There is a second coupling worth naming, running the other direction. The cross term describes detection changing what the efficiency tool has to work on, but this coupling is efficiency changing what detection catches. If a triage assistant changes how long analysts spend on cases, it changes what they find. Faster review may mean more cases worked and more fraud caught; it may also mean shallower review and lower recall. Either way, r1 is no longer independent of t1. Remember, r1 is the recall after the project, the share of attempted fraud you catch once the new system is in place. If your efficiency project affects investigative work, then you should measure recall during the efficiency pilot too. Do not assume it held.
The running example, part one: point estimates
A large retirement recordkeeping platform is evaluating two projects: a graph-based detection layer (Class A) and an LLM case-summarization assistant (Class B). This example continues through Parts 2 and 3.
| Parameter | Point value | Source |
|---|---|---|
| N: requests per year | 12,000,000 | Operations |
| p: fraud incidence | 0.0004 | Confirmed case history |
| L: average loss | $18,000 | Loss ledger, 3-year |
| ρ: recovery rate | 0.35 | Recovery operations |
| r0 → r1: recall | 0.62 → 0.74 | Shadow scoring, one quarter |
| π0 → π1: precision | 0.060 → 0.050 | Shadow scoring |
| t0 → t1: hours per alert | 0.60 → 0.42 | Case-level randomized pilot |
| w: loaded analyst cost | $70/hour | Finance |
| f: cost per false positive | $30 | Contact center analysis |
| η: savings realization rate | 0.55 | BPO contract + headcount plan |
Total fraud attempts: 12,000,000 × 0.0004 = 4,800 per year. Total annual exposure: 4,800 × \18{,}000 = \$86.4$ million.
Class A
Alert volumes, from the identity A = Npr/π:
That is 21,440 additional alerts, a 43% increase in queue volume, which is exactly the kind of operational consequence a detection business case tends to leave out.
Investigator cost at c = 0.60 × \70 = \$42$ per alert:
Customer friction: false positives rise from 49,600 × 0.94 = 46,624 to 71,040 × 0.95 = 67,488, an increase of 20,864:
Class B
Applied to the post-detection queue of 71,040 alerts, at 0.18 hours saved each:
Note how much the realization rate costs: gross value is $895,000, but only 55% of it converts. A business case claiming the gross number overstates by $403,000. Rather than one number, show η as a small table of contract outcomes. The base case above assumes BPO renewal plus partial hire deferral; internal-staff-only would put it near 0.30 and full BPO conversion near 0.80.
The cross term
Neither project’s standalone case contains this. Report it as its own line so it is claimed once and only once.
Every figure above is a point estimate pretending to a certainty nobody has. Part 2 is about admitting that honestly.
Four ways this goes wrong
False positives priced as a rate. A rate hides the scale multiplier. At forty million accounts, a two-point increase in friction can cost more than the entire loss reduction. Price it in dollars of downstream contact.
Savings realization rate assumed to be 1. Hours are not dollars. If nobody’s headcount changes and no invoice shrinks, the savings are slack. Name the conversion mechanism or discount the number.
The recovery rate left out. Preventing a loss you were going to recover anyway is worth the recovery cost, not the loss. Multiply by (1 – ρ).
Review assumed to taper. For anything with a regulatory consequence, human review is permanent. Model reviewed throughput from day one.
Where this leaves you
You now have a complete value model: avoided loss for detection, realized capacity for efficiency, and the cross term that belongs to both projects at once, each with its operating costs attached. Every number in it is a single point estimate, and several of the inputs cannot be looked up anywhere: they are judgments about things that have not happened yet. Part 2 is about handling that honestly, covering the rare, severe events that break expected-value math entirely, and the Monte Carlo simulation that turns the whole model from one number into a defensible range.
References
Citations are keyed by number to the bracketed markers in the text.
[1] Hubbard, D. W. (2014). How to Measure Anything: Finding the Value of Intangibles in Business, 3rd ed. Wiley.
[2] Hubbard, D. W., & Seiersen, R. (2016). How to Measure Anything in Cybersecurity Risk. Wiley.
[3] Davis, J., & Goadrich, M. (2006). “The Relationship Between Precision-Recall and ROC Curves.” ICML ’06.
[4] Federal Financial Institutions Examination Council. Authentication and Access to Financial Institution Services and Systems. August 2021.
[5] Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
[6] Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.
[7] Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012). “Large-Scale Validation and Analysis of Interleaved Search Evaluation.” ACM Transactions on Information Systems 30(1).
[8] Radlinski, F., Kurup, M., & Joachims, T. (2008). “How Does Clickthrough Data Reflect Retrieval Quality?” CIKM ’08.
[9] Dell’Acqua, F., McFowland, E., Mollick, E., et al. (2023). “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality.” Harvard Business School Working Paper 24-013.
[10] Brynjolfsson, E., Li, D., & Raymond, L. (2023). “Generative AI at Work.” NBER Working Paper 31161.
2 thoughts on “Estimating the Return on GenAI Projects in Enterprise Fraud Management, Part 1 of 3: The Value Model”