Real-Time Fraud Detection & Anomaly Monitoring

Payment fraud is a problem where every stakeholder wants a different thing and all of their wants are legitimate. The finance team wants chargebacks down. The product team wants the checkout flow to feel invisible. Customer support wants fewer angry emails from customers whose legitimate cards got declined. The fraud operations team wants tooling that lets them investigate suspicious cases without spending twenty minutes reconstructing what the system saw. Compliance wants an audit trail that will survive a regulator's questions. And underneath all of that, the platform has to score transactions in real time, in the critical path of the checkout, without adding perceivable latency. My job was to build the system that made all of those groups less unhappy at the same time.

The baseline when I started was a rules engine written over five years by a rotating cast of analysts. It had grown to about 340 rules, most of them useful once and never retired, some of them contradictory, and a handful that fired on almost every transaction and were therefore ignored by the review team. Chargeback rate was climbing, false-positive rate on rule-based blocks was above 20%, and every rule change required a manual review and a deploy. The team was drowning in maintenance, and the fraud loss line on the P&L was starting to be visible from the executive floor.

The company processed roughly 6 million transactions a day across e-commerce and marketplace flows, with a long tail of transaction shapes: card-present, card-not-present, wallet, buy-now-pay-later, marketplace payouts, and a growing volume of cross-border payments. Fraud patterns differed sharply across those shapes. Card-not-present fraud looked like classic account-testing and stolen-card patterns. Marketplace fraud looked more like collusion between buyer and seller accounts. Buy-now-pay-later fraud had a much longer feedback loop because the loss did not materialize for weeks. A single model would not fit all of these, and I did not try to make one.

The latency budget was hard. The checkout team gave me a 40 ms p99 budget for the scoring call, inclusive of feature fetching, model inference, and the round trip. If I blew that budget, the checkout page would slow down measurably and the product team would, correctly, roll me back. That constraint was the shape of the system before anything else was decided.

The constraints were: 40 ms p99 for the scoring call, 99.99% availability on the scoring path because a failure meant blocking a legitimate customer, full explainability on every score for the fraud operations team, no PII leaving the environment, and a hard rule that any model change had to be shadow-tested against at least two weeks of production traffic before it could take live decisions. That last constraint is the one that saved me the most trouble over the following year.

The non-goals were as important. I did not build a general anomaly detector. I did not build a model that tried to catch every fraud pattern ever documented. I built a system that caught the specific fraud patterns we were actually losing money on, in the specific transaction shapes that mattered most, with headroom to add new patterns as they emerged. I also did not build a model that decided on its own whether to block a transaction. The model produced a score and a reason set. A decision layer, driven by risk policy set by the fraud operations lead, turned that score into an action: approve, step up to additional authentication, hold for review, or block. Keeping the model and the policy layer separate meant policy could be changed without a model deploy, which was worth its weight in gold during the first months.

The other non-goal, and the one I had to defend most often, was avoiding deep learning for the initial launch. A gradient-boosted tree on well-engineered features was going to give me most of the lift with a fraction of the operational cost, and it would be far easier to explain to the review team and to compliance. I revisited neural models later for specific sub-problems, but the primary scoring model stayed a boosted tree throughout the first year, and I do not regret it.

The scoring path has three components in the hot path: a feature service, a model service, and a decision service. A transaction arrives at the ingress, which enriches the payload with the authenticated user context and forwards it to the feature service. The feature service fetches a fixed set of features from two backends: an online store built on Redis for sub-millisecond point lookups and a streaming feature layer built on Kafka plus a stream processor for velocity and aggregation features updated in near real time. The feature vector is handed to the model service, which runs a gradient-boosted tree via a compiled runtime for low-latency inference. The score and the feature attributions are passed to the decision service, which applies the current risk policy and returns an action. Total budget: 40 ms p99. Actual measured p99: 27 ms for six months, drifting to 31 ms as feature count grew.

The offline pipelines feed three things. First, the training pipeline, which pulls labeled historical transactions and rebuilds the model weekly. Second, the feature store's batch layer, which computes long-horizon features like a customer's 30-day spending profile and writes them to the online store overnight. Third, the label pipeline, which reconciles chargebacks and confirmed fraud reports against the transactions that produced them, with a delay of up to 120 days to account for chargeback windows. Label delay is a fact of life in fraud, and I designed around it rather than trying to shorten it.

The graph layer was the piece I added in month three, and it changed the quality of the system meaningfully. Each transaction touches several entities: a user, a device, a card, an IP, a shipping address, and often a merchant on the marketplace side. Those entities form a graph that persists across transactions. A user who has transacted from three devices, each of which is also used by five other users, is a very different risk profile from a user who has transacted from one device that is theirs alone. I built a lightweight graph store, updated in near real time, that exposed a fixed set of graph features to the model: entity fan-out, shared-device counts, shortest path to a known-fraud entity, community membership from a periodic community detection run. Those features moved model performance on account-testing and mule patterns by a large margin, and they gave the review team a visualization they could actually use during investigations.

Everything is instrumented. Every scoring call emits a structured event with the input features, the model version, the score, the reason set, the policy version, and the resulting action. Those events are the raw material for evaluation, model retraining, and analyst tooling. Without that event stream, the rest of the system is a black box you cannot debug or improve. With it, everything is a query away.

Features do the heavy lifting in this kind of system, and I treated feature freshness as an SLO on par with scoring latency. The online feature store served point features in under two milliseconds. The streaming feature layer produced velocity features — transactions per user per minute, distinct cards per device per hour, distinct shipping addresses per card per day — with a target end-to-end latency of under three seconds from event to feature. When that target slipped, and it did during a Kafka rebalance incident I will describe later, the model degraded quietly, because it was scoring on stale velocity features and missing patterns that only appear at short time horizons.

The feature catalog was a living document. Every feature had an owner, a definition, a source, a freshness target, a training-serving skew check, and a shadow test showing its incremental lift on the model. Features were retired as aggressively as they were added. A feature that no longer contributed to model performance was a liability, because it added latency, added surface area for drift, and added confusion for the review team when they looked at attributions. I ran a quarterly feature pruning that typically removed a fifth of the catalog.

Training-serving skew was the single most common source of quiet model degradation, and I fought it with three layers of defense. First, a shared feature-definition library used by both the batch training pipeline and the online feature service, so a change to a feature's computation had to be made in one place. Second, a nightly job that sampled live scoring events and recomputed the features from raw data, comparing against the served values and alerting on any drift beyond a tolerance. Third, a monthly retraining pass that used features as they were served in production, not as they were computed offline, so the model learned the actual distribution it would score against. Skew still crept in occasionally, but it was caught within a day rather than a quarter.

The primary scoring model was a gradient-boosted tree ensemble trained on roughly six months of labeled transactions with careful stratification by transaction shape and confirmed-fraud outcome. I trained separate models per major transaction shape — card-not-present, marketplace, buy-now-pay-later — because their feature distributions and fraud patterns were different enough that a single model underperformed all of them. A small router at the top of the model service picked the right sub-model per transaction. The router itself was a two-line lookup, not a model, because a router that can be wrong is a source of bugs I did not want.

I chose boosted trees over neural networks for four reasons, and I want to name them because the reflex to reach for deep learning is strong and often wrong for this class of problem. First, tabular data with mixed categorical and numeric features, moderate dimensionality, and heavy class imbalance is exactly the regime where boosted trees perform very well out of the box. Second, feature attributions from a tree ensemble are trustworthy and cheap to compute at inference time, which the review team needed. Third, training a tree ensemble to convergence takes minutes on commodity hardware and inference is measured in microseconds per record, which fit my latency budget with room to spare. Fourth, and least glamorous, boosted trees are boring, which meant I could hire and train analysts to work with them without a research-scientist bottleneck.

I did revisit neural models for two specific sub-problems: sequence modeling of a user's recent transaction history, and embedding of high-cardinality categorical features like merchant identifiers. Both showed lift in offline evaluation, but only the categorical embedding survived into production, as a set of precomputed embedding features that fed into the tree ensemble. The sequence model was more accurate on paper but added latency and operational complexity that were not worth the marginal lift on our traffic. I keep a note in my head that on a different transaction volume and pattern, the trade might flip.

Class imbalance was extreme — fraud rate was under a percent — and I handled it with a combination of class weighting during training, careful threshold tuning against a business cost function rather than raw accuracy, and a deliberate choice to optimize for area under the precision-recall curve rather than area under the ROC curve. The cost function encoded the actual dollar cost of a false negative and the estimated cost of a false positive, including the customer-support impact and the churn risk of blocking a legitimate customer. That cost function lived in a config file that the fraud operations lead could edit without touching model code, and it was edited several times a year as the mix of transaction shapes shifted.

The decision layer took the model score, the reason set, and a set of policy rules and produced an action. Policy rules were a small, human-readable DSL rather than a code deploy. A rule looked like 'if score above 0.85 and transaction amount above 500 and cross-border, hold for review.' The policy lived in a versioned config, was reviewed like code, and could be rolled forward or back independently of the model. Splitting policy from model was the single decision that most reduced the operational tension between the fraud team and the engineering team, because it gave the fraud team a lever they controlled without waiting on a deploy queue.

The review path was designed before the model was deployed, because I had watched enough fraud programs collapse under the weight of review queues to know that the queue is the actual product. Every transaction sent to review appeared in an analyst UI with the score, the reason set, the graph neighborhood, the customer's transaction history, and a set of one-click actions: approve, decline, escalate, mark as confirmed fraud. Every action was logged with the analyst identity, timestamp, and free-text notes. Confirmed fraud outcomes fed the label pipeline. Approve and decline outcomes fed a shadow evaluation of the model's precision at the review-tier threshold. The review UI was where the fraud team spent most of their day, and I treated it as a first-class product with its own roadmap.

Step-up authentication was the middle option that I initially underestimated and that turned out to be the highest-leverage action in the policy set. For transactions with a moderately elevated score, redirecting the customer through an additional verification step — a 3DS challenge, a one-time code — resolved most of them either into a confirmed legitimate transaction or a silently abandoned session. The abandon rate on step-ups was informative in itself: fraudsters abandon at much higher rates than legitimate customers when asked for additional verification. I treated the step-up path as a probe as well as a control, and its output fed the model as a feature over time.

Model evaluation ran on three timescales. Offline, on a held-out slice of historical transactions with labels reconciled to at least 60 days post-transaction, I measured precision at each decision threshold, recall against confirmed fraud, and area under the precision-recall curve. Online, in shadow mode, I ran candidate models on live traffic without acting on their scores and compared their decisions to the live model's for two weeks before any promotion. And in production, I tracked score distributions, action distributions, and downstream chargeback rate on a rolling 30- and 90-day window.

Drift monitoring watched three signals. Feature-value drift compared the distribution of each feature this week to its distribution during training. Score drift compared the distribution of model scores this week to the distribution from the shadow evaluation of the current model. Outcome drift compared realized chargeback rate at each score band to the expected rate at that band. Any signal beyond its tolerance produced an alert. In practice, feature drift fired the most often, usually because a marketing campaign or a merchant onboarding had shifted the traffic mix.

Retraining was scheduled weekly and triggered on drift outside of the schedule. Every retrain went through the same two-week shadow evaluation before it took live decisions, with no exceptions, including the retrain that happened during the first big incident when I was very tempted to skip the shadow window. I did not, and I am glad, because that retrain turned out to have a subtle regression on marketplace transactions that only appeared on live traffic and would have made the incident much worse if it had been deployed on the same day.

The first significant incident was a Kafka rebalance that stretched the streaming feature layer's end-to-end latency from three seconds to just over two minutes for about forty minutes on a Sunday morning. Velocity features went stale. The model, unaware, kept scoring transactions using the stale features. Fraud that relied on short-horizon patterns — account testing, card enumeration — slipped through at higher rates than usual for that window. The fix was two-part: an availability probe on feature freshness that would trip the decision layer into a more conservative fallback policy if freshness slipped past a threshold, and a set of alarms on stream-processor lag that would page rather than email. Both existed in some form before the incident. Neither was tight enough. The lesson was that feature freshness is a first-class SLO and needs SLO tooling, not just a dashboard.

The second incident was a feature-definition drift I mentioned earlier. A batch job that computed a customer's 30-day spending profile started using a slightly different aggregation window because of a timezone change in an upstream data source. The online feature was correct. The training feature was wrong. The model retrained on the wrong feature and its precision on high-value transactions degraded by two points before the skew check caught it. The skew check now runs hourly instead of daily on the top decile of features by importance, and the on-call playbook includes 'check the skew dashboard' as a first step for any unexplained precision drop.

The third incident was closer to a design lesson than an operational failure. A new marketplace category was onboarded whose transaction shape did not match any of my per-shape sub-models well. Precision on that category was significantly worse than average. The router quietly sent those transactions to the closest sub-model, which was not close enough. I added a fallback tier: transactions the router cannot confidently place go to a generalist model trained on all shapes, with a slightly more conservative policy default. The generalist model is worse than the specialists on their home shapes, but it is much better than the wrong specialist on an out-of-distribution shape.

The fourth was a small but instructive false-positive spike from a rule change. The fraud team, using their new authority to change policy without a deploy, tightened a threshold to react to a fraud pattern they were seeing. The tighter threshold caught more fraud but also blocked a specific segment of legitimate customers at a high rate. Customer support noticed within hours. The policy change was rolled back within a day. The lesson was that policy changes needed the same shadow-evaluation discipline as model changes. I added a required simulation step to the policy tooling: any policy change had to be simulated against the last two weeks of scored traffic before it could go live, showing expected impact on approve, review, and block rates by segment. The fraud team took to it quickly, because it gave them confidence to make bolder policy moves.

Chargeback rate dropped by 38% in the first quarter after the new system took primary decision authority, and by another 12% over the following two quarters as the model and policy matured. Fraud loss in absolute dollars dropped by more than half over the first year, against a background of steadily growing transaction volume. False-positive rate on blocked transactions dropped from above 20% under the legacy rules engine to under 6% by the end of the first year. Step-up authentication took over the middle tier and resolved most moderately risky transactions without either a block or a full manual review.

Review queue volume dropped by roughly 40% even as transaction volume grew, because the decision layer was catching clear-cut cases without human review and routing only genuinely ambiguous ones to the queue. Median time to disposition on reviewed cases dropped from around eleven minutes to under four, driven by the analyst UI showing the graph neighborhood and the reason set inline. The fraud team took on new categories of investigative work they had never had time for before, including proactive account-takeover monitoring.

The engineering team retired the old rules engine after a six-month coexistence period. About twenty of the original 340 rules were preserved as policy overrides in the new decision layer, and the rest were deleted. The maintenance burden of the fraud stack dropped substantially, and the on-call rotation for the scoring path went from three to one page per week on average.

I would build the shadow evaluation harness before I built the model. Every hard decision downstream was easier because that harness existed, and the version I first shipped was a rough one that took months to mature. Starting with a well-designed shadow harness would have paid back immediately.

I would introduce the graph layer earlier. I waited three months because I wanted to prove the tabular model first, and the tabular model was fine, but the graph features moved the needle on the patterns that were costing the most money. If I were starting over, I would prototype the graph store in parallel with the initial model.

I would treat the analyst UI as a launch-blocking product, not a fast-follow. The fraud team's willingness to trust the model was proportional to how well the UI explained the model's decisions. In the first weeks, when the UI was thin, the team second-guessed the model. Once the UI matured, they trusted it enough to let it take more actions autonomously. The UI is the interface between the model and the humans who own the outcome, and its quality bounds the model's real-world impact.

And I would document the cost function earlier and more publicly. The cost function encodes the business's actual preferences over false positives and false negatives, and every threshold decision references it. Making it a visible, versioned, discussed artifact — not a config file only the model owner touched — would have prevented several rounds of misaligned expectations between the fraud team, the customer-support team, and the finance team.