Build a Real-Time Fraud Detection Model with Python and Kafka

At 2:14 a.m. a card is charged €3.40 at a petrol station. Nine minutes later the same card spends €1,900 on electronics two hundred miles away. A batch model notices the pattern when it retrains the next morning. A streaming model flags it while the second transaction is still being authorised, and that timing is the whole point.

What follows is a working path to that second system: a Python service that reads transactions from Kafka, keeps a rolling window of history per card, scores every event with an isolation forest, and publishes suspicious ones to an alerts topic. It fits in a few hundred lines, and the structure holds up far beyond a tutorial.

The Pipeline in Four Pieces

Split the problem before writing code, so each piece can be tested and replaced on its own.

• Producer — generates or forwards transaction events into a transactions.raw topic.
• Feature stage — consumes raw events, maintains per-card rolling state, emits enriched records.
• Model — an isolation forest trained offline and loaded by the scoring service.
• Alert stage — applies a threshold to each score and writes flagged transactions to fraud.alerts.

Locally, one consumer can handle features and scoring together. The separation pays off when you want to replay a day of raw traffic through a new model without touching feature logic.

Setting Up Kafka and the Python Side

Run a single-broker Kafka with Docker Compose; one broker is enough to develop against. Then install the client and modelling libraries: confluent-kafka or kafka-python, scikit-learn, numpy, pandas, and joblib. Consumer group rebalancing, offsets and partition ordering are behaviours worth learning on a laptop rather than in production.

Choose your partition key carefully

Publish with card_id as the message key. Kafka hashes keys to partitions and preserves order within a partition, so every event for one card arrives in sequence — exactly what a rolling window needs. Key by transaction ID instead, and two events from the same card can land on different partitions and be processed out of order. Your "seconds since last transaction" feature becomes noise.

Producing a Realistic Stream

A useful simulator mixes ordinary behaviour with occasional bursts. Each event carries a card ID, timestamp, amount, merchant category, country code and a channel flag such as card-present or online. Add a few scripted anomalies — repeated small amounts, a sudden jump in value, several countries within minutes — so you can watch the model react.

Two producer settings are worth forming as habits: enable idempotence to avoid duplicates on retries, and set a small linger.ms so records go out in batches rather than one at a time.

Features That Actually Move the Needle

The forest is only as good as the numbers you feed it. Describe deviation from a card's own habits, not just absolute amounts, because €900 is unremarkable for one customer and alarming for another.

1. Log-transformed amount, to compress the long tail.
2. Transaction count for the card in the last 5 and 60 minutes.
3. Sum of amounts in the same windows.
4. Seconds since the previous transaction.
5. Whether country or merchant category differs from the card's recent history.
6. Hour of day, encoded cyclically.

Keep window state in a dictionary keyed by card ID, holding a bounded deque of recent events. Trim anything older than your longest window on every update. When one process is no longer enough, move that state to Redis and keep the same interface.

Compute features identically offline and online. Train on values produced by the same function the consumer calls. Reimplementing the logic in pandas for training is the most common reason a model looks fine in a notebook and behaves oddly in the stream.

Training the Isolation Forest

Isolation forest is unsupervised, which suits fraud work because labels arrive late and rarely. Random splits isolate points, and anomalies end up isolated after fewer splits because they sit away from the dense mass of normal traffic.

Fit on historical features that are mostly legitimate. You don't need to include known fraud; a random sample of normal traffic works. Set n_estimators between 100 and 300, leave max_samples at its default unless your data is small, and treat contamination as a rough expectation rather than a precise rate. Trees don't need feature scaling, though centring inputs keeps things tidy if you later swap in a linear model.

Use score_samples for continuous scores; lower means more anomalous. Then decide where to cut. With a labelled week of history, plot scores against labels and pick a threshold that keeps false positives at a level your review team can absorb. Persist the fitted model, the feature list and the threshold together — a threshold without its model means nothing.

Scoring Every Transaction

The consumer loop is short: poll, compute features, call the model, compare to the threshold, act.

• Load the model once at startup, never per message.
• Time the scoring call and track a rolling p95; a few milliseconds matters at volume.
• On a flag, publish to fraud.alerts with the score, the contributing features and the raw event, so a reviewer has context.
• Commit offsets after processing, and route malformed payloads to a dead-letter topic instead of crashing the loop.

Resist the urge to block the transaction inside the consumer. Score, publish, and let a downstream service decide. That keeps the model replaceable and your response time predictable.

What Breaks After Launch

Three things, usually. Behaviour drifts — new merchants, new countries, seasonal spending — so track the score distribution weekly and retrain on a rolling window. Feedback loops appear: a blocked card never produces a confirmed fraud, so training data slowly fills with whatever the model already approves. Sample and review some approved transactions to break that

Photo: Tumisu / Pixabay