DataKite Banking LLM V3: Benchmark Results, and Where It Fits in Rawi

Rawi is DataKite's governed AI platform for banks. It takes a customer request written in plain language and turns it into a safe, policy-checked action (a transfer, a bill payment, a balance query) without a person in the loop, and without the request ever leaving the bank's own systems. The DataKite Banking LLM is the part of Rawi that reads the customer.

We have just rolled out its latest version, DataKite 30B-FP4-V3, and it is now the model shipping in Rawi deployments. This post covers how the updated model performs: how accurately it reads a customer, how fast it completes the parsing stage, and what hardware footprint it uses. We benchmarked V3 on 2,462 hand-annotated banking tasks in Saudi Arabic and English against two open-weight models a bank could self-host.

The short version: the new version leads on every accuracy measure, with its widest margin on entity recognition, the step that matters most for grounding a banking request. It also keeps the parsing workload on bank-controlled infrastructure, which is central to how Rawi is designed for regulated deployment.

How Rawi works, and where the model fits

Rawi handles a request in three stages. The Banking LLM reads the customer's message into a structured parse. Rawi's knowledge graph then grounds that parse against the bank's own data, resolving a name like "Abdullah" to a beneficiary the customer is allowed to pay. Finally, a deterministic governance layer checks the request against policy before anything runs, and logs every step for audit.

The model proposes; the symbolic layers decide. That split is the point of a neurosymbolic design: the model can be flexible about how people phrase things, because nothing executes until the graph has grounded and approved it. It also explains why the three headline numbers are the ones we watch:

• Accuracy sets the ceiling. The graph can only ground what the model hands it, so a wrong field means a failed or rejected grounding. Better extraction means fewer of both.
• Latency is shared. Grounding and policy checks run on top of the parse on every request, so a fast parse is what leaves room for the rest.
• Deployment footprint matters. Self-hosted inference is what lets a bank keep every request inside its own approved operating boundary.

Improving how well the model handles that first stage is the main lever we have on the quality of the whole pipeline, and it is where most of the work in V3 went. Here is what that produced.

What the model has to do

That first stage is a structured extraction problem. Given a customer's message, the model returns a JSON object with the intent, the entities involved, and their values:

// "Transfer 200 riyals to Abdullah to cover the team lunch"
{
  "intent": "transfer_to_beneficiary",
  "entities": {
    "transfer_amount": 200,
    "currency": "SAR",
    "beneficiary": "Abdullah",
    "transfer_reason": "cover the team lunch"
  }
}

We score three things separately: whether it gets the intent right, whether it spots the right entities, and whether it pulls the correct value for each one. We keep them apart because the errors compound. A request is only safe to act on when all three are correct, so each score on its own is the optimistic view of the full task.

How we tested it

The test set is 2,462 tasks, each labelled by hand with the correct intent, entities, and values. The language mix mirrors how people in the Gulf actually write: 54% Saudi Arabic, most of it code-switched with English, and 46% English. Tasks run from simple single-intent requests to messy ones with several entities and details left implicit.

We benchmarked V3 against two other self-hostable open-weight models:

Model	Parameters	Quantization	Access
DataKite Banking LLM V3	30B	FP4	Self-hosted
GPT OSS 20B	20B	MXFP4	Self-hosted, open weights
Qwen3-4B-Instruct-2507	4B	None	Self-hosted, open weights

We picked the two open-weight models as realistic alternatives for teams evaluating a self-hosted architecture on the same class of on-premises hardware a bank uses for Rawi. The models are not the same size, which is worth keeping in mind and something we come back to at the end.

Results

Model	Intent	Entity	Value
DataKite 30B-FP4-V3	96.0%	96.3%	85.8%
GPT OSS 20B-MXFP4	89.0%	74.7%	77.5%
Qwen3-4B-Instruct-2507	82.9%	67.6%	67.8%

On intent recognition, V3 leads the open-weight alternatives by 7.0 to 13.1 percentage points. The real separation is on entity recognition, where V3 reads 96.3% of parameters correctly: 21.6 points ahead of GPT OSS 20B and 28.7 points ahead of Qwen3-4B. Value recognition is the hardest test for everyone, because it means turning free-form text into clean values, especially when Arabic and English are mixed. V3 leads there too, by 8.3 to 18.0 points.

For Rawi, entity recognition is the most important separation. It determines how cleanly the knowledge graph can ground a request before the deterministic policy layer decides whether anything can execute.

Operational profile

Metric	DataKite Banking LLM V3	Conditions
P90 parsing latency	340 ms	60 concurrent requests, self-hosted on 2x NVIDIA H200
Self-hosted hardware	2x NVIDIA H200	$2,500/month each in the reference cost model

We measure parsing latency from the moment a message arrives to the moment the JSON parse is finished. That number matters because grounding and policy checks run after the parse on every request. The hardware figure is a reference deployment assumption for the Banking LLM stage; running the full cluster is part of the Rawi platform.

A few caveats

This is our own benchmark on our own data, not a public one. We can share sample tasks and support an independent evaluation under NDA. The models differ in size, and the open-weight alternatives are smaller than V3, so weigh those comparisons accordingly. The tasks are single-turn extraction; they do not measure multi-turn conversation, retrieval, or reasoning. And the latency and hardware figures reflect a specific deployment profile, which any bank should validate against its own traffic and infrastructure requirements.

None of that moves the core result. On the job the Banking LLM actually does inside Rawi, reading a customer accurately enough for the rest of the pipeline to act, V3 is the strongest self-hosted option we tested. It is the version now running in Rawi, and the one we will keep building on.

---

Notes: (1) P90 parsing response time measured on a batch of 60 concurrent requests. (2) Reference hardware assumes two H200 GPUs at $2,500/month each. For evaluation access or sample tasks under NDA, contact us.