FxAI Decision Model vs Sarvam Prism

FxAI-decision-model f346832 · sarvamai/prism 417cf26 · both FP8 · both run by Futurix

The scorecard

Most model comparisons put one lab's measurements next to numbers copied from another lab's blog. Sarvam released Prism's weights, so we did better: we ran Prism ourselves, next to the FxAI Decision Model, on the same 51,368 questions, and scored both with the same code. Every row below is a head-to-head on identical inputs.

FxAI leads where agent builders need judgement: grading agent runs, catching prompt attacks, and reasoning over knowledge and code. It is also better calibrated and needs a quarter of the memory. Prism leads on Indian languages, the public Decision Index and context length, and it is order-invariant by construction. "Even" means the 95% interval of the difference includes zero.

Agent trace evaluation

Agent builders need to know which runs failed, which need a person, and where things went wrong, without paying an LLM judge for every trace. We used the four public datasets Sarvam used, with labels from human experts or each environment's own check: 698 runs from AgentRewardBench, τ-bench, Who&When and TRAIL.

FxAI is ahead on all four of Prism's own summary columns. The largest gaps are diagnostic: naming the agent responsible for a multi-agent failure (64.6% vs 54.7%) and finding the first wrong step (41.7% vs 32.8%). Both gaps are significant. FxAI did this while reading a shortened version of the 4% of traces that exceed its 8,192-token window; Prism read every trace in full.

Sarvam reports 75.0 / 86.4 / 69.6 / 78.5% for Prism on its own sample of these datasets. Its question wording and trace format are unpublished, so we used ours for both models.

Agent security guardrails

Sarvam's security set is internal, so we built a public one with the same shape: nine tasks and 5,510 questions. The two CVE tasks use only vulnerabilities published since 1 July 2026, and the ATT&CK task uses only procedure examples added since then, so neither model can have seen them in training.

An agent guardrail's main job is to stop text that tries to take over the agent. On that job FxAI is the stronger model. Its biggest lead is on instructions hidden inside tool results and web pages, the attack that matters most once an agent browses or reads email: 97.8 vs 89.4. It also leads on jailbreaks and SMS scams. Prism is better at spotting phishing URLs and emails, which pulls its nine-task average ahead (85.2% vs 84.1%). On the brand-new CVEs and ATT&CK procedures the two are level.

Reasoning and knowledge

Sarvam's second index recreates the Artificial Analysis Intelligence Index in decision form. Its version is unpublished, so we built a public one from the same components and ran both models on it.

FxAI's index is 39.4 to Prism's 37.0. It leads on code execution (75.6 vs 66.8), long-document reading, MMLU-Pro and Humanity's Last Exam. Prism leads on competition math and on the “I don't know” test, where it abstains on 39% of obscure questions against FxAI's 6.5%. FxAI's confidence still flags its guesses: answers it gives at ≥ 90% confidence on that test were all correct. The practical rule: threshold on FxAI's confidence rather than offering it an abstain option.

Sarvam reports 65.9 for Prism on its own recreation; on ours, Prism scores 37.0. The two sets differ, which is why we ran both models on the same one.

Calibration

A decision model's probabilities decide what gets automated and what goes to a person. They are only useful if a 90% answer is right about 90% of the time. We measured both models on the same questions.

FxAI's expected calibration error is lower in both settings: 0.017 vs 0.020 on the Decision Index questions, and less than half of Prism's in the Indian languages (0.012 vs 0.026). Prism's confidence runs low; in the Indian languages it predicts 4.5% errors among its ≥ 90% answers and makes 2.4%, so a threshold set on its confidence sends more work to people than it needs to. At that threshold on the Decision Index questions, FxAI leans the other way: its ≥ 90% answers err 3.6% of the time against 2.5% predicted, while Prism's err 3.2% against 3.8%. Across all confidence levels, FxAI's probabilities are the closer match. Prism's lower Brier score comes largely from its higher accuracy on these questions; Brier rewards accuracy and calibration together. On the full public suite (234,610 questions), FxAI's ECE is 0.010.

The use cases Prism was built for

Sarvam lists speech-to-speech harnesses, support intent routing, model routing and document intelligence as Prism's jobs. We built public tests for each and ran both models.

Across all eleven the two are level (69.9 vs 69.0). FxAI is clearly better at the routing decision teams care about most, whether a prompt needs the expensive model (63.0 vs 54.9 macro-F1), and at classifying documents from noisy OCR text. Prism is ahead on barge-in detection and two support and finance sets, including a perfect 100% on the Bitext support dataset, which suggests it saw that dataset in training. Voice intent, end-of-turn, contracts and LegalBench are statistically tied.

A quarter of the hardware

Prism needs one GPU with about 120 GB of memory: a B200 or H200. Its FP8 weights alone are 106 GB, so it does not fit on a 96 GB RTX PRO 6000. We had to split it across two. FxAI's weights take 29 GB.

For a team without B200s this is the difference between running a decision model and not running one. We capped FxAI at 44 GiB, what a 48 GB card leaves after the driver, and its heaviest cases still ran. That puts it on widely rented 48 GB cards, or two copies on one 96 GB card. Prism is faster per request on its target hardware: 44 ms on a B200 against our 125 ms on an RTX PRO 6000, a slower GPU. FxAI's floor of about 123 ms per call is fixed per-call overhead. CUDA graphs and fused FP8 kernels remove that kind of overhead; they are how Sarvam cut Prism from 664 ms to 44 ms, and they are next on our list.

Where Prism leads

• Indian languages. Prism's index is 81.8 to FxAI's 71.5 across ten languages and Hinglish, and it leads in every language. On the same questions in English the two are level (93.6 vs 93.3), so the gap is Indic language understanding, which Prism was built for and FxAI was not trained on.
• The public Decision Index. Prism's published public index is 67.0; FxAI's official run scores 62.26. On our 8,941-question sample Prism is more accurate (86.8% vs 84.0%) and wins 22 benchmarks to FxAI's 14. FxAI wins on BBH, MMLU-Pro, GSM8K, GPQA, HLE, CRUXEval and BPoMP.
• Option order. Prism sorts options before scoring, so shuffling changes 0.05% of its answers. FxAI changes 8.9% as sent; with the same sort applied in the client it changes 0.44%, which is the same as resending an identical request.
• Context length. 131,072 tokens against 8,192.

How we tested

Same inputs: every question went to both models with the same state, instructions and options. Prism read every agent trace in full; FxAI read a head-and-tail cut of the 4% over its window. Scoring: accuracy or macro-F1 per task, chance-corrected indices, and a paired bootstrap (1,000 resamples of the same items) for every difference; “Lead” requires the 95% interval to exclude zero. Test sets: AgentRewardBench, τ-bench, Who&When, TRAIL; nine public security sets plus NVD and MITRE ATT&CK data published after 1 July 2026; MMLU-Pro, GPQA Diamond, HLE, AIME/MATH-500, LiveCodeBench and CRUXEval, QuALITY, PopQA; Sarvam's own translated MMLU, ARC, BoolQ and TriviaQA, plus Belebele, IndicXNLI, IndicSentiment, IndicCOPA, MASSIVE and SentiMix; MInDS-14, Switchboard, Bitext, SPROUT, LEDGAR, LegalBench, Financial PhraseBank and RVL-CDIP OCR; and 3,938 requests from the public Decision Index.

Using FxAI

from futurix_decision import DecisionInput, load_model

model = load_model("/path/to/FxAI-decision-model", device="cuda:0")   # one 48 GB GPU is enough
result = model.decide(DecisionInput(
    state=trace_text,
    question="Did the agent fully complete the user's task?",
    options=["success", "failure"],
))
if result.confidence >= 0.9:
    act(result.decision)          # confident: automate
else:
    send_to_review(trace_text)    # uncertain: a person decides

Weights and runtime: huggingface.co/Futurix-AI/FxAI-decision-model, Apache 2.0. Sort options into a fixed order before calling to make answers order-independent.

Prism was run by Futurix from its public weights; numbers marked "reported" are from Sarvam's announcement. Code, test-set builders and raw outputs for both models are in our benchmark repository. Measured 3–4 November 2026.