FuturixAI
Reward-model training is the cost bottleneck in modern RLHF pipelines. It often needs tens of billions of parameters and an offline preference-tuning phase before any reinforcement learning happens.
A judge, not a reward model: A frozen, instruction-tuned 7B LLM with a one-line JSON rubric and a rank-16 LoRA adapter, touching just 0.8% of its parameters, fully replaces the heavyweight reward models used before.
State of the art at 7B: The judge scores 96.2% on RewardBench, ahead of specialised reward networks from 27B to 70B parameters.
Better policies, with no offline phase: A 7B actor trained with online PPO against the judge reaches 92% exact match on GSM-8K. The best 70B DPO baseline scores 61.8%.
What changes in the pipeline
A standard pipeline trains a reward model offline, then starts reinforcement learning. Our judge skips the first stage completely. What the judge rewards is set by a rubric in its prompt, so it can change without retraining.
Accurate as a judge
As a static evaluator, the plug-and-play judge reaches 96.2% accuracy on RewardBench, ahead of specialised reward networks from 27B to 70B parameters.
The ablations show where the gains come from. Six in-context demonstrations deliver most of the zero-to-few-shot improvement (+2pp). The LoRA adapter closes the remaining gap, especially on the safety and adversarial Chat-Hard segments.
Strong as a reward
A good judge only matters if it trains good policies. Used as the reward in online PPO, it lets a 7B actor go well past a much larger model trained with DPO.
Explanations you can trust
For an enterprise to rely on a reward function, it has to understand it. To measure interpretability, we introduce HH-Rationales: 10,000 pairs from Anthropic HH-RLHF, each with a human-written justification.
Using the judge
Prompt engineering plus a tiny LoRA gives you a cost-effective, transparent and easy-to-adjust reward function. It removes the offline phase entirely and sets new state-of-the-art results for both static evaluation and online RLHF. The full method and ablations are in the paper on arXiv.
