Explanatory Reinforcement Learning.
Our central research direction asks whether a model learns better when feedback includes reasons as well as scores. The method is still being tested with Hikaru-1. ERL has substantially improved Hikaru, although the gains have not yet reached the level we want.
Why a single reward score can be limiting
Many RLHF systems learn a scalar reward from human preferences. That score can guide training, but it does not by itself explain why one response was preferred over another.
A response might succeed on factual accuracy while falling short on clarity or tone. We want to test whether preserving those distinctions, and the reasons behind them, provides a more useful training signal.
Three changes to the reward signal
1. A graded scale
In our implementation, the grader assigns a value on an
eleven-point scale: −5, −4, … 0 … +4, +5. Zero marks
a response that was neutral, neither good nor bad. Conventional RL
can also use graded scalar rewards; the scale alone is not what
distinguishes ERL.
2. Per-section scores
A single score still hides too much. We grade dimensions separately, personality, helpfulness, and others, each on the same scale. A model can be told that it was helpful but cold, or accurate but evasive, and adjust each independently.
3. Grader explanations
The core of the method. A grader model writes an explanation for every score: where the response succeeded, where it failed, and why. The model receives that explanation to guide a retry. We train from the corrected attempt rather than teaching it to memorize the critique.
Feedback and retrying
In the approach described in our Hikaru update, the model receives feedback and tries again, using the grader’s scores and explanations to improve its response. We are testing how this process translates into durable improvements beyond the training examples.
Why we think this matters
Our hypothesis is that a correction with a reason may help a model generalize better than a score alone. We have observed substantial improvements in Hikaru, but a reproducible comparison against a clear baseline is still needed to establish how well the method generalizes.
Keeping the grader's explanation alongside each score can also give us a record to inspect. It may make training decisions easier to review, although those explanations still need to be checked for quality.
Where it stands
The method is still in development. Our first concrete test is
Hikaru-1, a finetune of Ministral-3 (8B)
trained with ERL, which entered development on June 16, 2026 and
continues today at a reduced pace. Its purpose is to measure
whether a model trained this way behaves measurably differently
from one trained with a scalar reward baseline.
- Active: the ERL pipeline and the Hikaru-1 training run
- Planned: evaluations comparing ERL against baseline RLHF
- Current limit: Hikaru’s improvements remain below our target level
- Additional testing: a small ERL experiment on the internal codename Cinder; see Deriva Update 3
- Future: testing whether larger base models improve the result
Full papers will be published here when ready. We would rather share something correct than something first.
Background reading
For the scalar reward model we are comparing against, see the InstructGPT paper. We will publish our own evaluation details when they are ready.
Sential Innovations research notes · Updated September 29, 2026