Research

ERL

Explanatory Reinforcement Learning. Grades a model can understand, and a chance to try again.

Most reinforcement learning tells a model how well it did. ERL also tells it why, shows it what it did to earn that grade, and then lets it try again.

How it works

We take the model we want to train, put it into training, and then run ERL. ERL is built on RLAIF (reinforcement learning from AI feedback), split into a few core parts:

  1. The model attempts a task.The model in training works through a task the way it normally would.
  2. A frontier model grades it.The attempt is scored on several separate scales at once, not boiled down into a single number.
  3. The grader explains the grades.Along with each grade comes an explanation of why the attempt earned it.
  4. The model sees what it did.We give the model its grades, the explanation, and a record of what it actually did, so it knows exactly which actions led to which grade.
  5. It tries again.With all of that in hand, the model retries the task. The retry should be better, and pass.

A multi-scalar reward

Each attempt is graded on several scales, such as personality and safety. Every scale has 11 points, from −5 to +5. Zero is neutral, because not every scale applies to every task.

Below +3 failsPass

To pass, an attempt needs +3 or higher on every scale that applies to it. Anything below +3 is an automatic fail.

That means strong scores can't carry a weak one. If a model does amazingly on personality but badly on safety, it doesn't pass just because most of its scores were good.

Why explain the grade

A number tells a model that something went wrong. An explanation tells it what went wrong, and seeing its own actions next to that explanation tells it where. That gives the model something it can actually correct on the next attempt.

The explanation is something to retry from, not something to memorise. In early versions of ERL we wrote the grader's explanations straight into the model, and it began memorising them instead of learning from them. Giving them back as the starting point for a retry fixed that: the model learns the correction, not the wording of the critique.

The grader matters just as much. When a grader is wrong, the model learns the wrong lesson faithfully, so improving the grader is part of improving the training. Read how we learned this.

Hikaru is the testbed

Hikaru-1 is where we put ERL to the test. Built on Ministral-3 (8B), it's small enough to train again and again, and large enough for the results to mean something.

ERL has substantially improved Hikaru so far, though the gains are still below the level we want. We share results in our news as we're confident in them.