Summary

A statistical model used to predict the outcome of pairwise comparisons between items. It is an easy way to convert pairwise comparisons into a differentiable loss function, allowing you to train a reward model to output the exact scores the RL algorithm needs.

Derivation

One natural model assigns each response a positive latent strength:

Then the probability that response A beats response B is:

Substitute :

Divide numerator and denominator by :

Recognize the sigmoid:

Now suppose the dataset tells us A actually won.

We maximize its predicted probability:

Or minimize:

where and denote winner and loser.

Derive the gradient

Let:

Then:

The sigmoid derivative is:

Therefore:

Thus:

Since:

the chain rule yields:

If , then and :

Under gradient descent, the winner’s score is encouraged upward and the loser’s downward.

Example

Rewards are not absolute. Suppose:

Then the preference probability is:

Now add 1,000 to both scores:

The probability is still:

Only the difference matters. In other words, pairwise comparisons alone don’t identify a unique absolute reward scale or offset.

From scratch

def reward_scores(backbone, head, input_ids, attn_mask):
    h = backbone(input_ids, attention_mask=attn_mask).last_hidden_state  # (B, T, H)
    last = attn_mask.sum(-1) - 1                                        # last real token
    pooled = h[torch.arange(h.size(0)), last]                           # (B, H)
    return head(pooled).squeeze(-1)                                     # (B,)
 
def rm_loss(r_chosen, r_rejected, margin=0.0):
    # r_*: (B,) scores for the same prompt, different completions
    return -F.logsigmoid(r_chosen - r_rejected - margin).mean()
 
@torch.no_grad()
def rm_accuracy(r_chosen, r_rejected):
    return (r_chosen > r_rejected).float().mean()   # the only metric that matters