Summary
A statistical model used to predict the outcome of pairwise comparisons between items. It is an easy way to convert pairwise comparisons into a differentiable loss function, allowing you to train a reward model to output the exact scores the RL algorithm needs.
Derivation
One natural model assigns each response a positive latent strength:
Then the probability that response A beats response B is:
Substitute :
Divide numerator and denominator by :
Recognize the sigmoid:
Now suppose the dataset tells us A actually won.
We maximize its predicted probability:
Or minimize:
where and denote winner and loser.
Derive the gradient
Let:
Then:
The sigmoid derivative is:
Therefore:
Thus:
Since:
the chain rule yields:
If , then and :
Under gradient descent, the winner’s score is encouraged upward and the loser’s downward.
Example
Rewards are not absolute. Suppose:
Then the preference probability is:
Now add 1,000 to both scores:
The probability is still:
Only the difference matters. In other words, pairwise comparisons alone don’t identify a unique absolute reward scale or offset.
From scratch
def reward_scores(backbone, head, input_ids, attn_mask):
h = backbone(input_ids, attention_mask=attn_mask).last_hidden_state # (B, T, H)
last = attn_mask.sum(-1) - 1 # last real token
pooled = h[torch.arange(h.size(0)), last] # (B, H)
return head(pooled).squeeze(-1) # (B,)
def rm_loss(r_chosen, r_rejected, margin=0.0):
# r_*: (B,) scores for the same prompt, different completions
return -F.logsigmoid(r_chosen - r_rejected - margin).mean()
@torch.no_grad()
def rm_accuracy(r_chosen, r_rejected):
return (r_chosen > r_rejected).float().mean() # the only metric that matters