VQV Signal

RESEARCH WATCH TECHNICAL

Berkeley AI Research reposts stress test of score centering in off-policy LLM RL

According to a post on X by @YichuanM, Berkeley AI Research reposted a thread by Yichuan Wang that stress-tests score centering, a popular technique in off-policy LLM reinforcement learning, with batches up to 127 steps stale. The thread reports that score centering removes a hidden drift term, with gradient growth at age 64 falling from 3.2x to 1.5x. It also reports that the method combines badly with importance sampling, and that a state-level KL mask plus random buffer sampling kept training stable. The source snippet is cut off, so further details are not available here.

Source: Scobleizer News Pulse (X) · x.com Found via Robert Scoble’s Scobleizer News Pulse · a post on X, not independently verified Published 2026-11-18T05:45:00+00:00 Detected 2026-11-18T20:02:27+00:00
View original source

According to a post on X by @YichuanM, Berkeley AI Research reposted a thread by Yichuan Wang that stress-tests score centering, a popular technique in off-policy LLM reinforcement learning, with batches up to 127 steps stale. The thread reports that score centering removes a hidden drift term, with gradient growth at age 64 falling from 3.2x to 1.5x. It also reports that the method combines badly with importance sampling, and that a state-level KL mask plus random buffer sampling kept training stable. The source snippet is cut off, so further details are not available here.

AI-assisted summary based on the listed source.

Berkeley AI Research reposted Yichuan Wang's thread testing score centering, a popular trick for off-policy LLM reinforcement learning, with batches up to 127 steps stale. It reports that score centering removes a hidden drift term (gradient growth at age 64 falls from 3.2x to 1.5x) but combines badly with...

Teams training LLMs with off-policy reinforcement learning, where batches can lag behind the current policy, may need to check how score centering interacts with importance sampling in their own setup. The reported stability fixes, a state-level KL mask and random buffer sampling, are the concrete options to evaluate. Watch for the full thread and any independent replication, since this rests on a social post and a truncated snippet.

Signal Strength 92% Technical label WATCH Public Interest 24 Category RESEARCH Reader Depth TECHNICAL

Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.

Public Interest components
Recognizable Entity Score 0 Practical Impact Score 26 Novelty Interest Score 48 Consequence Score 34 Curiosity Score 0 Shareability Score 42

VQV surfaced this signal because it is recent, relevant to Open Source LLMs, connected to Scobleizer News Pulse (X).