Summary
According to a post on X by @YichuanM, Berkeley AI Research reposted a thread by Yichuan Wang that stress-tests score centering, a popular technique in off-policy LLM reinforcement learning, with batches up to 127 steps stale. The thread reports that score centering removes a hidden drift term, with gradient growth at age 64 falling from 3.2x to 1.5x. It also reports that the method combines badly with importance sampling, and that a state-level KL mask plus random buffer sampling kept training stable. The source snippet is cut off, so further details are not available here.
AI-assisted summary based on the listed source.
What happened
Berkeley AI Research reposted Yichuan Wang's thread testing score centering, a popular trick for off-policy LLM reinforcement learning, with batches up to 127 steps stale. It reports that score centering removes a hidden drift term (gradient growth at age 64 falls from 3.2x to 1.5x) but combines badly with...
Why it matters
Teams training LLMs with off-policy reinforcement learning, where batches can lag behind the current policy, may need to check how score centering interacts with importance sampling in their own setup. The reported stability fixes, a state-level KL mask and random buffer sampling, are the concrete options to evaluate. Watch for the full thread and any independent replication, since this rests on a social post and a truncated snippet.
Signal Intelligence
Signal Strength 92%
Technical label WATCH
Public Interest 24
Category RESEARCH
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 26
Novelty Interest Score 48
Consequence Score 34
Curiosity Score 0
Shareability Score 42
Why this is here
VQV surfaced this signal because it is recent, relevant to Open Source LLMs, connected to Scobleizer News Pulse (X).