Summary
According to a post on X by @HeMuyu0327, in an AttnRes-style model the raw cosine similarity between layer queries and earlier layer outputs is negative for almost every layer except the token embedding. The post says MLP outputs are consistently rejected as well. The author argues that softmax-ed query weights hide this pattern and that such models may not use token embeddings to their full potential. The work is described as ongoing, with a blog write-up mentioned.
AI-assisted summary based on the listed source.
What happened
Muyu He reports that in an AttnRes-style model, the raw cosine similarity between layer queries and earlier layer outputs is negative for almost every layer except the token embedding, while MLP outputs are consistently rejected. He argues that softmax-ed query weights hide this, and that such models may not use...
Why it matters
The claim comes from a single social post describing ongoing work, so it has not been independently confirmed. Researchers designing attention-based and residual architectures are the readers most affected, since the claim points to a possible gap in how token embeddings are used. The promised blog write-up would show the underlying analysis and whether the pattern holds across models.
What this means for you
Hardware and robotics watchers may want to track whether this becomes a product, benchmark, or deployment signal.
Signal Intelligence
Signal Strength 88%
Technical label WATCH
Public Interest 23
Category ROBOTS & HARDWARE
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 0
Novelty Interest Score 70
Consequence Score 30
Curiosity Score 0
Shareability Score 41
Why this is here
VQV surfaced this signal because it is recent, relevant to AI Chips, connected to Scobleizer News Pulse (X).
