RWKV (Receptance Weighted Key Value)

RWKV (Receptance Weighted Key Value) is a neural network architecture that reimagines recurrent neural networks for the transformer era, achieving GPT-level language modeling performance while maintaining the efficient inference characteristics of RNNs.

The architecture combines parallelizable training—traditionally a transformer strength—with constant computational and memory complexity during inference, unlike transformers whose memory requirements grow quadratically with sequence length. RWKV achieves this through a linear attention mechanism that reformulates the attention operation, allowing the model to function either as a transformer or an RNN depending on the computational context. The largest dense RWKV model trained to date reaches 14 billion parameters.

This architecture resolves a fundamental tradeoff in sequence processing: transformers offer superior performance but scale poorly with context length, while traditional RNNs are efficient but struggle with long-range dependencies and training stability. RWKV eliminates the self-attention bottleneck entirely, enabling theoretically infinite context length with constant memory overhead during generation.

RWKV has gained significant adoption in the Chinese AI ecosystem, where compute constraints make efficient inference particularly valuable. The project joined the Linux Foundation in September 2023, and multiple architectural iterations (RWKV-4 through RWKV-7) have refined the approach with matrix-valued states and dynamic recurrence mechanisms. The architecture remains 100% free of self-attention mechanisms.