My Thoughts on DeepSeek R-1

There's already extensive coverage on DeepSeek R-1. I just want to share a few of my own thoughts:

So what is DeepSeek's most significant contribution with R-1?

DeepSeek demonstrates that with a sufficiently "strong" base model, even without initial SFT (supervised fine-tuning), pure reinforcement learning (RL) with only ORM (Outcome Reward Model) alone can lead to the emergence of complex reasoning. No fancy techniques such as PRM (Process Reward Model) and tree searches (e.g. MCTS, Beam) are necessary.

Prior to that, most labs believed this was unfeasible because ORM rewards were thought to be too sparse to guide models towards high-quality reasoning chains (CoT paths). Consequently, efforts have focused on creating sophisticated PRMs, reward shaping if you will, to guide the RL process.

Anyone with RL experience knows that reward shaping is a double-edged sword. While it provides dense reward streams, it also introduces bias if not handled carefully. Many studies have shown how difficult it is to attribute rewards to specific reasoning steps. For example, it's not uncommon for a LLM to arrive at the right answer but with flawed reasoning.

Therefore, creating a reliable PRM is extremely challenging. It risks introducing bias that degrade reasoning performance, and It also consumes significant computational resources.

With the R-1 model, DeepSeek has just empirically proved that pure RL alone is all we need!

Why do big labs such as Meta, Deepmind, and Qwen all missed this?

What I admire about DeepSeek is their commitment to first-principles thinking -- challenging assumptions rather than blindly following trends. Great scientific breakthrough often requires clarity of thoughts and questioning basic assumptions. Einstein achieved this with Special Relativity by discarding unnecessary complexities and blindly-followed assumptions like the existence of "ether". Unfortunately, modern physics has sort of gone astray, with efforts veered into increasingly esoteric mathematics rather than revisiting the core assumptions. Well this is for another topic.

I would also want to add that one important caveat for pure RL to work is the base model must be "strong enough" to compensate for the lack of dense reward signals. I suspect models like Llama 3 might fall short in this regard.

With R-1, DeepSeek has just revealed to the world a clear path to AGI. The R-1 paper will go down as one of the most pivotal papers in the history of AI

Subscribe now