12/11/2025 at 7:48:29 PM
by solsane
12/12/2025 at 1:49:12 AM
The Olmo team is AFAIK the only SOTA-ish model that has fully open source code and data. Their report is fantastic: https://www.datocms-assets.com/64837/1763662397-1763646865-o...It should give you an idea of how hard it is to do a SOTA model from scratch!
If you relax the SOTA aspect, Karpathy's nanochat has you covered: https://github.com/karpathy/nanochat
by huevosabio
12/12/2025 at 10:39:10 AM
Read this article: https://dl.acm.org/doi/10.1145/3712285.3759827 Training algorithms are relatively simple (base training, fine-tuning, RL), but the scale is critical. I.e., the engineering infrastructure. The authors recommend a 128 GPU cluster minimum and many petabytes of training data.by bjourne
12/12/2025 at 10:21:53 AM
I'd suggest you take a read on HuggingFace's writeup when they trained smolLM3https://huggingface.co/spaces/HuggingFaceTB/smol-training-pl...
rare detailed insight on the entire process
by walpurginacht