Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

7/14/2026 at 1:37:04 PM

AI trains AI already, agents are happy to spin up real training pipelines for deep learning or regression models or whatever you want right? I guess the advantage to your project is that it provides a framework to allow the agent to access extra compute?

by callamdelaney

7/14/2026 at 1:56:32 PM

Yes I'd heard the labs (Anthropic mostly) speaking about LLMs training LLMs, so I wanted to make things a little more concrete and test it out myself! Essentially you are correct though, my framework allows the agent access to compute, but also the agent itself is being trained to become better at training models with that compute.

by Danau5tin

7/14/2026 at 4:58:27 PM

i remeber reading in one of the release blog posts that that version was the "first that codex helped train"

by vorticalbox

7/14/2026 at 5:00:42 PM

Yes exactly, and also there is auto-research by Karpathy, Fable-5 on parameter golf etc which point to the direction of highly-assisted / autonomous AI training as the future. Wild.

by Danau5tin

7/14/2026 at 5:42:16 PM

I guess more important than LLMs training LLMs (which is mostly the same code over and over) is LLMs cleaning and curating training data.

by amelius

7/14/2026 at 8:34:30 PM

Very good point. Sifting through noisy data and creating curated datasets could be of great value. Perhaps worth a project by itself!

by Danau5tin

7/14/2026 at 5:49:48 PM

Not sure how much of it was marketing but M2.7 supposedly trained itself

https://www.minimax.io/blog/minimax-m27

by throwa356262

7/14/2026 at 6:28:12 PM

Very cool! Thank you

by Danau5tin

7/14/2026 at 1:21:21 PM

Lots of emoji in that readme. Was it mainly codex?

by bitbasher

7/14/2026 at 1:31:56 PM

Mainly Fable, but It was me who wanted to emojis added hah. I also of course edited the README by hand (crazy I know), but the code is entirely fable

by Danau5tin

7/14/2026 at 5:31:20 PM

i read around launch that anthropic will fallback to opus if fable is used for frontier LLM development. did you run into anything like that?

by rolls-reus

7/14/2026 at 5:55:42 PM

Strangely, I did not. I was expecting it and looking out for it the whole time. At least I never say a warning!

by Danau5tin

7/14/2026 at 8:50:01 PM

that's because they said they'll silently fallback in case of model training. Then they backtracked, and said they wont silently fallback, but how will anyone ever know?

by se4u

7/14/2026 at 2:50:36 PM

why...?

by r_lee

7/14/2026 at 2:56:37 PM

I guess we found the target audience for models that slather READMEs in emojis.

by snerbles

7/14/2026 at 3:56:57 PM

The thing we must keep in mind with any of the AI-slop style writing is that it was reinforced behavior because humans wanted it.

by sosodev

7/14/2026 at 5:42:35 PM

To circumvent distillation.

by piratebroadcast

7/14/2026 at 1:26:53 PM

“The Anthropic team for their incredible coding models (Fable-5 wrote every line of code in this project), and the Claude Code harness.” Source: the repo

by netvarun

7/14/2026 at 6:20:09 PM

Funny how I could tell without reading anything. I immediately lost interest upon noticing.

by bitbasher

7/14/2026 at 4:11:25 PM

I'm curious to whether the recursively trained models degenerate to troglodytes after a couple of generations.

by nnevatie

7/14/2026 at 4:21:17 PM

I am also curious! The inner-RL-loop models are only trained once, then disgarded. But the outer-RL-loop model is trained on the same tasks over and over again. I imagine it would overfit after many more steps, but perhaps with a larger set of diverse tasks, the model would simply improve.

by Danau5tin

7/15/2026 at 8:53:46 AM

How do you prevent the agent from reward-hacking the hidden eval? e.g. writing training data that effectively leaks the eval distribution rather than teaching a general skill?

by azertyvode

7/15/2026 at 8:58:13 AM

Great question. Today the answer is that there is nothing stopping this, but you are right this is a 100% valid reward hacking possibility.

No signs of it in the run I did, but for any future larger scale runs, it would be wise to dedicate some focus to prevent this.

by Danau5tin

7/15/2026 at 4:37:17 AM

I want to see a latent space of all neural network weights where each point represents an entire neural network

by UltraSane

7/14/2026 at 2:23:07 PM

What is RL?

by normie3000

7/14/2026 at 2:24:12 PM

Reinforcement Learning.

by jeffhuys

7/14/2026 at 5:50:44 PM

Thanks.

by normie3000

7/14/2026 at 1:53:42 PM

Can you explain how it works?

What problems would it do well on and why?

Where would it start to fail/break?

What are the limitations of a system like this?

When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail.

It's the blind leading the blind.

by saberience

7/14/2026 at 2:03:39 PM

I chose the key technical decision and direction (such as the system architecture, the tasks to train on, the stack of Tinker, Prime-RL & Runpod - all of which I know well) etc.

The problems it would do well on are training small agentic (multi-turn, tool use) task based models using the prime-rl stack, which are close to the distribution trained upon. It would likely not transfer to other training frameworks such as SLIME, ART or ROLL, it would also likely not transfer well to RL for complex agents such as coding agents etc.

It is limited due to its scale. As a single person, the resources required to train this on a more diverse dataset, with more complex tasks on a larger variety of models, is outside my abilities! I believe there are many avenues to explore to improve performance for this to be genuinely valuable. For now, this just a proof of concept to show the possible.

I would like to think I have a good understanding of RL, evaluations, and agentic systems after a few years of working on these areas. However, I will always have gaps. I use Fable to help accelerate me, and fill those gaps at the same time, from which I can learn from too.

by Danau5tin

7/14/2026 at 1:56:49 PM

I think the counter point for these projects is that you may not need a deep understanding if you can measure the outcome. While this may not be true every time today, it plausibly will be in the future - making the activity worthwhile.

by lumost

7/14/2026 at 2:22:43 PM

Yes I do agree with this. I believe we are shifting from "make the model good" (prompt/context engineering, etc) to "define good for the model" (success criteria/rubrics). Over time I believe this will become increasingly obvious (as long as model capabilities continue to increase).

by Danau5tin

7/14/2026 at 1:58:35 PM

Well, you say that, but when "measuring" anything in RL, that measurement itself is not always obvious.

That is, creating the scoring system/judge models etc for RL is not easy at all. You can easily create an RL loop which is getting better and improving its scores, but actually the result is totally garbage, because you're measuring the wrong thing.

by saberience

7/14/2026 at 5:03:37 PM

What do you see wrong with the OPs reward design?

by babelfish

7/14/2026 at 2:00:15 PM

Did you read the README?

by babelfish

7/14/2026 at 2:03:12 PM

The AI generated README?

by saberience

7/14/2026 at 5:55:59 PM

Your comments are casting aspersion without showing that you have looked into the work or its author. This project is a case study in good work done with heavy ai assistance while it is clear that there is a skilled person leading. I predict that this will become very common and welcome here.

by irickt

7/15/2026 at 2:38:40 PM

Why should anyone put serious time and effort into using/understanding a product when the author hasn't put serious time and effort into making it?

I could make this exact same thing over a weekend and post it on Hackernews. But I won't because I would be embarrassed to do so.

The bar for posting something to HN should be high, the bar for wanting people to read your code, your writing, should be putting serious effort and thought into it. Not just vibe coding something up with a vibe coded README and 100% vibe coded code and not even a novel idea or implementation.

by saberience

7/14/2026 at 1:52:29 PM

No idea why you got downvoted into oblivion with the context post. Cool idea!

by RamblingCTO

7/14/2026 at 3:57:09 PM

Because HN detects AI comments and automatically makes them dead. I vouched for it as it's important for this post as context.

by satvikpendem

7/14/2026 at 11:53:56 PM

That wasn't AI detection.. it was self-promotion detection, but hey.. you've helped a promoter out.

by gnabgib

7/14/2026 at 11:48:01 PM

Trying to understand, So you downvote the post because of AI comments in HN ?

by bicepjai

7/14/2026 at 11:54:21 PM

HN the software, not HN the community (and no.. that isn't what happened)

by gnabgib

7/14/2026 at 1:54:32 PM

Thank you!

by Danau5tin

7/15/2026 at 8:53:50 AM

[flagged]

by Sentino_API

7/15/2026 at 2:27:14 AM

[flagged]

by leonkatz

7/14/2026 at 5:37:28 PM

I think that En Dash is supposed to be Tilde

  –$1.3k -> ~$1.3k

by tantalor

7/14/2026 at 5:54:49 PM

Yes, thank you. Typo!

by Danau5tin

7/14/2026 at 4:16:55 PM

[dead]

by no7z

7/14/2026 at 12:42:06 PM

I RL-trained an agent whose job is to write RL training jobs for smaller models, and open-sourced the whole thing.

The setup is two nested RL loops:

- Outer loop: the trainer agent (Qwen3.6-35B-A3B, LoRA) is handed a task spec ("teach a small model to do X"). It works in a sandboxed workspace with file tools and writes a complete prime-rl training job: a verifiers environment + rubric, a dataset, and a hyperparameter config. Submitting triggers a validation probe with capped retries. - Inner loop: each validated job is dispatched to a warm pool of up to 16 Runpod GPU pods, where prime-rl GRPO-trains a small Qwen (0.6B or 1.7B). The checkpoint is scored pre/post on a hidden eval the agent never sees. - The inner model's improvement flows back up as the outer loop's reward (plus a validation-efficiency term and a small train-speed tie-breaker). The outer loop is tinker-cookbook's importance-sampling GRPO, run async off-policy so one slow episode doesn't stall a batch.

Results, over 54 outer-loop steps (~1,750 real GPU training jobs):

- Episode reward went from ~0.0 to a ~0.63 peak. - Learning came in two distinct rungs: first "stop failing validation and dying on GPUs", then "make better models". GRPO took the steepest gradient first — the entire early gain was process reliability, and only once that saturated did the hidden-eval scores of the trained models start climbing. - It transferred to a held-out task family that the agent never trained on: mean reward 0.399 untrained → 0.545 at step 34, easing to 0.49 by step 54 (n=10 per arm, so noisy — a rise then a plateau/dip). - The agent learned to stop picking the weaker 0.6B base model (1.7B share of its jobs: 42% → 95%) and started actually using the hyperparameter surface (21% → ~78% of episodes).

Cost: the headline arc was ~$1.3k all-in (~$810 Runpod, ~$465 Tinker). Each inner training job cost ~$0.13–0.30 — a benchmark matrix over GPU × base-model picked cheap pairs (mostly A40s in practice, since the cost-winner was rarely in stock).

Two honesty notes: the outer loop runs through Tinker's managed API rather than local GPUs — the inner loop is all open-source stack on rented pods. And ~$1.3k is the headline arc, not the project; the pilots and blind alleys that got me there cost a few hundred more, and every one of them is written up in the retros in the repo, including the failures.

I did this because I think agents that improve other AI systems are going to be a big part of the next few years, and I wanted to know what it actually takes to get the reward moving. Turns out: way more debugging of the process than the policy, and it's all more accessible than it looks.

Happy to answer questions about the reward design, the GPU orchestration, or the things that didn't work.

by Danau5tin

7/15/2026 at 10:02:50 AM

Curious whst was your prompt/spec design process. How did u maintain the goal?

by debpalash