5/18/2026 at 4:31:51 PM
Something I've had good progress with using local models and simple open-source harnesses is to repeat, in a new context, simple verification prompts.I'd run the following 5-10 times with one model, then again with a 2nd model.
"Verify the correctness and completeness of all security configs/rules in SETUP.md. Consider if anything is missing, and if anything is not needed. Do not modify any files; only write potential findings to report.txt"
"Verify all findings and claims in report.txt."
Replace "SETUP.md" with whatever you're working on.
It's both terrifying and incredible watching what the models get correct and what they get completely wrong.
However, after enough runs they tend to settle on a state they claim does not need any more edits. And that result is generally useful with much fewer errors/hallucinations compared to a single run.
by vibe42
5/18/2026 at 10:53:49 PM
I have also had positive experience with doing this multiple times via multiple model families, and then to recursively have the fixes reviewed too.It's called review-anvil and does find significant amount of problems that might pop up:
by mrshu
5/18/2026 at 10:29:18 PM
Don't you think "consider if anything is missing" leads them into adding something with sycophancy RL training and "if anything is not needed" making it remove something?Or does "verify all claims in report" counteract that?
by knollimar
5/19/2026 at 2:03:16 PM
It can indeed cause some models to try too hard to come up stuff, but the next verification prompt does counteract it.E.g. some findings first classified as moderate priority often get reclassified as low priority even if the finding itself is correct.
The exact phrasing doesn't seem to matter as much as keeping the prompts short, simple and to the point.
However some models seem to do a bit better when adding ", if any" to prompts such as "List potential improvements".
by vibe42