alt.hn

6/29/2026 at 5:05:16 PM

Qwen 3.6 27B is the sweet spot for local development

 https://quesma.com/blog/qwen-36-is-awesome/

by stared

6/29/2026 at 5:38:52 PM

The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0]

Some people will be happy to pay that premium for privacy, but at roughly 10X the cost of a MacBook Neo, that money could also buy a lot of credits on OpenRouter or frontier labs.

[0]: https://www.apple.com/shop/buy-mac/macbook-pro/14-inch-space...

by bensyverson

6/29/2026 at 5:50:11 PM

The maths there is pretty undeniable, but it is not where I'd make the split. Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it.

I don't know how much serious hands-free agentic coding I will ever do on my MacBook alone, but I do know that I would not have got so far into understanding this without tinkering with local models, llama.cpp, LM Studio, and LM Studio and all that.

I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled. Because it's just huge, exhausting, jargon-drenched, unknowable, and I am over the hill at fifty-plus.

Until, that is, I could poke around with setting it up on my own (secondhand) machine, watching the API calls, understanding some of the terminology. I didn't even buy the machine for that; it's just adequate to the task.

The Neo is too small to really get much benefit from this opportunity to make it more visceral and knowable.

by dofm

6/29/2026 at 6:33:24 PM

> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it.

Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem.

Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are currently an expensive toy.

When I try to program with a local LLM (I'm on a 32/128 GB system), I end up wasting time compared to a cloud LLM.

by pizza234

6/29/2026 at 6:56:10 PM

The key word there is 'currently'.

by AlpacaJones

6/29/2026 at 6:47:04 PM

Again, I would not argue against any of this.

And I can't say that I won't switch to openrouter (even just for the same models) at some point.

But one of the things I have found about my own process learning is that some lessons only come to you when you make yourself available to them. And if that means doing things the difficult way, that is what you should do.

by dofm

6/29/2026 at 6:11:00 PM

I just got Claude to download and install all the models and servers and agents and prepare all the launch scripts for me... no need to learn, just ask it to do it for you

by ddalex

6/29/2026 at 6:30:46 PM

Right, but I am a middle-aged bloke who is experiencing existential angst about whether I can carry on in this industry.

I have a pretty deep, maybe paranoid need to be confident I have an intrinsic understanding, and I have found in my life that lessons come to you when you make yourself open to learning.

So I need to build on top of what I know, taking as much of the hard way as I can bear to take at any one time — it has to be not quite difficult enough to put me off.

I can't really explain what I have learned this way that is different, but I feel it in a way that I wouldn't if I'd simply pushed a button.

For the same reason, I have a really basic 3D printer that I've set up myself, set up Klipper, configured how I want it, learned how to calibrate, all that. And now I can say that I feel I have an understanding of 3D printing. I could hold my head above water in a discussion with a real expert, maybe find work in an adjacent field where my insights would keep me grounded.

I can afford a really good printer that has all that set up, and more, has no problems. But I'd just be someone who has a 3D printer.

by dofm

6/29/2026 at 6:54:20 PM

>no need to learn, just ask it to do it for you

And that's how skills die.

by coldtea

6/29/2026 at 6:39:28 PM

Then what is the point of ddalex?

by sorokod

6/29/2026 at 6:23:05 PM

[dead]

by kdkdjduxnd

6/29/2026 at 6:00:21 PM

> I totally struggled to find the right frame of mind to explore any of this stuff without feeling defeated and bamboozled.

I found LM studio to be a nice starting point. Frindlier and more featureful than Ollama and not as intimidating as llama.cpp (though you will want to use that eventually)

by rusk

6/29/2026 at 6:34:05 PM

LM Studio is also nice because of the way the interface explains things; parameters have explanations and hints. It has been designed by people who really care about making it understandable.

I tried Ollama but I've settled on Unsloth Studio generally; once things really settle down I'll just run the llama-server UI, which is pretty nice.

A friend is tinkering with LLMs for amusement on a 16GB Raspberry Pi 5, and when I explained that llama.cpp now had a typical web chat interface he was so happy — it's amazing what the "table stakes" are now.

by dofm

6/29/2026 at 6:22:28 PM

I've setup to local paradigms for local coding:

- opencode with it's webui

- deer-flow with it's research/powered front end

They both run websites so you don't have to baby sit them (eg, keep your mac open). I've build a pdf compressor over a few days by first having deer flow try and research the frameworks and pipeline. It stalls out because its not really a fluid programmer. Once it stalls out, I transferred it (manually for now) to opencode and it's refactoring it because it's just a collective bundle of sticks and it needs a lot of testing to tweak out the limited scop context. LLMs can't really hold large scopes (locally anyway, from what I've read from HN, it's possible with longer context).

It'll complete in a few days with maybe 3-4 hours of full attention interaction, but it's running 3x that without my attention. Obviously, if I paid more attention it'd run quicker, but since it's local, it's not pumping out large volumes of code, it's mostly looping over tests and capabilities as observed.

It's running Qwen3.6 35B MoE on a AMD 128GB strix halo. If I switched to the dense models, perhaps it'd be smarter, but the trade off seems to be much slower gen.

by cyanydeez

6/29/2026 at 6:26:50 PM

> - opencode with it's webui

Have you tried Paseo?

I have opencode in a VM, and the paseo daemon running in the VM, and then the Paseo Mac app. Really nice.

(You can also use the Opencode GUI to frame a remote opencode web interface)

by dofm

6/29/2026 at 6:49:38 PM

You can also just add OpenCode web as a PWA, if that's what you mean by "frame".

I'm gonna check out paseo, but am not looking forward to all the ram the agent needs + all the ram paseo needs

by c-hendricks

6/29/2026 at 6:35:58 PM

But the tokens or credits are gone. MacBook stays. You can run other models on the same MacBook. What I read people burn every month on saas… for that money you break even on that MacBook in 5 months.

Edit: it’s not just “data privacy”, when you are using Claude, you are shipping EVERYTHING to Anthropic. It’s crazy.

by throw1234567891

6/29/2026 at 5:59:40 PM

You can also run Qwen 3.6 27B dense model on DGX Spark with comparable performance [1][2] for about $4000 (Asus Ascent GX10 is $3999 at various retailers).

In theory you can also get 48GB of VRAM with, say, two 3090s, but it will take up a lot of space and generate a lot of heat compared to the Macbook Pro and GB10.

[1] https://x.com/MiaAI_lab/status/2070859135399182444

[2] https://github.com/MiaAI-Lab/Qwen3.6-27B-NVFP4-vLLM

by porphyra

6/29/2026 at 6:02:06 PM

> 48GB of VRAM with, say, two 3090s

So like... $2000+ just for the used GPUs? Plus I assume it's considerably more effort to get it working.

by esperent

6/29/2026 at 6:22:22 PM

>Plus I assume it's considerably more effort to get it working.

Nah, not really. It is a little annoying in terms of space and power, though. Not every case and motherboard can support cards that big.

by fluoridation

6/29/2026 at 5:46:45 PM

The model they reference can be easily run with 24gb+ of VRAM, and there are other similar models capable of running easily on 16gb of VRAM. It's not like 128gb is a requirement here.

by Catloafdev

6/29/2026 at 6:54:08 PM

And if you go for actual GPUs it'll run much faster, I'd say 24gb may be pushing it for context, but my 5090 with 32GB VRAM is usually somewhere between 60 to 100 tok/s with mtp and 2-3k tok/s for prompt processing. I'm not sure what they cost now but it's definitely still quite far from the macbook, and there's also some other 32GB GPUs that are considerably more affordable

by Numerlor

6/29/2026 at 6:40:51 PM

For a MBP I have 48 GB of RAM M5 Pro. It runs at about 12-14 t/s at Q4, you could probably optimize it further. RAM is not a limitation but overall memory bandwidth. Q8 is slower. 35B A3B Qwen is quite speedy, but a little less accurate. With Qwen 3.6 27B dense I can squeeze a 9B parameter model and use that for fast analysis or code scanning while 27B is churning on a task in the background. It is tight, but totally reasonable.

The real sweet spot for Qwen 27B is getting it on something like a Dual 3090 system or some other config where it can blaze at 50-80 t/s and that costs well under 6K currently. It is a surprisingly capable model. Using something like GLM for orchestration, specs, task farming and then letting Qwen churn is relatively inexpensive.

Overall I recommend people try models of this class out using OpenCode and some for pay service to experiment with them and understand how they work. I find they are very useful.

Long term, I am convinced enough that if I wanted to use local models for any number of reasons I would be okay investing in a dual GPU box. The Mac is not fast enough for me and M5 Max is just too expensive relative to GPU linux box. Still, it is nice to have the models local ON the laptop and it is useful for what I care about locally.

by bitexploder

6/29/2026 at 5:57:31 PM

I'd go for at least 32GB+. It'll fit in 24GB but leaves you little to no room for context, and that's at 4-bit quantization.

If you want to run unquantized, you definitely need 128GB.

by thewebguyd

6/29/2026 at 6:01:58 PM

Nobody runs unquantized, there's literally no reason to. Q8 would be the largest anyone actually runs on consumer hardware for inference.

by Catloafdev

6/29/2026 at 6:43:02 PM

It also comes down to inference speed, not "can I run this". 8-bit quant is quite a bit slower on an M5 Pro.

by bitexploder

6/29/2026 at 6:42:58 PM

[dead]

by gchamonlive

6/29/2026 at 6:27:59 PM

a computer with 24 GB VRAM is at least $3000

by nok22kon

6/29/2026 at 6:44:29 PM

I can't speak for the US, but in Germany (where hardware is usually more expensive, not less), I got my 3090 3 months ago for 750 euro and have been running the iq4_nl 27B using q4 kv (which after recent patches in llama.cpp is in my xp indistinguishably accurate from q8 of f16) at full ctx, with MTP at 2, peaking around 70 t/s on small ctx, around 50 t/s when im around 64k and ends around 40 t/s near the cap. The rest of the PC is a 50 euro ddr3 16gb i5 4th gen box, absolutely nothing special. And this setup is often more useful than dsv4pro (and sometimes kimi, but not glm) for research and ML work.

by sleepyeldrazi

6/29/2026 at 6:43:50 PM

Isn't the directionality important. I.e. it is currently possible to run useful / great models locally, but on high end machines; and in a few years we will likely be able to run even better models on standard machines.

by montebicyclelo

6/29/2026 at 5:50:27 PM

Just putting it out there: I run Qwen 3.6 on my M1 Mac Studio with 64gb. It's quantized and all that, but I agree with TFA: it's the sweet spot for local development right now.

by nozzlegear

6/29/2026 at 6:09:48 PM

> The article is based on running Qwen 3.6 on a 128GB MacBook Pro. For reference, a 128GB MBP currently starts at $6699 USD [0]

Qwen3.6-27B would be faster on a 3090 that costs around $1000-1200 though so I don't think it's a good counter-argument.

Op just happened to have that MacBook, but it doesn't mean it's necessary to run the model.

by stymaar

6/29/2026 at 6:33:40 PM

That 3090 is going to burn 750W and it will still cap you at a 4 bit quant and ~48K context. Here's someone who worked through it:

https://github.com/noonghunna/qwen36-27b-single-3090

Flies though (50-70tps is impressive for a model this smart)

I went through roughly the same process to get it working on my M2 Macbook Pro... at awful speeds of course, since models like this one are mostly bound by memory bandwidth.

by boutell

6/29/2026 at 6:23:27 PM

For that price you can put together a PC with 128GB of ram ($2000) and an RTX 5090 ($3600) and get 70-100 tokens per second instead of 45

by dmayle

6/29/2026 at 5:53:23 PM

I’m running the same model on a 48GB MBP with a q4 quant and it’s pretty decent. You definitely don’t 128GB. That’s the scale for 70B models at q8 or something.

by dannyw

6/29/2026 at 6:26:06 PM

I've been running it on my 48GB MBP too and it's not particularly great. Super slow and not near enough to the quality provided by even Claude Sonnet.

by dom96

6/29/2026 at 6:06:24 PM

How much does one of those cost in the US? Here in Brazil, your notebook is worth as much as a used Honda Fit, which seems absolutely insane. For comparison, the ThinkPad I'm currently running cost me 1/20 of how much this MBP costs here, leaving me with over $8.000 to spend with LLM inference (if I actually spent money with that).

by doodlesdev

6/29/2026 at 6:16:50 PM

I purchased mine for approximately $4400 AUD before the price hikes. That unit is now ~$5100 AUD.

I use my MBP essentially as my workstation, it's almost always plugged in. I have a MBA (M4, 24GB RAM) that I picked up for ~A$1500 or so, and that's an amazing daily driver. I don't do local LLM inference on that unit, I can just hit my own APIs (via LM Studio) on the MBP over Tailscale.

by dannyw

6/29/2026 at 5:59:09 PM

I run Qwen 3.6 on my Framework Desktop 128GB, and it's very performant. I know Framework has had to raise the price since I preordered mine, but they're still well under half the cost of that Macbook.

by organsnyder

6/29/2026 at 6:04:17 PM

I get ~55 Tok/s on my framework desktop with the 35B A3B q8 model, and so far am also very happy with the coding performance.

by andy99

6/29/2026 at 6:24:17 PM

did you upgrade to MTP?

by cyanydeez

6/29/2026 at 6:45:01 PM

Runs fine on 2x4080s or on two 5060/5070s with 16GBVRAM... and faster than on the mac.

by trentor

6/29/2026 at 6:04:48 PM

Absolutely for the average developer the token speed is just going to be too slow for it to be workable. I think we’re looking at 2028 when memory becomes cheaper again and they’ll be a lot more people using local models.

by dvduval

6/29/2026 at 5:39:59 PM

But you have to factor in that this device will last you 5-10 years. That said, I wouldn't spend almost $7k USD on this macbook lol.

by Insanity

6/29/2026 at 5:44:38 PM

Memory requirements of newer models will increase, so while the hardware may last 10 years it won't be able to run the latest models for 10 years.

by petilon

6/29/2026 at 5:55:39 PM

My experience working in the open model space pretty deeply (both LLMs and diffusion models) for years now is that it is not quite as simple as that.

In the open model space an insane amount of effort goes into getting more powerful models to run with the same or less RAM. For example in the diffusion world many things that could not be run on easily under 24GB of VRAM actually run much better today with much less VRAM than they did a few years ago. You can do many things today with 8-16GB of VRAM that would not have been possible. At the same time the most advanced open models, like LTX 2.3 for video gen, still seem to respect 24GB of VRAM as the upper bound.

Similarly the standard "big" but localish open model for LLMs back in the day was Llama 3 70B, this was both a much worse and much larger model than Qwen 3.6 27B

So in two different spaces I've witnessed the "RAM required to run the best" decreasing or at least remaining stable, while the performance being achieved in both areas is astounding (LTX 2.3 is faster, better and more capable than the Wan 2.2 model that held popularity before it).

The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models.

by roadside_picnic

6/29/2026 at 6:47:33 PM

> insane amount of effort goes into getting more powerful models to run with the same or less RAM

The same can be said about operating system memory requirements. I am sure Linux and Windows kernel developers can confirm. Yet 30 years ago Solaris used to run comfortably in 16 MB of RAM, today you need 512 times that to run Linux.

by petilon

6/29/2026 at 5:47:59 PM

You raise a fair point, but I'm not convinced it'll offer a meaningful difference in performance as long as we're stuck with the current AI paradigm.

by Insanity

6/29/2026 at 5:53:58 PM

Will they? Or will we find ways to optimize models and need less? Only time will tell.

by bluGill

6/29/2026 at 6:25:49 PM

I think you have too much faith in context AGI.

at 128GB, you can find almost it's entire context for Qwen3.6 35B MoE.

Again, I think you have too much faith in extrapolation. It's like you got a baby at 0 months, then measured it at 12 months and expect it to be a giant.

by cyanydeez

6/29/2026 at 5:50:06 PM

It can't run the latest models today - GLM-5.2 class models already need 1TB+ of RAM.

... but, the models that WILL run on 128GB (or 64GB or even 32GB) models today are a huge improvement on the best models that would run in the same amount of memory six months ago.

by simonw

6/29/2026 at 5:43:05 PM

In 5-10 years, incremental cloud tokens will be far cheaper (likely but not guaranteed).

by someperson

6/29/2026 at 6:06:08 PM

[flagged]

by jubilanti

6/29/2026 at 5:54:14 PM

I have a 1500 dollar machine that can run it at 50 tok/s (3 V100s)

by georgeven

6/29/2026 at 6:25:33 PM

How did you buy 3 V100's for $1500??

by Dig1t

6/29/2026 at 6:39:27 PM

i like that people are taking the privacy argument seriously, after however many decades. i think there are other arguments to be made for running these locally which are less settled, but IMO the Fable debacle drives it home: the surest way to embrace this technology without worry that it will be taken away from you down the road is to physically own the compute.

by colinsane

6/29/2026 at 6:17:07 PM

AMD started their 128GB Halo Strix at a pretty damn good point at ~2.5k; I got mine after the first memory bump at $3k.

I think you might be a little to into the stew here.

by cyanydeez

6/29/2026 at 6:45:41 PM

I got mine at the same price point, and I've been pretty pleased with it. Tailscale lets me use it from my ultrabook / lightweight laptop, no burning lap or crazy fan noises. Desktops with the amd ai+ 395 are still fairly affordable for what they can do.

I haven't tried it with https://lemonade-server.ai/ yet but I just might give it a shot.

by zdragnar

6/29/2026 at 6:53:17 PM

I'm running Lemonade on Nixos on my Framework Desktop. I had been trying other tools out before finding Lemonade, but Lemonade really made it plug-and-play.

by organsnyder

6/29/2026 at 5:48:46 PM

a lot of credits? we can’t predict any price change for them

by oldfuture

6/29/2026 at 5:58:10 PM

How many credits would it buy? How long would it take to use them up? What's the payback period?

From what I understand, for a developer, $5000/month is maybe the high end, but $5000/year is fairly standard. (Is that accurate?) So if it pays back in 15 months, that's pretty decent. If it pays back in two months, that's spectacular.

by AnimalMuppet

6/29/2026 at 6:52:21 PM

Using some rough napkin (well, spreadsheet) math, if you ran Qwen 27B for every minute every day at the current price of $0.195/$1.56 with a 2/3 input to output ratio (eg. agentic coding) at the advertised 22 tps it would take you just about 11 years to get to ~$5000 dollars spent.

by dminik

6/29/2026 at 6:16:59 PM

Are you comparing the cost of hosted Opus to running Qwen 3.6 locally? That doesn't really seem fair.

by eli

6/29/2026 at 5:44:58 PM

What kind of narrative are you trying to push?

Do you know how much VRAM/unified is needed for the 27B model, which is generally regarded as better between the two compared in the article, is needed with little to no KLD loss and at 256k context?

Also, once you worked out how much memory is needed for that, maybe tell us how much a non-Apple system that you can run that (probably similarly or faster) would cost?

And when you have answered that, can you tell us how much privacy costs? Maybe also tell us how private OpenRouter is?

Edit: looking at other replies that are basically pointing out the same thing I did, I guess it's my wording. It's frustrating that people who misinform others in some nicely packaged ways or just simply uninformed get to keep doing that if they sound nice. Thanks.

by h4ny

6/29/2026 at 5:49:38 PM

> maybe tell us how much a non-Apple system that you can run that (probably similarly or faster) would cost?

Ryzen AI Max 395+ with 128GB of unified memory can be found around $3-4k.

But 27B isn't that large, either, especially if you are ok with the quantized models. So this laptop choice seems to more be a "because they had it" rather than "this is what's necessary for this particular workflow"

by kllrnohj

6/29/2026 at 5:55:18 PM

That's my point. You can run Qwen3.6 27B with MTP and whatever else you want to bolt onto it at 256k context for much less than even a Ryzen AI Max 395+ with 128GB would cost. Even unquantized you don't need 128 GB so given your comment and the downvotes maybe I didn't word my original comment properly for this?

by h4ny

6/29/2026 at 5:34:30 PM

None of the examples reflect 'real work', at least not what I'd consider real work. Being able to nail a zero-shot greenfield project is relatively easy even for a small model. There's not much context to build up and it can fall back to similar examples in the training data easily. So long as you're not asking it to invent something wholly new it'll probably manage.

The real test is whether or not it can work with your existing codebases. In my limited experiments Qwen 3.5 (maybe 3.6 is loads better) does OK on a Rust+React app, and less well on a C# monolith. Not to the point of being unusable but definitely poorly enough that I went back to Claude after 20 minutes. If I lost access to a cloud model and had to use Qwen instead I'd be visibly sad.

by onion2k

6/29/2026 at 6:20:35 PM

> Being able to nail a zero-shot greenfield project is relatively easy even for a small model

Not really germane to your comment but I hope I don’t sound old when I say I remember a time when spinning up a PoC was a week of work, and a statement like yours was pure science fiction.

by janalsncm

6/29/2026 at 6:26:52 PM

I love the ability to spin up any repo on github by pointing a local model at it with zero cost beyond the heat & electricity.

by cyanydeez

6/29/2026 at 6:02:19 PM

In my experience, even with basic project concepts the small models struggle to spin up greenfield stuff. There's just too many decisions to be made and they're not good at that.

Modifying existing code is way easier if you don't expect it to be smart about it. Don't say "add X feature" and let it explore the codebase and build its own understanding. Point it at the relevant files and say "the goal is to add X feature to this code, follow Y guidelines". Now you've done the hardest part of making the decisions and it just has to follow instructions while coloring within the lines.

by sosodev

6/29/2026 at 6:40:54 PM

>Point it at the relevant files and say "the goal is to add X feature to this code, follow Y guidelines".

Is that not how you would work with any model, local or not? I wouldn't trust it to make the right decisions unattended. I just know the moment I look away it's going to do something utterly braindead.

by fluoridation

6/29/2026 at 5:50:17 PM

> In my limited experiments Qwen 3.5 (maybe 3.6 is loads better)

1. Maybe you should tell us what those limited experiments are.

2. Maybe you should actually try 3.6 because it's huge difference in most cases. Don't forget to tell us quants and don't forget to tell us scope.

3. Maybe actually show us data compared to frontier models instead of this... vibe comment. Pretty tired of this kind of comments on HN that doesn't require logic or evidence. Just vibes. Like the pelican riding a bicycle crap that everyone has taken for granted but has no objective way of assessing goodness.

by h4ny

6/29/2026 at 6:13:50 PM

I feel like I'm going insane seeing people buy these 128gb MBP for thousands of dollars to run models that are objectively much worse than SOTA and spending so much more. The amount spent on a 128gb M5 MAX can buy you a damned new car here. What the hell am I missing? Are developers in other countries living in such different worlds?

(I'm aware the price is, in absolute terms, more expensive where I live compared to the USA. That reinforces what I think, because anyone sane that would've bought one of those in another country would sell them as soon as they landed here and save that money.)

by doodlesdev

6/29/2026 at 6:31:33 PM

I also don't understand why people in this price bracket are buying Mac laptops instead of desktop computers with GPUs? Just to flex that it's portable?

by JeremyNT

6/29/2026 at 6:55:13 PM

A mac with a boatload of RAM can run models that will exceed the limits of any GPU not worth at least twice the Apple hardware itself.

You get fewer tokens per second, but at some point the balance between quality and quantity makes the large model size worth the spend.

When you're spending this kind of money, you may as well treat yourself to a pretty screen and some decent speakers. Nothing the competition doesn't offer these days, but you get them for free with the car-priced RAM upgrade so why go for less.

by jeroenhd

6/29/2026 at 6:56:03 PM

What GPU can I buy with >100GB of memory?

by ilogik

6/29/2026 at 6:23:38 PM

Yes they are, 6k is peanuts to a lot of people.

by adamors

6/29/2026 at 6:43:28 PM

I think things are moving fast, tested that new vibethink-3B, works on many small tasks/fast, and playing with ornith-35B with a draft vibethinker-3b as a draft gave me some good speed/results.

Was just trying to see how small I could go and get acceptable results, but yeah, larger Qwen 3.6 with MTP is going to be better. Cant wait to see how AI model (unsloth/local-llm/heretic/reaper/etc communities) are tweaking/engineering quality down into smaller models. Lots of new things coming out.

by IronWolve

6/29/2026 at 6:10:43 PM

I've been working with local models for the past year. There's so many possibilities, but I don't think coding is one. Coding requires so many layers beyond inference; I spent so much time trying to replicate what Claude Code does end to end locally. Understanding all the layers and keeping up with the advancements feels like a slog. Even this article messes up and misunderstands what some of the settings are doing. Qwen in particular seems to work at first, then often gets stuck in thought loops when used for actual work.

However, text-to-speech, speech-to-text, and non-code LLM use cases are so useful to have local, and don't require big hardware.

Having a universal reliable inference engine interface, I think, is the big unlock that needs to happen before app devs can ship these features.

Personal concrete use case: meeting recording app. This uses Parakeet + Qwen to create local transcriptions and post-cleanup, respectively.

Right now this app has to download and manage all these models, then bundle an inference engine to run them. It's a lot of code that probably should belong to the OS, or at least a standard interface.

While apps can offload some of this to llama.cpp or a similar process over http, that's another set of setup for the user to do before they can have a useful app.

Anyway, if you're getting started on a Mac, I'd suggest trying out oMLX (https://github.com/jundot/omlx) before messing with llama.cpp. In particular they have community benchmarks so you can see what kind of performance you're likely to get: https://omlx.ai/benchmarks. I wished each one had more configuration details though.

by blopker

6/29/2026 at 6:16:00 PM

> I don't think coding is one

Certainly this is falsifiable easily by any of us doing it on a regular basis

> Qwen stuck in thought loops

This does happen when context is not managed effectively; creating plans, using subagents and compactions strategically resolves this

by iwontberude

6/29/2026 at 5:47:14 PM

FWIW I'm running gemma4 31b on my 5090 and it's pretty great as well.

QAT, MTP, 128k context.

I liked Qwen 3.6 27b too, it just seems that Gemma4 is a bit underrated.

by beastman82

6/29/2026 at 5:52:50 PM

My experience also aligns with this. I'm running gemma4 31B on a 4090 through llm.cpp with unsloth models. I also run Qwen 3.6. Qwen is good for thinking and planning as it is faster, but Gemma4's generated code is much higher quality in the first try (Rust, C++ and C#). so it needs less revisions to be at a level I'm comfortable for merging.

by kofu

6/29/2026 at 6:19:31 PM

I second unsloth models. I'm using them over blackwell-oriented nvfp4 models as they are (empirically) top quality and performance.

by beastman82

6/29/2026 at 5:52:21 PM

Nice. I flip flop between Qwen 3.5 9B Q6_M and Gemma4 12B Q4_K_M on a 4080 Super. They run at about the same speed and I can have them review each other's plan or diffs. For smaller projects I find them very capable, and I can step up to a better quant for slightly more challenging work.

by accrual

6/29/2026 at 6:31:03 PM

you can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B.

by nok22kon

6/29/2026 at 5:33:50 PM

> ... on my Macbook Max M5 128 GB

Local development for who? How many of y'all are rocking 128GB of memory? Am I reading Apple's site correctly that it's a $10,000 laptop?

by 0x0000000

6/29/2026 at 5:44:04 PM

You don't need nearly that much RAM to run Qwen 3.6 27B, though. qwen3.6:27b-q4_K_M is only 17GB, for example.

by kllrnohj

6/29/2026 at 6:13:42 PM

This is what I run on an M5 MacBook Air 32GB. Works great.

I’m not having it build whole features from scratch, though. I give it pretty explicit instructions closer to the class or function level, and it still saves me an immense amount of time, while I’m very connected to the code that’s written.

Definitely the sweet spot for me.

by DanHulton

6/29/2026 at 5:36:19 PM

It wasn't $10k a month ago

by wpm

6/29/2026 at 5:49:09 PM

I'm on 128GB ram strix halo, bought framework desktop for a few thousand CAD back when everyone was calling framework desktop overpriced

by __s

6/29/2026 at 6:09:12 PM

Think commercial. My company invested in a local rig since privacy is important to our customers and sometimes I want to use these models on private data.

by mr_mitm

6/29/2026 at 5:50:16 PM

A 27B model can fit easily on a 32GB VRAM card (e.g. 5090) or a 32GB computer in RAM at FP8/Q8 (unsloth have 28.6GB Q8 files).

For 24GB VRAM cards (e.g. 4090) you can use Q6_K (22.5GB) or Q5_K_M (19.5GB) quants, possibly offloading some of the weights to RAM.

by rhdunn

6/29/2026 at 5:44:21 PM

Certainly won't work on my M4 Pro with 24GB lol

by spike021

6/29/2026 at 6:18:55 PM

I’m using it on a 48GB machine and it causes some lag, so it might be worse on 24, but it should run.

Unsloth recommends 18GB of RAM for Qwen3.6-27B (for their version of the model).

https://unsloth.ai/docs/models/qwen3.6

by MatthiasPortzel

6/29/2026 at 5:54:33 PM

I feel you!

Sent from my 8gb M2 Mac mini.

by whynotmaybe

6/29/2026 at 6:16:34 PM

Is there any hope for people that cant even run 27B parameters, Qwen3.6 or otherwise? Are there any quantized models that do well with tool calling at smaller parameter sizes?

I do not have a crazy rig, a modest gaming one at that, but in trying to understand more about agents and their capabilities, I am SOL with my 16 GB of RAM and 8GB of VRAM. I can get most small, non tool calling models to perform well, but I've had major issues with anything over 9B doing anything more than reasoning (egregiously slow at higher parameter counts).

And so far, I cant get even Pi to extend itself or do any meaningful work with any of the models I currently can get to run.

by Otternonsenz

6/29/2026 at 6:26:34 PM

I suspect with those specs, you're not in the game right now for reliably using local models for code generation. The easiest way in is a MacBook with at least 32GB of RAM. This should be able to run a 4bit quantization of qwen 3.6 using the MLX format really well.

by fumeux_fume

6/29/2026 at 6:46:46 PM

I think at 16 GB you'd struggle to run the regular development tools nowadays, forget about any interesting inference.

by fluoridation

6/29/2026 at 6:20:51 PM

[dead]

by jadbox

6/29/2026 at 5:28:07 PM

I have been having pretty good success with Qwen 3.5 9B for "nontrivial but not challenging work all things considered" -- it runs great on my 24gb unified memory m4 pro MacBook Pro. What do the baseline specs look like Mac-wise for getting this model to run? Am I looking at a 96gb? 128? 256?

by rhgraysonii

6/29/2026 at 6:23:29 PM

I posted this elsewhere, but Unsloth says the 27B model should run in 18GB. That leaves little RAM for other tasks, but it depends on your tolerance for slowness I suppose. I haven’t tried it in 24GB so report back if you do.

https://unsloth.ai/docs/models/qwen3.6

by MatthiasPortzel

6/29/2026 at 5:31:12 PM

You might be interested in Ornith 1.0 9B, which is a new intriguing post-training of Qwen 3.5 9B.

Qwen 3.6 27B will run in full offload with a 4-bit quantisation in 64GB on an M1 Max. It is quite slow.

I don't know about 48GB but 64GB should be enough.

by dofm

6/29/2026 at 5:57:52 PM

I've been trying Ornith 1.0 35B, I'm pretty impressed with it: https://simonwillison.net/2026/Jun/29/ornith/

by simonw

6/29/2026 at 6:40:41 PM

It's the one I have loaded right now.

It got rather tangled up when I tried it with one of my coding tests, which is a simple wordpress plugin, but I frustrate the model by asking it to write code for older PHP, break WP coding conventions and use a rather bespoke method for arranging code in objects. So it is sort of a hybrid of a green field and brown field task; a bit muddy.

It did not do as well as Qwen 3.6 35B, but the way it worked through its thoughts was interesting.

TBH I struggled to understand what DeepReinforce are doing that is materially different; the explanation of their training technique goes over my head at this point.

by dofm

6/29/2026 at 5:33:02 PM

Thanks! I was thinking of doing the 128gb to have some future proofing. I figure at this point, it's akin to a mechanic keeping great tools around, when it comes to having this sort of homelab and exposing it for your own uses. And great practice for building the next era of user facing computing that will be around as this proliferates.

by rhgraysonii

6/29/2026 at 5:40:28 PM

I would not buy a 64GB model again, probably, if this were to remain particularly important to me. But I gather memory bandwidth is pretty important here.

So for example I'd favour a used M1 Max over a used M2 Pro, at least based on my naïve understanding. Not quite sure where the balance changes.

There appear to be some hardware improvements with the M3 and up regarding the Apple Neural Engine which I'd hope would show up in MLX performance; I remember seeing some optimisations in image generation models that are only possible on later hardware.

The GPU cores are progressively better I believe, but the memory bandwidth is lower. Though perhaps the M4 can get closer to actually saturating said bandwidth.

(And I must reiterate that my understanding of this stuff is pretty naïve.)

by dofm

6/29/2026 at 6:41:27 PM

What do folks use to keep on top of new model releases that are appropriate to their system? i.e. the models that will actually work on the MacBook Pro with 48GB of RAM or whatever their specs are.

I've seen sites here and there but they feel like quick little toys that don't get updated, so they always suggest old models.

by dom96

6/29/2026 at 6:13:58 PM

I'd also look at the qwopus distil if you're using qwen 3.6 27b. It's a nice refinement of the current 27b with slightly better stats.

Jackrong has a few different ones available depending on what you're trying to do: https://huggingface.co/Jackrong

by jjcm

6/29/2026 at 5:38:31 PM

Call me back when you can run these models on 16GB of RAM and any recent i5/i7. Until then, there’s no point on using these toy models.

by RedCinnabar

6/29/2026 at 5:39:21 PM

You need it to run in about 8 GB so you have extra space for the context window.

by giancarlostoro

6/29/2026 at 5:40:42 PM

Hello, it's the internet calling, today is that day.

https://github.com/ikawrakow/ik_llama.cpp

Edit: it's gonna be slow if you're not using any VRAM. But it's possible. Software isn't going to speed that up anytime soon, it's just a hardware bandwidth limit.

by Catloafdev

6/29/2026 at 6:40:56 PM

Has anyone considered a home server? Assuming mobility is not important if we pick components to match a similar hardware would it be more value for money?

by prasanthabr

6/29/2026 at 5:22:28 PM

I've come from the future to say Qwen 3.7 27B is just around the corner and slaps!

by aand16

6/29/2026 at 5:47:01 PM

Are RAM prices down?

by layer8

6/29/2026 at 5:24:15 PM

Do no give me hope like that.

by lor_louis

6/29/2026 at 5:29:28 PM

I am eagerly waiting!

by mendeza

6/29/2026 at 5:46:35 PM

I was interested to see that Qwen3.5-122B-A10B narrowly beat Qwen3.6-27B on Donato Capitella's SWEBench-verified-mini run with a similar 128GB UMA architecture.

https://pi-local-coding-bench.dev

by seemaze

6/29/2026 at 5:32:43 PM

> What it does:

>

> --jinja for tool calling support

Pretty sure this flag hasn't done anything for a while. It's enabled by default since ~November of last year

by kpw94

6/29/2026 at 6:15:49 PM

27-30B in general seems to be the level where you actually start having decent models. I just wish consumer hardware hadn't stagnated so much that we can't easily go higher than that, and that even running those requires a $5k machine now.

by SkitterKherpi

6/29/2026 at 5:22:40 PM

And AI companies will continue to buy up all the silicon to make this prohibitively expensive to run at home.

by HotGarbage

6/29/2026 at 5:28:07 PM

It will run (somewhat slowly) on a five year old M1 Max with 64GB RAM.

Personally I prefer the 35B MoE model, which is fast enough to be interactively useful, and capable, but I would probably use the 27B if I wanted to generate whole applications like that.

I am unconvinced that most "local" AI applications need anything much more powerful than the Gemma 4 12B model. Local agentic coding is a small niche, but there are plenty of ways a local model can help with development tasks.

I would really like to see a 12B or 16B Qwen 3.6.

I am currently playing with Ornith 1.0 in the MoE configuration, which is based on the 35B variant of Qwen 3.5; I am not sure if it is better than the 3.6 version.

Benchmarks say it is; my own silly tests either suggest otherwise or suggest that I have to talk to it a bit differently.

by dofm

6/29/2026 at 5:45:27 PM

I need to ask, since I have desperately wanted to make Gemma 4 12B work, but im not sure if its the quant (i usually up it to q8, which is a lot higher than iq4_nl that i use for 3.6 27B) or the model itself, but it just starts confusing itself really quickly when I give it coding tasks. And quickly starts failing tool calls.

I really want to have a model that i can run locally on my 24gb m4 pro mbp for when i don't have internet to connect to my 3090 running the qwen, and i love how gemma 4 models 'feel', but i can't make them be competent. I am in the middle of finetuning both qwen3.5 9B and gemma 4 12B just to try and make those bridge closer to 27B for coding/agentic tasks (and am trying to ternarize and DQT 27B so that it fits in ~9gb pre-KV).

How do you run the gemma? What do you use it for (and in what harness), maybe llama.cpp and pi-mono just aren't for this model and that's what i'm doing wrong.

by sleepyeldrazi

6/29/2026 at 6:23:25 PM

It sounds to me like you're further along on this than I am, if you are fine tuning?

I am still mostly tinkering/learning rather than spilling out code, and I feel quite slow on it. So it doesn't matter too much to me if it is really slow. More the journey than the destination if that makes sense. I'm stubborn.

I have tried the Gemma 4 12B model (Unsloth's QAT version) with search/browse tools in LM Studio and Unsloth Studio, when I am trying to understand a new thing.

Basically I get it to write introductory starter documentation for me to absorb, because my big personal problem, these days, is focussing enough to start a project and then digging in; I need the help.

I have found its limits on obscure packages (that it sometimes makes up) but before that it's a bit like stumbling on a blog post that happens to be really right for your particular need. Good enough to work through.

It's stuff I could ask Perplexity to do, or ChatGPT, to be fair, I just like LM Studio for this and have the inquisitiveness to want to run it locally.

In your case: I don't believe it's the quant. I'm sure it's the model — it has good coding knowledge but it's clearly not specialised. It might be good enough at writing Python/PHP/JavaScript at a novice level. It is also quite good on WordPress tooling and functions.

But I wouldn't bother with it for agentic coding if you've got experience elsewhere. Might be interesting to see what you can do with the 9B Ornith model?

Qwen 3.6 MoE in its Unsloth version is another matter. Impressive and I am trying to find ways to support my old brain doing what I've done before.

by dofm

6/29/2026 at 5:55:29 PM

Something I find really confusing from this post is the MLX versions of the model running much slower. As I understand it, these model versions are meant to take advantage of Apple Silicon and MacOS APIs, and should produce better/faster results. Any insight into what’s happening here?

by mbgerring

6/29/2026 at 6:14:52 PM

I've tested it extensively for actual local development for my job, and hard disagree here. It's a waste of time to use it. Wish it were not true.

by markdog12

6/29/2026 at 6:20:33 PM

I posted elsewhere but if you have more space try gemma4 31b

by beastman82

6/29/2026 at 5:29:50 PM

How does llama.cpp use the GPU efficiently as opposed to MLX?

Is there any way to use MLX and GPU at the same time? Or does memory become a big problem?

TBH, I never understood Apple hyping these neural cores because I didn't think anyone actually uses them except maybe certain photo/video editing software.

If I can generate voice at the same time as video, that would be useful.

by blobbers

6/29/2026 at 5:46:44 PM

Llama.cpp uses the GPU very effectively because inference of LLMs is very rudimentary and basically as simple as your GPU memory bandwidth. That's essentially the baseline performance ceiling, with model-specific optimisations like MTP potentially increasing it.

The neural cores aren't suitable for LLMs/transformers and isn't used in LLM inference. On the M5 and later chips, it comes with neural accelerators, aka Tensor Cores, which speed up the 'prefill' (i.e. processing your context window) part, but don't do anything for inference.

The MLX vs GGUF debate is mostly irrelevant. The GGUF pathways are optimised for apple silicon to the extent of practically identical performance to MLX. MLX is just one way of using Apple GPUs, it comes with many optimisations in the box, but they're not hard and they're no longer MLX-exclusive.

by dannyw

6/29/2026 at 5:35:57 PM

Strix Halo user here. While Qwen 3.6 27B exhibits remarkable intelligence density, I will still take unsloth's dynamic IQ2_XXS of Minimax M2.7 over Q8_0 Qwen 3.6 27B any day of the week, and this isn't just because of generation speed either. I wrote my own custom harness, and I get hallucinated tool call parameters and bizarre invocations with Q3.6 27B even at Q8_0, but no issues with the IQ2_XXS of M2.7.

by anonym29

6/29/2026 at 6:14:07 PM

> I get hallucinated tool call parameters and bizarre invocations

tweaking sampler might help

by BoredomIsFun

6/29/2026 at 5:35:01 PM

none of these local models are good for development, complete waste of time. nobody has $100k+ hardware sitting around at home to actually run a good model

by mikert89

6/29/2026 at 5:36:54 PM

skill issue

by jlongr

6/29/2026 at 6:06:02 PM

Gemma4 31B with MTP enabled is faster and I feel a bit stronger at coding. Either one can run in 32GB VRAM or unified RAM with some tuning (3 bit weights, 8 bit kv cache)

by cat_plus_plus

6/29/2026 at 6:15:50 PM

Local models are great for a lot of things past just software development. We need to move towards solving other real world problems vs just building software. I've been focused on that with TxtAI (https://github.com/neuml/txtai) for 6 years now.

by dmezzetti

6/29/2026 at 5:33:27 PM

Very capable lora adapters are surfacing but it seems they are very niche.

by ascii0eks84

6/29/2026 at 5:47:44 PM

Can you share more? It’s the first I hear of lora outside research papers. Practical applications would be great to see.

Lora if effective could be a great reason to run local models.

by DenisM

6/29/2026 at 5:25:07 PM

Spent a week trying to get sensible results out of llama 3.3 At one point it even simulated doing the work, log output and everything and when I challenged it about the missing artefacts it actually started questioning my intelligence. Seems appropriate for a Zuck enterprise.

Qwen on the other hand got straight to work with astonishing competency on the same system.

From what I read llama3 needs beefier compute to reliably invoke tools, which I presume relates to it focussing more on simulating AGI rather than being a useful tool.

by rusk

6/29/2026 at 5:26:30 PM

llama 3? Are you from 2023?

by am17an

6/29/2026 at 6:22:10 PM

[flagged]

by suthakamal

6/29/2026 at 6:14:53 PM

[dead]

by CurbStomper

6/29/2026 at 5:20:05 PM

This is kind of like saying grass is green to be honest

by 217

6/29/2026 at 5:29:43 PM

Like everybody got 128 GB RAM..

by madduci

6/29/2026 at 5:39:18 PM

I've been running it almost since launch on a 3090 (24gb vram), you really don't need that much. Second hand those are really cheap and i get 50-70 t/s (with MTP at 2), full ctx. IQ4_NL (unsloth) on this model seems suspiciously competent, and after the (by now not so recent) updates to q4 KV on llama.cpp, I just keep going back to it after dsv4pro disappointed me for the 100th time because it gave up on a task.

by sleepyeldrazi

6/29/2026 at 5:31:53 PM

Doesn't need it at Q4 at least; it'll run in 64GB.

by dofm