7/15/2026 at 5:44:05 PM
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
by dwa3592
7/15/2026 at 6:11:51 PM
Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
by smeeth
7/15/2026 at 11:01:32 PM
There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat!by flockonus
7/15/2026 at 10:46:36 PM
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?by acchow
7/15/2026 at 6:13:29 PM
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.by dwa3592
7/15/2026 at 6:13:56 PM
I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).by notnullorvoid
7/15/2026 at 5:51:49 PM
> on a decent speedBut you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
by reidrac
7/15/2026 at 6:00:01 PM
I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare!https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
by LeifCarrotson
7/15/2026 at 10:25:10 PM
> It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.This is how I used to think about my 3D printer, but FWIW the way my actual thinking and planning works, print speed really matters. Not for the final print, but for iterative work and test parts, it is obvious that either having a fast printer helps. Having multiple slow printers also helps, but there are only so many areas of a design you can iterate on at once.
At the moment my own LLM use is experimental and iterative, and I definitely favour the faster MoE models for much of what I am doing, even if I might in principle prefer to get the final work done in the slower ones.
by dofm
7/15/2026 at 6:33:37 PM
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.by RussianCow
7/15/2026 at 10:38:58 PM
I'd rather have slower and better output than worse and faster output.by hx8
7/15/2026 at 9:27:54 PM
In 1980s ibm has studied and said why sub-second response needed to maintain the mental flow. That time you send a whole screen unlike unix like character by character. This proves very true even when you deal with form processing. I think that we are dealing with the same issue here.Keep your mental context in your brain is critical
by ngcc_hk
7/15/2026 at 7:11:38 PM
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
by Aurornis
7/15/2026 at 7:43:41 PM
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.by zoobab
7/15/2026 at 7:56:06 PM
It’s not a cost savings relative to paying API prices even if you’re suspending it.This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
by Aurornis
7/15/2026 at 6:19:49 PM
We clearly have different goals. I want an LLM to review my code, not the other way around.by allknowingfrog
7/15/2026 at 9:37:37 PM
I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code"Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.
Go ahead and do your bespoke coding on your side-project loves and core libraries... The stuff that will last, anyway.
But if you're working for a corpo and still doing bespoke... That's... not gonna last, I'm afraid. Well, either you remaining there, or that, as it were.
by pmarreck
7/15/2026 at 10:37:57 PM
There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to.The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It's called a "programming language".
The mistakes are always in the transfer from human to machine. I still find a high-level programming language to be the best way to express my intent. Humans will make mistakes in the hand-off to AI just like they make mistakes in the hand-off to code, but at least code is deterministic.
by allknowingfrog
7/15/2026 at 6:28:56 PM
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're backby nolok
7/15/2026 at 9:20:25 PM
> another task a lunch break whatever and it will be done when you're backAt 5 tokens per second and unknown prompt processing speed, you may need a very extra long lunch break depending on your codebase.
by Aurornis
7/15/2026 at 7:46:55 PM
How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.by allknowingfrog
7/15/2026 at 8:20:02 PM
This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws.These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
by vitally3643
7/15/2026 at 10:18:03 PM
This thread started with me saying "we clearly have different goals" and then being told that I just need to hold the screwdriver differently...by allknowingfrog
7/15/2026 at 10:02:24 PM
You don't "maintain flow." You eat lunch.I swear, tech culture has gotten people wanting to work for the machines, rather than the other way round.
by fwip
7/15/2026 at 10:23:01 PM
This was a discussion about LLM usage patterns. I'm not opposed to lunch breaks. I'm opposed to being required to take the equivalent of 12 lunch breaks a day while I wait for slow responses.by allknowingfrog
7/15/2026 at 10:29:23 PM
Right? Tech should make my work easier. Not have me stressing out even more.Let the machine do its work while I relax, I’ll check up on it later.
by ClikeX
7/15/2026 at 6:05:32 PM
We aren’t there yet. Not for frontier development work at least.by adastra22
7/15/2026 at 8:26:17 PM
Except often queued agentic flows must be checked in on. Or to use the comparison, 3D printers are not immune to making spaghetti all night when something goes wrong. (I’m not a 3d printing expert so maybe that is solved now)It is common for agents to just stop because overload or some API error hijinks.
Or you get a TUI question that is blocking.
In general you’re right though, staring at tokens from agentic is not time well spent.
Some of these I’ve built custom harness around in iterm2 though.
by bredren
7/15/2026 at 6:01:10 PM
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?by bitpush
7/15/2026 at 8:47:48 PM
Filament snaps at 1am and then you have to run print again. 10 hours turn into many days potentially.I watch tokens to see if it goes in right direction. If model goes off the rails, then it is time to stop and adjust prompt.
by varispeed
7/15/2026 at 6:14:18 PM
[dead]by ctoth
7/15/2026 at 6:01:01 PM
For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe).Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
by aziis98
7/15/2026 at 10:43:09 PM
I think 10 t/s output is usable for many coding workflows if the input speed is much much higher (~100 t/s is a rough minimum line). The low output speed can really hurt for heavy reasoning output but it can still be used to get some jobs done as long as you don't expect highly interactive use.by kevincox
7/15/2026 at 10:31:27 PM
The slower models seem fine for home lab usecases such as processing document transcriptions and tagging them, for example. I don’t need that to be live, it can just churn overnight.by ClikeX
7/15/2026 at 5:55:12 PM
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.by dwa3592
7/15/2026 at 6:08:28 PM
Ignore the haters. What you've done is incredible!by jckahn
7/15/2026 at 7:13:16 PM
If you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibriIt says so right in the readme. They’re not hiding anything.
by Aurornis
7/15/2026 at 9:54:36 PM
How fast does a human write code?by Wowfunhappy
7/15/2026 at 9:59:47 PM
It's fast if you want to automate things that run independently or overnight. It's slow if you want to iterate code together with it.by sigmoid10
7/15/2026 at 5:56:55 PM
Yep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac.Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
by bigyabai
7/15/2026 at 6:59:12 PM
> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
by Aurornis
7/15/2026 at 9:04:29 PM
>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.What are the LLM standards?
Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
by dwa3592
7/15/2026 at 10:20:21 PM
I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.
It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
by Aurornis
7/15/2026 at 10:53:03 PM
Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.by dwa3592
7/15/2026 at 9:23:50 PM
That is awesome!I am curious about the decision to not use GPU since this is for Apple Silicon.
Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
by felineflock
7/15/2026 at 9:49:35 PM
It will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.by dwa3592
7/15/2026 at 8:00:04 PM
Downloading now just 'cause the repo nameby TYPE_FASTER
7/15/2026 at 5:59:37 PM
How are the thermals? I noticed that running any serious workload locally heats system fast.by haute_cuisine
7/15/2026 at 6:08:39 PM
i have been optimizing for that. for now samosa is capped at using half of the avaiable cores and switching between them, which keeps the system 'less hot' as it would have been. i will also release better thermal control in the next release. at this point its basically sacrificing about 20% of the speed to keep the hardware less stressed (and hot).by dwa3592
7/15/2026 at 7:50:15 PM
By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/sby typon
7/15/2026 at 8:54:13 PM
Literally the only way this is going to happen is if aliens come to earth and gift us some amazing technology.by veber-alex
7/15/2026 at 8:32:48 PM
Unless there are major improvements to how much hardware it takes to run a 1T model, this is deeply unrealistic. First because why release hardware that puts your biggest customers (data centers) out of business. Second because as I understand it the data centers have bought up all the high end chip production capacity for at least the next year and unless the bubble pops that'll continue for a while.by kennywinker
7/15/2026 at 9:04:49 PM
Because for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.by iwontberude
7/15/2026 at 10:47:37 PM
First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT.Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available.
Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
by kennywinker
7/15/2026 at 6:06:06 PM
I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.by Capricorn2481
7/15/2026 at 6:19:05 PM
You are comparing a 35B models to a 635B+ frontier model, of course thats not even closeby Azantys
7/15/2026 at 7:38:19 PM
I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.by Capricorn2481
7/15/2026 at 7:51:05 PM
To be fair, they don't have the text editor highlighting all the matching parens. I'd be lost too.by IsTom
7/15/2026 at 9:15:19 PM
Yeah prediction models and many parentheses are probably not a good combination, but we're not talking about anything exceptionally complicated here. I have had syntax issues in Python as well.by Capricorn2481
7/15/2026 at 8:30:16 PM
It really depends on the language, popular languages work pretty goodby Azantys
7/15/2026 at 8:01:28 PM
try q8, check your parameters. qwen3.6-35b-a3b should definitely be able to do so with no issues at all.by segmondy
7/15/2026 at 6:02:44 PM
> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second.That is no where near decent at all.
by rvz
7/15/2026 at 6:11:52 PM
it's a 16GB machine. i am proud of this machine so far.by dwa3592
7/15/2026 at 5:46:38 PM
> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chatIt's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
by embedding-shape
7/15/2026 at 5:55:22 PM
Nothing says they're using Qwen for local development. They could be using it to for conversations, knowledge, or "creative writing."by smith7018
7/15/2026 at 10:02:28 PM
> Nothing says they're using Qwen for local developmentI know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...
by embedding-shape
7/15/2026 at 5:54:02 PM
>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build thatthat would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
by dwa3592
7/15/2026 at 9:51:38 PM
> that would have run into a race condition unfortunately ;)Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
by embedding-shape