alt.hn

7/11/2026 at 10:38:57 PM

Mesh LLM: distributed AI computing on iroh

 https://www.iroh.computer/blog/mesh-llm

by tionis

7/12/2026 at 1:43:52 PM

I have a macbook pro, figured I'd see how easy it was to contribute some vram...

And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc.

It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!

by maccam912

7/12/2026 at 2:51:03 PM

[flagged]

by jjheuaao

7/11/2026 at 11:25:19 PM

I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.

Are we talking 1 token per second for a split model? Less?

Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.

by SwellJoe

7/12/2026 at 7:49:50 AM

> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk)

Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why:

- when offloading the weights to RAM or NVMe, you need to transfer the massive weights from your slow storage to the GPU for each layer being processed for each token. And as such you are being bottlenecked by the transfer bandwidth (which is either the men bandwidth of your DRAM or the read speed of your disk)

- when using a distributed setup, the weights stay in the VRAM on each machine, the it's the GPU memory bandwidth that matters for the weights, and it's much higher than the two other bandwidth discussed above and as such the bottleneck isn't here. You need to tranfert data from a group of layers sitting on one device to the next one another device, but the amount of data is much smaller than the weights (we're talking about kilobytes of data, not gigabytes) so the network throughput isn't a limiting factor.

The limiting factor is the network latency: if you split your model between 4 devices, you'll have 3 times the network latency per token. If you're on a network with 1ms latency, that means 3ms of latency per token. Which means the theoretical upper bound for your inference speed without speculative decoding is 30tps (this theoretical limit assumes the computation itself is instantaneous).

So this is unlikely to be practical over the internet (too high of a latency) but on a local/enterprise network with speculative decoding it could totally work.

Edit: note that all of the above is about token generation, for prefill/prompt-processing the distributed setup will almost certainly win (because in this case, the network latency doesn't add up)

by stymaar

7/12/2026 at 11:54:58 AM

I’m staring at this comment for a while now: With 3ms latency combined per token, wouldn’t that mean (1 / latency) = 333 token/s for the theoretical upper bound? I’m not trying to nitpick, just curious if I misunderstand something.

by ul5255

7/12/2026 at 12:27:55 PM

Indeed, I completely screwed my math up. Looks like 10am is too early in the morning for a Sunday.

33 tps max token generation speed would be for 10ms of network latency in the above example.

by stymaar

7/12/2026 at 9:20:22 AM

Ah, that's interesting. I though there was more data crossing the network. So, why does a DGX Spark come with super fast network if 10Gbps ethernet would be sufficient for splitting a model? I never bought a second Strix Halo on the assumption that the pipe between them would be a limiting factor to using larger models, so obviously there's something I don't understand.

by SwellJoe

7/12/2026 at 11:35:57 AM

The amount of data is only low for inference, not for training, and AFAIK DGX spark is supposed to be a researcher's machine that can do small-scale training.

by stymaar

7/12/2026 at 1:47:20 PM

A 1 gbe local network should have < 1 msec latency per hop so theoretical upper bound is substantially higher than 30 tps (again assuming instantaneous compute) => thus network latency should not be the limiting factor in reality, no?

by chromatin

7/12/2026 at 4:24:21 PM

On a fast network, yes. If you want to create a distributed network over the internet, then it will.

by stymaar

7/12/2026 at 12:36:37 AM

This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN.

The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.

I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.

by i386

7/12/2026 at 12:44:25 AM

What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?

by zdw

7/12/2026 at 1:12:44 AM

The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet.

We use a customized Q2 quantization that preserves sensitive tensors at Q8.

To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.

While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.

Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.

by i386

7/12/2026 at 1:37:10 AM

Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each

by wlesieutre

7/12/2026 at 6:29:48 AM

It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.

by drawnwren

7/12/2026 at 9:17:07 AM

> because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5

Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.

by embedding-shape

7/12/2026 at 9:20:53 AM

M5 max has 614GB/s, you mean the m4?

by khalic

7/12/2026 at 6:13:56 PM

Oh, I was looking at every M5 except for the 40-core M5 Max. They have 460.

by drawnwren

7/12/2026 at 12:59:49 AM

That sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.

by SwellJoe

7/11/2026 at 11:40:49 PM

Perf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.

by woadwarrior01

7/12/2026 at 2:53:25 AM

That's about the speed I get on a AMD Ryzen AI 9 HX 370 (inside a Framework 13), with Qwen3.6-35B-A3B, so doing the same on that much larger model...

by imrehg

7/12/2026 at 5:33:25 AM

The first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture.

As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.

by MattPerry

7/12/2026 at 12:21:48 AM

I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.

by i386

7/12/2026 at 5:24:57 AM

What is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?

by maxgashkov

7/13/2026 at 7:27:14 AM

Great question! We’ve had a lot of discussions about the direction we want to take this, and how to best generate some kind of incentive / fairness reward.

And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup).

When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described.

For now, the public mesh is totally open with no restrictions or limitations.

by sig_kill

7/12/2026 at 2:44:42 AM

Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.

A few questions:

1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.

2.) Any safeguards against malicious actors poisoning model activations?

by Creamsicle47

7/12/2026 at 2:54:22 AM

To be honest, both are very tough problems we don't have a good answer for yet. If that is something that concerns you, look into building a private mesh with trusted peers.

by i386

7/12/2026 at 5:28:02 AM

Thanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system?

P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.

by Creamsicle47

7/12/2026 at 1:14:58 AM

This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!

by iotapi322

7/12/2026 at 1:22:45 AM

Thank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.

by i386

7/13/2026 at 9:26:58 PM

This is surprisingly similar to what I'm trying to build (at least at the network level), down to the use of iroh. I have not attempted model splitting though, and the angle is slightly different. Major props for this, it seems like a great project. The more we can push local inference, the better for everyone. Interested to see if there's collaboration opportunities. https://github.com/dsegovia90/cocompute

by dsegovia

7/12/2026 at 2:08:03 AM

I have never really delved into kv cache implementation, do they run effectively separate caches per layer?

If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.

Is anyone doing best-of-n with a n stage pipeline running each query offset by one?

by Lerc

7/12/2026 at 2:52:14 AM

Each stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.

by i386

7/12/2026 at 3:22:35 PM

> A model gets partitioned by layer ranges into stages: layers 0 to 15 on one node, 16 to 31 on the next, and so on down the pipeline.

Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?

Is there a document explaining all constraints of this implementation?

by throw1234567891

7/13/2026 at 7:31:01 AM

It depends on the size / structure of the model… but generally you can check out the MeshLLM huggingface org for pre-computed model splits.

We have a job that will take popular models, and dice them up and post on HF.

When that specific model is served, the layers will be shovelled around the available nodes on the mesh.

by sig_kill

7/13/2026 at 9:52:51 AM

Do you have instructions how to dice models myself? Every model in your org is a measly 4bit. I kinda hoped it is as simple as taking a GGUF and throwing at Mesh LLM but I fear it's not going to be that straightforward :)

by throw1234567891

7/12/2026 at 8:12:32 AM

Is it a fully custom inference engine or are you reusing parts of an existing stack? (llama.CPP, vLLM, etc.)

by stymaar

7/12/2026 at 9:24:20 AM

Our skippy library is a patch queue on top of llama that allows us to access internal information, such as activations, and filter tensors on model load.

by i386

7/12/2026 at 6:11:13 PM

This really should be in the blogpost. It’s both useful info and basic courtesy to be explicit about which underlying inferencing engine you are using

by rao-v

7/13/2026 at 7:32:30 AM

We didnt post it, we use a library (iroh) who featured us - so we are here answering any Q’s instead :)

by sig_kill

7/12/2026 at 3:55:43 AM

The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?

by zmmmmm

7/13/2026 at 7:36:44 AM

Yeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties.

by sig_kill

7/12/2026 at 4:41:56 AM

I wonder how security is done in this engine, since it's accepting input from anyone. llama.cpp's RPC layer seems to says that you shouldn't run it in public (I assume because it is lower level and may result in RCE on your GPU)

by whs

7/12/2026 at 2:15:37 AM

Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?

by DerivativeBS

7/12/2026 at 2:39:22 AM

It can dynamically route. If a machine drops out of split, the topology is recalculated and the request is automatically retried.

by i386

7/12/2026 at 11:24:53 AM

Would this benefit from integrating with the Colibri project announced here just days ago?

by cromka

7/12/2026 at 6:56:06 AM

[flagged]

by keynha

7/12/2026 at 4:08:03 AM

I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.

Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.

Nicely done OP.

by Abishek_Muthian

7/12/2026 at 6:36:07 AM

Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.

by unrvl22

7/13/2026 at 10:33:55 AM

This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM

by sig_kill

7/13/2026 at 6:28:14 AM

True, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.

by Abishek_Muthian

7/12/2026 at 2:02:59 AM

I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow.

What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.

by dwoosley

7/12/2026 at 6:00:34 AM

I've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research.

AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.

I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.

by whs

7/12/2026 at 7:24:57 AM

I spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.

by kennywinker

7/12/2026 at 9:25:15 AM

I’d love a bug report - we can get it working for you!

by i386

7/12/2026 at 7:06:15 PM

Let's say I have a bunch of computers (Raspberry Pi 3+, MBP from 2017, Lenovo T420) sitting around that are on my local network. All are from the last 15 years and my main driver is a M2 MBP with 32 gb of RAM.

Would this let me utilize that spare hardware to do anything meaningful with a self-hosted coding-focused LLM at a somewhat-decent token generating speed, or is this still a pipe dream?

by lenerdenator

7/11/2026 at 10:58:20 PM

I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.

by jmercouris

7/12/2026 at 8:46:22 AM

Throughput is not a problem as you just share relatively small vectors (a few kilobytes in size), the key issue is network latency.

by stymaar

7/13/2026 at 7:04:05 PM

I didn't mean the throughput of the network, I meant the throughput of the calculations. Yes, the latency is the issue.

by jmercouris

7/12/2026 at 12:38:03 AM

That isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.

by i386

7/12/2026 at 3:20:25 AM

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.

by nullc

7/14/2026 at 4:52:47 AM

@null not as yet, but did a tiny bit of research before. If i understand it this would help mainly with batching (ie concurrent sessions/users/turns etc) to keep the pipeline more busy (vs a batch of a fixed size going together), which is interesting, but most work so far has been on latency hiding for straight line performance to start.

We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss).

tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!

by michaelneale

7/14/2026 at 6:49:16 AM

I'm looking at adopting a mesh network architecture for self-sovereign inference across longitudinal health records.

by enbarca

7/11/2026 at 11:35:36 PM

Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?

by darkpicnic

7/11/2026 at 11:54:54 PM

I'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.

by tekacs

7/12/2026 at 12:59:46 AM

Just wondering, why do you care about encryption in this context?

by metadat

7/12/2026 at 1:25:02 AM

If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.

by darkpicnic

7/12/2026 at 2:03:32 AM

Even if transport is encrypted, the LLM computations will always be clear text, right?

by oezi

7/12/2026 at 9:19:20 AM

Indeed, it's in-transit-encrypted so snoopers won't be able to see it, but it's not E2E encrypted nor in-process encrypted, the one's doing the inference could obviously see the input/output.

by embedding-shape

7/12/2026 at 11:39:52 AM

I knew this was possible, i asked chatgpt about a year ago and it said no the latency would be too big of a problem. I spent the best part of a year learning libp2p and was looking for a project to do with it at the time.

by dana321

7/12/2026 at 12:59:25 PM

Does this support Qwen 3.6 (e.g. 27B) and the myriad of llama.cpp options (batch sizes, quantization, etc.)?

I'd love to see some performance data.

by roger_

7/13/2026 at 10:34:53 AM

Yes! I’ve worked on the settings interface between our runtime and llamacpp, these are documented and available via our config.toml file

by sig_kill

7/12/2026 at 4:00:34 AM

The real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.

by whatjustin

7/12/2026 at 7:16:43 AM

All these ASICS being designed and specialized for AI but none seem to be being built for consumers. Reason?

by SubiculumCode

7/12/2026 at 10:20:43 AM

the https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.

by vigsterkr

7/12/2026 at 6:35:24 PM

They might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.

by derdi

7/11/2026 at 10:49:04 PM

It sounds like iroh enables distributed compute without having to finangle custom hardware.

by turtleyacht

7/12/2026 at 8:29:26 PM

So, how is this different then exo ? it does the same job as well

by s4saif

7/12/2026 at 8:56:40 PM

exo is Mac only.

by i386

7/13/2026 at 9:57:53 AM

no it isn't

by throw1234567891

7/12/2026 at 1:27:58 PM

Is this truly more secure though? The host can still see your data.

by Onavo

7/12/2026 at 7:36:36 AM

Is there a catch? If not, this would be super useful.

by josefrichter

7/12/2026 at 8:15:56 AM

The catch is that the token generation speed is going to be limited by network latency, making it unbearably slow to run over the internet.

It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).

by stymaar

7/13/2026 at 12:41:29 AM

How does this differ to darkbloom?

by brcmthrowaway

7/11/2026 at 11:19:33 PM

cocompute.ai is already doing this really well.

by darkpicnic

7/11/2026 at 11:22:06 PM

Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).

by SwellJoe

7/11/2026 at 11:28:43 PM

Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!

by darkpicnic

7/11/2026 at 11:23:23 PM

Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.

by dnoberon

7/12/2026 at 7:40:09 AM

distributed AI computing so your hallucinations can be geographically diverse too

by luciana1u

7/12/2026 at 12:36:47 AM

I just wish I had the hardware to try it out!

by _superposition_

7/14/2026 at 5:25:29 PM

[dead]

by guillaume_code

7/12/2026 at 11:19:42 AM

[flagged]

by tangsoupgallery

7/12/2026 at 7:31:18 AM

[flagged]

by bubbi

7/11/2026 at 11:01:00 PM

[flagged]

by nttylock

7/12/2026 at 8:01:55 AM

[dead]

by jkwang

7/12/2026 at 1:34:21 AM

[dead]

by jing09928

7/12/2026 at 1:03:04 PM

[dead]

by hydrogenbon007

7/12/2026 at 12:24:45 AM

[dead]

by tlongwell-block