2/15/2026 at 10:47:26 PM
by jxmorris12
2/16/2026 at 2:45:46 AM
good writeup but im curious about tail latency under mixed prompts. if one request has huge context and another is tiny, do you bucket by expected decode length or just fifo with continuous refill?also did you test fairness knobs? ive seen p95 improve while a few tenants get starved unless there is some aging policy.
by umairnadeem123
2/15/2026 at 11:21:19 PM
This article does not explain what happens if the multiple prompts need different experts. Does it try and schedule the maximum number experts into memory to try and run the maximum number of prompts at once? Scheduling gets very complicated and there are different trade offs around fairness of processing which prompts at which times.by charcircuit
2/16/2026 at 4:10:10 AM
Continuous batching is a nonsense term. Systems that behave this way are usually called streaming (as opposed to batch).Batch is started and stopped and produces one result (or set of results). Computation happens when the job is started until the job completes.
Streaming is on continuously and computation happens when new data arrives. It produces a stream of results.
This is a streaming system. And most of the analysis doesn't really understand how high throughput (as in >16Mb/s/core) are architected. It doesn't even use the correct language for what is being done to the data (relational algebra) so the proper optimization techniques can't be applied.
It is however a very nice explanation of the preprocessing used in LLM systems.
by hunterpayne
2/15/2026 at 11:55:22 PM
How long until “first principles” is a meme like “considered harmful”? Or are we there already?by asteroidburger
2/16/2026 at 10:40:13 PM
Ok, we've removed first principles from the title above.by dang
2/16/2026 at 2:54:42 AM
"from first principles" has been a common phrase in science and philosophy for a long time: https://en.wikipedia.org/wiki/First_principleby wavemode
2/16/2026 at 4:57:34 AM
Sure, but that's not the way it's being used by your daily twitter/X poster.by esseph