alt.hn

8/14/2026 at 5:58:16 AM

A simple fix for LLM tail latency

 https://engineering.myhoai.com/posts/a-simple-fix-for-llm-tail-latency/

by oskrim

8/17/2026 at 11:23:22 PM

You don't have to send every single request twice, just the ones that are haven't returned in time. Wait until some threshold, such as your p95 latency, and send your backup request after that. Return whichever request comes back first, and it should cut your tail latency without doubling your cost, since it only duplicates the small % of requests at the tail.

Google calls this a 'hedged request': https://cacm.acm.org/research/the-tail-at-scale/

by ak_t

8/17/2026 at 9:53:29 PM

Sending two identical parallel requests is the classic approach. But, logically speaking, it should also double the cost.

I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.

I wonder if higher-availability tiers of LLM providers do a similar thing internally.

by nine_k

8/17/2026 at 10:51:11 PM

Token caching might help here, but if it returns the same result, faster, for the same price as priority, seems good

by ImPostingOnHN

8/18/2026 at 1:40:01 AM

I wonder how parallel token caches are, like when exploring a tree of sample continuations.

by awwaiid

8/17/2026 at 9:49:31 PM

Nice turn around, does anyone has a benchmark regarding other types of requests (priority vs send twice) other than voice/call? Or the tests already test that?

by dvaplima

8/18/2026 at 4:59:58 AM

Is there a way to do this automatically when using claude/codex?

by sharts

8/17/2026 at 10:01:01 PM

Why not send it thrice?

by crisnoble

8/17/2026 at 11:29:46 PM

It turns out that best of 2 random draws outperforms best of 1, best of 3, and best of all in many load balancing scenarios: https://brooker.co.za/blog/2012/01/17/two-random.html

It's a bit unintuitive, but they key idea is roughly 'If you're working on stale load data (as always), best of 2 strikes the right balance between distributing load evenly and giving more work to less loaded hosts'. If you do 'best of k', you end up with herd behavior, overloading one host. 'best of 1' sends too much traffic to slow hosts.

by ball_of_lint

8/18/2026 at 12:31:34 AM

Be careful to not overstate the conclusion of that blog post - best of 3 wins when the cache update rate is more frequent than average task duration, and best of 2 still has herding behavior and loses to 1 random when the update rate is less frequent (which may be high load, or we may also be missing a dimension here). Since the plot chose the window where best of 2 wins, it might bias or prime the reader to believe best of 2 is usually or always best, forgetting that underloaded and overloaded scenarios are in reality more common than all servers having a nice balanced medium size load for long periods.

by dahart

8/17/2026 at 9:06:41 PM

I love this. Simple. Useful. To the point. If AI was used, I can't tell because it is clearly representing the author's beliefs.

by eigenblake

8/17/2026 at 9:44:15 PM

agreed, reads like a breath of fresh air, no fluff

by thousand_nights

8/17/2026 at 10:06:51 PM

If you want a controllable and predictable system, host it yourself. APIs will always have outages, delays and breaking changes every so often. That's the price you pay for not doing it properly and outsourcing your job.

by moffkalast

8/18/2026 at 2:03:37 AM

> host it yourself. APIs will always have outages, delays and breaking changes every so often.

Since you've solved all of these problems, including hardware, etc, you should expand this to a business! Many people would be very interested in an "Infinite 9's" (potential business name there) uptime service!

by nomel

8/18/2026 at 2:33:31 PM

Ah but as soon as you start serving clients there's maintaining a public API, juggling prompt caches from different people, etc. It's a completely different use case which vastly complicates your setup than just having one endpoint that's always there that does exactly what you need it to do with zero waiting or internet latency. It's not that complicated to set it up, a GPU or two, vLLM or llama.cpp and you're set, local APIs are drop in replacements.

Of course there's a better middle ground with renting a GPU VPS off Runpod or Vast or something, and you get most of the benefits already without having to buy currently overpriced hardware.

by moffkalast

8/17/2026 at 9:51:13 PM

for a tier thats twice the cost i would expect >2x the speed. somewhere 5-10x

e.g. 1.40m would become 0.30s.

do people really pay for these priority plans?

by ramon156

8/18/2026 at 12:39:32 AM

[flagged]

by alikhater30000

8/17/2026 at 10:33:43 PM

This is half the story; you should show performance per dollar. I doubt your 2x approach would fare well against the priority if you consider the costs.

by behnamoh

8/17/2026 at 10:52:07 PM

The article mentions that the priority tier costs 2x normal, so the costs of running normal twice should be fine.

by ImPostingOnHN