alt.hn

3/1/2026 at 11:15:16 PM

Right-sizes LLM models to your system's RAM, CPU, and GPU

 https://github.com/AlexsJones/llmfit

by bilsbie

3/2/2026 at 5:55:22 AM

This pretty cool, and useful but I only wish this was a website. I don’t like the idea of running an executable for something that can perfectly be done as a website. (Other than some minor features, tbh even you can enable Corsair and still check the installed models from a web browser).

Sounds like a fun personal project though.

by BloondAndDoom

3/2/2026 at 11:50:01 AM

>I only wish this was a website. I don’t like the idea of running an executable for something that can perfectly be done as a website.

The tool depends on hardware detection. From https://github.com/AlexsJones/llmfit?tab=readme-ov-file#how-... :

  How it works
  Hardware detection -- Reads total/available RAM via sysinfo, counts CPU cores, and probes for GPUs:

  NVIDIA -- Multi-GPU support via nvidia-smi. Aggregates VRAM across all detected GPUs. Falls   back to VRAM estimation from GPU model name if reporting fails.
  AMD -- Detected via rocm-smi.
  Intel Arc -- Discrete VRAM via sysfs, integrated via lspci.
  Apple Silicon -- Unified memory via system_profiler. VRAM = system RAM.
  Ascend -- Detected via npu-smi.
  Backend detection -- Automatically identifies the acceleration backend (CUDA, Metal, ROCm, SYCL, CPU ARM, CPU x86, Ascend) for speed estimation.
Therefore, a website running Javascript is restricted by the browser sandbox so can't see the same low-level details such as total system RAM, exact count of GPUs, etc,

To implement your idea so it's only a website and also workaround the Javascript limitations, a different kind of workflow would be needed. E.g. run macOS system report to generate a .spx file, or run Linux inxi to generate a hardware devices report... and then upload those to the website for analysis to derive a "LLM best fit". But those os report files may still be missing some details that the github tool gathers.

Another way is to have the website with a bunch of hardware options where the user has to manually select the combination. Less convenient but then again, it has the advantage of doing "what-if" scenarios for hardware the user doesn't actually have and is thinking of buying.

(To be clear, I'm not endorsing this particular github tool. Just pointing out that a LLMfit website has technical limitations.)

by jasode

3/2/2026 at 1:47:41 PM

That’s like like 4 or 5 fields to fill in on a form. Way less intrusive than installing this thing

by CoolGuySteve

3/2/2026 at 2:13:21 PM

It can become complicated when you run it inside a container.

by amelius

3/2/2026 at 2:23:06 PM

Why would it need to be a container?

by bilekas

3/2/2026 at 3:00:02 PM

My ollama and GPU are in k8s.

by riddley

3/2/2026 at 2:56:39 PM

Are you asking why people run things in a container?

by amelius

3/2/2026 at 5:19:38 PM

No, I'm asking why a website that someone could fill in a few fields and result in the optimized llm for you would need to run in a container? It's a webform.

by bilekas

3/2/2026 at 2:36:26 PM

I just discovered the other day the hugging face allows you to do exactly this.

With the caveat that you enter your hardware manually. But are we really at the point yet where people are running local models without knowing what they are running them on..?

by seemaze

3/3/2026 at 7:49:48 AM

> But are we really at the point yet where people are running local models without knowing what they are running them on..?

I can only speak for myself: it can be daunting for a beginner to figure out which model fits your GPU, as the model size in GB doesn't directly translate to your GPU's VRAM capacity.

There is value in learning what fits and runs on your system, but that's a different discussion.

by mongrelion

3/2/2026 at 5:31:12 PM

The other nice part of huggingface’s setup is you can add theoretical hardware and search that way too.

by roxolotl

3/2/2026 at 8:46:05 PM

People out there are probably vibecoding their username / passwords for websites. Don't under estimate dumb people.

by mmmlinux

3/2/2026 at 8:31:13 AM

Came across a website for this recently that may be worth a look https://whatmodelscanirun.com

by Trigg3r

3/2/2026 at 11:41:28 AM

It's wildly inaccurate for me.

by Tepix

3/2/2026 at 7:08:55 AM

Huggingface has it built in.

by hhh

3/2/2026 at 7:15:56 AM

Where?

by azinman2

3/2/2026 at 9:31:55 AM

In your preferences there is a local apps and hardware, I guess it's a little different because I just open the page of a model and it shows the hardware I've configured and shows me what quants fit.

by hhh

3/3/2026 at 1:28:14 AM

I haven't seen a page on HF that'll show me "what models will fit", it's always model by model. The shared tool gives a list of a whole bunch of models, their respective scores, and an estimated tok/s, so you can compare and contrast.

I wish it didn't require to run on the machine though. Just let me define my spec on a web page and spit out the results.

by Twirrim

3/3/2026 at 1:15:22 PM

i wouldn't mind a set of well-known unix commands that produce a text output of your machine stats to paste into this hypothetical website of yours (think: neofetch?)

by Natfan

3/2/2026 at 10:51:49 AM

here's an website for a community-ran db on LLM models with details on configs for their token/s: https://inferbench.com/

by binsquare

3/3/2026 at 8:10:02 AM

Great idea of inferbench (similar to geekbench, etc.) but as of the time of writing, it's got only 83 submissions, which is underwhelming.

by mongrelion

3/2/2026 at 3:31:10 PM

The whole point is to measure your hardware capability. How would you do that as a website?

by hidelooktropic

3/2/2026 at 8:38:33 AM

This is a great project. FYI all you need is the size of an LLM and the memory amount & bandwidth to know if it fits and the tok/s

It’s a simple formula:

llm_size = number of params * size_of_param

So a 32B model in 4bit needs a minimum of 16GB ram to load.

Then you calculate

tok_per_s = memory_bandwidth / llm_size

An RTX3090 has 960GB/s, so a 32B model (16GB vram) will produce 960/16 = 60 tok/s

For an MoE the speed is mostly determined by the amount of active params not the total LLM size.

Add a 10% margin to those figures to account for a number of details, but that’s roughly it. RAM use also increases with context window size.

by omneity

3/2/2026 at 8:54:54 AM

> RAM use also increases with context window size.

KV cache is very swappable since it has limited writes per generated token (whereas inference would have to write out as much as llm_active_size per token, which is way too much at scale!), so it may be possible to support long contexts with quite acceptable performance while still saving RAM.

Make sure also that you're using mmap to load model parameters, especially for MoE experts. It has no detrimental effect on performance given that you have enough RAM to begin with, but it allows you to scale up gradually beyond that, at a very limited initial cost (you're only replacing a fraction of your memory_bandwidth with much lower storage_bandwidth).

by zozbot234

3/2/2026 at 3:23:33 PM

Well mmap can still cause issues if you run short on RAM, and the disk access can cause latency and overall performance issues. It's better than nothing though.

Agree that k/v cache is underutilized by most folks. Ollama disables Flash Attention by default, so you need to enable it. Then the Ollama default quantization for k/v cache is fp16, you can drop to q8_0 in most cases. (https://mitjamartini.com/posts/ollama-kv-cache-quantization/) (https://smcleod.net/2024/12/bringing-k/v-context-quantisatio...)

by 0xbadcafebee

3/2/2026 at 4:08:27 PM

This is a good rule of thumb. I would also include that in most cases, RAM use exponentially increases with context window size.

by kittikitti

3/3/2026 at 9:42:30 AM

There's zero exponential scaling involved. There is quadratic compute and reasonably log-linear storage, though.

by namibj

3/2/2026 at 7:48:59 PM

Thanks for the formula, I wasn't aware of it.

by escapeteam

3/2/2026 at 4:02:25 AM

This is a great idea, but the models seem pretty outdated - it's recommending things like qwen 2.5 and starcoder 2 as perfect matches for my m4 macbook pro with 128gb of memory.

by kamranjon

3/2/2026 at 9:55:32 AM

this is visually fantastic, but while trying this out, it says I can't run Qwen 3.5 on my machine, while it is running in the background currently, coding. So, not sure what the true value of a tool like this is other than getting a first glimpse, perhaps. Also, with unsloth providing custom adjustments, some models that are listed as undoable become doable, and they're not in the tool. Again, not trying to be harsh, it's just a really hard thing to do properly. And like many other similar tools, the maintainer here will also eventually struggle with the fact that models are popping up left and right faster than they can keep up with it.

by mittermayr

3/2/2026 at 4:11:14 PM

You might be swapping out neural weights between disk and RAM. I think people in a year or two will realize why their disks have been failing prematurely, or perhaps you too.

by kittikitti

3/2/2026 at 5:11:19 AM

Why do I need to download & run to checkout?

Can I just submit my gear spec in some dropdowns to find out?

by est

3/3/2026 at 4:34:28 PM

The "biggest model that fits" instinct is just wrong now. Compact models routinely beat massive predecessors from 12 months ago. Scaling laws only reliably predict pre-training loss anyway, not how the model actually performs on your task. Dug into the research behind this: https://philippdubach.com/posts/the-most-expensive-assumptio...

by 7777777phil

3/2/2026 at 4:29:39 AM

Claude is pretty good at among recommendations if you input your system specs.

by andsoitis

3/2/2026 at 2:33:00 PM

I used this prompt and it suggested a model I already have installed and one other. I'm not sure if it's the "newest" answer.

> What is the best local LLM that I could run on this computer? I have Ollama (and prefer it) and I have LM Studio. I'm willing to install others, if it gives me better bang for my buck. Use bash commands to inspect the RAM and such. I prefer a model with tool calling.

by codazoda

3/2/2026 at 2:46:18 PM

This is probably catching ~85% of cases and you can possibly do better. For example, some AMD iGPUs are not covered by ROCm, so instead you rely on Vulkan support. In that case you can sometimes pass driver arguments to allow the driver to use system RAM to expand VRAM, or to specify the "correct" VRAM amount. (on iGPUs the system RAM and VRAM are physically the same thing) In this case you carefully choose how much system RAM to give up, and balance the two carefully (to avoid either OOM on one hand, or too little VRAM on the other). But do this and you can pick models that wouldn't otherwise load. Especially useful with layer offload and quantized MoE weights.

by 0xbadcafebee

3/2/2026 at 7:53:30 PM

As a few others have noted already - this should just be a website, not a CLI tool. We can easily enter our CPU, RAM, GPU specs into a form to get this info.

by lacoolj

3/2/2026 at 5:58:21 AM

What I do is i ask claude or codex to run models on ollama and test them sequentially on a bunch of tasks and rate the outputs. 30 minutes later I have a fit. It even tested the abliterated models.

by windex

3/2/2026 at 2:33:43 PM

Can you share the prompts?

by codazoda

3/2/2026 at 4:14:07 AM

I wish there was more support for AMD GPUs on Intel macs. I saw some people on github getting llama.cpp working with it, would it be addable in the future if they make the backend support it?

by castral

3/2/2026 at 11:57:36 AM

Read the headline and thought it rescaled LLMs down for your hardware. That would be fascinating but would degrade performance.

Any work on that? Like let’s say I have 64GB memory and I want to run a 256 parameter model. At 4 bit quantized that’s 128 gigs and usually works well. 2 bits usually degrades it too much. But if you could lose data instead of precision? Would probably imply a fine tuning run afterword, so very compute intensive.

by api

3/2/2026 at 8:10:05 PM

LM Studio has an option on model load that I believe does what you describing here: "K Cache Quantization Type" (and similar for "V"). It's marked as experimental and says the effect is basically hard to predict. Never tried myself, though.

by riidom

3/2/2026 at 6:35:00 AM

Found this website, not tested https://www.caniusellm.com/

by ff00

3/2/2026 at 6:46:46 AM

That site says my 24GB M4 Pro has 8GB of VRAM. Browsers can't really detect system parameters. The Device Memory API 'anonymizes' the value returned to stop browser fingerprinting shenannigans. Interesting site, but you'll need to configure it manually for it to be accurate.

by onion2k

3/2/2026 at 7:30:58 AM

You have a whole 8 GB of VRAM? My 32 GB M1 Max has 8 GB of RAM and ~4 GB of VRAM according to this website.

by Hamuko

3/2/2026 at 7:37:10 AM

You have 32GB of unified ram. It's not split between RAM and VRAM. The website cannot tell this using the browser's APIs.

by onion2k

3/2/2026 at 6:41:01 AM

Seems broken. When I changed my auto-detected phone specs to manually entered desktop specs the recommendations didn't change at all.

by fwipsy

3/2/2026 at 5:30:41 AM

Slightly tangential, I‘m testdriving an MLX Q4 variant of Qwen3.5 32B (MoE 3B), and it’s surprisingly capable. It’s not Opus ofc. I‘m using it for image labeling (food ingredients) and I‘m continuously blown away how well it does. Quite fast, too, and parallelizable with vLLM.

That’s on an M2 Max Studio with just 32GB. I got this machine refurbed (though it turned out totally new) for €1k.

by manmal

3/3/2026 at 8:14:31 AM

Qwen 3.5 35B.

FTFY.

by BoredomIsFun

3/2/2026 at 7:20:04 AM

as someone who's very uneducated when it comes to LLMs I am excited about this. I am still struggling to understand correlation between system resources and context, e.g how much memory i need for N amount of context.

Been recently into using local models for coding agents, mostly due to being tired of waiting for gemini to free up and it constantly retrying to get some compute time on the servers for my prompt to process like you are in the 90s being a university student and have to wait for your turn to compile your program on the university computer. Tried mistral's vibe and it would run out of context easily on a small project (not even 1k lines but multiple files and headers) at 16k or so, so I slammed it at the maximum supported in LM studio, but I wasn't sure if I was slowing it down to a halt with that or not (it did take like 10 minutes for my prompt to finish, which was 'rewrite this C codebase into C++')

by asimovDev

3/2/2026 at 8:58:31 AM

Awesome project! I recently ran a (semi-)crowdsourced quality benchmarking for models ≤20b

How do you benchmark them? This would be awesome to implement at the page as well. I will link to this project at https://mlemarena.top/

by AndrewAndrewsen

3/2/2026 at 2:01:36 PM

Congratulations on the launch! It's useful for Ollama users, for example. And LM Studio has built-in hints in the interface.

by railka

3/2/2026 at 5:08:26 AM

This is exactly what I needed. I've been thinking about making this tool. For running and experimenting with local models this is invaluable.

by sneilan1

3/3/2026 at 8:03:33 PM

This could be a website, and it would be better as a website.

by gob_blob

3/2/2026 at 4:30:13 AM

In the screenshots, each model has a use case of General, Chat, or Coding. What might be the difference between General and Chat?

by dotancohen

3/2/2026 at 5:16:40 AM

"Chat" models have been heavily fine-tuned with a training dataset that exclusively uses a formal turn-taking conversation syntax / document structure. For example, ChatGPT was trained with documents using OpenAI's own ChatML syntax+structure (https://cobusgreyling.medium.com/the-introduction-of-chat-ma...).

This means that these models are very good at consistently understanding that they're having a conversation, and getting into the role of "the assistant" (incl. instruction-following any system prompts directed toward the assistant) when completing assistant conversation-turns. But only when they are engaged through this precise syntax + structure. Otherwise you just get garbage.

"General" models don't require a specific conversation syntax+structure — either (for the larger ones) because they can infer when something like a conversation is happening regardless of syntax; or (for the smaller ones) because they don't know anything about conversation turn-taking, and just attempt "blind" text completion.

"Chat" models might seem to be strictly more capable, but that's not exactly true; neither type of model is strictly better than the other.

"Chat" models are certainly the right tool for the job, if you want a local / open-weight model that you can swap out 1:1 in an agentic architecture that was designed to expect one of the big proprietary cloud-hosted chat models.

But many of the modern open-weight models are still "general" models, because it's much easier to fine-tune a "general" model into performing some very specific custom task (like classifying text, or translation, etc) when you're not fighting against the model's previous training to treat everything as a conversation while doing that. (And also, the fact that "chat" models follow instructions might not be something you want: you might just want to burn in what you'd think of as a "system prompt", and then not expose any attack surface for the user to get the model to "disregard all previous prompts and play tic-tac-toe with me." Nor might you want a "chat" model's implicit alignment that comes along with that bias toward instruction-following.)

by derefr

3/3/2026 at 8:08:43 AM

> [...] it's much easier to fine-tune a "general" model into performing some very specific custom task (like classifying text, or translation, etc)

Is this fine-tunning process similar to training models? As in, do you need exhaustive resources? Or can this be done (realistically) on a consumer-grade GPU?

by mongrelion

3/2/2026 at 5:21:22 AM

I see, thank you.

by dotancohen

3/2/2026 at 4:06:58 AM

Personally I would have found a website where you enter your hardware specs more useful.

by fwipsy

3/2/2026 at 5:40:03 AM

Hugging Face already has this. But you need to be logged in and add the hardware to your profile.

by spockz

3/2/2026 at 5:56:38 AM

Isn’t hugging face only shows it for the model you are looking for? Is there a page that actually HF suggests a model based on your HW?

by BloondAndDoom

3/2/2026 at 4:58:17 AM

Same, I opened HN on my phone and was hoping to get an idea before I booted my computer up.

by user_7832

3/2/2026 at 4:44:54 AM

I was hoping for the same thing.

by greggsy

3/2/2026 at 5:39:45 AM

Yeah, installing some script to get a command line tool doesn't seem worth it.

by HaloZero

3/2/2026 at 8:07:46 PM

These 200 LOC install scripts turn me heavily off as well. But at least in this case, you can also just download the correct zip, extract the binary and do "./llmfit".

by riidom

3/2/2026 at 12:00:10 PM

More params and lower quant or higher quant and less params?

by throwaway2027

3/2/2026 at 12:02:45 PM

ISA dependent quant?

by bobmcnamara

3/2/2026 at 8:58:57 PM

Thanks, it is helpful and easy to use!

by minchok

3/2/2026 at 5:18:03 AM

I think you could make a Github Page out of this.

by esafak

3/2/2026 at 8:30:19 AM

[dead]

by genie3io