Expand the replies to this comment to learn how AI was used in this post/project.

The truth is, if you are comparing to companies that spend billions on r&d, your self hosted options are not going to be 'better'. But there also is something to be said for specialization. Coding focused LLM's can punch above their expected weight. Claude would still probably be 'better', but if you mainly want some autocomplete and debugging help, or want to not rely on a cloud service that can drasticaly change the price or go down.. self hosted is one alternative. What i would actually suggest here? look at the cloud rental space to really test your options. There are services like vast or runpod, where you rent the ability to run an opensource model, usually for fairly cheap for a short period. You can rent a 3090 for like, 20 cents an hour to test with, see if any models running on it produce work you like.. and then figure out what your actual use case for LLMs is., if it's worth spending money on hardware or not. I went the hardware route and regret it a bit- i haven't touched my locallms' in months now, just because i've had a lot of real life stuff distract me. For the amount i played with them, i probably could have spent 10% of the money on cloud rental space... but who knows, maybe i'll get back into it and spend some heavy time on them.

Thanks for that , I do wonder about the privacy of renting though…kind of the main reason I wanted to self host

At what price does it become interesting or comparable to at least last year's model? I think for organizations that deal with personal data and wouldn't want to use cloud models, onprem might be interesting.

There are a lot of differing opinions on this. People have different reasons to self host AI and I think they are all very valid. I personally invested in a GPU solely for AI coding tasks, it was the most expensive PC component I've ever bought. It was an expensive panic purchase, due to Github Copilot pricing/credit changes, thinking that prices would rise rapidly across the industry. But now, for me and my current situation, it makes more financial sense to use API services like Deepseek or Qwen than to spend the electricity running my own GPU, not even considering the cost of the card.

Do you fear the privacy loss of using api ? That’s my main concern . Already using chat gpt, it’s scary how much it knows about me …

It depends on the use case. I know with my 3090 I would highly recommend self-hosting your AI. It still won't be Claude level by any means, but its good enough IMO for day-to-day use, and you aren't giving all your information, conversations, etc to whoever is hosting the LLM.

It depends on your goals and budget. I have a cheap 16GB card, I can run small translations locally, it can do small code changes, it can do AI research and generate answers based on results, it can summarise/group/categorize, generate similar words, it can OCR files and process text for you, all the things you expect from an LLM But it wasn't good for larger projects and integration with coding tools. Main issue is model capability and context window that is very small when compared to paid versions. I also ran image generation without big issues, even created an auto-prompt maker + image generation local app. So it depends on what you want to do with it. I think 24GB cards won't get you much more, they might provide you 2-4% better results?

It really depends on you, your setup which isn’t necessarily just hardware, and your experience. First.. it doesn’t matter what you build.. it will NEVER do what Claude et all can do. You can dump $100,000 into a private AI and while it’ll be awesome, it still isn’t going to match these $Billion dollar systems. Additionally with a local AI YOU need to both learn how to properly feed it info that’s relevant to you (RAG, Notes, System Prompts) but also how to communicate with it. Open a chat with Claude and ask it specifically how to use and why the System Prompt is so important. Ask it about RAG and how that can assist you with your network, work, school, etc. That said, even a small model with the proper setup can be damn nice and useful. For example… An N100 based mini PC 😆 with a 500GB NVME, 16GB ram, running Debian 13, Ollama and Open WebUI takes about 20 minutes to setup. Add RAG to add manuals, documents, notes, system description, detailed network information, hardware, software, etc.. with qwen3 4B model becomes an amazing little network assistant for you. Add in a solid and detailed System Prompt and that’s a surprisingly serious little $300 CPU/Ram based AI that can do a lot with added tweaking. Bump the ram to 32GB and it gets faster and better context length also. You LEARN a lot with this kinda setup. I specifically used a BeeLink S12 for this project. This is not a fast machine but it lets you learn. Jumped to a Minisforum NAB9 i9 w/64GB ram and the same Debian setup. I then went with Proxmox as the Host OS and added VMs for the “front door” Open WebUI, another VM for the inference engines… ollama and lama.cpp, and a 3rd and 4th for Vector Qdrant and Indexer OCR / scratch. 4B models are instantaneous, 7B models are very good and 12B models are usable but slow. RAG/document indexing here gets really fast. Qwen3 8B as a daily assistant and Gemma 3 12B when you want higher-quality reasoning and don’t mind a bit more latency. Add a small embedding model like bge-small or nomic-embed for RAG. Light automation is possible. Add in AnythingLLM for more learning. Dropping a 12-16GB GPU into your Desktop with 32GB-64GB ram 6-10+ cores and NVMEs becomes very useful. 3-9B models are excellent, 12B models are very good, 14B with appropriate quantization become usable. Some 20B quantization models can be used but are slow. You’re able to start using local agents, image embeddings and vision models, even occasional image generation with smaller diffusion models though this will be slow. We’re talking Ryzen 7 7700 or Core i5-14600K systems with 64GB ram here with say a RTX 5060 TI. My son is run a Gigabyte Z790 Aorus Elite AX, Intel Core i7-13700KF, 64GB Ram and the ASUS Prime RTX 5060 Ti (16GB) on a Debian 13 KDE desktop. Ollama and Open WebUI installed. Qwen3 4B and Gemma 4B are always loaded, Qwen3 8B is his daily driver with Gemma 3 12B for higher quality along with Qwen Coder 7B and 14B. Mirrored NVMEs are absolutely better then single NVMEs and I’d suggest 2TB NVMEs. He’s even running ComfyUI on this system for learning though is ahead of me here as I haven’t even touched that yet. All that said.. I would strongly suggest if you’re going to spend money on a GPU… Just order a RTX 3090 24GB vRam (or any 24GB) GPU! Practically everyone I know who has purchased an 8-16GB GPU regrets inside of a single year. While they do work well and are capable, nearly everyone wants that 24GB sooner than later. Everything you’ll wanna do is doable with 24GB vRam for a long time. 64GB ram really helps here and 128GB you’ll love. System Ram is extremely useful so don’t skimp on it. My own current and main AI server.. Supermicro H12SSL-i, 256GB ram in with an EPYC 7502P and 2 A6000 48GB GPUs. Proxmox with Debian 13 VMs. VM1 AI - 2x mirrored 2TB Samsung 990 Pro, all model files (Ollama, llama.cpp weights). VM2 Vector - 1 1TB WD SN770, Qdrant storage + snapshots. VM3 Indexer - 1 1TB WD SN770, working scratch, temp OCR output. VM4 Storage - 2x 8TB WD Red NAS, raw document archive, nightly Qdrant snapshots, Proxmox VM backups. VM4 is also rsynced to our NAS nightly. I’m actually redoing this system with added NVMEs and VMs along with added vLLM, ComfyUI, LibreChat and AnythingLLM. If you’re seriously looking to start into AI.. here is an AI Hardware analogy that might really help as many people don’t understand what everything does. Think of an AI system like a person working at a desk. VRAM is the size of the desk.. it holds the AI model, its working memory (KV cache), and everything needed to answer questions without constantly reaching elsewhere. GPU compute is the person’s brain.. it performs the mathematical work that generates each new token, so it largely determines how fast the AI responds. System RAM is the filing cabinets behind the desk storing supporting data such as embeddings, vector databases, application memory, and anything not actively on the desk. The SSD is the library down the hall. It’s used to quickly retrieve and load models and data before work begins, but it has little effect once everything is loaded into memory. The network is the people walking up to the desk asking questions and receiving answers. Together, the SSD loads the information, RAM organizes supporting resources, VRAM keeps everything immediately accessible, the GPU does the thinking, and the network delivers requests and responses. Hope this helps.

Youre never going to get a frontier capability on a consumer card. Best use for consumer cards as of Aug 2026 is for an aux model inside your harness to reduce token spend of your frontier. (Vision, title generation, compaction, web search etc)

I am not sure I understand what that means … you use api for the hard stuff but let your card do the smaller things? Does that really reduce cost and is privacy / ownership kinda out then ?

It depends on what you are going to use it for. But if your first sentence is “resembling….”, it’s not going to happen. I tried 12 and 14b models and I can’t say there is much resemblance at all. They could definitely serve a purpose of experimenting and playing around. Or even a set and forget task that you don’t care about taking forever. I wouldn’t purchase anything with the hopes of using it in comparison of the other big ones. Doubling you gpu doesn’t seem like it would double your experience with what you e tested. So, if you are going for 10x your experience now, unfortunately you won’t get it.

Depends on what. I have a local model running for my tagging in the Karakeep instance and a Whisper model for transcribing through a web app of mine

I used my rtx 3090 to extract keywords from movies and images. Vibe coded the thing with a cloud clanker though.

You can get the performance of frontier models from ~6 months ago with ~24GB of memory (Qwen 3.6 27b - Hopefully Qwen 3.8 27b (releasing next week) improves that a bit). That is no small price. Generally, it is more efficient (and cheaper) to buy tokens outright. The main reason most people do it is privacy. It is no secret AI companies use all the data you give them for training their next model. I personally don’t want to have to worry about tokens or have a subscription service to run automations every day.

GPUs are insanely expensive. From purely a financial and ability perspective, no not worth it. But if you're coming at it from a privacy and hobbyist angle, it may be worth it to you.

The hyperscalers are losing money hand over fist right now. They lose money on most customers who are on a monthly plan. A $20/month plan costs $240 over 2 years. Do you think you can build an equivalent rig for $240 including electricity costs?

Depends on the LLM capabilities. If you want something that could genuinely come close to current frontier models, then no, selfhosting is absolutely not worth it. Depending on your specific model choices, if you're looking for general purpose usage same as ChatGPT etc., then you're looking at the 96GB - 256GB range. With how fast models evolve and the usage the giants are able to shell out at a fraction of the price you'll pay for your own inference, you won't ever come even remotely close to recouping that hardware investment.

With 16GB of VRAM you still need a substantial amount of RAM to be able to offload some of the weights for the smallest LLMs that are capable of anything substantial. And those models are waaaay more capable than GPT-3 which is absolutely ancient by LLM standards. Qwen3.6-35B-A3B is sort of the golden standard for the GPU poor nowadays, and personally I wouldn’t use anything smaller than a Q4 quantisation, which should be around ~21GB in size. And hosting LLMs locally really isn’t worth it financially unless you make substantial amounts of money with the inference you bought.

I think it is just getting fun if you have more then 128gb vram. And still then it is more an expensive hobby. For privacy it would also make sense to just rent some GPUs and run the models there, that is way cheaper

Yes, but you need to use the right tool for the job. My home server has no gpu, and I'm able to run 4b up to 14b models on 32gb of ram. For scheduled scripts that check my personal data like my investments, banking, and health data, it can consistently generate insights and text summaries back. Because I run these overnight, speed is not a factor (and it is slow). Don't expect it to do complex reasoning, handle vague tasks, or handle a whole code base/file system but if you give it a straight forward task with a minimal set of data, even the small models do great now.

Open weight LLMs are never going to be as good as frontier models because those models require frontier hardware by definition. Assuming you drop your goal of 'getting anything resembling llm competing with chat gpt, Gemini or Claude', then you can run something like gemma4:12b on a 16 GB card, but even then, you're barely scratching the surface of what local models can do. In other words, you would be disappointed. Now, what if you want to push local inference on consumer hardware to the limit? Like, imagine your homelab is a gaming desktop. Now we're talking; I can get two of ASRock Creator Radeon AI Pro R9700 R9700 CT 32GB 256-bit GDDR6 PCI Express 5.0 x16 Graphics Card for a total of 4 369 CAD. With this, you could be running qwen3.6:35b, nemotron3:33b, gpt-oss:20b. But you won't be running DeepSeek. Now, consider that GitHub Copilot Pro+ costs 53.44 CAD / months. In other words, these two GPUs will net you inferior performance but cost 6 years 9 months of GitHub Copilot Pro+. Not only that, but frontier models are improving every week. For this to be worth it, you would need to make usage of it that surpasses the every day cost, every day, for 6 years and 9 months, without upgrading GPU. And that's just counting the GPU cost; I would need to change power supply to run these cards because of the Fire Hasard connector. And that's just for these cards, because it's possible to get workstation hardware (like Threadripper CPU) that can handle 2 more GPUs, but by then you'd be more than doubling that cost and the time to make them cheaper than buying a subscription.

It is not. Really nothing else to add. No nuances. It is not. If you want to tinker, use your 6gb. If you want performance, $20 a month.

I have a 16gb card. It's usable but nowhere near claude . I use it for some automation and experimentation, but I bought my card before prices went crazy. Not sure I'd do it today