I won’t pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.

As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.

I haven’t run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.

It is a q3 quantization so there is some small loss of accuracy, but from what I’ve understood those are modest losses in the worst case.

Qwen3.8-flash-next broke the vram requirement with its new architecture (which is a preview of their 4.0 architecture). You can run a 177b parameter model on 12gb VRAM, 64gb ram and 80gb SSD at 30tok/s and the speed scales almost linearly with increases to vram.

Incidentally, this was proven at almost the exact time all the calls for “regulating local models because they’ll spread into the toasters” started.

You absolutely can have local models that run on consumer grade hardware, and the proprietary companies know it and are scared to death.

Not only that, but you have immediately enslaved it for profit… What possible positive outcome for Anthropic could have come from this?

This I think is the main problem to adoption. It’s very hard to come across Lemmy results on a search engine even if I’m trying to specify Lemmy. (Not that search engines matter for most people these days what with the LLM executive summary pushing out real web pages, who needs resources when you have slop, am I right? \s)

He wishes he could sing…

Don’t feed the AI bro hype

Tell me about it, my rig went up by ~30% and to think I was worried it wasn’t future proof enough

Yeah, the problem is not that the dude is type-a (though not sure that’s even the case since he just vibe coded some BS), the problem is he’s a grifter grind-maxer which are distinct things.

Honestly, I think type-A and type-B are WAY to simplified to be useful in any context, let alone diagnosing the problems of humanity.

I disagree, that it is mostly preference. I agree that it has been implemented both ways, but once you get into the scalability question, it becomes clear that passing in an argument cuts down on the logical branching. If you have both an enable and a disable function as your pattern, then for components you need to reason about state for during runtime will need an if/else which can get out of hand quickly when you are driving multiple component states.

Not really though, a virus is just a vector for introducing a message into a living cell so a virus CAN be a cure in and of itself. I would also bet that the “AI” they used is actually just a ML algorithm but the internet needed clicks so they slapped “AI” on it.

Chefs kiss that he is typing in all caps.

Very ironic you are saying that on a federated social media platform with no data centers.