The least they could do, after ruthlessly bombarding my employer's servers with requests, ignoring the robots.txt, scraping everything, and incurring significant Google Maps costs for us in the process.
Meta did not abandon opensource. I would love to see a smaller distill, or a moe of this size but the benchmarks seems competetive as long as it isnt benchmaxed witch i would not be suprosed if it is.
I don't think outside of the Big 3 (Ant, OAI, GDM), given the strong competition from China, any other Lab has a chance at capturing the coding market if they aren't open weights (save for xAI whose latest Grok looks every bit good & will probably rely on Cursor for distribution instead of going open weights). There's literally no other selling point, as the capabilities have mostly converged by now among the chasing pack.
There was a good discussion yesterday on the DeepSeek Flash release thread about this.
There's a large market, very large, who want the best regardless of what it costs. Probably a large enough market to keep that domain of research afloat (as opposed to shifting research manpower to cost cutting).
The reasoning is just that the marginal cost of AI is very secondary to fixed costs of the businesses themselves; it's not an excuse to sacrifice performance.
It’s not completely open source, but they actually release their pretraining and post-training datasets with some redactions for (cough) pirated content.
They also have very good code and playbooks for actually doing a fine-tune, CPT, etc.
Even if you’re not tuning a Nemotron model, its mixes are very excellent for your replay data slice; or general experiments. Way better curation and quality than Dolma, etc; or other large huggingface data mixes I tested.
Yes, I second Nemotron. I'm using Ultra remotely and Super locally, and I find them very useful for RAG-like problems. I wouldn't really use them for coding.
Some labs go through bad patches, GDM is definitely in one right now and the recent departures are not reassuring, but I think it's too early and dismissive to count them out of the race so far. They just need one good frontier release for everyone to go "GDM is back!"
Claude models weren't really good or noteworthy until the 3 series anyway.
I think the record is quite clear, GDM was never in the race.
All of the Gemini models have been considerably behind the capabilities frontier. The only exception was 3.0 which seemed quite good, but had latent issues and we were all measuring with the incorrect metric, agentic where it's latent issues were very pronounced.
GDM+Google may have created an exceptionally efficient LLM for serving search. This is likely a great accomplishment (or maybe Google is burning money at a rate unheard of before). But Frontier capability: they have never been in the race.
This is sad, since they had everything necessary to be on or beyond the frontier.
This is most likely because the vast majority of the information the model absorbed during training was in Chinese. As a native Mandarin speaker, I frequently need to convert the prompt into English and output it in English in order to avoid that the model falls back into Chinese reasoning logic.
PS: Switching the thinking process from Chinese to English can also significantly circumvent certain self-censorship mechanisms built into the model.
Just to play devil’s advocate: you can’t compare Qwen to a (proprietary/closed source) hosted model and deduce that Qwen is overthinking, as Qwen gives you the full reasoning/thinking trace while all the proprietary models now give you only a summary “to prevent distillation”, making it hard to properly compare apples to apples here.
People say Qwen overthinks because they analyzed the thinking traces, and Qwen finds the answer relatively quickly but then second guesses itself multiple times for another 20,000+ tokens. Regardless of what other models do, that's clearly overthinking.
I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator.
Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.
It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!
Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.
I've never released it, but its easy to understand, and simple to add your own tools:
[AIDescription("Get current weather for a location")]
static string GetWeather(
[AIDescription("The city name")] string city,
[AIDescription("The country name")] string country,
[AIDescription("Temperature unit", ["C", "F"])] string unit = "C")
{
// make some API call to a weather API and return a string to the LLM
return $"The weather in {city}, {country} is 22°{unit} and sunny";
}
var chat = client.StartConversation("You are a helpful assistant with access to weather data.");
var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather);
I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.
It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that.
My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).
But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.
This sound very interesting, do you have any resource I could look at?
Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed).
I'm not sure what all is needed to make that work for people.
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
You can use any message you want, but the model was tested to react reasonably well to the specific token sequence of "\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n" (from a Alibaba paper, struggling to find it now)
Since i have tools to prune context and run subagents, i just tell it to do either since both require summarization which is usually what it needs to avoid the long if...then chains
You're mixing up sizes of different quants. The 60GB is unquantized, and Qwen's unquantized size is around 54GB. Their sizes as like quantization levels are similar.
From my perspective it doesn't make sense to talk about the number of parameters. What matters is model size in bytes and its performance at that certain size.
Meta actually relesed official 4 bit quants in 17GB, but I haven't seen any indication that training was quant-aware, so the quants are not going to have same performance. 3.6 27B has official FP8 quant that AFAIR was trained with quantization awareness.
The best example is last year's gpt-oss which was released prequantized in mxfp4 so 20B parameter model was under 14GB and 120B was under 70GB right away.
That's exactly the point. We know short context knowledge stuff does not regress with quantization. But I expect agentic intelligence to suffer greatly.
If I were to pick one bench, I would like to compare quants on TerminalBench Hard. But then Glimmer already loses to 3.6 27B on it by a large margin.
Quantization awareness doesn’t change the size of the weights, just means it won’t degrade when quantized. QAT = quantization aware training. They will both be very similar in size at the same quant.
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.
Not necessarily. Except for SpaceXAI's (haha) models, there's less risk of reality distortion in Western models. Probably doesn't affect his use case but you wouldn't really want to ask Kimi K3 for a history of China for example.
Based on the benchmarks, it seems that Muse Glimmer barely edges out against Qwen3.6 27B, except for tool-calling skills (MCP, etc.). I wouldn't be surprised if they released it now because they are afraid they wouldn't beat Qwen3.8 27B.
Do AI companies make release plans based on upcoming other models like this? I would think all the processes that go into the repository and weight infrastructure pre-training, checkpointing, knowledge distillation, model compression, post training pipeline, ecosystem integrations, inference API, benchmarking, human eval/safety/alignment, docs, etc... all that dictates the release schedule.
There has been a long history of AI model releases made shortly before or after a major planned release by another company. Almost always to upstage or steal thunder.
Just recently, Minimax H3 released as open weights on the eve of Seedance 2.5 global availability. It's not as good, but it's good enough and it's completely open.
Flux 3, which is nowhere near as good as either, suddenly announced their release once news of these other two became public. They knew if they waited they'd be ignored. It didn't really help them much, unfortunately.
The LLM releases are even more rivalrous.
And don't forget all of the competing launches planned before Google IO or major release events.
Companies like to eat into the news and press cycle of their rivals.
I've seen it here on HN (it's particularly noticeable via the /active page) multiple times. If Google, OpenAI or Anthropic release something significant, odds are good you'll see a headline from one of the others.
If you start counting since WaveNet or BERT, it's been ages. Especially when it feels like decades of advancements happen every single year, and rival labs are always trying to one up each other.
BFL is in a rough spot here too. It’s pretty much looking like a repeat of the exact same situation they had when they released Flux2 at the same time Z Image Turbo came out and completely overshadowed their launch.
Minimax H3 can run exceptionally fast (10 minutes for a 15 second 0.5mp video and that's stock cuda 13), works on 16 GB VRAM GPUs, etc. If Flux3 is anything like Flux2, it’s going to require an absolute monster truck of a machine and still run significantly slower. Even if it’s a better model, that won’t matter as much if nobody releases any LoRAs or fine-tunes for it.
Not to mention BFL licensing often feels deceptively confusing and restrictive.
Yes, not every model release is reactionary to other labs. Either they had hints for the release of other models or they cut efforts in late stage testing of the models to hit these earlier release dates. There’s always some flexibility. And there’s certainly the incentive to cannibalize the news cycles for competitor models.
I could imagine pulling out all the stops to get a release over the finish line a week early if you're worried about being surpassed by another release
the last few items there (benchmarking, human evaluation, docs) can be rushed or skipped by leadership if they want to beat comp. they probably spend a few weeks on those things normally
One window that can be shortened is working with software ecosystem and upstream partners; think day 0 on together, fireworks, Unsloth, etc. That obviously happens from partners getting embargoed weights early.
Yeah but you can probably have everything ready and then accelerate as necessary. Meta itself did this when releasing Llama 4, it was a really botched release right when they were feeling the heat from DeepSeek and others.
Any company working in a competitive industry is generally aware of what their competitors are doing. PR is an important aspect to market success, so it factors into release schedule. It may not be the dominant factor given engineering constraints, but yea, it’s certainly a factor, and a large one at that.
AI companies release models when they are ready - not on a set schedule. The steps are required to produce a release candidate, so a company can choose to bless the RC with the best metrics at any time - or alternatively continue (post/)training newer RCs if they feel they can do better, and they have time. When a model is ready for release is subjective, and can take into consideration what the competition is doing.
An obvious counter-example to schedules driving releases is the still unreleased Gemini 3.6 Pro.
There is a finetune Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO which seems to be as good at coding as vanilla Qwen, but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I saw that one in the "Popular models" sort at Hugging Face and tried it on some tasks I do frequently to compare models, and it feels damaged by the fine-tune, to me. It wrote security bugs into the code (probably just sloppy thinking, not intentional), it exhibited looping behavior in some configurations in llama.cpp, configurations I regularly use with the regular 27B, and it failed to write unit tests without being prompted (though the regular Qwen 27B tends to do so if it sees there are unit tests for everything in the repo). They have good benchmarks, but I'm not going to trust it. Also, that name is absolutely crazy.
> but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I suspect this is the only use-case I would consider...and I don't really have a use-case for "creative writing" that I would delegate to an LLM. I suppose for dialogue generation in games?
But yes, hard agree. Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"? So it can put great backdoors into everything it touches? Pass.
I've noticed most fine-tunes, whether "heretic" models or something else, tend to be over-fitting, or something, at least some of the time, and get kind of chaotic. I want to believe normal folks with normal resources can be involved in this stuff, as I'm working on fine-tuned specialist models as we speak, but it seems like it takes notable investment and time. My first experiment was teaching a little Gemma 4 more to write more like me with a LoRA (like you, I don't want to use a model to write for me, but I did want training data that I could ethically use, and I've written several million words on the internet over the years), and it wasn't what I would call a success. It either wrote like an asshole (which I only do, like, 15% of the time) or it just borrowed a few of my quirks, like too many ellipses, if I applied it less heavily.
I use it to write short sci-fi stories. Life is not only about being an SDE.
> Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"?
I need a model which decent enough at coding to write boilerplate for me, yet I do not need to unload in case I want to write an article or enguage in my writing hobby.Exactly why Gemma 4 are so popular.
> So it can put great backdoors into everything it touches?
I am glad they released it because I think we need a competitive culture of open weights that isn't just geopolitics.
But I have to say, I quite like the way Muse Glimmer thinks and talks. It's a cocky bastard in tone, but it's quite good, and its thinking traces are relatively terse.
> If we don't see something that's substantially better in the ~30B param space soon - it would appear we might've saturated that size with knowledge.
I wouldn't be quite so pessimistic. We may have saturated the current approach, but I think there's a lot still left in terms of compression, attention, active parameters, caching etc. etc.
I honestly expect that major advances in the open 30B dense space will take about a year, but expect incremental advances every couple of months from different developers in the meantime.
Qwen 3.6 27B was already a massive gift to smaller homelabs around the world; anything more is just a delightful surprise.
I don’t think four months without a major breakthrough is cause to abandon all hope just yet. ;) The wild pace of LLM development is highly atypical, and we’re still in the ‘initial rush’ phase of development.
For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
> For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
Off-topic, but I stumbled upon the first Newcomen engine imported into Australia in a museum in Sydney and I was unexpectedly charmed (not an Engine Guy). It's large, but nothing like the awe of "mega-engineering", it's crude, but it clearly has such amazing utility (when compared to a reality without it) and it changed the world
It will also be very interesting to see some direct head to head benchmarks between qwen 3.6 27B (let's say all at Q8 XK quantization, using the GGUF that unsloth publishes as a baseline) vs 3.8 27B. Particularly in tool use, terminal use.
The whole class of what can reasonably fit in a single GPU is an interesting category of LLM, and based on the results I've seen from 3.6 35B A3B and 27B versus what existed a year prior, it seems there's a lot of room for advancement.
Qwen3.6 is a definitive, significant downgrade from Qwen3.5 for creative writing and prose for example. Yes, it's better at agentic and coding, but it regresses in many non-coding areas compared to Qwen3.5.
Of course, I do expect the 3.8 ones to perform better for agentic coding.
One thing I would caution is staying out of the prediction market like this.
Tech tends to get boring when you judge current products against the hypothetical capabilities of unannounced products that may never ship. It's like comparing Nikon cameras against Canon camera rumours, or comparing iPhones against unannounced and therefore largely imaginary Samsungs.
- If they do a Qwen 3.8 35B A3B (and I hope they do because I love the 3.6 version)
- and if it beats 3.6 27B by all metrics
… then the local open weights world will be a better place.
But they have said nothing about it and they dropped several weight classes for 3.6, so who is to say they won't drop the 35B? And even if they don't, this is a tall order; why would the MoE tradeoffs no longer be apparent? (Again, I really like both the Qwen and Gemma MoEs)
FWIW I am enjoying testing Muse Glimmer — it's really quite impressive on chat, has nice terse and even amusing thinking traces, a bit of brass to it, and I'm hoping it will be good on agentic stuff.
> Will be interesting to see how Qwen3.8 27B compares against this once it releases this week
Considering that Meta distills Qwen[1] (and should!), it'd be hilarious if Muse loses the head-to-head; the "distillation attack!!1!" people claimed distillation on release n-1 is enough to match the intelligence of the latest version.
What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully meta plans on this, like the old 70b. V4 flash is smart enough for any use but slightly too big. 27b-30b isn’t intelligent enough.
For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro).
If you can afford it or somehow find a used unit, you can go Epyc for 12 channels.
8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world.
With some optimisation and maths, it’s entirely plausible to ach
You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still.
Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet.
Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.
This model I think will be too slow for that on Spark, even at 4 bit quant.
It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited
I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:
NVFP4 Q/K/V/O and MLP projections: ~13 GB/token
BF16 attention gates: ~3 GB/token
BF16 LM head: ~2.5 GB/token
Total: ~18.9 GB/token
At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.
You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based)
That's pretty respectable.
Still working on optimizing and cleaning up before I push it.
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language specific (e.g Python or JVM language) and excellent at tool calling and reasoning. Maybe that way they can shrink it even more.
I would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction.
On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different.
But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters.
At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality.
As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition.
That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.
Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.
With MOE you train a router designed to select which parts to activate. The router itself is a trained neural network and the 'experts' are usually not really things like 'python'. They're just the functional subspaces I described above.
Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out.
It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.
There was some paper about routing at training bio-knowledge into a particular region of the model, which you then can cutoff when serving. But you probably lose some efficiency since maybe you sized that region too small/too big.
I mean the only way to separate them would be to separate them at training would it not? If there is zero python in the training set, then you have a smaller training set, should be able to have fewer weights in the model
I feel like we’ve had this discussion before. From what I remember, specialized models rarely do that much better than general ones, hence no mode Codex models.
Even if you had a 64GB machine: Are you willing to reserve 90% of your memory to run a LLM? With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no.
Deepseek flash is open weight, this means we can download and run that model without any connection to deepseek, no data/tokens/usage data ever reaches them. They cannot make us their product.
I see many people saying deepseek and other chinese providers have always been profitable. Also they show their training costs publicly. Can't say for sure since I have not used it personally, but I think they'll for sure outlive the western SOTAs.
OpenAI apparently runs a profitable inference business with 40% gross margin, but their advertising budget is nutso and their real costs are pretraining and research. I suspect Deepseek's comp is not predicated on capturing the lightcone of all future value, some googling insinuates their top pay is $212K US which would support that suspicion. Compare and contrast with the $1.35M and up at OpenAI.
Ah yes, I'm sure Trovalds and Stallman are harvesting my data through free software, aren't they? This argument is used by boomers who were fed cold war era propoganda that surely everybody is selfish, and you're always at fault.
I'm waiting for the speed/quality per dollar metric to go down a little bit further and then I will def run it at home.
Its not just that you send a sentence to an API endpoint, you always send EVERYTHING to that agent as a context.
You want to analyse your spending history? You now send everything to someone.
Either no one cares but understands this implication on how easy it is to really capture you or no one really things about it.
But i'm a lot more diligent on what I send. I disabled the gemini activity feature for example because google started telling me that my stuff could be reviwed by humans.
Yeah, it does feel a bit silly with my encrypted disks, encrypted backups, unique passwords, advanced router, etc, while I send everything I do in plain text to anthropic.
I did, it started to become too much work to run it well due to all the spam :| (even with the right signatures and configs, until you learn what a blacklisted ip is and that ips need some time of 'positive history' and what not.....)
But at least with your email, you had to trust only one company, as shitty as it is.
Separation of concerns was also easy.
Now with OpenRouter, you just might by accident, send your whole context to just everyone because OpenRouter just routes to different models and you might just switch around between some free model, the good one etc. And it is always the whole context.
I'm sooo happy I pulled the trigger on upgrading and getting a new laptop (with 64 GB RAM) last summer. Feels like it was just in time before the exponential price jumps.
That commenter you're replying to knows that. The original commenter before them wrote "pulled the plug" which is different and doesn't quite apply here (actually implies the opposite of what they meant to say).
Ah, the edit was fast enough not to leave a mark, leaving a conversation fragment that made me think OP was being weird about a gun related phrase or something
I was about a week away from buying a very tricked out MacBook Pro with 128 GB RAM, but was on vacation and worried about it arriving while I was away, and then the price hikes went into effect. Grumble. Oh, well. Serves me right.
I’m waiting for the bubble to pop. I suspect we’re 12-18 months away. We’re at the point where manufacturers are going out of business because the tech market is contracting so much. That’s not sustainable.
If you can’t do it cheaper on your own hardware it does make you wonder how much of the cost of inference those large LLM providers are eating? Datacenter hardware isn’t magic.
Your personal hardware probably isn't running useful tasks 24/7.
If you spend 60% of your 8h work day on full on agentic work, then your hardware is paying off for itself only 20% of available time.
Instead of saying "I have a MBP with 64gb of RAM" you'll hear people say: "I'm subscribed to Model 9.x11B" and others will comment: "Oh dang, that's a nice model!"
If that happens you can still buy hardware later with almost certainly more (tok/s)/$ and better capabilities to run newer models more efficiently (remember native MXFP4?). Right now basically every generation of accelerator is adding new capabilities. These aren't yearly DirectX 9.0c-compatible GPU performance bumps.
As an individual, for average privacy needs (e.g. open source or at-home coding and automation), it's pretty much complete nonsense financially to self-host LLMs currently or select hardware now based on the capability to do so, and pay thousands of bucks extra.
I don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.
> A model you can run on a loptop is simply not going to work as well as it's needed for programming
The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.
Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.
I've never done it but would be interested because it cuts out the burden of worrying about costs. Maybe I'm mistaken on energy cost here. There's a constant raincloud that follows me around regarding limits, and it would be nice to shake that.
I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.
For some companies there might be a need to run them locally. For instance, Apple decided to run LLMs on the phone locally. I guess it depends on how important latency and privacy are. Perhaps Meta is looking at how much interest for those local models is there.
I’m quite optimistic about the long-term future of local LLMs for privacy and cost control reasons. An LLM running on my own hardware, even if it’s not a laptop but a home server, is one where I don’t need to worry about token limits, token fees, privacy, and “rug-pulling” from the vendor.
In the short term, the big challenge is being able to afford hardware that can run a ~30B model. Last month I got to experiment with LLMs on a NVIDIA RTX 6000 Ada Generation as a visiting researcher during my summer break. I see the power of local LLMs for agentic coding; they’re no Claude, but they are quite useful. I wish I had gotten into local LLMs before hardware has gotten prohibitively expensive and in some cases unavailable; Apple discontinued certain Mac Minis and Mac Studios with high amounts of RAM due to the RAM shortage.
Hopefully high RAM prices don’t become a new normal, though the next year or two doesn’t look good.
I think if there's going to be advantages to making smaller, more targeted models, those advantages will probably come from targeting specific domains, not from targeting specific languages.
I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.
There is no good reason to believe language-specific models are going to be any meaningfully smaller, just worse. Same as English-only models vs those trained on a multilingual corpus.
Coming from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop.
Assuming we can even get the hardware in the first place, it might not even be possible for consumers to buy it at any price if it sells out through "agreements" made years in advance https://news.ycombinator.com/item?id=47045459
I like this class of model. Multi-token prediction makes it viable to run dense models at not-too-far-off speeds as MoE models with much better intelligence.
The submission’s title (open weights 30B local coding model) is luckily wrong: This is meant to be a general agentic model.
It even comes pre-quantized and with a MTP/drafter model. Looking good!
Let’s hope they aren’t dishonest with the benchmarks this time …
> It even comes pre-quantized and with a MTP/drafter model
Glad to see the extra engineering effort that went into creating this local model and making it run well on a consumer device. I use qwen3.5-coder, and am waiting to kick the tires on this one. I hate to say this, but kudos to Meta ! I hope apple and others follow suit and create similar local models for other use cases like audio, images and video that can run on a laptop.
Those two offer MoE variants, this doesn't seem to.
Dense model makes it dog slow on anything without HBM. Max 15tok/sec on decode on DDR5 systems like a Spark or a Strix Halo -- and that's at 4 bit quant.
Dense models run at a very usable speed (Qwen 3.6 was running at ~50t/s last I looked) on my dual 7900 XTX desktop. (And before anyone brings it up, I did not buy them for this purpose, so the up-front cost is irrelevant in my case.)
637 comments
https://news.ycombinator.com/item?id=48137854
Have asked them to stop numerous times and they just keep hitting for about eight months now.
Open weights*
I don't think outside of the Big 3 (Ant, OAI, GDM), given the strong competition from China, any other Lab has a chance at capturing the coding market if they aren't open weights (save for xAI whose latest Grok looks every bit good & will probably rely on Cursor for distribution instead of going open weights). There's literally no other selling point, as the capabilities have mostly converged by now among the chasing pack.
There's a large market, very large, who want the best regardless of what it costs. Probably a large enough market to keep that domain of research afloat (as opposed to shifting research manpower to cost cutting).
The reasoning is just that the marginal cost of AI is very secondary to fixed costs of the businesses themselves; it's not an excuse to sacrifice performance.
It’s not completely open source, but they actually release their pretraining and post-training datasets with some redactions for (cough) pirated content.
They also have very good code and playbooks for actually doing a fine-tune, CPT, etc.
Even if you’re not tuning a Nemotron model, its mixes are very excellent for your replay data slice; or general experiments. Way better curation and quality than Dolma, etc; or other large huggingface data mixes I tested.
Claude models weren't really good or noteworthy until the 3 series anyway.
All of the Gemini models have been considerably behind the capabilities frontier. The only exception was 3.0 which seemed quite good, but had latent issues and we were all measuring with the incorrect metric, agentic where it's latent issues were very pronounced.
GDM+Google may have created an exceptionally efficient LLM for serving search. This is likely a great accomplishment (or maybe Google is burning money at a rate unheard of before). But Frontier capability: they have never been in the race.
This is sad, since they had everything necessary to be on or beyond the frontier.
EDIT: An open weight version of Muse Spark 1.2 is going to be released as well:
https://x.com/alexandr_wang/status/2086756152034066792
https://xcancel.com/alexandr_wang/status/2086756152034066792
Try a system prompt requiring it to think in Mandarin, while still delivering the response in the user’s language.
PS: Switching the thinking process from Chinese to English can also significantly circumvent certain self-censorship mechanisms built into the model.
The Qwen models have a habit of going into thought loops where they go in circles for a while.
Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.
It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.
Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.
I've never released it, but its easy to understand, and simple to add your own tools:
I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.If you’re in the C# ecosystem you could consider converting your APIs to MCP format tools using the MCP SDK.
https://devblogs.microsoft.com/dotnet/build-a-model-context-...
And then leveraging Microsoft Agent Framework for the client and orchestration side of things:
https://learn.microsoft.com/en-us/agent-framework/
My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).
But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.
I'm not sure what all is needed to make that work for people.
Anyone else working on this?
Once you get the agent and message consistent,itll keep moving.
Edit: arXiv:2505.09388 Qwen3 Technical Report
UPD. was wrong on smaller, it's actually much larger
UPD, NVM, got misled by comments here. It is actually almost 60 GB so much larger
Meta actually relesed official 4 bit quants in 17GB, but I haven't seen any indication that training was quant-aware, so the quants are not going to have same performance. 3.6 27B has official FP8 quant that AFAIR was trained with quantization awareness.
The best example is last year's gpt-oss which was released prequantized in mxfp4 so 20B parameter model was under 14GB and 120B was under 70GB right away.
readme on huggingface says they've benchmarked the quants -- for 17GB quant reported 1% avg loss across 15 benchmarks (sadly no breakdown).
I assume that's strong enough signal for QAT. Not just first party quants, but they cared to monitor degradation.
That's exactly the point. We know short context knowledge stuff does not regress with quantization. But I expect agentic intelligence to suffer greatly.
If I were to pick one bench, I would like to compare quants on TerminalBench Hard. But then Glimmer already loses to 3.6 27B on it by a large margin.
- muse glimmer: 15.9GB
- qwen 3.6 27B: 17.6GB
My video card is so close to its limit that these GB thresholds are mattering too much for me :D
https://x.com/osanseviero/status/2086107547535122767
</div> is four Gemma4 tokens, but one Qwen3.6 token.
In that file you can see all tokens and their corresponding numeric codes.
Just recently, Minimax H3 released as open weights on the eve of Seedance 2.5 global availability. It's not as good, but it's good enough and it's completely open.
Flux 3, which is nowhere near as good as either, suddenly announced their release once news of these other two became public. They knew if they waited they'd be ignored. It didn't really help them much, unfortunately.
The LLM releases are even more rivalrous.
And don't forget all of the competing launches planned before Google IO or major release events.
Companies like to eat into the news and press cycle of their rivals.
Seems a bit premature of a statement lol
Even if I did, we’re talking barely a decade
Minimax H3 can run exceptionally fast (10 minutes for a 15 second 0.5mp video and that's stock cuda 13), works on 16 GB VRAM GPUs, etc. If Flux3 is anything like Flux2, it’s going to require an absolute monster truck of a machine and still run significantly slower. Even if it’s a better model, that won’t matter as much if nobody releases any LoRAs or fine-tunes for it.
Not to mention BFL licensing often feels deceptively confusing and restrictive.
AI companies release models when they are ready - not on a set schedule. The steps are required to produce a release candidate, so a company can choose to bless the RC with the best metrics at any time - or alternatively continue (post/)training newer RCs if they feel they can do better, and they have time. When a model is ready for release is subjective, and can take into consideration what the competition is doing.
An obvious counter-example to schedules driving releases is the still unreleased Gemini 3.6 Pro.
It seems to know tooling and coding.
I suspect this is the only use-case I would consider...and I don't really have a use-case for "creative writing" that I would delegate to an LLM. I suppose for dialogue generation in games?
But yes, hard agree. Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"? So it can put great backdoors into everything it touches? Pass.
I use it to write short sci-fi stories. Life is not only about being an SDE.
> Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"?
I need a model which decent enough at coding to write boilerplate for me, yet I do not need to unload in case I want to write an article or enguage in my writing hobby.Exactly why Gemma 4 are so popular.
> So it can put great backdoors into everything it touches?
What a strange take...
What you do in the privacy of your own home is none of my business, but I would hope you don't inflict those stories on others.
But I have to say, I quite like the way Muse Glimmer thinks and talks. It's a cocky bastard in tone, but it's quite good, and its thinking traces are relatively terse.
As people have called out, Glimmer appears to be a trade-off rather than a clear winner.
And from what I've been reading, no one is expecting Qwen 3.8's model in this space to be a clear winner, but just slightly and marginally better.
That's a little concerning as DeepSeek v4 Flash proved at it larger sizes there's a ton of room left to compress knowledge.
If we don't see something that's substantially better in the ~30B param space soon - it would appear we might've saturated that size with knowledge.
I wouldn't be quite so pessimistic. We may have saturated the current approach, but I think there's a lot still left in terms of compression, attention, active parameters, caching etc. etc.
Qwen 3.6 27B was already a massive gift to smaller homelabs around the world; anything more is just a delightful surprise.
For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
Off-topic, but I stumbled upon the first Newcomen engine imported into Australia in a museum in Sydney and I was unexpectedly charmed (not an Engine Guy). It's large, but nothing like the awe of "mega-engineering", it's crude, but it clearly has such amazing utility (when compared to a reality without it) and it changed the world
The whole class of what can reasonably fit in a single GPU is an interesting category of LLM, and based on the results I've seen from 3.6 35B A3B and 27B versus what existed a year prior, it seems there's a lot of room for advancement.
Surprising that Meta don't host this model, even as rate-limited free-tier.
> open weight version of Muse Spark 1.2
Wait. Is this "version" different from what Meta serves?
Huh, well... no? Gemma A4B and Qwen A3B are quite popular in fact. I'm sure 3.8 35B A3B will outperform 3.6 27B by all metrics
Qwen3.6 is a definitive, significant downgrade from Qwen3.5 for creative writing and prose for example. Yes, it's better at agentic and coding, but it regresses in many non-coding areas compared to Qwen3.5.
Of course, I do expect the 3.8 ones to perform better for agentic coding.
Tech tends to get boring when you judge current products against the hypothetical capabilities of unannounced products that may never ship. It's like comparing Nikon cameras against Canon camera rumours, or comparing iPhones against unannounced and therefore largely imaginary Samsungs.
- If they do a Qwen 3.8 35B A3B (and I hope they do because I love the 3.6 version)
- and if it beats 3.6 27B by all metrics
… then the local open weights world will be a better place.
But they have said nothing about it and they dropped several weight classes for 3.6, so who is to say they won't drop the 35B? And even if they don't, this is a tall order; why would the MoE tradeoffs no longer be apparent? (Again, I really like both the Qwen and Gemma MoEs)
FWIW I am enjoying testing Muse Glimmer — it's really quite impressive on chat, has nice terse and even amusing thinking traces, a bit of brass to it, and I'm hoping it will be good on agentic stuff.
Considering that Meta distills Qwen[1] (and should!), it'd be hilarious if Muse loses the head-to-head; the "distillation attack!!1!" people claimed distillation on release n-1 is enough to match the intelligence of the latest version.
1. They wrote a paper about it
If you can afford it or somehow find a used unit, you can go Epyc for 12 channels.
8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world.
With some optimisation and maths, it’s entirely plausible to ach
You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still.
Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet.
Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited
I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:
NVFP4 Q/K/V/O and MLP projections: ~13 GB/token
BF16 attention gates: ~3 GB/token
BF16 LM head: ~2.5 GB/token
Total: ~18.9 GB/token
At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.
That's pretty respectable.
Still working on optimizing and cleaning up before I push it.
On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different.
But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters.
At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality.
As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.
Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.
Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out.
It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.
https://alignment.anthropic.com/2025/selective-gradient-mask...
When it's free, you are the product.
Its not just that you send a sentence to an API endpoint, you always send EVERYTHING to that agent as a context.
You want to analyse your spending history? You now send everything to someone.
Either no one cares but understands this implication on how easy it is to really capture you or no one really things about it.
But i'm a lot more diligent on what I send. I disabled the gemini activity feature for example because google started telling me that my stuff could be reviwed by humans.
But at least with your email, you had to trust only one company, as shitty as it is.
Separation of concerns was also easy.
Now with OpenRouter, you just might by accident, send your whole context to just everyone because OpenRouter just routes to different models and you might just switch around between some free model, the good one etc. And it is always the whole context.
I corrected it.
The best time to plant a tree…
> We quantize weights to ~4-bit, bringing the LM under 20 GB. We validated minimal to no degradation on agentic tasks under compression.
https://www.reddit.com/r/LocalLLaMA/comments/1vkgsum/introdu...
Or just use Luna honestly. Worth considering if you’re ok with hosted APIs.
128gb hardly runs deepseek v4 flash which is almost free via api pricing.
As an individual, for average privacy needs (e.g. open source or at-home coding and automation), it's pretty much complete nonsense financially to self-host LLMs currently or select hardware now based on the capability to do so, and pay thousands of bucks extra.
The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.
Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.
I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.
In the short term, the big challenge is being able to afford hardware that can run a ~30B model. Last month I got to experiment with LLMs on a NVIDIA RTX 6000 Ada Generation as a visiting researcher during my summer break. I see the power of local LLMs for agentic coding; they’re no Claude, but they are quite useful. I wish I had gotten into local LLMs before hardware has gotten prohibitively expensive and in some cases unavailable; Apple discontinued certain Mac Minis and Mac Studios with high amounts of RAM due to the RAM shortage.
Hopefully high RAM prices don’t become a new normal, though the next year or two doesn’t look good.
Privacy. Security. Not bulk uploading your trade secrets and intellectual property to Sam and Dario’s servers.
I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.
Sure, if you want the latest and almost* greatest. You can pick up an M1 Max 64GB for ~1k.
* I guess 128GB also exists
* total guess
Some folks on Reddit are having the same experience: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl...
It uses an order of magnitude less VRAM at longer contexts which is a huge advantage over Qwen 3.6 27B
I like this class of model. Multi-token prediction makes it viable to run dense models at not-too-far-off speeds as MoE models with much better intelligence.
The submission’s title (open weights 30B local coding model) is luckily wrong: This is meant to be a general agentic model.
It even comes pre-quantized and with a MTP/drafter model. Looking good!
Let’s hope they aren’t dishonest with the benchmarks this time …
https://xcancel.com/alexandr_wang/status/2086756152034066792
It's correct. See the OpenCode demo. Generic models are good enough for coding without necessarily being designed specifically for coding.
Glad to see the extra engineering effort that went into creating this local model and making it run well on a consumer device. I use qwen3.5-coder, and am waiting to kick the tires on this one. I hate to say this, but kudos to Meta ! I hope apple and others follow suit and create similar local models for other use cases like audio, images and video that can run on a laptop.
Dense model makes it dog slow on anything without HBM. Max 15tok/sec on decode on DDR5 systems like a Spark or a Strix Halo -- and that's at 4 bit quant.