Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
openHenry from Cactus here!
We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2.
The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series.
On the tool call and mobile device use benchmarks, Needle 2 trades wins with closest small models like LFM2.5 230M and Apple Foundation Model, at 5x to 70x smaller, both at f16 vs Needle 2 at 2bit. Needle is based on Simple Attention Networks from our paper (https://arxiv.org/abs/2607.18363).
Edge AI has lately meant Macs and PCs, but that is just 1.5 billion of over 21 billion connected IoT devices in the world today, and in emerging markets most phones ship under $200, no NPU, cheap GPUs. These include budget phones, Raspberry Pis, microcontrollers, wearables, small robots like Reachy Mini, and connected home devices.
A conventional transformer of Needle's width and depth spends 164 MFLOPs per token, and even one squeezed down to Needle's parameter count spends 87, Needle spends 70. Even on a high-end phone, an always-on assistant lives inside a power budget; every MFLOP is milliwatt-hours, and Needle spends 7x to 85x fewer of them per token than the smallest performant LLMs. More about the architecture in the link.
When we structure intelligence for consumer devices as functions with typed parameters, the only hard part is mapping a messy sentence onto them; which function, with which values. Our research found that when framed that way, the problem needs no world knowledge and no open-ended prose, which is why 45M parameters suffice.
Needle 2 expands to structured extraction where the schema can be passed in-place of tools and the model returns structured output. You can use Needle as a text-classification model with an enum field, as a summarization model by providing a schema that extracts key fields, everything but free-range decode.
Every product has its own tool vocabulary and fine-tuning needle helps it achieve frontier-level performance on custom tasks, so using the python package (https://github.com/cactus-compute/needle), Needle can be fine-tuned Needle on a Mac/PC in minutes to a few hours, with automated data-generation pipeline, just pass a couple samples.
Nonetheless, every response carries a learned confidence score based our Cactus Hybrid technique. If above your threshold, act, below it, escalate to the cloud or bigger model. Combining Needle 2 with a private DeepSeek-v4-Flash deployment works particularly well for enterprise-level tasks at barely any cost, we can help with this setup.
We have put a lot of thoughts into Needle 2 but might still be missing quite a lot, please use the playground in the provided link to test Needle and share your thoughts, always appreciated!
172 comments
But yeah, in terms of “physical” AI, robotics definitely comes to mind for me as well, where tool calls/structured “device” use in a “realtime”/edge application are highly beneficial (if you wanted to go with LLMs), but beefy hardware can’t be easily used.
yes you're right, there's only so much a 14MB model can do.
Needle excels at in-conext inference, with tightly defined environments. In our experience:
accurate descriptions + narrow tool scope = success
Query: HN
Result:
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evaluate your releases on some usability criteria (including false positives). Any serious integrator or adopter of these models would want to have that information available.
OP and the linked page talk about the confidence score and using it as an action threshold, so it looks like an appropriate total response to me.
I am not sure how a micro model will fundamentally solve it. Would love to understand what dannyw and team did there?
> I'm hungover
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. 'hungover' implies a security door. No specific door named, so use 'front door' as default.", "confidence": 0 }
Prompt: "Unlock my doors"
{ "function_calls": [], "reasoning": "No tool for unlocking or locking door exists. lock_door only performs lock/unlock actions, not unlock.", "confidence": 0.9492 }
The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device.
ETA: pun not intended
That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions.
If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes (https://huggingface.co/openai/whisper-tiny). You can definitely get something like this running on a Pi 5. There are plenty of other STT models out there, some of which are built specifically for this context (again, see the Home Assistant stuff), but Whisper comes up a lot as a good default choice.
So with something like Whisper, you could just have a simple script which is constantly listening to a rolling window of audio and transcribing it. When the transcription includes a key phrase, you can pass the rest of the transcription to Needle2 (or anything else for that matter). From there, you take the results and execute the necessary tool calls.
There's a bit more to all of this to make it work smoothly, but fundamentally this is all there is to it. All this would work very fast on a Pi 5 (although I wouldn't expect the results to be particularly good without some serious hand-crafted logic, fine-tuning, etc.). If you want to mess around this stuff, handing all of this to Claude, Codex, etc., can get you something spun up and functional very quickly.
Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup?
Also, why mHC over attention residuals?
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.
I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.
For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.