yeah fair point. it's always tricky to get the whole idea across within HN's title limit. tldr: we ran the same workload in the same Lume macOS VM on the same Apple Silicon host, first with stock Metal capability reporting and then with our process-scoped dynamic library. The 11.08x figure is prompt processing, while 16.36x is token generation. the mechanism technically extends to graphics workloads too but these figures are specifically from llama.cpp
So those generation numbers aren't really anchored to Apple's hardware designs. It's just counting from when Apple introduced the Metal API, and the first several generations were when the GPU cores Apple was using were still nominally PowerVR designs.
yeah the naming is confusing. Apple family 9 isnt M9, it's a Metal GPU feature family. Apple maps family 7 to M1, family 8 to M2, family 9 to M3/M4, and family 10 to M5
> this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM.
correct. these figures apply to llama.cpp inside the macOS guest configuration we tested. Lume is the VM frontend we used, while Apple's Virtualization.framework provides the virtual GPU. bare-metal llama.cpp is unaffected.
> The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.
mostly, with one nuance: llama.cpp is selecting the correct kernels for the capability answers it receives. the stock guest reports an older Apple GPU family and a 32 KB threadgroup memory limit, so llama.cpp chooses slower kernels. Our process-scoped layer reports the tested Apple 9 and 64 KB values while allowing llama.cpp to select newer paths that the paravirtual GPU successfully execute
the layer itself though works at the Metal API boundary, independently of llama.cpp. other Metal compute and graphics apps now may select newer paths from the same capability answers, although this is still preliminary and each app needs separate testing. for example, MLX-LM stayed flat in our tests
That makes sense. The title initially sounded like a general llama.cpp speedup on Apple Silicon, but if the improvement comes from fixing kernel selection inside Virtualization.framework VMs, that distinction is pretty important.
The unified memory architecture is interesting for toying with medium size models but will never offer as much bandwidth as a dedicated GDDR memory bank. Conversely, GDDR can't be used for general CPU purposes because access latency is just too high. Unless someones also comes up with dynamically programmable memory banks, something I'm not sure would even be possible.
What I don't get, which this article doesn't talk about, why would Apple’s Virtualization.framework expose a lesser Metal profile instead of reporting all capabilities supported by the host GPU?
Apple doesn't let you "pass" the GPU through to a VM like most other ARM/x86_64 processors (forwarding interrupts and PCIe memory regions). There are symbols defined to do this within the kernel (if you dump the binary) but they aren't used in retail macos.
Instead you end up creating a paravirtual device that emulates the GPU acting like a 'normal PCI device' which you give to clients. This is usually reserved (by other hardware vendors) for when you're doing multi-tenat time sharing of higher end GPUs (like Nvidia enterprise cards can do).
These paravirtualized GPUs then just have 'less features' and Apple (being Apple) states no reason why.
QEMU/kvm does this as a default, because keeping a more generic CPU / etc makes moving VMs between machines with different hardware possible. If you try to move a VM it won't work, of course, if the new machine doesn't support what the old did.
Not sure of this is why Apple does it. With KVM, you tend to pick a baseline that all your machines support.
43 comments
So this was the comparison, for me the title was a bit confusing
The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.
correct. these figures apply to llama.cpp inside the macOS guest configuration we tested. Lume is the VM frontend we used, while Apple's Virtualization.framework provides the virtual GPU. bare-metal llama.cpp is unaffected.
> The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.
mostly, with one nuance: llama.cpp is selecting the correct kernels for the capability answers it receives. the stock guest reports an older Apple GPU family and a 32 KB threadgroup memory limit, so llama.cpp chooses slower kernels. Our process-scoped layer reports the tested Apple 9 and 64 KB values while allowing llama.cpp to select newer paths that the paravirtual GPU successfully execute
the layer itself though works at the Metal API boundary, independently of llama.cpp. other Metal compute and graphics apps now may select newer paths from the same capability answers, although this is still preliminary and each app needs separate testing. for example, MLX-LM stayed flat in our tests
historically related limitations have been coming up across Apple Silicon VM frontends for a while e.g. Tart tracked MPS/GPU support back in 2023: - https://github.com/openai/tart/issues/501 - https://github.com/openai/tart/issues/1032
UTM also has related cases where apps detect the Apple paravirtual Metal device but falls back to software rendering: https://github.com/utmapp/UTM/issues/7671
Because it is not allowed here, that's why. See the guidelines.
I wonder if their work is related?
PCIe card is the way forward IMO, AI keeps changing so you don't want static hardware.
"Static hardware" is still fully featured computer with lots of RAM, could be easily reused for other purposes.
Apple doesn't let you "pass" the GPU through to a VM like most other ARM/x86_64 processors (forwarding interrupts and PCIe memory regions). There are symbols defined to do this within the kernel (if you dump the binary) but they aren't used in retail macos.
Instead you end up creating a paravirtual device that emulates the GPU acting like a 'normal PCI device' which you give to clients. This is usually reserved (by other hardware vendors) for when you're doing multi-tenat time sharing of higher end GPUs (like Nvidia enterprise cards can do).
These paravirtualized GPUs then just have 'less features' and Apple (being Apple) states no reason why.
Not sure of this is why Apple does it. With KVM, you tend to pick a baseline that all your machines support.