llama.cpp generated Qwen3.5-0.8B at 22.09 tokens/s on our 8GB KiwiPi 5, against 15.95 tokens/s through RKLLM on the NPU. With Qwen3.5-2B, the short-input medians were 10.76 and 9.17 tokens/s.
We reran both engines on the same day, following our earlier RKLLM installation and model tests. The CPU produced text faster once generation started. But both CPU stability blocks were stopped by our temperature guard; both NPU blocks finished all ten requests.
I’d keep those results together. A fast completed request and ten requests that finish consecutively answer different questions.
CPU Q8_0 and NPU W8A8
Everything ran on the same RK3588S board under Ubuntu 24.04.3 LTS and Linux 6.1.141. The heatsink and fan were installed, the fan was spinning, and the desktop and KiwiDrop stayed open. Swap was off. We didn’t record room temperature.
The CPU path used upstream llama.cpp source, built at commit 2e0d86bebc210947680e8f0d67daa79f3260c106 with GCC 13.3 and native ARM optimizations. Generation and batch processing each used four threads on Cortex-A76 cores 4–7, with zero GPU layers. We didn’t search every thread configuration for a faster result, but later checked that choice in a separate llama.cpp CPU thread test. Four Cortex-A76 threads reached 20.29 tokens/s, compared with 7.90 using eight threads across both core types. Each setting received one request; these are separate measurements from the repeated results below.
For models, we downloaded the ggml-org 0.8B GGUF and Unsloth 2B GGUF in Q8_0, checked their publisher hashes, and verified them again on the board. The NPU reused Qengineering’s W8A8 conversions with RKLLM Runtime 1.3.1 and RKNPU 0.9.8. These are conversions of the same named models, not identical tensors or interchangeable quantization formats.
Our measurement harness was written for this experiment. It used the stock llama-server on loopback only, with one slot and no web UI; RKLLM used our existing C API measurement client. This wasn’t a new chatbot application or an externally exposed service.
Each combination received the same request texts independently, without retained history. We disabled thinking, turned off CPU prompt caching and explicitly cleared the RKLLM KV cache. The main series used context capacity 2048 and a 128-new-token output cap, with three repeats per input length.
Generation speed and first-text latency
All twelve performance rows have three complete repeats. The table shows their medians; model loading is excluded.
| Model | Engine | Input | First text, s | Decode, tokens/s |
| 0.8B | llama.cpp CPU | Short | 0.29 | 22.09 |
| 0.8B | RKLLM NPU | Short | 0.40 | 15.95 |
| 0.8B | llama.cpp CPU | Medium | 1.69 | 21.64 |
| 0.8B | RKLLM NPU | Medium | 1.93 | 15.94 |
| 0.8B | llama.cpp CPU | Long | 4.65 | 21.16 |
| 0.8B | RKLLM NPU | Long | 4.26 | 14.97 |
| 2B | llama.cpp CPU | Short | 0.65 | 10.76 |
| 2B | RKLLM NPU | Short | 0.50 | 9.17 |
| 2B | llama.cpp CPU | Medium | 3.52 | 10.69 |
| 2B | RKLLM NPU | Medium | 2.15 | 9.02 |
| 2B | llama.cpp CPU | Long | 9.76 | 10.42 |
| 2B | RKLLM NPU | Long | 5.20 | 8.84 |
Short, medium and long identify the same input texts across engines. They do not imply identical token counts after each engine applies its template and tokenizer.
Waiting for the first text
The CPU was faster at decode for both model sizes. On the long 2B input, though, RKLLM started returning text after 5.20 seconds, against 9.76 seconds for llama.cpp. For a question with a long document attached, that difference comes before you see any answer.
The 0.8B results were closer: 0.29 versus 0.40 seconds on the short input, and 4.65 versus 4.26 seconds on the long one. I’d look at both columns before choosing an engine for an interactive application.
There is a measurement detail here. CPU first-text timing includes the loopback HTTP request and streamed delivery, starting after template preparation. RKLLM timing runs from the API call to its first nonempty text callback. Decode speed uses each engine’s native counters, whose token accounting and timer boundaries can differ. These are practical software-stack measurements, not isolated CPU-versus-NPU hardware speedups.
The bounded performance prompts deliberately asked for more text than the output cap allowed. Their cutoffs were expected, rather than failed quality tests. We also ran three requests per combination with context capacity 4096, using the same long input; we didn’t test a full 4096-token prompt.
Temperature stops and process memory
The plan contained 116 requests. We obtained complete validated metrics for 106, with two incomplete CPU stability blocks retained in the results rather than quietly retried.
Each CPU block completed five requests before the guard stopped the next one. The highest sampled thermal-zone reading at the stops was 85.0°C for 0.8B and 85.888°C for 2B. The guard’s policy was to terminate active inference when any sampled zone reached 85°C. That isn’t a report of a kernel critical-temperature event, an engine crash or damaged hardware.
The SoC sensor tells only part of that story. Its maximum was 77.61°C in both CPU combinations, while another zone reached the guard threshold. We didn’t log CPU clock frequencies, so these measurements don’t quantify throttling.
After the first stop, we added an explicit wait for every thermal zone to fall below 55°C before the remaining phases. Telemetry confirms that the earlier main CPU 0.8B phases also started below 55°C. Completed measurements stayed in the series; the failed stability block wasn’t rerun.
Memory and completed stability requests
| Model | Engine | Peak process, MiB | Max SoC, °C | Stability completed |
| 0.8B | llama.cpp CPU | 1,963.56 | 77.61 | 5 of 10 |
| 0.8B | RKLLM NPU | 1,176.57 | 60.08 | 10 of 10 |
| 2B | llama.cpp CPU | 4,249.27 | 77.61 | 5 of 10 |
| 2B | RKLLM NPU | 2,259.64 | 61.00 | 10 of 10 |
Peaks cover the monitored engine/model combinations, including phases with larger context capacity. The guard checked every thermal zone, not just the SoC sensor.
RKLLM’s process memory was lower, but the two host processes do different work: stock HTTP server versus measurement CLI. The GGUF and RKLLM files also differ in size and mapping behavior. VmHWM is resident process memory, not dedicated NPU VRAM or total system allocation.
The memory table uses MiB. Linux reports CPU VmHWM in KiB; during this run, a live check also matched RKLLM’s field labeled “MB” to KiB divided by 1024. That gives us comparable units, without making the host-process overhead identical.
Mean whole-system CPU use ranged from about 46–53% for CPU inference and 33–34% for RKLLM. Those samples include the desktop, pauses and idle time. Our CPU and NPU workload guide covers the work that remains outside the accelerator. Four heavily loaded performance cores can account for roughly half of an eight-core system, so 50% doesn’t mean half the selected cores were unused. We didn’t measure wall power; lower temperature alone doesn’t give an energy-efficiency figure.
Answers under a larger output budget
We gave both engines the same seven practical quality tasks with a 384-token output cap, increased from 192 in our earlier series. That matters for arithmetic answers which previously stopped before finishing.
The 0.8B RKLLM model now reached the correct result: nine chickens and five rabbits. Both 2B configurations did too. The 0.8B CPU answer went through incorrect reasoning and reached its limit without a correct final answer. This small set doesn’t establish that NPU inference is generally more accurate.
All four raw JSON responses parsed directly and matched the requested object. All four extracted Python functions passed four checks, although their full responses included Markdown fences. A correct extracted function and strict code-only output are separate requirements.
The grounding task also exposed a smaller error: 0.8B on CPU declined to invent an enclosure color, then added an Ubuntu version that wasn’t among the supplied facts. Technical explanations elsewhere made unsupported claims about learning, overheating or power savings. Those answers still need checking, whatever engine produced them.
For this board, I’d investigate cooling before relying on the CPU’s higher decode rate for repeated work. The NPU completed the short stability series and returned the long 2B input sooner; it didn’t make every part of the workload faster. Our separate Qwen3.5-4B test covers a larger model, but 4B wasn’t part of this CPU comparison. The next useful run is the intended application under sustained load, with cooling verified and the complete response time measured.
FAQ
Did llama.cpp use all eight CPU cores?
No. It used the four Cortex-A76 cores, with four generation threads and four batch threads. Other thread counts and a GPU backend weren’t tested.
Can the engines use the same model file?
These paths used GGUF Q8_0 for llama.cpp and converted W8A8 .rkllm files for RKLLM. Neither the file layouts nor the quantization implementations were identical.
Did the CPU stability tests crash?
The harness deliberately stopped active inference at its 85°C thermal-zone threshold. Each CPU block retained five completed requests. Both NPU blocks completed all ten; none of these short runs establishes round-the-clock reliability.
