Qwen3.5-4B ran at 4.30 tokens/s on our 8GB KiwiPi 5, using a W8A8 conversion on the RK3588S NPU. It completed all 29 requests without an inference error or triggering the memory guard.
We’d already run the smaller Qwen3.5 models through the same RKLLM setup. This time we kept the board, measurement client and Runtime, downloaded the 4B file, and checked whether the extra memory and waiting bought us better answers.
On these few tasks, it didn’t.
The model got the animal-count problem wrong and added Markdown around an object we’d requested as raw JSON. That’s worth keeping beside the speed results, though seven prompts aren’t enough to rank the whole Qwen family.
Model and test setup
We used Qengineering’s converted language-model file, not the original Qwen checkpoint. The original Qwen3.5-4B model supports image and text input; this run used text only, without downloading the vision encoder.
The board still had Ubuntu 24.04.3 LTS, Linux 6.1.141, RKNPU 0.9.8 and RKLLM Runtime 1.3.1. GNOME and KiwiDrop stayed open, swap was off, and the existing NPU governor stayed in place. It was the same RK3588S board, with a heatsink and fan, rather than a new RK3588 test system.
The converted file is 5,578,247,044 bytes (about 5.58GB). We checked its SHA256 against the publisher’s LFS hash, then checked it again after transferring it to the board. The older files stayed on eMMC, but their models weren’t loaded alongside 4B.
You need room for the download as well as enough RAM to run it. Before starting, the board had about 40GiB of free eMMC space and 6.6GiB of available RAM. Those are separate checks; a model fitting on the drive says nothing about what remains for Linux after loading.
Qwen3.5-4B benchmark results
The model revision was ee09c44b420e031dbe96fcb489ef30c5f3e0d1a5. After copying its file into the existing lab directory, we checked it on the board:
Bash
cd ~/kiwipi-rkllm-lab
sha256sum models/qwen3.5-4b_w8a8_rk3588.rkllm
Expected SHA256 for this file:
Text
31ebf6a003675c82c72a2893bf46136720a0ae052a40bb12814e1bf67314b8ca
For measurements, we reused the small C++ client written specifically for these tests. It calls the official RKLLM API; it isn’t a chatbot application or a released inference server. The archived hashes confirm that both its executable and the Runtime library matched the earlier test.
Each input length ran three times, with a 2048-token context capacity and a 128-new-token output cap. We cleared the KV cache before every request and disabled conversation history and thinking. The saved decoding settings included top_k=1, top_p=0.95, temperature=0.8 and repeat_penalty=1.1.
The first-text measurement starts at rkllm_run and ends at the first nonempty text callback, excluding model loading. Generation speed uses the actual token count returned by Runtime. The table reports medians of three repeats; memory is the highest Runtime-reported process VmHWM in each group.
| Input tokens | First text, s | Decode, tokens/s | Decode range | Peak process MB |
| 39 | 1.03 | 4.30 | 4.29–4.31 | 4,806 |
| 266 | 6.42 | 4.21 | 4.10–4.23 | 4,806 |
| 746 | 16.71 | 4.09 | 4.08–4.10 | 4,806 |
Prompt length and waiting
The short prompt started returning text in 1.03 seconds. With 746 input tokens, that became 16.71 seconds, before the answer started appearing. Once generation began, throughput only fell from 4.30 to 4.09 tokens/s.
That’s a long pause if you attach a document to every question. For a short command you can wait for, it might be fine. I’d measure the inputs the application actually sends before deciding that four tokens a second is usable (and a token isn’t necessarily a word).
We deliberately requested longer answers than the performance cap allowed, to keep generation bounded and repeatable. Those cutoffs were expected. The separate quality tasks had a 192-token budget and need to be read as actual answers.
Earlier 0.8B and 2B results
| Model | Test date | Decode, tokens/s | First text, s | Peak process MB |
| Qwen3.5-0.8B | 22 October | 15.01 | 0.35 | 1,153 |
| Qwen3.5-2B | 22 October | 8.57 | 0.52 | 2,235 |
| Qwen3.5-4B | 27 October | 4.30 | 1.03 | 4,806 |
All three rows use the 39-token input. Speed and latency are medians of three repeats; process memory is the maximum reported in that group. These are historical comparisons on the same board, not a simultaneous benchmark.
The earlier 0.8B run reached 15.01 tokens/s on the short input, and 2B reached 8.57. Both started returning text sooner than 4B. The comparison chart includes those runs as historical references, not fresh measurements collected alongside the new model.
We used the same physical board, prompts, CLI and Runtime, but the reference runs were five days earlier. Room temperature and all background activity weren’t controlled across dates. I’d use the chart to judge the practical size of the delay, not to claim a tightly controlled three-model ranking.
Memory and model loading
Memory was a more immediate constraint. In the main performance series, the 4B process peaked at about 4,806MB according to Runtime. Across all phases, its highest reported peak was 4,870.4MB, while sampled available system RAM fell as low as 1.82GiB.
Those process readings aren’t dedicated NPU VRAM or the board’s total RAM usage. Linux and the desktop still need memory. We also set context capacity to 4096 for three requests, but kept the 746-token input, so that phase doesn’t show a full 4096-token prompt fitting or running well.
Model initialization took 18.34–18.88 seconds across the four processes. That’s another wait if you launch a process for each job instead of keeping it loaded. Files could already be cached by Linux; we didn’t measure controlled cold starts.
Arithmetic JSON and Python
The quality tasks were small: explanations, a summary, a question with missing information, arithmetic, JSON and a Python function. They’re useful examples of what the application would receive, not an accuracy benchmark.
For the arithmetic prompt, the model wrote down the right equations for 14 animals and 38 legs, then answered two chickens and six rabbits. That answer gives eight animals and 28 legs. The correct result is nine chickens and five rabbits.
The JSON object contained the requested values, but the response wrapped it in a Markdown code block. A JSON parser rejected the complete response. You can strip the fences, of course, but then the application needs to do that explicitly rather than treating the output as ready-to-use JSON.
The extracted Python max(a, b, c) function passed four cases. The response was also fenced code, rather than a directly executable Python file. And when given only the board’s processor and RAM, the model correctly said it didn’t know the enclosure color.
So there were working answers too. But for this converted artifact, these prompts and these decoding settings, we didn’t find a reason to accept the extra delay on quality grounds. We haven’t tested an unquantized baseline or another conversion, so I wouldn’t blame parameter count or quantization alone for the mistakes.
Repeated requests and temperature
Ten consecutive independent requests in one loaded process produced 4.48–4.57 tokens/s. That’s a separate stability series, not the 4.30-token/s median from the main table. The models weren’t running concurrently.
The monitor sampled once per second. Maximum sampled SoC temperature was 61.0°C, and NPU temperature reached 58.2°C; the highest reading across all thermal zones was 65.6°C. CPU activity included the desktop, and room temperature wasn’t measured.
A memory guard would have stopped the active test process after three consecutive samples below 384MiB available RAM. Each phase also had a 900-second watchdog. Neither triggered, but 29 completed requests aren’t an endurance test.
FAQ
Does Qwen3.5-4B fit in 8GB RAM?
The tested 4B file ran on this 8GB board without swap, with available system RAM staying above the guard threshold. That doesn’t establish enough room for larger contexts or other applications.
Did you test image input?
The original model supports visual input, but this experiment only loaded its converted language-model file. Image handling would require the appropriate encoder and execution path, neither of which we tested.
Can I load a GGUF file instead?
This run used a converted .rkllm file for the RK3588 target. A GGUF file is a different format and cannot be substituted directly into this measurement client.
