RKLLM on KiwiPi 5 NPU

RKLLM ran Qwen3.5-0.8B at 15.01 tokens/s on our 8GB KiwiPi 5, with Qwen3.5-2B reaching 8.57 tokens/s on the same short input. Both used W8A8 models on the board’s RK3588S NPU.

We completed 58 requests across the two models without an inference error. Some answers were still wrong. That’s a useful distinction for a local assistant: getting text back is the first test, and checking what it says comes next.

The board and RKLLM setup

The test board was a KiwiPi 5 V20 running Ubuntu 24.04.3 LTS and Linux 6.1.141. Models lived on its eMMC system disk. The RK3588 platform specifications cover the wider hardware; these results belong to this RK3588S board and software image.

Component	Tested configuration
Board	KiwiPi 5 V20, RK3588S, 8GB RAM
System	Ubuntu 24.04.3 LTS, ARM64, Linux 6.1.141
NPU software	RKNPU driver 0.9.8; RKLLM Runtime 1.3.1
Model files	Qwen3.5-0.8B and 2B, W8A8, converted with Toolkit 1.3.0
Cooling	Heatsink and fan; no enclosure
Background applications	GNOME desktop and KiwiDrop
Storage and swap	eMMC with ext4; swap disabled

We left the desktop running and used the existing NPU governor. No frequency-setting script, kernel replacement or driver upgrade was needed. The room temperature wasn’t recorded, so the thermal results describe this run with its fan, rather than a cooling comparison.

Runtime on the board

Rockchip’s official RKLLM SDK separates model conversion from execution. The Toolkit converts and quantizes models on a computer; the Runtime provides the C/C++ API used on the board. We installed the Runtime and compiled its example. We didn’t convert models ourselves.

Instead, we downloaded the Qengineering 0.8B model and Qengineering 2B model as ready-converted .rkllm files. Their sizes were approximately 1.34GB and 2.98GB. Only the language-model files were used—no vision encoder or image inference. Check free space on Linux before downloading both files and unpacking the SDK.

Build and run the example

We connected to the board over SSH and checked the running image before installing anything:

Bash

uname -a
free -h
df -h /
cat /sys/module/rknpu/version
readlink -f /sys/class/drm/renderD129/device/driver

On this image, renderD129 resolved to the RKNPU driver. Another image may use a different render-node number; check its driver association before treating the filename as proof that the NPU is available.

The build used the existing GCC 13.3 and CMake. Download the SDK archive at commit f7390530443bf84f0394255a449d7cbe81e69d1c, place it in ~/kiwipi-rkllm-lab as rkllm-sdk-1.3.1.tar.gz, then run:

Bash

cd ~/kiwipi-rkllm-lab
sha256sum rkllm-sdk-1.3.1.tar.gz
mkdir -p sdk-release
tar xf rkllm-sdk-1.3.1.tar.gz --strip-components=1 -C sdk-release
cmake -S sdk-release/examples/rkllm_api_demo/deploy \
  -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4
ldd build/llm_demo

Download qwen3.5-0.8b_w8a8_rk3588.rkllm from the model repository, check its publisher hash, and put it in the models subdirectory. The test package includes the pinned revisions and SHA256 values for both models and the SDK. From the same working directory, start the official example:

Bash

RKLLM_LOG_LEVEL=1 build/llm_demo \
  models/qwen3.5-0.8b_w8a8_rk3588.rkllm 64 2048

Here, 64 limits new output tokens and 2048 sets the context capacity. Initialization reported Runtime 1.3.1, driver 0.9.8, W8A8 and three NPU cores. The example then accepted prompts and generated text using that execution path.

RKLLM benchmark results

We wrote a small C++ measurement client specifically for this experiment, using the same official API and shared library. It’s a test utility, with no web server or graphical interface. 

For the main measurements, each model received three input lengths, repeated three times. We cleared the KV cache before every request, disabled conversation history and thinking, set context capacity to 2048, and capped generation at 128 new tokens. Decode speed uses the actual token count returned by the Runtime.

Model	Input tokens	First text s	Decode tokens/s	Peak process MB
Qwen3.5-0.8B	39	0.35	15.01	1,153
Qwen3.5-0.8B	266	1.78	14.86	1,153
Qwen3.5-0.8B	746	4.44	14.52	1,153
Qwen3.5-2B	39	0.52	8.57	2,235
Qwen3.5-2B	266	2.12	8.47	2,235
Qwen3.5-2B	746	5.18	8.25	2,235

Latency and throughput are medians of three requests. First text measures time from rkllm_run to the first non-empty text callback, excluding model loading. Memory is the highest Runtime-reported process VmHWM in each group, in its reported MB units.

Longer inputs make you wait

The smaller model generated text considerably faster. At roughly 15 tokens/s, I’d start with it for an interactive demonstration (a token isn’t necessarily a whole word).

But the wait before generation grew much more than the decode rate changed. Feeding 746 tokens to the 0.8B model meant waiting 4.44 seconds for its first text, against 0.35 seconds for the short input. A chat application that sends a large document with every question will expose that delay immediately.

Memory and context

The 2B model’s reported peak was about 2,235MB in the main series, against 1,153MB for 0.8B. This is process memory, including the process’s other allocations; it isn’t dedicated NPU VRAM or the board’s total memory consumption.

We also raised configured context capacity to 4096. Peak process readings reached 1,177MB and 2,260MB, respectively. Those requests still used the 746-token input, so they don’t measure processing a full 4096-token prompt.

Loading is another separate wait. The main performance processes initialized in 1.91 seconds for 0.8B and 6.96 seconds for 2B. Later 2B processes loaded in roughly 2.8 seconds, but files could already be cached by Linux. We didn’t control cold-cache conditions.

What the models actually answered

Both models returned the requested JSON object with the correct keys and values. Both also wrote a simple Python function using max(a, b, c), which passed four limited checks. That gives us two small working examples; it doesn’t establish general coding ability.

The arithmetic task was less tidy. The 2B answer derived nine chickens and five rabbits correctly, then hit the output limit before finishing its last sentence. The 0.8B answer stopped before reaching a result. These quality requests allowed 192 new tokens, and a larger budget might change what finishes.

And both produced technical claims we wouldn’t publish as facts. The 0.8B model said the device learns from its own experiences; 2B suggested offloading prevents lag or overheating. Neither follows from this test. The grounding task went better: neither invented an enclosure color when given only the board’s RAM and processor.

Repeated requests and temperature

Each model then answered ten consecutive requests in one loaded process, again with KV cache cleared. The 0.8B model ranged from 14.69 to 15.28 tokens/s; 2B ranged from 9.27 to 9.33. These are separate stability runs, not the medians in the main table.

Across the final series, the highest sampled SoC temperature was 61.9°C and available system memory never fell below 4.23GiB. We sampled once per second. CPU readings include GNOME and other running applications, so they can’t be attributed entirely to RKLLM.

We didn’t run a CPU-only model or measure wall power. Our CPU and NPU workload comparison explains their different jobs, but this experiment cannot put a number on NPU speedup or energy savings.

FAQ

Can RKLLM load a GGUF file

The tested Runtime loads converted .rkllm files. A GGUF or original Transformers checkpoint needs a supported conversion workflow; renaming the file won’t make it load.

Did the test prove offline operation

Inference ran locally through the NPU, but we didn’t disconnect the network. Downloading the SDK and models required network access; a disconnected operation test would be a separate check.

Is 8GB RAM enough

Both tested models ran on this 8GB board without swap. Larger models, longer contexts and other applications need their own memory checks; these results don’t establish a maximum supported model size.