How to Install the C01zaut WebUI for running LLM's on the RK3588 NPU

This video will provide step by step instructions on how to setup a WebUI to run LLM's on an Orange Pi 5 (or other Rockchip RK3588 devices from Radxa, etc.) using the built-in NPU (RKLLM). This tutorial will cover the installation of the operating system, setting up c01zaut's script for the WebUI, finding models in the RKLLM format, making the necessary configurations, and finally, of course, running the LLM.

The Github repository for an OS with NPU drivers pre-installed:
https://github.com/Joshua-Riek/ubuntu...

Lots of converted Rockchip models:
https://huggingface.co/c01zaut

The Github repository for the WebUI:
https://github.com/c0zaut/RKLLM-Gradio

Intro – 00:00
Operating System – 00:57
WebUI Installation – 01:50
Upload Models to the device – 02:30
Where to get models for the Rockchip NPU – 02:50
Update the model_configs.py file – 04:00
Maximum context length – 05:00
Running the script – 05:32
Bonus 1: Changing the WebUI title – 06:44
Bonus 2: Some more settings in the config file – 07:20

RKLLM, rknn-llm, OrangePi 5 OrangePi 5 Max, OrangePi 5 Pro, OrangePi 5 Plus, Radxa Rock 5, Rockchip RK3588, Rockchip RK3588S2, Khadas Edge2, ROCKPro64