LocalAI Testing: Viewer Question LLM context size, & quant testing with 2x 4060 Ti's 16GB VRAM

Описание к видео LocalAI Testing: Viewer Question LLM context size, & quant testing with 2x 4060 Ti's 16GB VRAM

Attempting to answer a good viewer question with a bit of testing in the lab.
We will look at how context size effects VRAM usage, and also address speed testing with different quant sizes with Codestral 22B.

We will using LocalAI, and two Nvidia 4060 Ti's with 16GB VRAM each.

Grab your favorite after work or weekend enjoyment tool and watch some GPU testing

Recorded and best viewed in 4K

Комментарии

Информация по комментариям в разработке

Похожие видео