4090 Local AI Server Benchmarks

How many Tokens per Second could you expect to hit with a RTX 4090 GPU on modern LLMs and which would be the best models to consider? Let's test out the performance in openwebui and ollama of the top new models that can run in a single 4090.
RTX 4090 https://geni.us/4090GPU_5N_s

AI Home Server Quad 3090 Build    • INSANE Ollama AI Home Server - Quad 3...  
AI Playlist    • AI Rigs, Testing, Reviews, Use Cases ...  

QUAD 3090 AI SERVER BUILD
GPU Rack Frame https://geni.us/GPU_Rack_Frame
RTX 3090 24GB GPU (x4) https://geni.us/GPU3090
Gigabyte MZ32-AR0 Motherboard https://geni.us/mz32-ar0_motherboard
Supermicro H12ssl-i Motherboard https://geni.us/MBD_H12SSL-I-O
Kritical Thermal GPU Pads https://geni.us/Kritical-Thermal-Pads
256GB (8x32GB) DDR4 2400 RAM https://geni.us/256GB_DDR4_RAM
PCIe4 Risers (x4) https://geni.us/PCIe4_Riser_Cable
AMD EPYC 7702p https://geni.us/EPYC_7702p
iCUE H170i ELITE CAPELLIX https://geni.us/iCUE_H170i_Capellix
(sTRX4 fits SP3 and retention kit comes with the CAPELLIX)
ARCTIC MX4 Thermal Paste https://geni.us/Arctic_ThermalPaste
CORSAIR HX1500i PSU https://geni.us/Corsair_HX1500iPSU
4i SFF-8654 to 4i SFF-8654 (x4) https://geni.us/SFF8654_to_SFF8654
HDD Rack Screws for Fans https://geni.us/HDD_RackScrews

Be sure to 👍✅Subscribe✅👍 for more content like this!

Join this channel    / @digitalspaceport  

Digital Spaceport Website
🌐 https://digitalspaceport.com

Chapters
0:00 4090 LLM Performance
0:50 Installing the 4090
2:55 Ollama LLM Model Shopping
6:38 Qwen 2.5 32b Tokens
9:12 Llama 3.1 8b Tokens
10:27 Llama 3.2 3b fp16 Tokens
12:38 Conclusion

*****
As an Amazon Associate I earn from qualifying purchases.

When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
*****