All You Need To Know About Running LLMs Locally

RTX4080 SUPER giveaway!
Sign-up for NVIDIA's GTC2024: https://nvda.ws/48s4tmc
Giveaway participation link: https://forms.gle/2w5fQoMjjNfXSRqf7
Please read all the rules & steps carefully!!
1. Sign-up for NVIDIA's Virtual GTC2024 session between Mar 18 - 21st
2. Participate the giveaway DURING Mar 18 - 21st
3. ???
4. Profit


TensorRT LLM
[Code] https://github.com/NVIDIA/TensorRT-LLM
[Getting Started Blog] https://nvda.ws/3O7f8up
[Dev Blog] https://nvda.ws/490uadi

Chat with RTX
[Download] https://nvda.ws/3OHPRHE
[Blog] https://nvda.ws/3whKZTb

Links:
[Oobabooga] https://github.com/oobabooga/text-gen...
[SillyTavern] https://github.com/SillyTavern/SillyT...
[LM Studio] https://lmstudio.ai/
[Axolotl] https://github.com/OpenAccess-AI-Coll...
[Llama Factory] https://github.com/hiyouga/LLaMA-Factory
[HuggingFace] https://huggingface.co/models
[AWQ] https://github.com/mit-han-lab/llm-awq
[ExLlamav2] https://github.com/turboderp/exllamav2
[GGUF] https://github.com/ggerganov/ggml/blo...
[GPTQ] https://github.com/IST-DASLab/gptq
[LlamaCpp] https://github.com/ggerganov/llama.cpp
[vllm] https://github.com/vllm-project/vllm
[TensorRT LLM] https://github.com/NVIDIA/TensorRT-LLM
[Chat with RTX] https://www.nvidia.com/en-us/ai-on-rt...
[LlamaIndex] https://github.com/run-llama/llama_index
[Continue.dev] https://continue.dev/

Model recommendations:
[Nous-Hermes-llama-2-7b] https://huggingface.co/NousResearch/N...
[Openchat-3.5-0106] https://huggingface.co/openchat/openc...
[SOLAR-10.7B-Instruct-v1.0] https://huggingface.co/upstage/SOLAR-...
[Google Gemma] https://huggingface.co/google/gemma-7b
[Mixtral-8x7B-Instruct-v0.1] https://huggingface.co/mistralai/Mixt...
[Deepseek-coder-33b-instruct] https://huggingface.co/deepseek-ai/de...
[Madlad-400] https://huggingface.co/jbochi/madlad4...
[Colbertv2.0] https://huggingface.co/colbert-ir/col...


This video is supported by the kind Patrons & YouTube Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon

[Discord]   / discord  
[Twitter]   / bycloudai  
[Patreon]   / bycloud  

[Music] massobeats - magic carousel
[Profile & Banner Art]   / pygm7  
[Video Editor] maikadihaika