NExT-GPT: The first Any-to-Any Multimodal LLM

NExT-GPT is the first end-to-end Multimodal Large Language Model (MM-LLM) that can take inputs in arbitrary combinations of text, image, video, and audio and generate outputs in any of the same modalities. In short, it is the first any-to-any MM-LLM model.

In this video I go through some of the NExT-GPT model architecture, the proposed alignment techniques like Encoding-side LLM-centric Alignment,
Decoding-side Instruction-following Alignment, Modality-switching Instruction Tuning, and the MosIT dataset.

Hope it's useful. Please leave your comments for any clarifications.

Website: https://next-gpt.github.io
Paper Link: https://arxiv.org/pdf/2309.05519
Code: https://github.com/NExT-GPT/NExT-GPT

MY KEY LINKS
YouTube:    / @aibites  
Twitter:   / ai_bites​  
Patreon:   / ai_bites​  
Github: https://github.com/ai-bites​

🛠 🛠 🛠 MY SOFTWARE TOOLS 🛠 🛠 🛠
✍️ Notion - https://affiliate.notion.so/aibites-yt
✍️ Notion AI - https://affiliate.notion.so/ys9rqzv2vdd8
📹 OBS Studio for video editing - https://obsproject.com
📼 Manim for some animations - https://www.manim.community
🎵 My music - https://www.bensound.com and

📚 📚 📚 BOOKS I HAVE READ, REFER AND RECOMMEND 📚 📚 📚
📖 Deep Learning by Ian Goodfellow - https://amzn.to/3Wnyixv
📙 Pattern Recognition and Machine Learning by Christopher M. Bishop - https://amzn.to/3ZVnQQA
📗 Machine Learning: A Probabilistic Perspective by Kevin Murphy - https://amzn.to/3kAqThb
📘 Multiple View Geometry in Computer Vision by R Hartley and A Zisserman - https://amzn.to/3XKVOWi

WHO AM I?
I am a Machine Learning Researcher/practitioner who has seen the grind of academia and start-ups equally. I started my career as a software engineer 15 years ago. Because of my love for Mathematics (coupled with a glimmer of luck), I graduated with a Master's in Computer Vision and Robotics in 2016 when the now happening AI revolution just started. Life has changed for the better ever since.

#machinelearning #deeplearning #aibites