Скачать или смотреть Running AFM-4.5B on Intel CPUs with OpenVINO

Unlock the full potential of your Intel CPU server processor! In this demo, we transform an Amazon EC2 r8i instance equipped with the powerful Intel Xeon 6 (Granite Rapids) processor into a local AI powerhouse, running cutting-edge language models like AFM-4.5B by Arcee AI with exceptional performance and efficiency.

⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. You can also follow me on Medium at   / julsimon   or Substack at https://julsimon.substack.com. ⭐️⭐️⭐️

First, we optimize the AFM-4.5B model using the OpenVINO toolkit, quantizing it to 4 bits and 8 bits, and then run inference with a simple Python example written using Hugging Face transformers and Optimum-Intel. Next, we install the OpenVINO Model Server to serve the optimized model, invoking it with both curl and the OpenAI Python client.

Intel's Granite Rapids architecture delivers enterprise-grade AI performance with its advanced P-cores and E-cores working in perfect harmony. With OpenVINO's Intel-optimized inference engine, you'll see how to leverage the Xeon 6's substantial compute power for maximum AI performance.

Resources:
Code and commands: https://github.com/juliensimon/arcee-...
Amazon EC2 r8i: https://aws.amazon.com/blogs/aws/best...
Docker on Ubuntu: https://docs.docker.com/engine/instal...
Arcee AI AFM-4.5B
https://huggingface.co/arcee-ai/AFM-4.5B
https://huggingface.co/arcee-ai/AFM-4...
Intel OpenVINO:
https://openvinotoolkit.github.io/ope...
https://docs.openvino.ai/2025/model-s...
Hugging Face Optimum Intel: https://huggingface.co/docs/optimum/i...