Building a Real-Time AI Assistant: Ainara Framework Demo with Local LLM

This video demonstrates Ainara, an open-source AI assistant framework that aims to implement the aspirational characteristics we've seen in sci-fi AI assistant characters.

The first foundation stone I'm working on is the ability to interact with the real world in real-time. For this purpose, I'm developing what I call Ainara's Orakle server - a REST API that provides 'skills' (concrete actions) and 'recipes' (chained actions of skills potentially combined with LLM processing).

Another component of the system is 'kommander', a CLI application designed to chat with the LLM, process Orakle commands in real-time, and make the LLM aware of the feedback from those commands.

As a proof of concept, I've developed some skills for Orakle: the capability of searching news that allow the AI assistant to search for real-time news (using newsAPI by now), and I've also created a more complex recipe that can download a web page, extract any articles within it, and process the content according to the desired profile - whether that's a layman's summary, easy-to-read language, or even content adapted for a young child, which works remarkably well (each of these actions is another skill which the recipe processes in a chained way).

This video demonstrates using Ainara to search for recent news about a major company and quickly assess its investment potential.

An important aspect to highlight in this video: I'm not using any commercial LLM, but rather running llama.cpp with a 5-bit quantized version of Qwen 14B 2.5 on my own server, which features a 'humble' NVidia RTX 3060 card. I'm quite impressed by how well it understands the instructions to interact with my local Orakle server, as demonstrated in this video.

Source code available at: https://github.com/khromalabs/Ainara