Today I discovered that Mistral AI has just released Mistral 7B, “a small, yet powerful model adaptable to many use-cases”. The LLM is around 13.5 Gb and can be run locally.
Has anyone considered building a plugin? I know it would mostly be for entertainment purposes, but seeing as it’s “the year of the voice”, I think it would be fun to have a digital companion that isn’t as dumb as rocks like Alexa. It could be made to look for commands before determining if a chat response is expected instead. Bonus points if it can remember details from conversations and build a “personality”.
I wish I were more tech savvy so I could set it up myself and share my work, but I can barely make proper Home Assistant scripts as it stands. 
Yeah sounds interesting, but even this “small” model would need a robust hardware to be ran, far away from what our HA hosts are honestly.
But some tools like this one https://jan.ai/ are coming closer to something usable, maybe.
The model can’t run as component, it’s too big and wouldn’t fit in a haos VM, but it would be enough to create a llama.cpp integration. It would need to take an hostname and port in the configuration. Then it would be able to chat and take voice commands from any model that runs on llama.cpp including mistral 7B.
This would be a more interesting integration to add than openai. It runs fast enough on my laptop, so it will surely run fast enough on the server where I run haos on top of kvm (with PCI serial and no USB passthrough).
https://llama-cpp-python.readthedocs.io/en/latest/
I just noticed there is also an API compatibile with openai, so maybe it’s enough to just add host/port to the current openai integration to make this work.