Mnemos: A fully local, low power, self hosted face recognition system

Share your Projects! Custom Integrations
1

I wanted to share a project I’ve been working on for a while that I’ve finally gotten to a place where I’m comfortable releasing into the wild.

It’s called Mnemos.

The Backstory

This all started when I realized Google knew who was at my front door better than I did. I wanted a self-hosted, local solution, but ran into a few roadblocks:

  • Frigate is fantastic, but it requires continuous video streams. Running standard video streams on three battery-powered cameras drained them in under a week.
  • Ollama image integrations is great for describing a photo but not who is in the photo.
  • Double Take seemed like the ideal middle ground, but the project is abandoned.

So, I decided to build something specifically tailored for snapshot-based / low power setups.

What is Mnemos?

Mnemos is a lightweight, locally run application that operates via API endpoints. It is split into two main parts:

  • Frontend: A clean interface where all user and person management takes place.
  • Backend: Left free to handle all the heavy image processing and lifting.

Instead of requiring a 24/7 video stream, Home Assistant is in charge of sending snapshot images to its /identify endpoint. The endpoint will perform analysis and return a list of names & confidence. In the event the person is unknown. They are automatically added to the user interface for tagging later.

Inference & Hardware Acceleration

Mnemos relies on InsightFace’s Buffalo model suite across the board for consistent accuracy, but handles execution based on your available hardware:

Current

  • CPU Inference: Uses standard onnxruntime with the Buffalo model pack out of the box.

Beta

  • Rockchip NPU Support: Employs librknnrt from rknn-toolkit2 alongside pre-converted RKNN Buffalo models to offload processing directly to Rockchip NPUs.

Planned

  • GPU Support: Nvidia acceleration support will also use onnxruntime-gpu to support buffalo models.

Raspberry Pi Warning: Standard Raspberry Pis are likely too underpowered to handle this smoothly. You will want hardware with a dedicated NPU/GPU or a more capable CPU. Personally i use a rk3588

The entire stack is docker first to reduce end user headaches configuring the containers.

Home Assistant Integration

To make setting it up painless, I created a companion HACS integration: Mnemos-HA.

It includes state sensors and an action you can use directly in your Home Assistant automation’s to send snapshots from your cameras whenever motion is detected. The action will respond back with identites in standard JSON format allowing easy parsing within Home Assistant.

What’s Next

I am currently working towards V2 of Mnemos which will include both Rockchip NPU and Nvidea GPU support in stable.

image
image1223×1268 137 KB

image
image1223×1268 102 KB

image
image1223×1268 48.3 KB

image
image1223×1268 39.4 KB

image
image979×1042 73.6 KB

I would love for the community to try it out, poke around, and share any feedback!

2

This certainly looks interesting, I’m keen to try it out!
What’s our timeline on Nvidia GPU support?

I’m already running Frigate so I wonder if the two could possibly coexist?

3

Sounds like you can send a frigate snapshot to Mnemos for recognition.

4

I haven’t personally tried Frigate. I stopped when i realised it only worked with video streams.

As long as its able to provide HomeAssistant with snapshots. It will work.

Nvidea support is next on my list. It will be released in stable within the V2 release.

5

Just as a update on this. I think it might be a good use case to build out proper support for this. Frigate currently can be used with Compreface and Double Take so it makes sense for me to add that support as well. I’ll add it as a roadmap item.

I will aim to include this in v2.1.0 which may be a couple of weeks while i work on adding the backend variant support for v2.0.0

1 Like