FaceID — trainable face recognition for Frigate with a proper review UI (Home Assistant app + standalone)

Share your Projects!
, ,
41

Three releases since the RAM post, and the first one matters most if you have ever had
FaceID call the wrong name.

v0.15.0 — one bad reference photo caused a misidentification, and the old check missed it

A different person walked in here and was published under my name at 0.576, well over
the threshold. The cause was a single photo in my own gallery: taken from above, showing
mostly the back of a head.

Version 0.14 already checked references against the other people in the gallery. That
check saw this photo and let it pass at 0.368, just under its limit — and it was not
wrong. It compares stored photo against stored photo, while a live face comes far
closer than any stored photo of the same person does. Gallery-vs-gallery is a lower bound
on the real risk, and this case shows the gap: 0.368 measured, 0.576 in reality.

The stronger signal needed no second person at all:
that photo’s average similarity to its own person’s other 49 photos −0.011
normal for that person 0.265
its rank among those 49 photos last

A reference that barely resembles the person it belongs to holds almost no face
information — and an embedding without face information matches everyone a little. It
cannot help its own person; it can only attract other people’s faces.

So FaceID now checks both directions in one pass — on new photos, at startup, and via
Settings → Keeping the gallery clean → check gallery now. Replayed against the real
event: with that photo set aside, the same face no longer matches me at all (I drop out of
the top three) and is correctly recognised as the right person.

The limit is relative to each person (self_outlier_ratio, default 0.25 of that person’s
median, minimum 5 photos), because how alike someone’s photos normally are varies a lot —
0.23 to 0.47 across the people here. Unusual angles land around 0.10–0.20 and are kept
on purpose; what falls out is the range where there is no face left to compare. Across 188
photos of 13 people it set aside 3. Nothing is deleted — they land in the set-aside pile
with the numbers, and one click restores them.

If you have ever seen a wrong name, run the check once. It costs nothing and it is
reversible.

v0.16.0 — faceid/event now carries the Frigate zones (thanks @phipe for the use case)

{"person": "unknown", "score": 0.446, "camera": "entrance",
 "event_id": "1787041175.591208-sg2em5", "ts": 1787041180.7, "zones": []}

Deliberately data rather than a filter — details a few posts up. An example automation
(alert for strangers, but only in a chosen zone) is in
docs/ha-automation-example.yaml.

That payload above is a real one from this morning, and it is worth a closer look: it is an
unknown person, with an empty zone list, forty seconds after a known person at the same
camera came through with the zone filled in. Exactly the case a zone filter would have
silently dropped — and the one you actually want an alert for. Write conditions as zone is
present, not zone is absent.

v0.16.1 — correcting something I wrote here

I said the same payload also appears as the last attribute on the presence sensor. True,
but it used to survive only as long as the presence window: after that the finalizer
rewrote the retained topic without it, so for most of the day the attribute was simply
empty. Found while verifying the release — all four sensors here had no last hours after
correct recognitions.

It now persists until the next recognition replaces it. The sensor state still falls back
to nobody; last keeps answering who was seen there most recently. Automations
triggering on faceid/event were never affected — only templates reading the attribute.

42

Just wanted to say I’ve been updating to every version you put out and it works great so far!

43

@skycryer, thanks for adding the zone stuff to the event. Been using it for a couple of days now and it works as it should!

44

Is there a way to have FaceID ignore a zone? It keeps getting faces from my TV and I would like to filter those out. Thanks for the great app.

45

No, and I would push back gently on doing it in FaceID at all — because by the time FaceID
sees the event, the cost you actually care about has already been paid.

Where the TV problem is best solved

Your television produces genuine person detections: Frigate is not wrong, there really is
a person shape on the screen. So this is not a threshold or a recognition problem, it is a
“do not detect there” problem, and Frigate has exactly the tool for it — an object filter
mask over the screen area:

cameras:
  living_room:
    objects:
      filters:
        person:
          mask: 0.35,0.10,0.62,0.10,0.62,0.48,0.35,0.48

Draw it in the UI rather than by hand: Settings → Masks & Zones → Object Mask, pick
person, outline the screen. Any detection whose anchor point falls inside is discarded.

I run the same thing here for a different nuisance — an object mask for car on my
entrance camera, so passing traffic never becomes an event.

Why that beats filtering inside FaceID

If the event never exists, then there is no snapshot, no face crop, no CPU spent, and
nothing lands in your review queue. FaceID only acts on events that have a snapshot, so
the mask stops the whole chain at the source.

A zone filter inside FaceID would run at the far end of that chain: the detection still
happens, the snapshot is still written, the face is still extracted — and all you would
suppress is the final notification. Your review queue would keep filling with strangers
from last night’s film. That is the part that actually annoys people over time.

One thing to watch when you draw the mask

The filter uses the object’s anchor point (bottom-centre of the box), not the whole box.
For a wall-mounted TV that is ideal — the anchor of a person on screen sits inside the
screen area, while a real person standing in front of it has their feet below it, outside
the mask.

If your TV sits on a low stand or reaches the floor, those two overlap, and a mask that
covers the full screen can start swallowing real people who walk in front of it. In that
case keep the mask to the upper part of the screen and check it against a few real events
before trusting it.

Why the ignore list is not the answer here

FaceID’s ignore list is built for recurring people — the neighbour, the mail carrier —
where the same face keeps coming back and becomes an anchor. A TV shows a different face
every few minutes, so there is nothing stable to learn. You would be adding anchors
forever and never catch up.

If you would rather keep the snapshots

Two alternatives, in the order I would try them:

  1. snapshots: required_zones: with a zone covering the part of the room you do care
    about. It is an include rather than an exclude, so you draw around the sofa instead of
    around the TV — same effect, and events outside it never reach FaceID.
  2. Since v0.16.0 every faceid/event carries the Frigate zones the person was in, so an
    automation can ignore anything from a TV zone. This only silences the notification; the
    detection and the review queue are unaffected. Useful as a stopgap, not as the fix.

Start with the object mask. It is the smallest change and it removes the problem rather
than hiding it.

46

OK, I had a motion mask but not an object one. I will try with object. Thanks

47

I’m having trouble getting faceid recognised faces/names to show up in the SgftBatten blueprint for frigate. Are the names being rewritten back to the sub_label? The documentation also says there is a separate event at faceid/events which makes no sense for the blueprint/automation.

Anyone have this working so that notifications are updated with the face/name recognised by faceid?

48

Yes, FaceID writes the name back as sub_label — and no, that blueprint cannot show it.
I went and measured why, because the answer turned out to be more definite than I expected.

Frigate does not announce a sub_label change

This is the part that decides it. I took an event that already carried a sub_label, set the
same value again through the API, and listened on frigate/events for 8 seconds:

sub_label set -> HTTP 200 {"success":true, ... "sub label set to Eli"}
MQTT updates for that event in 8s: 0

Nothing. So a blueprint that reacts to frigate/events — SgtBatten’s included — has no way
of learning the name, however long it waits. It is not a bug in the blueprint and not
something I can fix from FaceID’s side; the notification simply never hears about it.

And the name is not ready when the notification fires

Over 132 real recognitions in a week here, measured from the start of the Frigate event:
median until the name is published 7.0 s
fastest case (first snapshot, no detour) 3.0 s
over 10 s 44 %

Those 3 seconds are not FaceID being slow — that is how long Frigate itself takes before a
snapshot exists at all. 62 of those recognitions came from the very first attempt and still
could not be faster. So even a perfectly patient blueprint would be waiting on the wrong
thing.

So do it the other way round

You were right that faceid/event looks out of place for the blueprint — it is not meant
for it. It is meant for an automation that runs after the fact and replaces the
notification you already got, using the same notification tag. Your phone then swaps the
message in place instead of stacking a second one.

I have packaged exactly that as a blueprint, released this morning:

Import blueprint

The one setting that matters is Notification tag — set it to the same tag your Frigate
blueprint uses (the event id by default, which is what most of them use). Get that wrong and
you get two notifications instead of one updated one. There are pickers for the target
device and for the cameras, filters for zones, and a switch for whether strangers should be
announced at all.

Two honest caveats. First, this replaces the text of a notification that has already
arrived — it cannot make the name appear any earlier than the 7 seconds above. Second, I
have imported it and verified its logic against real payloads, but it has not yet run
through a live arrival here, so treat the first day as a trial. If it misbehaves, tell me
what you see and I will fix it.

One more thing I am still checking: whether Frigate’s end-of-event message carries the
sub_label. If it does, there is a second route for anyone who would rather stay inside their
existing blueprint. I will report back either way.

49

Thanks was able to use this to modify the blueprint to get subsequent updates via the faceid/event topic. I think it would be good to provide native sub_label updates similiar to how doubletake worked. With doubletake, the blueprints available worked seamlessly without having to monitor a separate topic.

50

Correction: I got the central claim in my last post wrong.

I wrote that Frigate never publishes an MQTT update when a sub_label is set, and that a
blueprint listening on frigate/events could therefore never see the name. That is not
true, and the test I based it on was flawed.

What was wrong with the test. I set a sub_label on an event that had already ended and
watched for 8 seconds. Frigate sends nothing further about a finished event, sub_label or
not — so the silence proved nothing at all. A negative claim needs a test run under the
conditions where the thing would actually happen, and mine was not.

What really happens. On a running event, Frigate 0.17.2 forwards the name in the very
same second FaceID writes it. Log and MQTT capture side by side:

07:30:53  faceid: attempt 1, best match Eli (0.514) — published
07:30:53  mqtt:   after.sub_label=['Eli', 0.514]
07:30:56  mqtt:   after.sub_label=['Eli', 0.514]
07:31:02  mqtt (end): after.sub_label=['Eli', 0.514]

Confirmed across several real arrivals, not one lucky case.

And this is the part that actually helps @crunchynuts — and probably others who assumed
FaceID was not writing anything back:
Frigate 0.17.2 sends after.sub_label = ["Eli", 0.514] — an array of name and score
SgtBatten’s blueprint reads after.data.sub_labels
which on 0.17.2 is null

So the name is in the payload — it is simply not at the field being read. If your
notification stays nameless, check after.sub_label before concluding that nothing was
written. Note the shape too: it is an array, so the name is after.sub_label[0], not the
whole value.

That looks like a Frigate-version difference rather than anyone’s mistake, and it is worth
raising with the blueprint author so it can handle both shapes.

What this changes about my blueprint: nothing about how it behaves, only why I said you
need it. Listening to faceid/event still has real advantages — it does not depend on which
Frigate version puts the name where, and it carries the score and the Frigate zones along
with the name. But “a Frigate-side blueprint can never work” was simply wrong, and if that
sent anyone down the wrong path, my apologies.

The README, the blueprint description and the changelog are corrected as of v0.19.5.

51

To help others and future reference, if you want to retrofit your blueprint.
Attached is a working blueprint based off your input and sgtbatten’s blueprint that I am using that:

  1. Updates the sub_label and notifies correctly
  2. Also triggers off faceid/event and notifies
  3. Additional notification method via Signal Messenger which calls a script - you will need your own script as it is tailored for my own build.

You should be able to see the options/logic in the blueprint to retrofit as need.
Thanks again for the responses.

Blueprint
Blueprint Frigate 0.17.2 + Signal + FaceID

Supporting script
Dispatcher script for Notifications

52

That is the best possible outcome for this thread — thank you for publishing it rather than
just fixing your own setup.

I have read through it (not run it), and it does the thing I deliberately did not: it keeps
SgtBatten’s full feature set and adds both routes on top — the Frigate payload and
faceid/event as an override. That is a genuinely different trade-off from the blueprint in
my repo, which stays small and does one thing, and both are worth having.

So I have linked yours from the README, with a plain table for choosing:

  • already running SgtBatten’s blueprint and want its full feature set → your merged version
  • want the name added with as little machinery as possible → the one in this repo

Marked as a community contribution, with the honest caveats: read but not run here, and the
Signal path is written for your setup. The link points at a pinned commit so it cannot shift
under people.

One thing worth spelling out for anyone retrofitting their own: on 0.17.2 the field is
after.sub_label and it is an array — ["Alice", 0.51] — so the name is
after.sub_label[0], not the whole value. That detail cost me a wrong forum post; it should
not cost anyone else one.

Thanks also for pushing back on the “separate topic” point in the first place. Without that
I would have left a wrong claim standing in the README and in a blueprint description.