Local realtime person detection for RTSP cameras

Blake, a big thanks for Frigate! I tried a few other systems before this and they clogged my NUC completely, whereas Frigate runs smooth and CPU utilization stays well below 20%.

I have the HA integration installed and get clips recorded from motion events just fine. My only issue is with the thumbnails. For some reason they are green, even if the stored video clips seem to be just fine.

Anything I can do to solve this?

Can you open an issue on GH and include the info requested in the template?

Ive got it working, at least camera.back_last_person shows detects person! Woohoo!
But in the frigate UI i only see this:

{"back":{"camera_fps":5.1,"capture_pid":40,"detection_fps":0.0,"pid":38,"process_fps":5.1,"skipped_fps":0.0},"detection_fps":0.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":9.53,"pid":36}}}

For some reason i thought there going to be some kind of interface in there. Or is this a correct view?

Btw is there a way to have a camera in ha that would show boxes around people in real tiime (rather than just when object is in the camera) as n this video : https://www.youtube.com/watch?v=nqHbCtyo4dY
Looked carefully through github readme but didnt find how to do that and whether it is possible at allā€¦

HA is the UI, so thatā€™s what I would expect. Beta 0.8.0 adds better integration with HA too. You could setup the mjpeg stream as a camera in HA to see a feed with bounding boxes, but it wonā€™t support HA stream and I wouldnā€™t recommend it since it takes extra resources that slow frigate down. My camera entities are setup with the new features in 0.8.0.

My camera entities are setup with the new features in 0.8.0.

im on v0.8.0 beta3, does it mean no need to setup sensors and cameras in HA manually or still need to do that?

You could setup the mjpeg stream as a camera in HA to see a feed with bounding boxes, but it wonā€™t support HA stream

do you mean using /<camera_name>/latest.jpg endpoint? I just wanted to play with it a little bit.
is this going to show video only when object is in it or all the time + when object detected boxes around it?

How can i say that frigate works fine, coral utilised, etc, ie does this look fine:
{"coral":{"detection_start":0.0,"inference_speed":9.53,"pid":36}

Thanks for your replies! And sorry about all the newb qs. Just really want to make it work and i think im very close now :slight_smile:

To get the tighter integration with HA, you need to add the integration as a custom_component. No need to setup the mqtt entities manually anymore. It also gives you the media browser for clips and recordings.

And those stats are great. Exactly what I would expect from the GK41.

Morning all,

I upgraded to the beta version 0.8.0-beta3-amd64 with the hass integration. Everything is working great. Detections are fast, last detection imagine shows up in the hass UI, clips are available in media browser, etc.

But for the life of me I canā€™t figure out why my notifications donā€™t include the image. I get the notice on my iphone. but there is no image embedded.

notification

I also tried replacing the public Nabu Casa URL with my internal IP like this:
image: ā€˜http://10.0.0.16:8123/api/frigate/notifications/{{trigger.payload_json[ā€œafterā€][ā€œidā€]}}.jpgā€™

but that doesnt work either.

Here is my notification. Any help in figuring out why this doesnā€™t work would be super appreciated. I am also getting a notification every second or so which didnā€™t happen prior to upgraded to beta 3, so is there a setting I missed to consolidate all those notifications? (basically if someone is walking up the front sidewalk, I get 3-4 notifications of that event.

- id: "1609552193688"
  alias: Notify on Frigate detection
  description: Notify when Frigate AI detects a person or car
  trigger:
    - platform: mqtt
      topic: frigate/events
  condition: []
  action:
    - service: notify.mobile_app_bw_max
      data_template:
        message: A {{trigger.payload_json["after"]["label"]}} was detected.
        data:
          image: 'https://<UUID>.ui.nabu.casa/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}.jpg'
          tag: '{{trigger.payload_json["after"]["id"]}}'
  mode: single

This whole system is amazing, thanks for the obviously significant amount of time you put into this Blake. Itā€™s very much appreciated!

-B

Those look like the format for Android notifications. Update according to the iOS format: https://companion.home-assistant.io/docs/notifications/notification-attachments

Also, the ā€œduplicateā€ notifications are actually ā€œupdatedā€ notifications when a better image is found. You can use the apns-collapse-id feature for the iOS notifications to replace it with the updated one. You can rate limit however you want using standard homeassistant automation features.

I had to use my internal IP for the photo to come through. I donā€™t think it will send if Iā€™m away from home but not managed to test yet

I finally switched to the 0.8.0 beta tonight (from 0.7.0).

One new thing I immediately noticed in the log was a lot of watchdog entries ā€“ looks like every 10 seconds it terminates the ffmpeg processes:

2021-01-02T19:35:28.352915980Z watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.353105977Z watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.370121880Z watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.370564321Z watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.382726835Z watchdog.cars_ludwig           INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.383290920Z watchdog.cars_ludwig           INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.389498947Z watchdog.path                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.389900333Z watchdog.backyard_tvroom       INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.390207974Z watchdog.backyard_tvroom       INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.390495195Z watchdog.path                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.391448270Z watchdog.cars                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.392933739Z watchdog.cars                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:28.394971234Z watchdog.trees_storey          INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:28.395263537Z watchdog.trees_storey          INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.372794497Z watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.372972961Z watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.390165575Z watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.390744230Z watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.413430770Z watchdog.cars_ludwig           INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.414505799Z watchdog.cars_ludwig           INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.414775848Z watchdog.backyard_tvroom       INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.414983901Z watchdog.backyard_tvroom       INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.415172601Z watchdog.path                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.415353570Z watchdog.path                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.417451307Z watchdog.trees_storey          INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.417777184Z watchdog.trees_storey          INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:38.420101829Z watchdog.cars                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:38.420330145Z watchdog.cars                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.392904954Z watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.393570953Z watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.403321192Z watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.403871449Z watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.424836484Z watchdog.cars_ludwig           INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.425627186Z watchdog.cars_ludwig           INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.437186470Z watchdog.path                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.437715310Z watchdog.path                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.438140333Z watchdog.trees_storey          INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.438530632Z watchdog.cars                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.438912939Z watchdog.cars                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.439489388Z watchdog.backyard_tvroom       INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:48.439880181Z watchdog.trees_storey          INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:48.442474411Z watchdog.backyard_tvroom       INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.420809586Z watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.420967601Z watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.428914766Z watchdog.front                 INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.429153731Z watchdog.front                 INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.458041597Z watchdog.cars_ludwig           INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.458111933Z watchdog.cars_ludwig           INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.464572889Z watchdog.backyard_tvroom       INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.464606269Z watchdog.backyard_tvroom       INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.476357177Z watchdog.trees_storey          INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.476585298Z watchdog.trees_storey          INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.480548367Z watchdog.cars                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.480576279Z watchdog.cars                  INFO    : Waiting for ffmpeg to exit gracefully...
2021-01-02T19:35:58.484298955Z watchdog.path                  INFO    : Terminating the existing ffmpeg process...
2021-01-02T19:35:58.484539819Z watchdog.path                  INFO    : Waiting for ffmpeg to exit gracefully...

If I enable clips, it deletes the temporary files every time too so the logs look even worse.

I reworked my config based on the new config example by working through it line by line, but my Intel based hardware acc is the same as it has always been and thereā€™s no per-camera settings other than the RTSP path.

Any ideas what this could be? I am trying the RTMP feature.

I use the ONVIF integration and that gives me a main and sub stream for each of my cameras. I was getting frequent CPU spikes on my Pi4 4GB up to 50/60%. Iā€™m not sure if it was the integrations with stream support or if it was because I had ā€œpreload streamā€ enabled but I removed them all and itā€™s CPU hasnā€™t went above 20% with frigate - excellent job!

Is the RTMP role (and the created camera entity) of frigate so that I can still live view my camera without re-adding the ONVIF integration?

Moreover, I have two stable configs for my camera 1 utilises rtsp and the other utilises rtmp - in your expert opinion, which config should I be using?

Thanks @blakeblackshear, I am not sure why I never thought to check that.

I updated to the IOS format and it still didnā€™t work but after some more messing around I eventually got it, so thanks very much for your help!

One thing I just wanted to note is that it only works with ā€œdata_template:ā€ in the message (like your example). When I was putting that in before, the built-in linter said that ā€œdata_templateā€ was deprecated and I should just use ā€œdataā€. I just wanted to document that here in case anyone else has that issue.

Thanks again, this is a truly amazing project!

-B

I believe the issue might be that I was using H.265 / HEVC encoding on my main streams, which isnā€™t supported by RTMP (as per Frigateā€™s Github readme).

I saw this when I enabled the FFMPEG details logs:

2021-01-02T22:09:59.895429615Z ffmpeg.path.clips_rtmp         ERROR   : Input #0, rtsp, from 'rtsp://hass:<redacted>@172.16.16.216/Streaming/Channels/1':
2021-01-02T22:09:59.895570540Z ffmpeg.path.clips_rtmp         ERROR   :   Metadata:
2021-01-02T22:09:59.895838759Z ffmpeg.path.clips_rtmp         ERROR   :     title           : Media Presentation
2021-01-02T22:09:59.895983484Z ffmpeg.path.clips_rtmp         ERROR   :   Duration: N/A, start: 1609625398.494767, bitrate: N/A
2021-01-02T22:09:59.896134099Z ffmpeg.path.clips_rtmp         ERROR   :     Stream #0:0: Video: hevc, yuvj420p(pc, bt709), 1920x1080, 90k tbr, 90k tbn, 90k tbc
2021-01-02T22:09:59.897730796Z ffmpeg.path.clips_rtmp         ERROR   : [flv @ 0x557a351b4100] Video codec hevc not compatible with flv
2021-01-02T22:09:59.897885730Z ffmpeg.path.clips_rtmp         ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented
2021-01-02T22:09:59.898075999Z ffmpeg.path.clips_rtmp         ERROR   : Stream mapping:
2021-01-02T22:09:59.898574983Z ffmpeg.path.clips_rtmp         ERROR   :   Stream #0:0 -> #0:0 (copy)
2021-01-02T22:09:59.898879880Z ffmpeg.path.clips_rtmp         ERROR   :     Last message repeated 1 times
2021-01-02T22:09:59.900340224Z ffmpeg.cars.clips_rtmp         ERROR   : Input #0, rtsp, from 'rtsp://hass:<redacted>@172.16.16.215/Streaming/Channels/1':
2021-01-02T22:09:59.901495602Z ffmpeg.cars.clips_rtmp         ERROR   :   Metadata:
2021-01-02T22:09:59.901670494Z ffmpeg.cars.clips_rtmp         ERROR   :     title           : Media Presentation
2021-01-02T22:09:59.901926689Z ffmpeg.cars.clips_rtmp         ERROR   :   Duration: N/A, start: 1609625398.498733, bitrate: N/A
2021-01-02T22:09:59.902074718Z ffmpeg.cars.clips_rtmp         ERROR   :     Stream #0:0: Video: hevc, yuvj420p(pc, bt709), 1920x1080, 90k tbr, 90k tbn, 90k tbc
2021-01-02T22:09:59.903224290Z ffmpeg.cars.clips_rtmp         ERROR   : [flv @ 0x55df0a2668c0] Video codec hevc not compatible with flv
2021-01-02T22:09:59.903779506Z ffmpeg.cars.clips_rtmp         ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented
2021-01-02T22:09:59.903926455Z ffmpeg.cars.clips_rtmp         ERROR   : Stream mapping:
2021-01-02T22:09:59.904133191Z ffmpeg.cars.clips_rtmp         ERROR   :   Stream #0:0 -> #0:0 (copy)
2021-01-02T22:09:59.904802825Z ffmpeg.cars.clips_rtmp         ERROR   :     Last message repeated 1 times

Removing those seem to have gotten rid of the watchdog issue!

Any outlook for the beta4 release date?

The documentation looks promising,e.g. I like the per camera ffmpeg, so I could try rtmp with the slightly unstable reolink. It also seems that the yaml format will change in some areas.

Its typically not the best idea to hurry development - but showing interest might be taken as a compliment :slight_smile:

ItsMee

I have been tied up with other projects at home for the past few weeks, so itā€™s not going to be for a few more weeks.

Sorry if Iā€™m missing this somewhere, but how does one increase the size of the RTMP streams in Home Assistant?

Iā€™m passing through my full resolution (1080p) streams via the RTMP feature. Iā€™m not using the stream: component.

I can open the RMTP streams that Frigate provide (rtmp://<frigate_host>/live/<camera_name>) in VLC and can see full resolution images:

and the cameras that I used to add to Home Assistant (generic camera, displaying the still image every 1 sec) shows up the full resolution stream:

But the Frigate streams are much smaller:

EDIT: OK, I tried the stream: function and the cameras are now full resolution and full FPS, although they do take a while to load. I guess it must be a limitation in Home Assistant.

1 Like

How do you go about huge media folder? I just left it overnight with 3 cameras and it filled up 13gb.
Is there a way to save these on external usb drive? ie mount it as media?

Did you turn on 24h recording?

ohh im blind! :man_facepalming:

Iā€™ve just copied it from a config i found somewhere else and thought this record thing is about recording when object detected rather than 24/7. ALso i didnt realise until now there is a modified config manual on release-0.8.0 branch so was looking at current stable config describtion all the time. lol.

Thanks!

1 Like

I just ordered mine ā€¦ Wondering already how it will perform :slight_smile: I ordered PCIe module to install directly in my Intel NUC :slight_smile: