Local realtime person detection for RTSP cameras

biggen1684 · December 14, 2020, 10:54pm

So if I use this as my docker-compose file:

version: "3.6"
services:
  frigate:
      container_name: frigate
      restart: unless-stopped
      privileged: true
      image: blakeblackshear/frigate:0.8.0-beta1-amd64
      volumes:
      ...

That is still the same beta image that we are working with, correct? so a docker-compose up -d should pull the new image this way?

blakeblackshear · December 14, 2020, 11:06pm

That image won’t be updated. The new version will be published as 0.8.0-beta2-amd64 .

biggen1684 · December 14, 2020, 11:09pm

Ok I see. How does one keep up with your scheduled pushes? Your 0.8.0-beta2 tree still has the older image listed in the readme.

MRobi · December 14, 2020, 11:22pm

On reboot it re-appears right away. Although it doesn’t seem to affect how things work.

I do have a few questions on the new version.

I’m trying to wrap my head around the new HA integration that works with the beta.
In the stable release, the mqtt topic of frigate/camera_name/object_name controlled a binary sensor that was either on or off. I used this as an outdoor security motion sensor. When an object was detected, the binary sensor turned on, the lights turned on. When it turned off, it would start a timer to turn the lights off. If this happened while the alarm was armed (either home or away), it would send notifications to our phone with the best.jpg snapshot.

With the new beta and HA integration, this topic now gives an object count. I’m sure I could make an automation that when the count increases, turn on lights, but I’m not seeing a path anywheres that could act as “off” to start the timer to turn off the lights…

I love the idea of 2 streams. I’d like to use the 4k stream to capture images. How do I differentiate the width/height of the 2 streams? I have one stream that is 640x320, the other is 3840x2160. Looking at the ffmpeg settings example, it appears there’s only the option to set 1 width/height.

blakeblackshear · December 14, 2020, 11:23pm

You can watch the repo on github and tell it to notify you for new releases.

blakeblackshear · December 14, 2020, 11:26pm

When they value is 0, turn off. When the value is >0 turn on. You can make a template binary sensor if you prefer.
The width and height in the config are specific to the stream with the detect role. The other streams can be whatever you want. Frigate doesn’t care.

technikhaus · December 14, 2020, 11:52pm

Just thought I’d say a massive thanks for this!
I recently got it running on my setup, all done through HA, integration + addon.
Just shifted my media directory through to my OMV NAS, and thought I’d post here as I saw others having issues with the database locked with saving to a CIFS share, all I had to do to get around that, was to set nolock when mounting the share. I have mine mounted at the Home Assistant OS level, replacing /mnt/data/supervisor/media with my NAS. All appears to be working well now.

My only consistant issue is the bad/broken frames, but I don’t think there’s much I can do there?
This all started when I first set it up, and hasn’t changed since shifting to the NAS storage.

frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
[h264 @ 0x55b67650f0] left block unavailable for requested intra mode
[h264 @ 0x55b67650f0] error while decoding MB 0 47, bytestream 55451
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x558c93ac10] moov atom not found
/media/frigate/recordings/FrontDoor-20201215124330.mp4: Invalid data found when processing input
frigate.record                 INFO    : bad file: FrontDoor-20201215124330.mp4
watchdog.FrontDoor             INFO    : No frames received from FrontDoor in 20 seconds. Exiting ffmpeg...
watchdog.FrontDoor             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : FrontDoor: ffmpeg sent a broken frame. something is wrong.
frigate.video                  INFO    : FrontDoor: ffmpeg sent a broken frame. something is wrong.
frigate.video                  INFO    : FrontDoor: ffmpeg sent a broken frame. something is wrong.
frigate.video                  INFO    : FrontDoor: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. something is wrong.
frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. something is wrong.
frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...

I do still need to buy a Coral for mine, as my interface_speed is around 300, and everything is quite delayed, so hopefully that’ll improve that.
the setup is:
Raspberry Pi4 8Gb
2x Degang’d Wyze cameras.

{"FrontDoor":{"camera_fps":5.1,"capture_pid":43,"detection_fps":1.8,"pid":39,"process_fps":0.8,"skipped_fps":0.0},"detection_fps":3.6,"detectors":{"coral":{"detection_start":1607989860.579606,"inference_speed":278.81,"pid":35}},"driveway":{"camera_fps":5.0,"capture_pid":41,"detection_fps":1.8,"pid":38,"process_fps":1.0,"skipped_fps":0.0}}

blakeblackshear · December 15, 2020, 12:10am

The wyzecams and any other wifi camera generally have issues providing a reliable stream for ffmpeg to consume error free. I run dahua wired cameras and often go weeks without a hiccup. There is quite a bit of reconnect and retry logic in frigate to recover in this situation.

technikhaus · December 15, 2020, 12:11am

It honestly appears to run really well apart from those disconnects, so I can’t complain at all!

Jafterdark · December 15, 2020, 1:31am

has anybody gotten this to run with like an Nvidia card trying to figure it out

haip · December 15, 2020, 2:36pm

Installed 0.8.0-beta1 and updated my config.yml.

No local port on 5000

7927e60dc1f9:/opt/frigate$ ss -tlpn
State                Recv-Q               Send-Q                             Local Address:Port                              Peer Address:Port              Process               
LISTEN               0                    4096                                  127.0.0.11:34643                                  0.0.0.0:*                                       
LISTEN               0                    128                                    127.0.0.1:5001                                   0.0.0.0:*                  users:(("python3",pid=1,fd=52))

TaperCrimp · December 15, 2020, 2:54pm

There are a lot of breaking changes with the 0.8.0 release. I found it easier just to rebuild my configuration. The 0.7.X configuration will not work.

haip · December 15, 2020, 3:11pm

Fount the issue… As I start the container as non-root user, nginx could not be started. A few changes to the compose file and it starts now.

TaperCrimp · December 15, 2020, 6:13pm

@blakeblackshear here’s an easier method in the MQTT binary sensor that gives the best of both worlds:

Take this, which worked with the original ON/OFF payloads:

- platform: mqtt
  name: "Motion BYC Person"
  state_topic: "frigate/byc/person"
  payload_on: "ON"
  payload_off: "OFF"
  device_class: motion
  availability_topic: "frigate/available"

…and change it to this:

- platform: mqtt
  name: "Motion BYC Person"
  state_topic: "frigate/byc/person"
  payload_on: "ON"
  payload_off: "OFF"
  device_class: motion
  availability_topic: "frigate/available"
  value_template: >
    {% if value == "0" or value == '' %}
      OFF
    {% else %}
      ON
    {% endif %}

Tested it quite a bit and it seems to work fine. If it’s NULL or 0, the binary sensor stays off. Anything else, e.g. 1 or 10 objects detected, and it turns to on.

FYI your work on Frigate has been outstanding. I basically replaced the brains in Blue Iris with Frigate via MQTT integration. It works amazingly well. Thanks much!

MRobi · December 15, 2020, 8:06pm

@blakeblackshear my issue with camera’s jumping around keeps coming back. If I watch the actual source rtsp stream it’s good, but through frigate it jumps around to different streams.

I uploaded a quick clip to show what i mean. This is while watching the re-broadcast rtmp stream, I get the same if I hit IP_address:5000/camera_name

blakeblackshear · December 15, 2020, 10:42pm

Interesting. I have no idea why this would be happening. Frigate does zero processing of the RTSP feed before re-broadcasting it as RTMP. The ffmpeg process streams that directly to nginx without even decoding, so it is basically just a pipe for what ffmpeg read from the rtsp stream. Not a line of code touches that stream.

The fact that it happens at the mjpeg endpoint is also interesting because that is what frigate reads from ffmpeg’s other output. I’m guessing you are using the same feed for both rtmp and detect.

Because this happens in both places, I don’t believe the issue originates in the actual frigate code. Somehow the ffmpeg process is receiving frames from 2 different cameras intermittently.

What type of cameras are these and what does your RTSP input look like?

MRobi · December 15, 2020, 10:54pm

They are Reolink B800 cameras with the RTSP stream coming from the NVR. And yes, same feed for rtmp, detect and clips. I actually noticed this happening a few days ago with the stable release which is what prompted me to move to the beta.

The RTSP link is: rtsp://user:[email protected]:XXX/h264Preview_02_sub

While watching the feed bouncing around through the mjpeg endpoint on my laptop, I loaded the same RTSP link into Tiny Cam Pro on my phone and it’s solid. I then created a camera entity with the same RTSP link and it’s also solid. If I reboot frigate enough times, it works as it should until I edit anything in the config again.

blakeblackshear · December 15, 2020, 11:04pm

What if you setup the same RTSP feed as an ffmpeg camera directly in HomeAssistant? Editing the config on frigate won’t really change anything other than cause ffmpeg to restart. I wouldn’t be surprised if running ffmpeg anywhere with that RTSP endpoint shows the same behavior.

MRobi · December 15, 2020, 11:09pm

That’s what I mean when I said I created a camera entity.

camera:
   - platform: ffmpeg
     name: Reolink1
     input: !secret reolink1

And it’s a solid stream.

Jafterdark · December 16, 2020, 12:14am

?? Now what if you’re using a GPU like an Nvidia card would you put GPU instead of CPU and then PCI if you have the answer thanks