Local realtime person detection for RTSP cameras

You don’t have to install HA supervised on a Pi. You can install it in a docker container. You don’t get HA add-ons that way, but you can still install whatever you want to the host outside the container.

I’m using the docker-compose file at near the bottom of this page to run HA on a RPi4 with the stock Raspberry OS: https://www.home-assistant.io/docs/installation/docker/

Sorry, I wasn’t trying to take this off topic. I was just noting that it looked like you would be using some hardware that had much more power than a raspberry pi and could do it all on one with overall better performance if you wanted to.
Not sure of your level of experience with HA so sorry if you already know this.
Home Assistant OS is the standalone version with the supervisor. I can be run on bare metal or in a virtual machine.
You can also install plain debian and then HA (with supervisor) on that for a supported install.
Start by reading here: https://www.home-assistant.io/getting-started/
and if you want to go the HassOS route then look here: https://www.home-assistant.io/hassio/installation/
There is also an installation sub-forum here that may be of more help: https://community.home-assistant.io/c/installation/17

I get the same complaint:

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:473
Integration: Sensor (documentation, issues)
First occurred: 0:40:54 (8 occurrences)
Last logged: 0:40:54

  • Platform frigate does not generate unique IDs. ID frigate_front_garden_car_person already exists - ignoring sensor.front_garden_car_person
  • Platform frigate does not generate unique IDs. ID frigate_front_garden_person_person already exists - ignoring sensor.front_garden_person_person
  • Platform frigate does not generate unique IDs. ID frigate_front_garden_car_car already exists - ignoring sensor.front_garden_car_car
  • Platform frigate does not generate unique IDs. ID frigate_front_garden_person_car already exists - ignoring sensor.front_garden_person_car

It was set to default, strangely. Thought I’d set it to 1G however it seemed to have reverted. I set it again and it’s been running very well indeed, despite a number of additional cameras @ full resolution.

Absolutely phenomenal work!

I’ve been watching this for a while and finally got round to setting it up yesterday. It worked great with minimal tweaking - massive thanks to @blakeblackshear for creating it!

In terms of optimising, I fear I might be a bit limited by my setup. I run Home Assistant OS on a VM within Proxmox, on a machine with an AMD Ryzen 3 3200G CPU. I pass through 4 cores to the HA VM. I have 3 Reolink RLC-410 POE cameras.

I am currently using CPU-based detection on the 640x480 fluent RTSP substream at 7fps. This works fine - CPU usage in the VM is around 25-30%. Detection is a little slow and sometimes misses people if they walk past the camera quickly.

The limitations of my current setup are:

  1. Lack of Coral hardware detection
  2. Lack of hardware acceleration (as far as I can see, it is next to impossible to pass the Vega GPU through to my VM in proxmox)
  3. Inability to increase the stream quality (my only options are 4k (main stream) and 640x480 (fluent stream).

The question for those with more demanding setups would be, if I were to get a Coral to pass through to the VM, would this allow me to run detection on the 4K streams, or would the lack of hardware acceleration then be a bottleneck in decoding the streams?

I ended up with windows 10 on my gk41 (as it was preinstalled there) + VirtualBox (with Linux Other option) + Hassos.
The plan was to use frigate as add on but for some reason when i install frigate add on it says:

Starting detection process: 20
Attempting to load TPU as usb
No EdgeTPU detected. Falling back to CPU.

I wonder if this is because i of that ‘Linux other’ instead of ‘Linux Debian’ or for any other reason? Is there a way to fix that?

Anyway i read here that others experience slow inference time running it through VM so i think im going to try and run frigate in a docker in parallel with Virtual box. Just wonder if Is there any obvious downsides with that?

I would strongly suggest not running this in a VM on top of windows 10. You will have multiple layers of operating systems to maintain and performance will be worse than a raspberry pi 4.

i see thanks! Would running docker (with frigate) + vm (with ha) on windows 10 be ok then?
Or shall i go back to original plan and leave ha on rpi3b and use gk41 for frigate only?
I think i really like the idea of having add ons.

hmm docker and virtual box dont seem to like each other. After installing docker the virtual box stops working.

Hi! Is it possible to change “shm-size” -parameter when using Frigate HassOs -addon? I’m running latest beta -version. Normally you could change the parameter from Docker, but don’t know if it possible if running this way?

Can you give me a hint how to do this (passing the environment variable to container)? Running Home Assistant Supervised on Debian with the frigate beta AddOn.

Thanks in Advance

Mike

I am running Debian server with Home Assistant supervised on my NUC and Frigate. The CPU usage for 3 cameras @ 1080p is 8%. I recommend this install type as you will have supported HA installation plus other add-ons.

Frigate does not support windows. I would just install debian directly on the gk41 and setup the supervised version of homeassistant.

Hey guys,

Version: Beta on RPI 4 32 bit (Had same issue on 64bit)

I have been using Frigate for a long time now. I was running it on a laptop via linux but decided I wanted my laptop back so I bought a RPI4 and im running it on 32 bit. (I tried 64 bit first). Everything works but two cameras. It keeps saying the following:

watchdog.carport               INFO    : No frames received from carport in 20 seconds. Exiting ffmpeg...
watchdog.carport               INFO    : Waiting for ffmpeg to exit gracefully...
watchdog.backyard              INFO    : No frames received from backyard in 20 seconds. Exiting ffmpeg...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...

Looking through the logs I found the following:

peg.carport.detect          ERROR   : Input #0, rtsp, from 'rtsp://admin:[email protected]/cam/realmonitor?channel=2&subtype=1':
ffmpeg.carport.detect          ERROR   :   Metadata:
ffmpeg.carport.detect          ERROR   :     title           : Media Server
ffmpeg.carport.detect          ERROR   :   Duration: N/A, start: 1609546034.460389, bitrate: N/A
ffmpeg.carport.detect          ERROR   :     Stream #0:0: Video: h264, yuv420p(progressive), 704x480, 90k tbr, 90k tbn, 180k tbc
ffmpeg.carport.detect          ERROR   : [h264_v4l2m2m @ 0x177c1f0] Using device /dev/video10
ffmpeg.carport.detect          ERROR   : [h264_v4l2m2m @ 0x177c1f0] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
ffmpeg.carport.detect          ERROR   : [h264_v4l2m2m @ 0x177c1f0] requesting formats: output=H264 capture=YU12
ffmpeg.carport.detect          ERROR   : Stream mapping:
ffmpeg.carport.detect          ERROR   :   Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> rawvideo (native))
ffmpeg.carport.detect          ERROR   : Press [q] to stop, [?] for help
ffmpeg.backyard.detect         ERROR   : Input #0, rtsp, from 'rtsp://admin:[email protected]/cam/realmonitor?channel=4&subtype=1':
ffmpeg.backyard.detect         ERROR   :   Metadata:
ffmpeg.backyard.detect         ERROR   :     title           : Media Server
ffmpeg.backyard.detect         ERROR   :   Duration: N/A, start: 1609546034.771778, bitrate: N/A
ffmpeg.backyard.detect         ERROR   :     Stream #0:0: Video: h264, yuv420p(progressive), 704x480, 90k tbr, 90k tbn, 180k tbc
ffmpeg.backyard.detect         ERROR   : [h264_v4l2m2m @ 0x1a76380] Using device /dev/video10
ffmpeg.backyard.detect         ERROR   : [h264_v4l2m2m @ 0x1a76380] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
ffmpeg.backyard.detect         ERROR   : [h264_v4l2m2m @ 0x1a76380] requesting formats: output=H264 capture=YU12
ffmpeg.porch.detect            ERROR   : frame=  913 fps=5.0 q=-0.0 size=  451935kB time=00:03:02.60 bitrate=20275.2kbits/s dup=0 drop=4561 speed=1.01x    
ffmpeg.backyard.detect         ERROR   : Stream mapping:
ffmpeg.backyard.detect         ERROR   :   Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> rawvideo (native))
ffmpeg.backyard.detect         ERROR   : Press [q] to stop, [?] for help

Does anyone know what’s going on here? These cameras worked fine before moving to the RPI4. I am running 6 cameras so is that to many for the PI4? I cannot seem to get more than 4 working at a time. If I change the feed I can get one of them working but then another camera will stop working. The cameras are outputting 704 x 480 so they are low quality streams.

Here is my config:

detectors:
  coral:
    type: edgetpu
    device: usb
mqtt:
  host: hass.z
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: mqttuser
  password: mqttpassword
ffmpeg:
  global_args: -hide_banner -loglevel info
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m
cameras:
  porch:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
################
  carport:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=2&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
################
  basement:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=3&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
        - car
        - truck
################
  backyard:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=4&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
        - car
        - truck
################
  mailbox:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=5&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
        - car
        - truck
################
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]/cam/realmonitor?channel=6&subtype=1
          roles:
            - detect
    width: 704
    height: 480
    fps: 5
    snapshots:
      show_timestamp: true
      draw_zones: false
      draw_bounding_boxes: false
    objects:
      track:
        - person
        - car
        - truck
################
objects:
  track:
    - person
  filters:
    person:
      min_area: 5000
      min_score: 0.60

Any help would be appreciated.

Try removing your hwaccel args to see if that’s the issue.

1 Like

wow that was it… I have no idea why I didnt try that myself. I spent 5 hours trying everything else but not removing that… Thank you so much. All my cameras are working correctly now. I really do appreciate everything you are doing.

Thank you.

You will want to get them back, but now we know where the issue is. Did you disable protection mode for the addon? Did you increase the GPU memory in raspi-config?

I couldn’t figure out the add-on (I have it installed on the hass system but didnt understand the rest) so I’m not using that. The GPU memory is on 76 Megs. What should I set that too?

cool, i just installed debian + supervised HA + Frigate addon. phew what an adventure it was :slight_smile:

Still haven’t manage it to work, i get these errors though for 2 of my cameras:

Fontconfig error: Cannot load default config file
ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.86.36/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=admin&password=xxx"
Starting detection process: 21
Attempting to load TPU as usb
On connect called
TPU found

and then these repeatedly:

back: ffmpeg sent a broken frame. something is wrong.
back: ffmpeg process is not running. exiting capture thread...
timelapse: ffmpeg sent a broken frame. something is wrong.
timelapse: ffmpeg process is not running. exiting capture thread...
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtmp://192.168.86.36/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=admin&password=xxx -f rawvideo -pix_fmt yuv420p pipe:
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtmp://192.168.86.31/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=admin&password=xxx -f rawvideo -pix_fmt yuv420p pipe:

And the Frigate UI looks like this:

{"back":{"camera_fps":0.0,"capture_pid":34,"detection_fps":0.0,"frame_info":{"detect":0.0,"process":0.0},"pid":36,"process_fps":0.0,"skipped_fps":0.0},"detection_fps":0.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":10.0,"pid":21}},"timelapse":{"camera_fps":0.0,"capture_pid":38,"detection_fps":0.0,"frame_info":{"detect":0.0,"process":0.0},"pid":40,"process_fps":0.0,"skipped_fps":0.0}}

my config is:

web_port: 5000
detectors:
  coral:
    type: edgetpu
    device: usb
save_clips:
  clips_dir: /media/frigate
mqtt:
  host: 192.168.86.xx
  user: xxx
  password: 'xxx'
ffmpeg: {}
cameras:
  back:
    ffmpeg:
      input: >-
        rtmp://192.168.86.36/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=admin&password=xxx
  timelapse:
    ffmpeg:
      input: >-
        rtmp://192.168.86.31/bcs/channel0_sub.bcs?token=sdasdasd&channel=0&stream=1&user=admin&password=xxx
objects:
  track:
    - person
    - car
  • tried it with **rtsp://admin:[email protected]:554//h264Preview_01_sub** with no luck. Just wonder if im missing anything obvious?

Looks like you are using rtmp streams. The default input args support rtsp, so you will need to override them. If you are using reolink cameras, there are several discussions and issues where you should be able find the right args.

I would set it as high as you can and try to back down from there if you don’t want it that high.

1 Like