Local realtime person detection for RTSP cameras

[blakeblackshear]

Just published the final 0.7.0 release. I also went ahead and created a supported addon repository so those using hass.io can get it running easier. Next I will be working on a frigate component that creates camera/sensor/switch entities and exposes recordings and clips through the new media browser feature.

[moto2000 (David)]

Great work on this; much respect and appreciation! Can’t wait to for the future features you have planned!

[andynbaker (Andy)]

So awesome! I’m not sure how other are doing it, but I am running the samba add-on and have my HA media folder automount to the my pi that runs frigate on startup. I then mount that samba share to the frigate container clips folder. Now i get my clips automatically in my HA media folder!

To keep it from growing out of control, I use this node-red flow to purge files after like three days or something like that:

[{"id":"697cca45.abaa94","type":"fs-file-lister","z":"14b58c56.2703e4","name":"List all the files","start":"/media/security","pattern":"*.*","folders":"*","hidden":true,"lstype":"files","path":true,"single":false,"depth":"1","stat":true,"showWarnings":true,"x":437,"y":1819,"wires":[["5331d716.338ea8"]]},{"id":"db9814af.b5eca8","type":"inject","z":"14b58c56.2703e4","name":"Everyday at 2:00AM","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"00 02 * * *","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":219,"y":1819,"wires":[["697cca45.abaa94"]]},{"id":"5331d716.338ea8","type":"function","z":"14b58c56.2703e4","name":"Identify the old ones","func":"var now = new Date();\nvar created = new Date(msg.payload.stat.created);\nvar ageinmilliseconds = now - created;\nvar ageinhours = Math.floor(ageinmilliseconds / 3.6e6);\n\nif (ageinhours > 48) {\n   msg.payload.action = \"delete\";  \n} else {\n   msg.payload.action = \"ignore\";\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":661,"y":1819,"wires":[["ec4710c2.c99bd"]]},{"id":"ec4710c2.c99bd","type":"switch","z":"14b58c56.2703e4","name":"Delete?","property":"payload.action","propertyType":"msg","rules":[{"t":"eq","v":"ignore","vt":"str"},{"t":"eq","v":"delete","vt":"str"}],"checkall":"true","repair":false,"outputs":2,"x":842,"y":1819,"wires":[[],["8394721c.452dd"]]},{"id":"8394721c.452dd","type":"fs-ops-delete","z":"14b58c56.2703e4","name":"Delete the old ones","path":"","pathType":"str","filename":"payload.name","filenameType":"msg","x":1025,"y":1819,"wires":[[]]}]

[charredchar]

@S1eepy

I am using q35 on my windows VM. Works overnight so far, but it seems there are some hickups until they move to newer linux kernel.

I am very interested in this as I NEED to run on Windows Server 2019 bare metal in order to utilize my GPU with TensorFlow. My server Host OS is Windows and it is not going to be possible for me to move to a Linux based system and Hyper-V does not support GPU passthrough. Other then ensuring all of the python dependencies were met did you have any issues getting Frigate to run?

@blakeblackshear

Docker is a very thin layer and introduces an insignificant amount of overhead. It’s performance is nearly identical to bare metal. I am not aware of any reason to go through the hassle of trying to maintain it directly in a VM, but you should be able to follow the dockerfile. You will have to rebuild your VM from scratch with each update since many system packages change with each release.

I would be very surprised if it was possible to get it working on windows given how many dynamically linked linux libraries are required in the underlying python packages. I won’t ever add or maintain windows support myself.

I have been dealing with trying to get multiple TensorFlow based Camera Object Detectors to run on a Windows based system this week. While I do agree that Docker is usually a fine solution instead of a full blown VM it is not always the best use case, this is the most apparent with Docker on Windows which just runs as a Linux Hyper-V VM anyway so you aren’t really benefiting from Docker like you would on a Linux system.
The only way to utilize a GPU on a Windows system is going to be to run your python directly on bare metal which is something I would like to do. In all cases, when it comes to TensorFlow anyway, I have been successful in getting it running as long as Python on Windows has everything it needs just like it would need for Linux. All failures I’ve had this week had to do with old code (over a year in each case) and the issues not coming from running TensorFlow but the other items needed for using TensorFlow and some method of alerts. Unfortunately I don’t actually know how to write code so I have been doing the best I can and moving on to another project in hopes I could get it working.
I will be working on using Frigate this week and I have high hopes since it uses Python and is still in active development. Thank you so much for still being active, all your work so far and any future help.

[S1eepy]

I am running 0.7.0 now, and facing 30% CPU usage with my 3 streams.
EDIT: CPU now back to normal value: 10-12%

Can you know me what is UID/GID of your rendering device?
Here is mine:
crw-rw---- 1 root uuidd 226, 128 Oct 13 11:35 **/dev/dri/renderD128**

Command run by frigate in the logs show no error:

ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://xxxx

[zeliant (Terence Yung)]

this is my dri directory inside the LXC container

[S1eepy]

Not sure if this changes anything, but seems I have an issue with UUID.
I suppose I would receive error msg if HWaccell was not working…
Here is what I have in the mounting section of the lxc:

root@docker:~# mount|grep render
udev on /dev/dri/renderD128 type devtmpfs (rw,nosuid,relatime,size=16358996k,nr_inodes=4089749,mode=755)

[zhide.p (dp9023)]

Hello,

i am getting below greenish output on 0.7.0 with the config below. May i know is there any thing wrong? At version 0.6.0, it’s working well.

ffmpeg:
  global_args:
    - -hide_banner
    - -loglevel
    - panic
  hwaccel_args: 
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
  input_args:
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -vsync
    - drop
    - -rtsp_transport
    - tcp
    - -stimeout
    - '5000000'
    - -use_wallclock_as_timestamps
    - '1'
  output_args:
    - -vf
    - mpdecimate
    - -f
    - rawvideo
    - -pix_fmt
    - rgb24

[blakeblackshear]

Scroll up a few posts

[zhide.p (dp9023)]

hi Blake,

wow thanks. this is incredible!! I am using a 2.0 Ghz, 4-core cpu, and it drop from 30% to 20% now running two 1080p and two 720p streams with coral!!

[dave122 (Dave)]

Hey,

Thank you for creating this, it looks like exactly what I need.

I’m having a few issues getting it working on my i5 ubuntu server with hassio.

if I use the hassio addon the debug stream updates every 5 mins or so and I don’t get any detections.

If I install it directly on the ubuntu docker then it works well for 30 seconds then freezes and updates every 30 seconds or so.

What am I doing wrong?

{"back":{"camera_fps":12.0,"detection_fps":0.1,"ffmpeg_pid":17,"frame_info":{"detect":1603195450.159611,"process":0.0,"read":1603195491.241285},"pid":19,"process_fps":0.1,"read_start":0.0,"skipped_fps":12.0},"detection_fps":0.1,"detectors":{}}

web_port: 5000
detectors:
  coral:
    type: null
    device: null
save_clips:
  clips_dir: /media/frigate
mqtt:
  host: 10.0.0.10
  topic_prefix: frigate
  user: mqtt
  password: 
ffmpeg: {}
cameras:
  back:
    take_frame: 1
    fps: 5
    snapshots:
      show_timestamp: true
    ffmpeg:
      input: >-
        rtmp://10.0.0.87/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=
      input_args:
        - '-avoid_negative_ts'
        - make_zero
        - '-fflags'
        - nobuffer
        - '-flags'
        - low_delay
        - '-strict'
        - experimental
        - '-fflags'
        - +genpts+discardcorrupt
        - '-vsync'
        - drop
        - '-use_wallclock_as_timestamps'
        - '1'
objects:
  track:
    - person
    - car
    - truck
  filters:
    person:
      min_area: 1
      max_area: 100000000
      min_score: 0.1
      threshold: 0.1

Thank you

[blakeblackshear]

If you don’t have a Coral device, just remove the detectors section from your config entirely. Can you post some logs from when you run it with ubuntu docker directly?

[dave122 (Dave)]

That was it, thanks! I I have been scratching my head for hours on this.

If I delete the detector section in hassio it automatically adds back the below and works.

detectors:
  coral:
    type: edgetpu
    device: 'usb:0'

[benmarshall]

Hi Blake,

First of all many thanks for putting together this awesome software! I’ve been using a manual docker build for the past few months on an RPi running hassio and it’s been rock solid. Migrated today to 0.7.1 using the new addon and got it working reasonably quickly with a few tweaks to config settings. I don’t even have a Coral but for just 2 low res streams the CPU usage on an RPi4 is fine and the detentions are quick enough for me.

One question though - i tried the hw acceleration flags suggested in the new addon doc here but they cause frigate to break. If i add the RPi flags (32 bit for me) then I get ffmpeg errors about broken stream, constant ffmpeg restarts and the reported camera fps drops to zero. Does hassio support these flags?

Thanks you!

[blakeblackshear]

I assume you mean HassOS. I am not sure honestly. I don’t run it on any of my devices.

[NewToThis]

Hi,
Just having a go with the add-on version Frigate NVR but I’m having a few issues and I was hoping someone could help please

web_port: 5001
detectors:
  coral:
    type: edgetpu
    device: 'usb:0'
save_clips:
  clips_dir: /media/frigate
mqtt:
  host: 192.168.1.201
  user: auser
  password: apassword
  topic_prefix: frigate
ffmpeg: {}
cameras: {}

1, I have the mqqt info in the ‘config.yml’ file, should i remove it?

2, for the docker version it says to have the file in /config/frigate/config.yml but for the add-on it should be /config/config.yml ? I tried again by removing both files one at a time but got the same info in the logs

On connect called
Starting detection process: 34
Attempting to load TPU as usb:0
No EdgeTPU detected. Falling back to CPU.
 * Serving Flask app "detect_objects" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

So I am not sure if they are being ‘read’ ?

3, I seem to have a port conflict with something on 5000. so i have changed the add-on configuration and in the Network sensor section. Also changed in the config.yml and in sensors.yaml. (and restarted home assistant) I can see in MQTT Explorer and HA that the offline/Online sensor is active and correct I am not getting anything for my frigate/garage/person. Am i missing some config in the add-on that i have in my config.yml file?

4, no hardware acceleration at the moment but if i remove the text in the configuration ‘detectors’ and hit save it comes back or change detectors: null and then the add-on crashes

Appreciate any help please

I think Dave122 has answered most of my questions above. I take it the config.yml file is NOT used now.

[NewToThis]

Thanks for posting how you formatted the add-on, I have been scratching my head with that!

[blakeblackshear]

For the addon, you just enter your frigate config directly into the configuration tab in the UI. No need to store/manage a separate file at /config/config.yml.

[blakeblackshear]

Also, just leave the default detectors in there. It will fallback to CPU if no Coral is detected.

[NewToThis]

Thanks https://github.com/blakeblackshear/frigate has this that was throwing me a curve ball
A config.yml file must exist in the config directory. See example here and camera specific info can be found here.