Local realtime person detection for RTSP cameras

Hi would you mind to share info how did you manage to mount the share? I was not able to figure it out.
How did you poing frigate to use the mounted folder?

Thank you

I have a Raspberry Pi 4 w/ a google Coral USB, running on Raspbian with the latest stable frigate (0.8.4-armv7) from docker compose file. I havenā€™t been able to get hardware acceleration working.

If I donā€™t comment out the hardware accel part of this frigate.xml config:

cameras:
  # Name of your camera
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://user1:[email protected]:554/live
          roles:
            - detect
            - rtmp
            - clips
            - record
          hwaccel_args:
            - -c:v
            - h264_mmal
    width: 2688
    height: 1520
    fps: 5

I get this error below. Is there something wrong with my hwaccel_args? When this happens, I get a green screen where the video should be.

pi@raspberrypi:~/docker $ docker-compose up
Creating network "docker_default" with the default driver
Creating frigate ... done
Attaching to frigate
frigate    |  * Starting nginx nginx
frigate    |    ...done.
frigate    | frigate.app                    WARNING : Camera driveway has record assigned to an input, but record is not enabled.
frigate    | Starting migrations
frigate    | peewee_migrate                 INFO    : Starting migrations
frigate    | There is nothing to migrate
frigate    | peewee_migrate                 INFO    : There is nothing to migrate
frigate    | frigate.mqtt                   INFO    : MQTT connected
frigate    | detector.coral                 INFO    : Starting detection process: 33
frigate    | frigate.app                    INFO    : Camera processor started for driveway: 36
frigate    | frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate    | frigate.app                    INFO    : Capture process started for driveway: 37
frigate    | frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x2248350 (EINVAL)
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:in:0(H264)(0x2248350) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:out:0(0x2256e50) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:ctr:0(0x2248040) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : Error while opening decoder for input stream #0:0 : Unknown error occurred
frigate    | frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
frigate    | ffmpeg.driveway.detect         ERROR   : [rtsp @ 0x8cd710] DTS discontinuity in stream 1: packet 18 with DTS 25896235559131, packet 19 with DTS 25896235572122
frigate    | ffmpeg.driveway.detect         ERROR   : [rtsp @ 0x8cd710] DTS discontinuity in stream 1: packet 21 with DTS 25896235572126, packet 22 with DTS 25896235575536
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x8eafe0 (EINVAL)
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:in:0(H264)(0x8eafe0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:out:0(0x91dbd0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:ctr:0(0x8eacd0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : Error while opening decoder for input stream #0:0 : Unknown error occurred
frigate    | frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate    | frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
frigate    | ffmpeg.driveway.detect         ERROR   : [rtsp @ 0x152f710] DTS discontinuity in stream 1: packet 16 with DTS 25896235718573, packet 17 with DTS 25896235732523
frigate    | ffmpeg.driveway.detect         ERROR   : [rtsp @ 0x152f710] DTS discontinuity in stream 1: packet 19 with DTS 25896235732527, packet 20 with DTS 25896235735938
frigate    | ffmpeg.driveway.detect         ERROR   : [rtsp @ 0x152f710] DTS discontinuity in stream 1: packet 32 with DTS 25896235736660, packet 33 with DTS 25896235748403
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_info_set: failed to set port info (2:0): EINVAL
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_vc_port_set_format: mmal_vc_port_info_set failed 0x15e4ed0 (EINVAL)
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:in:0(H264)(0x15e4ed0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:out:0(0x15e51e0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : mmal: mmal_port_disable: port vc.ril.video_decode:ctr:0(0x15e4bb0) is not enabled
frigate    | ffmpeg.driveway.detect         ERROR   : Error while opening decoder for input stream #0:0 : Unknown error occurred

Iā€™m getting multiple ā€œerror while decoding MBā€ immediately after startup, no matter if Iā€™m running with hwaccell options or without them. Anybody has ideas why is this happening?

2021-04-15T19:30:07.891336583Z frigate.mqtt                   INFO    : Turning off clips for hall via mqtt
2021-04-15T19:30:07.915211788Z frigate.mqtt                   INFO    : Turning off detection for hall via mqtt
2021-04-15T19:30:07.926828493Z detector.cpu1                  INFO    : Starting detection process: 32
2021-04-15T19:30:07.947388166Z detector.cpu2                  INFO    : Starting detection process: 34
2021-04-15T19:30:07.949864552Z frigate.app                    INFO    : Camera processor started for hall: 37
2021-04-15T19:30:07.972015000Z frigate.app                    INFO    : Capture process started for hall: 38
2021-04-15T19:30:48.445318806Z [h264 @ 0x8b80e0] error while decoding MB 55 21, bytestream -5
2021-04-15T19:31:08.907334082Z [h264 @ 0x19cf0e0] error while decoding MB 29 6, bytestream -5
2021-04-15T19:31:39.514317572Z [h264 @ 0x1a8e0e0] error while decoding MB 18 1, bytestream -23
2021-04-15T19:32:20.285039352Z [h264 @ 0x13580e0] error while decoding MB 25 30, bytestream -13
2021-04-15T19:32:30.532597425Z [h264 @ 0x12870e0] error while decoding MB 39 11, bytestream -6
2021-04-15T19:32:50.978076592Z [h264 @ 0x12ae0e0] error while decoding MB 26 18, bytestream -5
2021-04-15T19:33:01.196071224Z [h264 @ 0x1b270e0] error while decoding MB 69 41, bytestream -7
2021-04-15T19:33:21.670614605Z [h264 @ 0x1ae90e0] error while decoding MB 1 22, bytestream -27
2021-04-15T19:33:52.339451074Z [h264 @ 0x146f0e0] error while decoding MB 67 25, bytestream -8
2021-04-15T19:34:02.602071485Z [h264 @ 0xf660e0] error while decoding MB 25 6, bytestream -7

Iā€™ve been testing the temps for a couple of days, trying to give the M.2 Coral as much of a sustained ā€˜real worldā€™ load as possible from multiple high res cameras at 20fps, as well as all the other cameras at 5fps.

Temps could be a non-issue as the maximum Iā€™ve been able to hit is 50.6c @ 117fps with an 18c ambient temp. I think the PCI Coral starts to throttle @ 85c, so thereā€™s plenty of headroom for ambient temperature increases etc. Maybe Iā€™ll test again when ambient temps hit 30c during hot summer days.

Interestingly, the max detection FPS Iā€™ve hit is 128.6fps, when the inference speed was about 7.69ms. When working the Coral hard, the inference speed seems to increase suddenly, reducing the fps. So perhaps ~130fps is the best we can expect from Frigate with a single Coral, as a burst. The maximum inference speed logged was 15.62ms and the minimum was 6.6ms.

Either way, Iā€™ll get Home Assistant to notify if the Coral temp hits 75c :sweat_smile:

2 Likes

I have a Nano too. Iā€™m not sure this will perform better than a PI 4 with regards to detection, but HW acceleration for FFMPEG should be better. I was never able to get it working though, but Iā€™d really like to.

The Jetson has an M.2 slot, so there could be the possibility to fit an M.2 Coral and also use the Jetsonā€™s HW acceleration - which could make it an excellent performer (not sure anyoneā€™s tried this, though).

On the PI side, I found my PI 4 to perform quite well with a USB Coral. I just did a quick vs:

This is during the night with little movement, I havenā€™t tested it as CPU detector during the day with lots of movement. It is using the PI HW acceleration and the videos during day tests with Coral are great.

i used to run frigate on pi4 with hwaccel from hik stream , but i only use 640x480 with 4 camera. . no prob. it is possible.

Morning All,

Couple of newb questions regarding this.
I currently have HA running on an i7 mini PC using virtualbox on windows (iā€™m not too clued up with linux, proxmox, docker etc). So I need to keep it as simple as possible.
I got hold of a coral m.2 TPU but from the frigate docs it looks like VM cannot pass through the PCIe.

Should I ditch windows and go for a direct HASSos install and if so, can I use the NUC install for that?
If anyone has any suggestion on the best way to have a working HA box with Frigate using the M.2 Coral Iā€™d be very grateful on some tips.

My suggestion is to use proxmox. If it is possible.
The advantage is that you can put automatic backup for HASS and relax your mind.
I was having HASS on RPI and whenever i was updating or going away for a while i was wondering if anything happen i need to strat from tha last backup which was 1-2month ago. So i ended up using proxmox and automatic backup :slight_smile: with PCI patchrough.
There are a lot of tutorials that i used to install proxmox and HASS.

1 Like

if you want to ditch windows, and you are not familiar with proxmox, you can try to install debian as the host OS.

once debian is installed, install docker.

Then run home assistant supervised installer script to install home assistant

Then install coral m.2 drivers

Then proceed to add frigate addon and customs components.

2 Likes

Thanks for the advice.
Which do you think is best/easiest setup as Iā€™m not familiar with linux or proxmox but I will learn whichever one needs learning.

With virtualbox I liked being able to clone the whole install as a backup, I couldnā€™t find a way to do that in Proxmox or linux.

With the NUC installation method, is that itā€™s own operating system and HA in one? Would that be a tidier install?

You know never mind, to make it easier just install on NUC directly, and then make a setup to make snapshots backups.
That should be the easiest solution for you.
Once in a while make a local copy of the backup to your PC

If u can use the nuc install it will save you much hassle to install docker and home assistant supervised.

Itā€™s the easiest route for u since you are not familiar with Linux.

I got the 2x PCI Corals running :grin:. It seems that protection mode needs to be disabled for it to work - did you find that? Inference speed for the pair seems to be about 7ms in my case.

coral_triple_17-04-2021

Now testing triple Corals.

For kicks I got the double coralā€™s working too. One in the wifi slot and one in my other NVME slot. Ready to load it down once I get my other NIC in to dedicate to just the camera VLAN.

Did little video on this Frigate thing recently. Might help with some of the YAML struggles of the getting over the first camera ā€œhumpā€.

4 Likes

My advice would be to learn linux. It isnā€™t difficult to start, just install Debian OS and follow the guide here: Installing Home Assistant Supervised on Debian 10

You then have a handy linux GUI to play with, as well as Home Assistant. If you stick with it, you will find that anything is possible and everything becomes easier as you continue to learn.

The easy route isnā€™t always the best route, particularly when you have the will to learn.

2 Likes

Hello there,

while detecting people, the software generates a couple of pictures. Then, after a while, it deletes all but one picture and present this as the best image.

I would like to keep all of the generated images.

Where do I define this?
Here is my config:

#cameras:
  Nextone:
    ffmpeg:
      inputs:
        - path: rtsp://xxx:[email protected]:2554/axis-media/media.amp
          roles:
            - detect
    width: 1920
    height: 1080
    fps: 5

    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: False
      required_zones: []
      retain:
        default: 10
        objects:
          person: 15
          car: 15

    objects:
      track:
        - person
#        - cat
#        - dog
#        - car
      filters:
        person:
          min_area: 100
          max_area: 1000000
          min_score: 0.5
          threshold: 0.7
detectors:
  cpu1:
    type: cpu

Thank you very much.
Cheers
Chris

Any ideas? Recommendations?

Can anyone give me some advise on this post I did the other day please?

I believe I cannot get Frigate to run due to my CPU not having AVX support.

So does this mean I cannot use my existing system (HP Proliant N36l) if I want to use Frigate? I HAVE TO swop my hardware?

Or any other options please?

"Frigate working fine in a docker on my Dell R710, thank you very much but unfortunately that thing eats electric!

So I have tried the exact same set up on a older smaller pc to run 24/7, a HP Proliant nl36.
but I am having a major issue!
AMD Athlonā„¢ II Neo N36L Dual-Core @ 1300 MHz
Unraid set up
Home Assistant VM
Trying either Docker or Home Assistant addon, I am getting the message below,

The NL36 is old and I found this thread that I ā€˜thinkā€™ is related but could do with some help please
Fatal Python error: Illegal instruction Ā· Issue #695 Ā· blakeblackshear/frigate Ā· GitHub but I donā€™t think its been implemented in frigate yet, if it is this issue, anyway round it please?

I do have a PCI and also a USB coral plugged in and i have tried setting detectors as coral, PCI and USB
I have also tried removing the hardware config input arguments.

Any suggestions appreciated please?

 * Starting nginx nginx
   ...done.
Fatal Python error: Illegal instruction
Current thread 0x00007f11fdbc6740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1101 in create_module
  File "<frozen importlib._bootstrap>", line 556 in module_from_spec
  File "<frozen importlib._bootstrap>", line 657 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 36 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/opt/frigate/frigate/edgetpu.py", line 15 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/opt/frigate/frigate/app.py", line 19 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 783 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/opt/frigate/frigate/__main__.py", line 7 in <module>
  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

"

This is brilliant and works fantastically, thank you.
Just one query, should Frigate record audio with its clips and recordings, as mine doesnā€™t?