Local realtime person detection for RTSP cameras

Question :slight_smile:
It is almost impossible to get a google coral usb accelerator due to the ww chip shortage.
However I see that some suppliers have the coral dev board mini.
Is it possible to use?
I have a HA blue setup and want to use it in a usb config setup.
Anybody instructions/experience?

Thanks and happy new year

hschroyen

Well, I do not think we have the same issue.
I have the frigate camera entities and they actually works fine on my PC, android phone and Tab A7 tablet (via fully Kiosk).
However the Frigate live view via the custom:frigate-card does not work in my Tab A7 tablet (although it works fine on my PC and android mobile phone) . I assume it might have something to do with the custom:frigate-card in combination with Fully Kiosk and/or Tab A7?

Hi Folks. I wanted to try frigate with a coral tpu - but it was very hard to find one - the only one available is an rather exotic M.2 Type E chip - so i searched and searched ā€¦ and now i have this sandwich card running on my proxmox with an lxc container running a frigate instance:

This is an PCIe to mini-pcie adapter having a mini-pci to m.2 e typ adapter on top - which holds the TPU. Currently i only have access to one of the tpu chips - this is the only limitation of the pcie adapter.
But with my ā€œtinyā€ amount of cameras its more then enough for me.

2 Likes

I had a similar issue in my NUC where I had to use an adapter to use the same TPU with the same single lane limitation. I also have a USB coral on the same box which runs at 9ms infrencing speed vs 18ms of the m.2 card.

Good job finding a workable solution!

I am just getting around to upgrading my system from 2021.10.1 to the most recent version, and to do so, I need to upgrade Frigate. I updated the breaking change for the detect portion, but do I need to add in the detectors section, or will it default to my coral?

Edit: I am also getting an error when trying to switch over from clips to record. I donā€™t see an input section in the sample config
WARNING : Camera back_porch has record enabled, but record is not assigned to an input.

So this might be more of a HA general question, but maybe Frigate can do something.
Is there a way to store the clips/recordings to an external NAS?

Yes there isā€¦ But I suspect that you have to run Frigate on an external docker.
Im running Frigate on a small nuc ubuntu machine. By doing this way I have ful control over Frigate and take the load of from HA.
This is how my docker-compose file locks likeā€¦

version: "3.6"
services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: blakeblackshear/frigate:0.10.0-beta6-amd64

    shm_size: '4GB'
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${PWD}/config.yml:/config/config.yml:ro
      - /mnt/syn_frigate/media:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 2000000000

    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds

My media (Synology) folder is mounted to my ubuntu machine.
And all the clips and database is saved thereā€¦

Hi anyone has this type problem with Double Take and Deepstack does not connect to double take

DeepStack: Version 2021.09.01
/v1/vision/face
---------------------------------------
/v1/vision/face/recognize
---------------------------------------
/v1/vision/face/register
---------------------------------------
/v1/vision/face/match
---------------------------------------
/v1/vision/face/list
---------------------------------------
/v1/vision/face/delete
---------------------------------------
/v1/vision/detection
---------------------------------------
---------------------------------------
v1/backup
---------------------------------------
v1/restore
[GIN] 2022/01/06 - 20:59:14 | 200 |    5.949594ms |     172.30.32.1 | POST     "/v1/vision/face/delete"
[GIN] 2022/01/06 - 20:59:22 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 20:59:53 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:00:14 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:00:29 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:00:44 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:00:59 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:01:14 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:01:29 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:01:44 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/register"
[GIN] 2022/01/06 - 21:02:18 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:02:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:03:19 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:03:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:04:19 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:04:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:05:19 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:05:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:06:19 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:06:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:07:19 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"
[GIN] 2022/01/06 - 21:07:49 | 500 |          1m0s |     172.30.32.1 | POST     "/v1/vision/face/recognize"

Hi All,
Finally I have working setup. it crosses all my expectations ! In fact I have now shutdown my synology survilance station and only using Frigage. Thanks a ton for such a funtastic work.
I have Synology DS716+II along with Coral USB TPU. It is getting stream from Reolink RLC520 (I want to expand it to 4) Currently used docker image is blakeblackshear/frigate:0.10.0-beta6-amd64
Now what I notice is, Frigate docker consumes almost 20% CPU (no issues for me). But is it correct ? Does someone else is also having simmilar setup and can post the configuruation ? Or can some one verify my configuration for any possible improvements. Here is my Camera config. And here is my Docker Compose

Are you on DSM 7?
If so, your Synology probably isnā€™t using the Coral USB.
They disabled using USB except for storage and UPS.

If the Coral is detected you should see the following in the logs

2022-01-07T02:37:18.461033182Z [2022-01-06 21:37:18] frigate.edgetpu                INFO    : Attempting to load TPU as usb
2022-01-07T02:37:21.172793717Z [2022-01-06 21:37:21] frigate.edgetpu  

@surge919

Yes. I am using Synology with DSM 7. And I am sure it is recognizing the coral USB (I have loaded the drivers manually). Else, container wont start as I am passing it as a device to container.
Also I could also see this in the log

> 2022-01-06 20:12:47] frigate.app                    INFO    : Starting Frigate (0.10.0-db1255a)
> [2022-01-06 20:12:48] frigate.config                 WARNING : The 'retain_days' config option has been DEPRECATED and will be removed in a future version. Please use the 'days' setting under 'retain'
> [2022-01-06 20:12:48] frigate.config                 WARNING : The 'retain_days' config option has been DEPRECATED and will be removed in a future version. Please use the 'days' setting under 'retain'
> Starting migrations
> [2022-01-06 20:12:48] peewee_migrate                 INFO    : Starting migrations
> There is nothing to migrate
> [2022-01-06 20:12:49] peewee_migrate                 INFO    : There is nothing to migrate
> [2022-01-06 20:12:50] frigate.mqtt                   INFO    : MQTT connected
> [2022-01-06 20:12:50] frigate.app                    INFO    : Output process started: 219
> [2022-01-06 20:12:50] frigate.app                    INFO    : Camera processor started for Front: 221
> [2022-01-06 20:12:50] frigate.app                    INFO    : Camera processor started for Back: 223
> [2022-01-06 20:12:50] ws4py                          INFO    : Using epoll
> [2022-01-06 20:12:50] frigate.app                    INFO    : Capture process started for Front: 226
> [2022-01-06 20:12:50] frigate.app                    INFO    : Capture process started for Back: 228
> [2022-01-06 20:12:50] ws4py                          INFO    : Using epoll
> [2022-01-06 20:12:50] detector.coral                 INFO    : Starting detection process: 218
> [2022-01-06 20:12:50] frigate.edgetpu                INFO    : Attempting to load TPU as usb
> [2022-01-06 20:13:12] frigate.edgetpu                INFO    : TPU found

Amazing project!

My friend is wondering which is preferable, HA OS on bare metal and frigate as an add-on, or a machine running proxmox with HA OS in a VM, and frigate in its own LXC?

He originally anticipated using this NUC for more applications, but now is going to use it for ā€œjustā€ HA and frigate.

I also have the following entry

-v /dev/bus/usb:/dev/bus/usb

(It this what you mean by ā€˜loading the drivers manuallyā€™ or did you do an extra step?)

but itā€™s not detecting my device.

2022-01-07T16:55:34.720659381Z [2022-01-07 11:55:34] frigate.edgetpu                ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.

@surge919

No, this is passing already connected and detected USB device to container.

Create a separate ā€œ.shā€ file. E.g. startup.sh. Add below lines into it as per your USB Devices. And then schedule newly created script to execute at every startup. This shall load the USB drivers for different USB devices.

# Below 2 lines shall load USB serial derivers to kernal. This shall help to load the correct drivers if non block devices (harddisk) are connected on USB. Ex. Z wave stick.
#sudo insmod /lib/modules/usbserial.ko
#sudo insmod /lib/modules/ftdi_sio.ko


# below lines shall be added on DSM 7 to load conbee II and Opto Meter probes
/sbin/modprobe usbserial
/sbin/modprobe ftdi_sio
/sbin/modprobe cdc-acm

Hope this helps.

I have been running Frigate on a docker container for a while now with success. I recently decided to install HAOS on my bare metal and run the add-on. I canā€™t seem to get HA to see the USB accelerator. I see some people were having this problem before, anybody know what the issue is? Do you need to manually install the packages for the accelerator to work?

detectors:
  coral:
    type: edgetpu
    device: usb:0

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_cuvid

mqtt:
 ...

cameras:
  gate:
    ffmpeg:
      inputs:
        - path: rtsp://XXX:[email protected]:554/Streaming/Channels/1/
          roles:
            - detect
    detect:
      width: 3840
      height: 2160
      fps: 12

Iā€™ve been seeing frequent (every few second) crashes from cameras that have been working flawlessly for ages now, something like this:


[2022-01-10 14:43:34] frigate.app INFO : Starting Frigate (0.9.4-26ae608)
Starting migrations
[2022-01-10 14:43:34] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-01-10 14:43:34] peewee_migrate INFO : There is nothing to migrate
[2022-01-10 14:43:34] frigate.mqtt INFO : MQTT connected
[2022-01-10 14:43:34] detector.coral INFO : Starting detection process: 225
[2022-01-10 14:43:34] frigate.app INFO : Starting Frigate (0.9.4-26ae608)
Starting migrations
[2022-01-10 14:43:34] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-01-10 14:43:34] peewee_migrate INFO : There is nothing to migrate
[2022-01-10 14:43:34] frigate.mqtt INFO : MQTT connected
[2022-01-10 14:43:34] detector.coral INFO : Starting detection process: 225
[2022-01-10 14:43:34] frigate.app INFO : Output process started: 226
[2022-01-10 14:43:34] ws4py INFO : Using epoll
[2022-01-10 14:43:34] frigate.app INFO : Camera processor started for gate: 232
[2022-01-10 14:43:34] frigate.edgetpu INFO : Attempting to load TPU as usb:0
[2022-01-10 14:43:34] frigate.app INFO : Output process started: 226
[2022-01-10 14:43:34] ws4py INFO : Using epoll
[2022-01-10 14:43:34] frigate.app INFO : Camera processor started for gate: 232
[2022-01-10 14:43:34] frigate.edgetpu INFO : Attempting to load TPU as usb:0
[2022-01-10 14:43:34] frigate.app INFO : Capture process started for gate: 242
[2022-01-10 14:43:36] frigate.edgetpu INFO : TPU found
[2022-01-10 14:43:34] ws4py INFO : Using epoll
[2022-01-10 14:43:37] frigate.video INFO : gate: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-10 14:43:37] frigate.video INFO : gate: ffmpeg process is not running. exiting capture thread...
[2022-01-10 14:43:38] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59200]
[2022-01-10 14:43:38] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59200]
[2022-01-10 14:43:54] watchdog.gate ERROR : FFMPEG process crashed unexpectedly for gate.

[2022-01-10 14:43:54] watchdog.gate ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-10 14:43:54] watchdog.gate ERROR : You may have invalid args defined for this camera.

[2022-01-10 14:43:54] ffmpeg.gate.detect ERROR : [rtsp @ 0x561c0a423980] DTS discontinuity in stream 1: packet 6 with DTS 147764323295526, packet 7 with DTS 147764323314068

[2022-01-10 14:43:54] ffmpeg.gate.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-10 14:43:54] ffmpeg.gate.detect ERROR : [flv @ 0x561c0a42b700] Failed to update header with correct duration.

[2022-01-10 14:43:54] ffmpeg.gate.detect ERROR : [flv @ 0x561c0a42b700] Failed to update header with correct filesize.

[2022-01-10 14:43:54] ffmpeg.gate.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/gate: Connection reset by peer

[2022-01-10 14:43:54] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59200]
[2022-01-10 14:43:55] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58152]
[2022-01-10 14:43:55] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59236]
[2022-01-10 14:43:57] frigate.video INFO : gate: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-10 14:43:57] frigate.video INFO : gate: ffmpeg process is not running. exiting capture thread...
[2022-01-10 14:43:57] frigate.video INFO : gate: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-10 14:43:57] frigate.video INFO : gate: ffmpeg process is not running. exiting capture thread...
[2022-01-10 14:43:57] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:59236]
[2022-01-10 14:44:04] watchdog.gate ERROR : FFMPEG process crashed unexpectedly for gate.

[2022-01-10 14:44:04] watchdog.gate ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-10 14:44:04] watchdog.gate ERROR : You may have invalid args defined for this camera.

[2022-01-10 14:44:04] ffmpeg.gate.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-10 14:44:04] ffmpeg.gate.detect ERROR : [flv @ 0x559386bbd700] Failed to update header with correct duration.

[2022-01-10 14:44:04] ffmpeg.gate.detect ERROR : [flv @ 0x559386bbd700] Failed to update header with correct filesize.

[2022-01-10 14:44:04] ffmpeg.gate.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/gate: Connection reset by peer

[2022-01-10 14:44:07] frigate.video INFO : gate: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-10 14:44:07] frigate.video INFO : gate: ffmpeg process is not running. exiting capture thread...
[2022-01-10 14:44:14] watchdog.gate ERROR : FFMPEG process crashed unexpectedly for gate.

[2022-01-10 14:44:14] watchdog.gate ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-10 14:44:14] watchdog.gate ERROR : You may have invalid args defined for this camera.

[2022-01-10 14:44:14] ffmpeg.gate.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-10 14:44:14] ffmpeg.gate.detect ERROR : [flv @ 0x55c082f33700] Failed to update header with correct duration.

[2022-01-10 14:44:14] ffmpeg.gate.detect ERROR : [flv @ 0x55c082f33700] Failed to update header with correct filesize.

[2022-01-10 14:44:14] ffmpeg.gate.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/gate: Connection reset by peer

[2022-01-10 14:44:16] frigate.video INFO : gate: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-10 14:44:16] frigate.video INFO : gate: ffmpeg process is not running. exiting capture thread...

Iā€™ve stripped the config back to barebones (as above, with just one camera), but Iā€™m still seeing frequent crashes. The camera feed config is correct (resolution and FPS), so unsure whatā€™s wrong.

If you just updated to 0.9.4, then this is probably because rtmp is being enabled by default. High resolutions like this often cause rtmp to fail.

Thatā€™s a very high resolution and frame rate. Unless you are tracking very small fast objects, you will be better off with a lower resolution and frame rate.

Hi,

I started to experience issues with the camera stream, I added the same camera with 2 different streams to test different resolutions; but both of them are failing,
Any ideas/suggestions?

frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : [h264 @ 0x55a8bad040] decode_slice_header error
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : [h264 @ 0x55a8bad040] no frame!
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : [h264 @ 0x55a8bad040] non-existing PPS 0 referenced
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   :     Last message repeated 1 times
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : [h264 @ 0x55a8bad040] decode_slice_header error
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : [h264 @ 0x55a8bad040] no frame!
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : Guessed Channel Layout for Input Stream #0.1 : mono
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera1.record          ERROR   : 
frigate_camera_tpu | [2022-01-12 08:22:00] watchdog.camera1               INFO    : Terminating the existing ffmpeg process...
frigate_camera_tpu | [2022-01-12 08:22:00] watchdog.camera1               INFO    : Waiting for ffmpeg to exit gracefully...
frigate_camera_tpu | [2022-01-12 08:22:01] frigate.video                  INFO    : camera2: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate_camera_tpu | [2022-01-12 08:22:01] frigate.video                  INFO    : camera2: ffmpeg process is not running. exiting capture thread...
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera2.detect          ERROR   : [h264 @ 0x555f7ef180] decode_slice_header error
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera2.detect          ERROR   : [h264 @ 0x555f7ef180] no frame!
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera2.detect          ERROR   : Guessed Channel Layout for Input Stream #0.1 : mono
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera2.detect          ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
frigate_camera_tpu | [2022-01-12 08:22:00] ffmpeg.camera2.detect          ERROR   : 

regards

Good people of this thread, who didnā€™t vote on this Feature Request, please vote on regards of the mount of NAS to HA.
If we will have big vote numbers, we will be seen. Thank you for your time.

@Morphy What NUC are you using?

Also, in general, is there a good way to identify what mini PCs would have a free M.2 pcie slot that would work with Coral. Iā€™ve been searching for days and am struggling to find a solution.

Thanks!