Local realtime person detection for RTSP cameras

Hello,
is anybody using M.2 version of Coral TPU in NUC? Is NUC able to recognize it in this slot in Linux?
If yes, which version? For B+M key slot or A/E key slot, or both are compatible?
Thanks.

2 Likes

I use Hikvision and my url is:

rtsp://user:[email protected]:554/ch1/sub/av_stream

Have you tested the url in VLC?

Ordered a few from these people, and they arrived within a day or so. Iā€™m in the UK. https://www.okdo.com/p/coral-mini-pcie-m-2-accelerator-b-m-key/

2 Likes

I would start by commenting out the hwaccel_args: and all of the lines below it before inputs and see if that makes a difference.

As others have mentioned, try removing the hardware acceleration. Also confirm what your cameraā€™s codec settings are set to. E.g. are you trying to send a H265 stream or H264, an Intel CPU from circa 2012 is unlikely to support hardware decode of H265. Disabling hardware acceleration in your Frigate config will confirm at least if thatā€™s the issue.

For reference my Hikvision cameras work without issue using the following config excerpt, which is on an Intel i5-4300U and the DS-2CD2342WD-I camera.

  laundrydoor:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@x.x.x.x/Streaming/Channels/102
          roles:
            - detect
            #- rtmp
    width: 640
    height: 360

ffmpeg:
     hwaccel_args:
          - -hwaccel
          - vaapi
          - -hwaccel_device
          - /dev/dri/renderD128
          - -hwaccel_output_format
          - yuv420p

Wikipedia have a good table that shows what is supported by each generation of CPU/Quicksync

I have Hikvision, mine working fine. Be aware that you need to change codec to H264 without the + and look out for the right resolution
CAM1:

ffmpeg:
  inputs:
    - path: rtsp:/xxx:[email protected]/Streaming/channels/101/?transportmode=unicast
      roles:
        - detect
        - clips

Yeah, they dont ship to my country. I need to wait when i will be in another :smiley: thx

Availability is 16. 7. 2021, but atleast the ship to my country. Thank you very much

And be sure to set rtsp authentication to digest/basic.

Fock, wil I ever get this stable?

I have a coral but to no avialā€¦ I cant see (dont know where to look and never seen a message no tpu detected fallling back to cpu) it does not use coral usbā€¦

I tried:

detectors:
  coral:
    type: edgetpu
    device: 'usb3:2'
detectors:
  coral:
    type: edgetpu
    device: usb
detectors:
  coral:
    type: edgetpu
    device: 'usb0'

This is the device in the ubuntu os:
root@frigate:/etc/frigate/config# lsusb

Bus 003 Device 002: ID 18d1:9302 Google Inc.
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

How can I be sure this coral is used?

I get this errors:

root@frigate:/etc/frigate/config# docker attach 8facad07ebb2
frigate.http                   DEBUG   : Received mqtt message on frigate200/stats.
frigate.events                 WARNING : More than 90% of the cache is used.
frigate.events                 WARNING : Consider increasing space available at /tmp/cache or reducing max_seconds in your clips config.
frigate.events                 WARNING : Proactively cleaning up the cache...
frigate.events                 WARNING : More than 90% of the cache is used.
frigate.events                 WARNING : Consider increasing space available at /tmp/cache or reducing max_seconds in your clips config.
frigate.events                 WARNING : Proactively cleaning up the cache...
frigate.video                  INFO    : schuurachter: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : schuurachter: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : voordeur: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : voordeur: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : terras: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : terras: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : parkeerplaats: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : parkeerplaats: ffmpeg process is not running. exiting capture thread...
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55d6447a0140] moov atom not found
/tmp/cache/parkeerplaats-20210505135854.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: parkeerplaats-20210505135854.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55bfd1aa6140] moov atom not found
/tmp/cache/terras-20210505135854.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: terras-20210505135854.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55fd53f23140] moov atom not found
/tmp/cache/voordeur-20210505135853.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: voordeur-20210505135853.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55569f3dd140] moov atom not found
/tmp/cache/schuurachter-20210505135853.mp4: Invalid data found when processing input
frigate.events                 INFO    : bad file: schuurachter-20210505135853.mp4
frigate.events                 WARNING : More than 90% of the cache is used.
frigate.events                 WARNING : Consider increasing space available at /tmp/cache or reducing max_seconds in your clips config.
frigate.events                 WARNING : Proactively cleaning up the cache...
ffmpeg.schuurachter.detect     ERROR   : [swscaler @ 0x55ab9896b540] deprecated pixel format used, make sure you did set range correctly
ffmpeg.schuurachter.detect     ERROR   : av_interleaved_write_frame(): No space left on device
ffmpeg.schuurachter.detect     ERROR   : [segment @ 0x55ab987e6640] Failure occurred when ending segment '/tmp/cache/schuurachter-20210505135853.mp4'
ffmpeg.schuurachter.detect     ERROR   : Error writing trailer of /tmp/cache/schuurachter-%Y%m%d%H%M%S.mp4: No space left on device
ffmpeg.parkeerplaats.detect    ERROR   : [swscaler @ 0x55d160396b80] deprecated pixel format used, make sure you did set range correctly
ffmpeg.parkeerplaats.detect    ERROR   : av_interleaved_write_frame(): No space left on device
ffmpeg.parkeerplaats.detect    ERROR   : [segment @ 0x55d1601f2c40] Failure occurred when ending segment '/tmp/cache/parkeerplaats-20210505135854.mp4'
ffmpeg.parkeerplaats.detect    ERROR   : Error writing trailer of /tmp/cache/parkeerplaats-%Y%m%d%H%M%S.mp4: No space left on device
ffmpeg.terras.detect           ERROR   : [swscaler @ 0x5576e4db6f80] deprecated pixel format used, make sure you did set range correctly
ffmpeg.terras.detect           ERROR   : av_interleaved_write_frame(): No space left on device
ffmpeg.terras.detect           ERROR   : [segment @ 0x5576e4162000] Failure occurred when ending segment '/tmp/cache/terras-20210505135854.mp4'
ffmpeg.terras.detect           ERROR   : Error writing trailer of /tmp/cache/terras-%Y%m%d%H%M%S.mp4: No space left on device
ffmpeg.voordeur.detect         ERROR   : [segment @ 0x55db40a8cc40] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
ffmpeg.voordeur.detect         ERROR   : [segment @ 0x55db40a8cc40] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
ffmpeg.voordeur.detect         ERROR   : [swscaler @ 0x55db409b8340] deprecated pixel format used, make sure you did set range correctly
ffmpeg.voordeur.detect         ERROR   : av_interleaved_write_frame(): No space left on device
ffmpeg.voordeur.detect         ERROR   : [segment @ 0x55db40a8cc40] Failure occurred when ending segment '/tmp/cache/voordeur-20210505135853.mp4'
ffmpeg.voordeur.detect         ERROR   : Error writing trailer of /tmp/cache/voordeur-%Y%m%d%H%M%S.mp4: No space left on device

how to solve?

@karldonteljames
I actually added the hardware acceleration when I couldnā€™t get the display working the first time
Iā€™ve removed the h/w acceleration lines and itā€™s still the same issue
Hereā€™s the log btw, I forgot to add it on my previous post

frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...
ffmpeg.backyard.detect         ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.backyard.rtmp           ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.backyard              INFO    : Terminating the existing ffmpeg process...
watchdog.backyard              INFO    : Waiting for ffmpeg to exit gracefully...
ffmpeg.back_lawn.detect        ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
ffmpeg.back_lawn.rtmp          ERROR   : rtsp://user:[email protected]:554/Streaming/channels/102/: Invalid data found when processing input
watchdog.back_lawn             INFO    : Terminating the existing ffmpeg process...
watchdog.back_lawn             INFO    : Waiting for ffmpeg to exit gracefully...
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : back_lawn: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : back_lawn: ffmpeg process is not running. exiting capture thread...

On my hikvision camera settings, itā€™s on basic.

For me it was this url and working fine on streaming on vlc

@ukro

Mineā€™s already set to H264, I never did use H265 or the + codec on the camera settings. Using the subchannel since I donā€™t really wanna put too much strain on the cpu yet.

@NoFace
Iā€™ve got a similar question, just that my mac mini (running on debian) doesnā€™t have any nvme slots
Would something like nvme to sata adaptor would work? Just throwing it out there
Thereā€™s still no coral usb stock at my place but plenty of those m.2 coral boards

I have portainer and when looking at container log in portainer UI I will get ā€œTPU foundā€ and ā€œattempting to load TPUā€ message when working and TPU error message when not.

Your logs show many Cache error messages so this may be bigger issue for you. You should maybe look at increasing cache size.

How would I need to do that?

Ok, got the TPU messageā€¦ although my NUC CPU goes wild (fan) after restart of container and during the time it ā€œdoesā€ work (before crashing).

frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.edgetpu                INFO    : TPU found

@sender

reduce input image size and that may resolve your remaining issues.
after try reduce size you can look at increase memory if issues remain

Sorry, answered in the other threadā€¦

Where did you stick the Coral inside of the Argon case? I wonā€™t get my Coral for a few weeks but would be interested in how you snuck it in there and how you cabled the USB.

Jason

Thank You! when i originally set up home assistant on a raspberry pie i tested frigate as a home assistant custom component and was able to use all the sensor info. if i do it as a separate docker container in unraid will i still be able to get the frigate sensors and controls in home assistant? and if i want to add it as a ha addon and not as a separate docker will it be able to use the mini pcie coral?