Local realtime person detection for RTSP cameras

Never investigated why my automation fires to often. The one from the blueprint is working perfectly.
Decided not to need a critical notification when someone is at the front door.

Will retry the critical notification when I add my indoor cam.

This thread is nuts itā€™s so popular!

Cannot find anything or track stuff so easy

So I created a discord for anyone who whatā€™s to discuss

Totally unofficial and unconnected!

Iā€™m getting a lot of melting vids, I donā€™t see this happen in my DVR software.

Like this:

image

I receive regular error messages like these in the logs.

[h264 @ 0x55d9ea237d00] error while decoding MB 38 2, bytestream -31
    Last message repeated 2 times
[h264 @ 0x55d9ea237d00] cabac decode of qscale diff failed at 16 5
[h264 @ 0x55d9ea237d00] error while decoding MB 16 5, bytestream 20315
[h264 @ 0x55d9ea237d00] left block unavailable for requested intra4x4 mode -1
[h264 @ 0x55d9ea237d00] error while decoding MB 0 3, bytestream 243949
[h264 @ 0x5644b7580d00] error while decoding MB 22 0, bytestream -32
[h264 @ 0x5644b7580d00] error while decoding MB 22 0, bytestream -29
[h264 @ 0x5644b7580d00] left block unavailable for requested intra mode
[h264 @ 0x5644b7580d00] error while decoding MB 0 2, bytestream 25945
[h264 @ 0x5644b7580d00] top block unavailable for requested intra mode -1
[h264 @ 0x5644b7580d00] error while decoding MB 25 0, bytestream 276116
[h264 @ 0x55e600204d00] error while decoding MB 11 8, bytestream -36
[h264 @ 0x55e600204d00] top block unavailable for requested intra mode -1
[h264 @ 0x55e600204d00] error while decoding MB 23 0, bytestream 39859

Using Coral:

  coral:
    type: edgetpu

Configuration is as follows for this particular camera:

drive:
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED
          roles:
            - clips
            - record
            - detect
    width: 640
    height: 480
    fps: 5
    motion:
      mask:
        - 391,480,640,480,640,422,385,427
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 270
      retain:
        default: 10
    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      retain:
        default: 10

Brand of camera?
Had same issue when using RTSP streams from REOINK camsā€¦ Switched to RTMP and it went away.

1 Like

Is anyone successful in using the USB Coral with Home Assistant OS running in Virtualbox on macOS? I just got mine today and canā€™t get Frigate to find it. It always shows this:

frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.edgetpu                INFO    : No EdgeTPU detected.

Iā€™m running a 2013 Mac Pro and followed the instructions here to set up the Coral but itā€™s still showing in System Information as Composite Device from 0x1a6e (Global Unichip Corp.).

I set up Virtual Box with the USB Filter as well as found this article that talks about setting up a filter for a Google device. I tried that also but it didnā€™t seem to make a difference.

I can tell that Home Assistant sees it by using the SSH add-on and typing lsusb I get Bus 002 Device 003: ID 1a6e:089a and that disappears if I unplug the Coral.

Lastly, the Frigate addon works fine if I switch to a CPU detector instead of the EdgeTPU detector. Iā€™ve tried with Protection Mode enabled and disabled, doesnā€™t seem to make a difference. Anyone have any ideas of other things to check, either macOS or in Home Assistant?

Relevant frigate.yml:

detectors:
  coral:
    type: edgetpu
    device: usb

Logs:

* Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
frigate.app                    WARNING : Camera front_yard_1 has rtmp enabled, but rtmp is not assigned to an input.
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.app                    INFO    : Camera processor started for front_yard_1: 41
frigate.app                    INFO    : Capture process started for front_yard_1: 43
frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 36
frigate.edgetpu                INFO    : Attempting to load TPU as usb
Process detector:coral:
frigate.edgetpu                INFO    : No EdgeTPU detected.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector
    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)
  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__
    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...
frigate.app                    INFO    : Stopping...
frigate.record                 INFO    : Exiting recording maintenance...
frigate.object_processing      INFO    : Exiting object processor...
frigate.events                 INFO    : Exiting event processor...
frigate.events                 INFO    : Exiting event cleanup...
frigate.watchdog               INFO    : Exiting watchdog...
2 Likes

Reolink. How did you switch to RTMP?

Try this url:

rtmp://ip adress/bcs/channel0_main.bcs?channel=0&stream=0&user=usenam&password=password

For substream use channel0_sub

Works in VLC, but Frigate throws a fit:

frigate.video                  INFO    : drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : drive: ffmpeg process is not running. exiting capture thread...
drive:
    ffmpeg:
      inputs:
        - path: rtmp://REDACTED/bcs/channel0_sub.bcs?channel=0&stream=0&user=REDACTED&password=REDACTED
          roles:
            - clips
            - record
            - detect
    width: 640
    height: 480
    fps: 5
    motion:
      mask:
        - 391,480,640,480,640,422,385,427
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 270
      retain:
        default: 10
    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      retain:
        default: 10

Try adding this to ffmpeg

ffmpeg:
....
....
  input_args:
    - '-avoid_negative_ts'
    - make_zero
    - '-fflags'
    - nobuffer
    - '-flags'
    - low_delay
    - '-strict'
    - experimental
    - '-fflags'
    - +genpts+discardcorrupt
    - '-use_wallclock_as_timestamps'
    - '1' 
1 Like

you sir are a saint.

Thanks

Glad to helpā€¦ :slight_smile:

Recently changed to a google coral, but the inference speed is still at 30. Are there any ways to improve it?

Picked one up, itā€™ll be here today. Iā€™ll be attempting to load HassOS+Frigate on it and run it that way for a bit to see how well it works. My last venture with an Atom x5 was not well so curious to see how well this Celeron can handle both. Coral will be later this month after I run it for a bit to evaluate performance. What a great find for $39, thank you!!

@unrly you bought t430 TC for 39$? Where , I will buy few.

This device is awesome, runs 4W and HA Supervised is using ~5% of the CPU.

1 Like

Picked it up on eBay, normal price is $45. Refurbished, so Iā€™ll be curious to see the condition it comes in.

Glad to hear youā€™re still liking it! That is impressive!

Edit: HA up and running on the emmc, works fantastic. Just have Frigate running for now. Iā€™ll restore my VM snapshot soon. Inference speeds are similar to a 6th gen i3 in VirtualBox, about 350ms for 1 cam at 1080p.

2 Likes

if I use Frigate in a docker container, is there a way to retrieve the following image from within hassio? Like a button on Lovelace that will open a pop-up of the image:

http://192.168.x.xxx:5000/api/cam_name/person/best.jpg

Add it as camera

1 Like

Im not sure, but i think nuc wifi adapter is using m.2 E key? Why dont u just remove wifi adapthee and use coral m.2 E? Would it work with rc2 hassos?

Anyone tried chuwi larkbox? It has 128gb emmc with m.2 2242 expansion slot. Im not sure intel UHD in the celeron is good enuff for h254 decoding. If the m.2 expansion does work with coral m.2 in upcoming HassOs it could be a perfect cheap smallbox for frigate and hass. Any idea?

On an ODROID-N2+ with just 4 camera feeds I get 50% cpu usage even when using a Coral USB device.

Is there any capability of hardware accelerated h264 decoding on this platform?