Local realtime person detection for RTSP cameras

Yes, still does seem to have ay affect :slight_smile:
I’ll continue my fault finding

Hi,

so I tried lowering the “max_disappeared” setting with no visible change. Also the lowering the “best_image” did not produce more snapshots.

Could you post your node-red code for me please? Have not worked with node red before, this is new land for me.
Thank you very much.

Cheers Chris

Do you know where I can get a copy of the firmware? The original firmware it was shipped with only gave me the option for h265 too, which is one of the reasons I swapped.

How would I go about transcoding the stream to include the frigate detection boxes? I have tried this:

ffmpeg:
  global_args: 
    -hide_banner 
    -loglevel warning
  input_args: 
    -avoid_negative_ts make_zero 
    -fflags +genpts+discardcorrupt 
    -rtsp_transport tcp 
    -stimeout 5000000
  output_args:
    clips: 
      -f segment 
      -segment_time 300 
      -segment_format mp4 
      -reset_timestamps 1 
      -strftime 1 
      -c copy 
      -an

but that give me the error:

Unable to create clip for 04_Side_Gate and event 1619622504.318758-5yi399. There were no cache files for this event.

I can’t find the information to tell me what each of the arguments do, or what the options are to change them too. I’ve looked here. Configuration | Frigate (blakeblackshear.github.io)

Also my camera is quite high up, and some objects can be quite small, is there a way of increasing person detection when they are smaller in the frame? I have the following at the moment:

          min_area: 0
          min_score: 0.2
          threshold: 0.3

Increase the ‘-detect’ resolution stream worked for me in a similar situation, I ended up using 3072x2048 (max resolution mode from hikvision fisheye). This way, a person can easily fill 300x300px.

Thanks. I’ll wait for my coral adaptor to arrive tomorrow before doing that.
I’m trying to reencode the stream, but i’m getting these errors:

ffmpeg.06_Rear_Garden.clips ERROR : Guessed Channel Layout for Input Stream #0.1 : mono
ffmpeg.07_West_Fence.clips ERROR : [rtsp @ 0x5630f1244840] max delay reached. need to consume packet
ffmpeg.06_Rear_Garden.clips ERROR : [hevc @ 0x5584da18b700] Could not find ref with POC 0
ffmpeg.07_West_Fence.clips ERROR : [rtsp @ 0x5630f1244840] RTP: missed -63704 packets
ffmpeg.07_West_Fence.clips ERROR : Guessed Channel Layout for Input Stream #0.1 : mono
ffmpeg.07_West_Fence.clips ERROR : [hevc @ 0x5630f12cec40] Could not find ref with POC 4
ffmpeg.07_West_Fence.clips ERROR : [rtsp @ 0x5630f1244840] max delay reached. need to consume packet
ffmpeg.07_West_Fence.clips ERROR : [rtsp @ 0x5630f1244840] RTP: missed 1864 packets

my current ffmpeg looks like this:

ffmpeg:
  global_args: 
    -hide_banner 
    -loglevel warning
  input_args: 
    - -avoid_negative_ts
    - make_zero
    - -fflags
    - nobuffer
#    - -rtsp_transport tcp
#    - -stimeout 5000000
  output_args:
    clips: 
      -f segment
      -segment_time 300
      -segment_format mp4
      -reset_timestamps 1
      -strftime 1
      -c:v libx264
      -an

first, awesome work!

Everything works with only CPU processing, but I have a problem with the USB Coral Stick.
I’m not sure if the Coral stick is damaged or if it’s some other kind of error.
I am using Proxmox as host system running HassOS.

lsusb on the host looks like this:



however, it is not completely displayed to me, is this correct?

in the log, it doesn’t look very good, anyway:

* Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.app                    INFO    : Camera processor started for back: 49
frigate.app                    INFO    : Capture process started for back: 50
detector.coral                 INFO    : Starting detection process: 46
frigate.mqtt                   INFO    : MQTT connected
Process detector:coral:
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.edgetpu                INFO    : No EdgeTPU detected.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector
    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)
  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__
    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...
frigate.app                    INFO    : Stopping...
frigate.events                 INFO    : Exiting event processor...
frigate.record                 INFO    : Exiting recording maintenance...
frigate.object_processing      INFO    : Exiting object processor...
frigate.watchdog               INFO    : Exiting watchdog...
frigate.events                 INFO    : Exiting event cleanup...
frigate.stats                  INFO    : Exiting watchdog...
peewee.sqliteq                 INFO    : writer received shutdown request, exiting.
root                           INFO    : Waiting for detection process to exit gracefully...

the config file I have changed to that:

detectors:
  coral:
    type: edgetpu
    device: usb
mqtt:
........

is there anything I could check? or is it possibly the host machine?

thanks a lot.

Haven’t tried it myself, but I want to if the PCIe corals ever come back in stock. I also run proxmox and HassOS and from what I’ve read in the past you need to install the drivers on the proxmox host. I never did understand if the passthrough to HassOS worked without any other modifications though.

I can’t find it on the fly, only in use with the PCIE card.

do you know where you read that?

does not make much sense for me… i mean the deconz USB stick you can also simply pass through.

@sender - While I’m not on LXC, I am running Frigate on Unraid/Docker and HA in a VM with Frigate addon. I am thinking though your problem lies somewhere within the Frigate deployment and i can share my working configuration config.yml file if you like to compare. I started from scratch and now have pretty much every option turned on and working. I also use 2 cpu option (no Coral or any such device). Let me know.

Yes please share, I can use all help I can get :+1:

Hi @basejob,

and what is the ram size of the pi?

Thanks!

I guess the first thing to do is access your rtsp streams directly in VLC and make sure you have continuous stream with correct frames from the cameras giving you grief just to rule out any network bottlenecks etc.

mqtt:
  host: 192.168.200.132
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  topic_prefix: frigate
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate
  # Optional: user
  user: mqtt-user
  # Optional: password
  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
  password: mqtt-user
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60
cameras:
  backyard:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.200.50:554/s1
          roles:
            - detect
            - clips
        - path: rtsp://192.168.200.50:554/s2  
          roles:         
            - rtmp
        - path: rtsp://192.168.200.50:554/s0
          roles:
            - record
      output_args:     
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy      
    width: 640
    height: 360
    fps: 5
    motion:
      mask: 320,106,640,209,640,0,191,0,0,0,0,65
      threshold: 10
      contour_area: 50
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
    record:
      enabled: true
      retain_days: 15
    objects:
      track:    
       - person
       - dog
      filters:
       dog:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.6
       person:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.7
  Deck:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.200.51:554/s1
          roles:
            - detect
            - clips
        - path: rtsp://192.168.200.51:554/s2  
          roles:         
            - rtmp
        - path: rtsp://192.168.200.51:554/s0
          roles:
            - record
      output_args:     
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    width: 640
    height: 360
    fps: 5
    motion:
      mask: 368,51,640,205,640,0,0,0,0,62
      threshold: 10
      contour_area: 50
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
    record:
      enabled: true
      retain_days: 15
    objects:
      track:    
       - person
       - dog
      filters:
       dog:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.6
       person:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.7
  Side:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.200.52:554/s1
          roles:
            - detect
            - clips
        - path: rtsp://192.168.200.52:554/s2  
          roles: 
            - rtmp
        - path: rtsp://192.168.200.52:554/s0
          roles:
            - record
      output_args:     
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy      
    width: 640
    height: 360
    fps: 5
    motion:
      mask: 399,40,640,0,378,0,257,0,0,0,0,117
      threshold: 10
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
    record:
      enabled: true
      retain_days: 15
    objects:
      track:    
       - person
       - dog
       - car
       - bus
      filters:
       dog:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.6
       person:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.7
  Frontyard:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.200.53:554/s1
          roles:
            - detect
            - clips
        - path: rtsp://192.168.200.53:554/s2  
          roles: 
            - rtmp
        - path: rtsp://192.168.200.53:554/s0
          roles:  
            - record
      output_args:     
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy      
    width: 640
    height: 360
    fps: 5
    motion:
      mask:
       - 640,0,640,70,0,57,0,0,0,0,0,0
       - 610,161,604,201,640,227,640,138
       - 410,360,468,304,519,276,569,311,610,360
       - 52,287,104,255,173,282,195,318,218,360,24,360
      threshold: 10
      contour_area: 50
    zones:
      yard:
        coordinates: 0,360,640,360,640,91,640,91,0,79
      road:
        coordinates: 0,73,640,85,640,0,0,0
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
    record:
      enabled: true
      retain_days: 15
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      crop: false
    objects:
      track:    
       - person
       - dog
       - car
       - bus
      filters:
       dog:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.6
       person:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.7
  Driveway:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.200.54:554/s1
          roles:
            - detect
            - clips
        - path: rtsp://192.168.200.54:554/s2  
          roles:
            - rtmp
        - path: rtsp://192.168.200.54:554/s0
          roles:
            - record
      output_args:     
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    width: 640
    height: 360
    fps: 5
    motion:
      mask: 640,78,640,0,0,0,0,76
      threshold: 10
      contour_area: 50
    zones:
      drive:
        coordinates: 640,267,470,209,275,162,108,157,0,156,0,360,387,360,640,360
      road:
        coordinates: 0,145,375,153,389,73,0,72
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
    record:
      enabled: true
      retain_days: 15
    snapshots:
      enabled: true
    mqtt:
      enabled: true
      crop: false
    objects:
      track:
       - person
       - dog
       - car
       - bus
      filters:
       dog:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.6
       person:
        min_area: 5000
        max_area: 100000
        min_score: 0.5
        threshold: 0.7
detectors:
  cpu1:
   type: cpu
  cpu2:
   type: cpu
 
1 Like

Did you ever get this to deploy on portainer? Can’t seem to find the right settings.

Frigate:
How do I change mode to CPU? Seems to be trying to find TPU.
I’m running it in docker desktop.
sneighbour/frigate


Starting detection process: 36


    ffmpeg_input = get_ffmpeg_input(ffmpeg['input'])


Attempting to load TPU as usb


KeyError: 'input'

I did… but not using it anymore.
Sorry but i dont have the setup anymore. So I cant be of much help.

Have 2 cams running in the homeassistant addon successfully but want to offload them to an ubuntu vm (or lxc).

Yeah if this not the config.yaml (check mine against your setup) and those cameras work fine with the addon directly in HA, then possibly something to do with your virtual NICs / drivers on your LXC perhaps or allocation of memory/ vCpu to the VM/container. I would start troubleshooting there first -transfer a large file over the network to that container (10-15GB) and see if you get a sustained 100/MB per sec.

Needed the detectors:

detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu    

But now plenty of other errors. entry endpoint for api is missing for a start

I suspect I have the problem noted in the documentation ‘I have clips and snapshots in my clips folder, but I can’t view them in the Web UI.’ but struggling to get it working.

Running NUC/Proxmox/HA OS/Frigate NVR (through supervisor). Setup went well, Frigate is capturing snapshots and clips without issue, I can view/play them through SMB but going through Media Browser causes issues:

Works:
Media Browser>Local Media>Frigate>Clips - I can view all the files, all be it without thumbnails for some reasons. I can then launch the clips without issue. I noticed it’s being played through: https://mydomain.com:8123/media/local/frigate/clips/front_door-1619691568.383451-smnacg.mp4?authSig=(long auth string)

Doesn’t work:
Media Brower> Frigate>Clips - Again I can see all the files, this time with thumbnails! However when launching the file I get 503 error. This time the URL is: https://mydomain.comt:8123/api/frigate/clips/front_door-1619749886.286841-4tpuco.mp4?authSig=(long auth string)

If accessing the media through the Frigate UI also get 503.

Guessing I need to mount the folder as per the FAQ, but struggling to understand exactly what is needed: 'try mounting a volume to /media/frigate inside the container instead of /media/frigate/clips

The current and I’m guessing default mount within the Frigate container is: /mnt/data/supervisor/media > /media

Any suggestions what I have done wrong? Thx.

So if you have h264 i have no idea why you have this problem with the permission.
You have 2 options with my tutorial
1:
If you have a NAS you can try to mount media to network share
2:
Go to the host machine and check if there is a permission error, if it is , just do as in tutorial but don’t change fstab, but change the /mnt/data/supervisor/media permission with chmod +x
Tutorial:
https://community.home-assistant.io/t/solved-hassos-mount-nas-network-share/

I’m pretty sure you cant change anything in frigate addon as it recreates it self each time.I spend to much time ti figuring it out.