Frigate - cpu detector running on on Proxmox can't get it stable - please help

Share your Projects!
14

In this case, try increase memory available for /dev/shm in VM or at least check current size

run “df” in vm command line should provide current size.
I have (6) cam with 60mb and doesnt even blink but when using larger input image it freeze
I assuming larger size for /dev/shm will help based on “recommended” from frigate github install guide and the errors you receive that point to this. You should aslo see current use % so this may provide insight of if this is causing error or not. Error says “warning 90%” so I guess maybe it just crash or frigate stop at 100

15

Thank you @tmjpugh this is very constructive help and feedback!

I have now changed the VM from 2 cores/vcpu
to:
image

and also:
tmpfs_cache_size: 512m

I have 3 cams on “low res” and 1 on high res and it seems to run more quiet/smooth now. not sure about that /dev/shm size but this is the output:

ubuntu@frigate:~$ df
Filesystem                        1K-blocks     Used Available Use% Mounted on
tmpfs                               1197448     1188   1196260   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  20511312 12408068   7038284  64% /
tmpfs                               5987228        0   5987228   0% /dev/shm
tmpfs                                  5120        0      5120   0% /run/lock
tmpfs                                  4096        0      4096   0% /sys/fs/cgro                                                              up
/dev/sda2                            999320   218392    712116  24% /boot
overlay                            20511312 12408068   7038284  64% /var/lib/doc                                                              ker/overlay2/c27c113c473c78c3d8cdd0fd669787ff79935dd2586b481713db659e427f7c65/me                                                              rged
overlay                            20511312 12408068   7038284  64% /var/lib/doc                                                              ker/overlay2/6966c1a657864b0f349ddd4508ac8cd414877f35a21c88baa5a87e31fcd6bd9d/me                                                              rged
tmpfs                               1197444        4   1197440   1% /run/user/10                                                              00
16

Any chance you know how to get rid of these:

frigate.app                    WARNING : Camera voordeur has clips assigned to an input, but clips is not enabled.
frigate.app                    WARNING : Camera schuurachter has clips assigned to an input, but clips is not enabled.
frigate.app                    WARNING : Camera parkeerplaats has clips assigned to an input, but clips is not enabled.
frigate.app                    WARNING : Camera terras has clips assigned to an input, but clips is not enabled.

I see that when I restart frigate and go with another browser/session to the interface the “clips” icon is not turned on. I can turn it on by hand then… how can I “keep it on”?

17

maybe only needed to increase cores. Did that remove memory warning?
I not sure about /dev/shm as well but that look more than sufficient

set clips >> enabled: True in your config to permanently enable ON

18

Tried that but when I do that frigate wont start and says:

 * Starting nginx nginx


   ...done.


Error parsing config: extra keys not allowed @ data['clips']['enabled']


 * Starting nginx nginx


   ...done.


Error parsing config: extra keys not allowed @ data['clips']['enabled']

this is config:

clips:
  enabled: True
19

Not sure if that did it… but for sure it is now more “easy”. But I still wonder… if I have a Coral, why is cpu so high?

20

ffmpeg still need cpu
it is used to retrieve, decode, encode and create videos/clips/images
TPU is only for detection

Part of my config below
use this for example of how clips should be. Must set clips config for each camera and enable clips as role for an available image source


################################
#           MQTT               #
################################
mqtt:
  host: 172.17.0.3
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: frigate
  password: frigate
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60


################################
#         DETECTORS            #
################################
detectors:
  coral_pci:
    type: edgetpu
    device: pci

################################
#          CAMERAS             #
################################
cameras:
#Each role can only be assigned to one input per camera. The options for roles are as follows:
###detect	Main feed for object detection
###clips	Clips of events from objects detected in the detect feed. docs
###record	Saves 60 second segments of the video feed. docs
###rtmp	Broadcast as an RTMP feed for other services to consume. doc


  #######################################
  #              GATE-02                #
  #######################################
  Gate-02:
    ffmpeg:
      inputs:
        - path: rtsp://10.88.20.92:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp
            - clips
            #- record
#    width: 2592
#    height: 1944
    width: 704 
    height: 480
    fps: 10
    # Optional: timeout for highest scoring image before allowing it
    # to be replaced by a newer image. (default: shown below)
    best_image_timeout: 60
    record:
      # Optional: Enable recording (default: global setting)
      enabled: False
      # Optional: Number of days to retain (default: global setting)
      retain_days: 30

    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - animal
        - car
        - motorcycle
        - person
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          car: 20
          animal: 3
          person: 20
          motorcycle: 10
21

Oh I see, the clips is intended and need to be set per camera?

no, that doe not work for me…


 * Starting nginx nginx


   ...done.


Error parsing config: extra keys not allowed @ data['cameras']['terras']['clips']['max_seconds']


 * Starting nginx nginx


   ...done.


Error parsing config: extra keys not allowed @ data['cameras']['terras']['clips']['max_seconds']

I am really not sure how to get that in my config file…

22

Bingo!!

Frigate is amazing but the config file is like Homeassistant
I really want to know if this can be split like homeassistants. I only be using frigate about 8 hrs and have shutdown but motioneye and doods in favor of this

23

the error says you have set something wrong in the config for the “terras” camera for “clips” “max_seconds” value. Check this part of config and make sure no extra spaces or incorrect spacing

extra keys not allowed @ data['cameras']['terras']['clips']['max_seconds']

post current config if you cant find error

EDIT
max_seconds was not in my config example and I not find it in documentation
maybe it not belong in config

24

Think I got ‘that’ part now running… all cams are now saving clips.

although yesterday it was running more smooth up until this morning and then it froze again… I could not see ‘any’ sensible thing in log so I restarted the container.

will update on this. thanks so far! very much appreciated.

EDIT: Update

It is hopeless… the reason for having a more stable situation yesterday seems to be the fact there was less motion due to wind. This morning ther eis more motion and it keeps crashing…

part of the log I was able to capture:

  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap


  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch


  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__


  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen


  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen


  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start


  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes


  File "/opt/frigate/frigate/app.py", line 228 in start


  File "/opt/frigate/frigate/__main__.py", line 15 in <module>


  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code


  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main


Fatal Python error: Bus error

More info:


root                           INFO    : Detection process didnt exit. Force killing...


detector.coral                 INFO    : Starting detection process: 3528


frigate.edgetpu                INFO    : Attempting to load TPU as usb


frigate.edgetpu                INFO    : TPU found


frigate.http                   DEBUG   : Received mqtt message on frigate200/stats.
25

Bus error is likely caused by memory issue according to docs

Try “df” at command line of vm again and see if results change from yesterday

Also based on, it’s ok with little motion but crash when a lot of motion, it sound like resource/memory issue.

26

You don’t use addon correct?
Remove this from config

tmpfs_cache_size: 512m

Also, you have FPS set between 4-6 depending on camera. Is this set the same in camera? Just curious, not sure if this matter.

27

This is my final config with still the “when a lot of motion it crashes”

#logger:
  # Optional: default log level (default: shown below)
  #default: debug
ffmpeg:
  # Optional: global ffmpeg args (default: shown below)
  global_args: -hide_banner -loglevel warning
  # Optional: global hwaccel args (default: shown below)
  # NOTE: See hardware acceleration docs for your specific device
  hwaccel_args: []
  # Optional: global input args (default: shown below)
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
  # Optional: global output args
  output_args:
    # Optional: output args for detect streams (default: shown below)
    detect: -f rawvideo -pix_fmt yuv420p
    # Optional: output args for record streams (default: shown below)
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    # Optional: output args for clips streams (default: shown below)
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an

# MQTT Config ##########################################################################################################
mqtt:
  host: 192.168.1.142
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  topic_prefix: frigate200
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate200
  # Optional: user
  user: mqtt
  # Optional: password
  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
  password: mqtt
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60

# Cameras Config ######################################################################################################
cameras:
# FRONT Camera ########################################################################################################
  voordeur:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/102/
          roles:
            - detect
            - clips
    width: 704
    height: 576
    fps: 6
    snapshots:
      enabled: true
      crop: false
    objects:
      track:
        - person
        - dog
        - cat
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
        - dog
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 200
          dog: 20
          cat: 200

# SCHUUR Camera ########################################################################################################
  schuurachter:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/102/
          roles:
            - detect
            - clips
    width: 704
    height: 576
    fps: 6
    snapshots:
      enabled: true
      crop: false
    objects:
      track:
        - person
        - car
        - dog
        - bicycle
        - cat
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
        - car
        - dog
        - bicycle
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 200
          car: 200
          dog: 20
          cat: 200
          bicycle: 200

# PARKING Camera ########################################################################################################
  parkeerplaats:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/102/
          roles:
            - detect
            - clips
    width: 640
    height: 480
    fps: 6
    snapshots:
      enabled: true
      crop: false
    objects:
      track:
        - person
        - car
        - dog
        - bicycle
    motion:
      mask: 402,30,461,47,455,170,640,230,640,0,474,0,56,0,58,45,207,57,206,34,233,32,229,88,326,77,354,54,379,38
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
        - car
        - dog
        - bicycle
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 200
          car: 200
          dog: 20
          cat: 200
          bicycle: 200

# TERRAS Camera ########################################################################################################
  terras:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/101/
          roles:
            - detect
            - clips
    width: 3840
    height: 2160
    fps: 4
    snapshots:
      enabled: true
      crop: false
    objects:
      track:
        - bird
        - dog
        - cat
    motion:
      mask: 3840,0,3840,1290,3193,1383,3069,769,3001,378,2723,0
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 5
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 5
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - bird
        - dog
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          bird: 200
          dog: 3
          cat: 200
detectors:
  coral:
    type: edgetpu
    device: usb
28 
ubuntu@frigate:~$ df
Filesystem                        1K-blocks     Used Available Use% Mounted on
tmpfs                               1197448     1216   1196232   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  20511312 13540276   5906076  70% /
tmpfs                               5987228        0   5987228   0% /dev/shm
tmpfs                                  5120        0      5120   0% /run/lock
tmpfs                                  4096        0      4096   0% /sys/fs/cgro                                                                                               up
/dev/sda2                            999320   218392    712116  24% /boot
overlay                            20511312 13540276   5906076  70% /var/lib/doc                                                                                               ker/overlay2/6966c1a657864b0f349ddd4508ac8cd414877f35a21c88baa5a87e31fcd6bd9d/me                                                                                               rged
tmpfs                               1197444        4   1197440   1% /run/user/10                                                                                               00
overlay                            20511312 13540276   5906076  70% /var/lib/doc                                                                                               ker/overlay2/c27c113c473c78c3d8cdd0fd669787ff79935dd2586b481713db659e427f7c65/me                                                                                               rged
29

Yes, it matches the camera

30

Help it keeps crashing…

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdea3fff700 (most recent call first):,
  File "/opt/frigate/frigate/log.py", line 71 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdecc247740 (most recent call first):,
  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock,
  File "/usr/lib/python3.8/threading.py", line 1011 in join,
  File "/opt/frigate/frigate/video.py", line 245 in capture_camera,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start,
  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes,
  File "/opt/frigate/frigate/app.py", line 228 in start,
  File "/opt/frigate/frigate/__main__.py", line 15 in <module>,
  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code,
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main,
Fatal Python error: Bus error,
,
Thread 0x00007fdeb2be0700 (most recent call first):,
  File "/usr/lib/python3.8/threading.py", line 302 in wait,
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed,
  File "/usr/lib/python3.8/threading.py", line 870 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Current thread 0x00007fdeb23d6700 (most recent call first):,
  File "/opt/frigate/frigate/video.py", line 117 in capture_frames,
  File "/opt/frigate/frigate/video.py", line 231 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdeb1b8d700 (most recent call first):,
  File "/opt/frigate/frigate/video.py", line 205 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdea3fff700 (most recent call first):,
  File "/opt/frigate/frigate/log.py", line 71 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdecc247740 (most recent call first):,
  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock,
  File "/usr/lib/python3.8/threading.py", line 1011 in join,
  File "/opt/frigate/frigate/video.py", line 245 in capture_camera,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start,
  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes,
  File "/opt/frigate/frigate/app.py", line 228 in start,
  File "/opt/frigate/frigate/__main__.py", line 15 in <module>,
  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code,
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main,
Fatal Python error: Bus error,
,
Thread 0x00007fdeb1b8d700 (most recent call first):,
  File "/usr/lib/python3.8/threading.py", line 302 in wait,
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed,
  File "/usr/lib/python3.8/threading.py", line 870 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Thread 0x00007fdea3fff700 (most recent call first):,
  File "/usr/lib/python3.8/threading.py", line 302 in wait,
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed,
  File "/usr/lib/python3.8/threading.py", line 870 in run,
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner,
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap,
,
Current thread 0x00007fdecc247740 (most recent call first):,
  File "/opt/frigate/frigate/edgetpu.py", line 214 in detect,
  File "/opt/frigate/frigate/video.py", line 295 in detect,
  File "/opt/frigate/frigate/video.py", line 380 in process_frames,
  File "/opt/frigate/frigate/video.py", line 273 in track_camera,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch,
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen,
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start,
  File "/opt/frigate/frigate/app.py", line 173 in start_camera_processors,
  File "/opt/frigate/frigate/app.py", line 227 in start,
  File "/opt/frigate/frigate/__main__.py", line 15 in <module>,
  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code,
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main,
detector.coral                 INFO    : Starting detection process: 361,
frigate.edgetpu                INFO    : Attempting to load TPU as usb,
frigate.edgetpu                INFO    : TPU found,
31

I think problem lies in VM setup but I not use VM so cannot provide suggestion there. I do have ideas below however

I think your error cause changed but you need to confirm. Before you getting memory warning then crash but now Tstate_lock then restart just before crash. I think original memory issue may be resolved but now you have new issue where frigate not finish some process but try start new process then crash as result. You could try reduce all frame rate to 1 to see if this help just as confirmation step.

Also, do you need VM?
VM setup always seem difficult to me where docker is simpler solution. Unless windows/Linux mixed needs.

32

Good tip, where to do?

I have 1 NUC with proxmox and run some workloads on it. I want to use the rest of capacity for frigate. So yes I need to use VM. Although I can also try the LXC route but I have had totally no succes with that yet… (tried a lot).

33

for exampe, change fps from 6 to 1.
do this for all camera
this is not solution, only step to help identify cause.

cameras:
  voordeur:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/102/
          roles:
            - detect
            - clips
    width: 704
    height: 576
    fps: 6

EDIT
Proxmox has some VM dashboard that provide summary of individual VM?
What does that show. Low storage, high memory use, cpu high use, anything stand out?