Local realtime person detection for RTSP cameras

mr6880 · December 24, 2020, 2:42am

I’m seeing frequent crashes and strange things in the log now.

This is running 4 cameras @ 1080p 5fps and 1 camera @ 2048*1536 5fps on the latest beta version of Frigate. It runs just fine for around 15 minutes, and then several (but not all) cameras freeze and I see gibberish in the log. The frozen cameras do not recover and require a restart of Frigate.

One thing I noticed is that detection_fps reported by Frigate appears to be locked to the sum of detection_fps of the cameras when they failed, and then increases from that figure if motion is detected on cameras that did not fail. In the example below, no motion is being detected by the working cameras:

{"south":{"camera_fps":5.0,"capture_pid":61,"detection_fps":0.0,"pid":51,"process_fps":5.0,"skipped_fps":0.0},"detection_fps":15.1,"detectors":{"coral":{"detection_start":0.0,"inference_speed":8.09,"pid":45}},"front_gate":{"camera_fps":5.0,"capture_pid":55,"detection_fps":4.8,"pid":48,"process_fps":4.5,"skipped_fps":0.0},"garage_front":{"camera_fps":5.1,"capture_pid":56,"detection_fps":0.0,"pid":49,"process_fps":5.1,"skipped_fps":0.0},"overview":{"camera_fps":5.0,"capture_pid":67,"detection_fps":2.5,"pid":53,"process_fps":4.8,"skipped_fps":0.0},"porch_beta":{"camera_fps":5.1,"capture_pid":63,"detection_fps":7.8,"pid":52,"process_fps":4.7,"skipped_fps":0.0},"rear_garden_overview":{"camera_fps":5.1,"capture_pid":71,"detection_fps":0.0,"pid":54,"process_fps":5.1,"skipped_fps":0.0}}

I’m trying to work out what exactly causes the issue (it’s likely a change I’ve made), but here’s what I’m seeing in the log:

 in   File r"u/nu
s  File r/"li/bu/spry/tlhiobn/3p.8y/tthhorne3a.d8i/ntgh.rpeya"d, line i932n in g_.bpoyo"ts, line t932r in ap__bionontesrt
r  File a"/pu_sirn/nleirb
/p  File y"/thusorn3/.li8b//tphyrtehaodnin3.g8./ptyh"re, line a890d in ing_.bpoyo"tst, line ra890p in 
_
Thread 0xb00007fa31cb7d700o (most recent call first):
o  File ts"t/orpatp/
fr
igThread 0xate00007fa31cb7d700/ (most recent call first):
fr  File ig"at/eo/pvti/dferoi.gpayte"/f, line r201i in grautne
/  File v"i/dueso.rp/yl"ib, line /201p in ytrhuonn
3.  File 8"/t/hursera/dliinbg/.ppyyt"h, line o932n in 3_.b8o/otthsrteraadp_iinngne.rp
y  File "", line /932u in sr/_lbiobo/tpystthroanp3_.i8n/ntehrr
e  File ad"i/ngu.spry/l"i, line b890/ in p_ybtohootns3t.r8a/pt
h
rThread 0xea00007fa31f37e700d (most recent call first):
in  File g.p"y"/o, line p890t in /_frbiogoattster/afpr
i
Thread 0xg00007fa31f37e700a (most recent call first):
t  File e/"l/oogp.tp/fyr"ig, line a67t in e/rufnr
i  File g"a/tues/rlo/gl.ipby/"py, line t67h in onr3u.n
8  File /"t/hursre/aldiibn/gp.yptyh"o, line n9323. in 8_/btoohtrsetardaipn_gin.npeyr"
, line 932  File  in "/_ubosort/sltirba/pp_yitnhnoen3r
.8  File /t"/hurseard/ilnibg/.ppyy"t, line h890o in n_3b.o8o/ttshrteradaipng
.
pThread 0x00007fa34cfab740y (most recent call first):
"  File , line "890/ in u_sbro/oltisbt/rpaypt
ho
nThread 0x3.00007fa34cfab7408 (most recent call first):
/t  File h"r/eusard/ilnigb/.pyptyh"on3, line .10278 in /t_hwraeaidti_nfgo.pr_y"tst, line a1027t in e__wlaiotc_kf
or  File _"t/sutsart/el_ilbo/cpykt
h  File o"n/3u.8sr//tlhireba/dpyitnhong3..8p/yt"h, line r1011e in adjionign.
py  File "", line /1011o in ptj/ofiring
at  File e/"fr/oipgta/tfer/ivgiadtee/of.rpiyg"ate, line /241 in vicadepot.uprye"_c, line a241 in mecraap
tu  File re_"ca/muersar
/li  File b/"p/yutshorn/l3i.b8//pymtuholnt3i.p8r/ocmeuslstiiprnogc/preoscseisnsg/process.py", line 108 in run
  File "/usr/lib/python3..8p/ym"ult, line i108p in rorcuesns
in  File g/"p/ruoscre/slsi.bpy/"p, line y315t in ho_nb3o.o8t/smturalpt
ipro  File ce"s/suisngr//plriob/cpeystsh.opny3".8, line /315m in u_lbtoiportosctersasipn
g  File /po"p/euns_rf/loirbk/.ppyy"tho, line n75 in 3._8la/umnuchl
tip  File r"oc/eusssri/nlgi/bp/oppyetnh_ofonr3k..8p/myu"lt, line i75p in ro_clesasiunngc/hp
op  File en"_/ufosrrk/.lpiyb"/py, line 19t in ho_n_3i.n8it/_m_
ult  File ipr"oc/eusssri/nlgi/pbo/ppeynt_hfono3r.k8./pmyul"t, line i19p in r_oc_esisniintg_/_c
o  File nte"xt/.upsyr"/li, line b277/ in py_tPhoopne3n.
8/m  File ul"ti/pursorc/elsisbi/npgy/tchoonnt3ex.t8./mpuyl", line t277i in pro_cPoepsesnin
g/  File c"o/nutserx/tl.ipby"/py, line t224h in on_3P.o8p/emnu
lt  File i"p/roucsers/sliinbg/p/yctohnotne3x.t8.p/ym"ul, line ti224p in r_oPcoepsesni
ng  File /"p/rusocre/ssl.ipby/"py, line t121h in on3s.ta8r/tm
ul  File ti"p/roopcte/sfsriignagt/ep/rforciegsast.ep/ya"p, line p121. in psyta"rt, line 
161 in   File st"a/rotp_tc/farmiegraat_ec/afprtiugratee_/parppoc.epys"s, line e161s
 in s  File ta"r/to_ptc/afmreirgaa_tcea/pftruirgea_tper/oacpeps.psy"es, line 
202  File  in "s/toaprtt/
fr  File iga"t/eo/fptr/ifgraiteg/aatpep/.fpryi"ga, line t202e in /_s_tmaairnt_
_  File .p"y/"opt, line /15fr in i<gamtoed/ulfer>i
gat  File e"//_u_msra/inl_i_b./ppyyt"ho, line n153 in .<8m/orduunlpey>.
p  File y""/, line 87u in sr_/lruinb_/cpoytdheo
n3  File .8"//ruusnrp/y.plyib"/p, line yt87h in o_nr3un._8c/roudnep
y.  File py""/, line u194s in r/_lirbun/_pmyotdhuolne3_.a8s/_rmuanipny
.py", line 194 in _run_module_as_main
Fatal Python error: Bus error
Thread 0x00007fa33b43c700 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 302 in wait
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed
  File "/usr/lib/python3.8/threading.py", line 870 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap
Current thread 0x00007fa31a37c700 (most recent call first):
  File "/opt/frigate/frigate/video.py", line 115 in capture_frames
  File "/opt/frigate/frigate/video.py", line 227 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007fa31cb7d700 (most recent call first):
  File "/opt/frigate/frigate/video.py", line 201 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007fa31f37e700 (most recent call first):
  File "/opt/frigate/frigate/log.py", line 67 in run
  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap
Thread 0x00007fa34cfab740 (most recent call first):
  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock
  File "/usr/lib/python3.8/threading.py", line 1011 in join
  File "/opt/frigate/frigate/video.py", line 241 in capture_camera
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start
  File "/opt/frigate/frigate/app.py", line 161 in start_camera_capture_processes
  File "/opt/frigate/frigate/app.py", line 202 in start
  File "/opt/frigate/frigate/__main__.py", line 15 in <module>
  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

blakeblackshear · December 24, 2020, 2:56am

What is your shm size? That “Bus error” typically means you ran out of shm storage.

blakeblackshear · December 24, 2020, 3:50am

That’s possible. It will require a rework of the retention settings so that the retention in the database is independent from the clips. I may need to do that for a few other reasons anyway.

S1eepy · December 24, 2020, 4:13pm

I am facing some FFMPEG issues with beta3. Not tried other betas.
Strange because I am not seeing video interruption here: http://ip:5001/cam_name?motion=1&regions=1
No clips gets recorded, but detection seems to work since I am getting MQTT snapshots.

FFMPEG config:

ffmpeg:
  global_args: -hide_banner -loglevel fatal
  hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
  input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
  output_args:
    detect: -f rawvideo -pix_fmt yuv420p
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    rtmp: -c copy -f flv

Here are the logs:

2020-12-24T16:09:59.323328757Z  * Starting nginx nginx
2020-12-24T16:09:59.334874776Z    ...done.
2020-12-24T16:10:00.043096597Z frigate.app                    INFO    : Creating tmpfs of size 256m
2020-12-24T16:10:00.074442441Z frigate.app                    INFO    : Camera processor started for garageptz: 39
2020-12-24T16:10:00.077589468Z frigate.app                    INFO    : Camera processor started for japonais: 41
2020-12-24T16:10:00.083251467Z frigate.app                    INFO    : Camera processor started for z5: 43
2020-12-24T16:10:00.087452269Z frigate.app                    INFO    : Camera processor started for garage: 44
2020-12-24T16:10:00.091042005Z frigate.app                    INFO    : Capture process started for garageptz: 45
2020-12-24T16:10:00.094602348Z frigate.app                    INFO    : Capture process started for japonais: 47
2020-12-24T16:10:00.104497553Z frigate.app                    INFO    : Capture process started for z5: 50
2020-12-24T16:10:00.113517320Z frigate.app                    INFO    : Capture process started for garage: 54
2020-12-24T16:10:02.736627369Z detector.coral                 INFO    : Starting detection process: 36
2020-12-24T16:10:02.740790957Z frigate.edgetpu                INFO    : Attempting to load TPU as usb
2020-12-24T16:10:02.740855152Z frigate.edgetpu                INFO    : TPU found
2020-12-24T16:11:43.977883480Z [mov,mp4,m4a,3gp,3g2,mj2 @ 0x564f7b1b2140] moov atom not found
2020-12-24T16:11:43.977912564Z /tmp/cache/z5-20201224171141.mp4: Invalid data found when processing input
2020-12-24T16:11:43.979577155Z frigate.events                 INFO    : bad file: z5-20201224171141.mp4
2020-12-24T16:11:43.992332925Z [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55e27330c140] moov atom not found
2020-12-24T16:11:43.992386315Z /tmp/cache/garage-20201224171140.mp4: Invalid data found when processing input
2020-12-24T16:11:43.993949254Z frigate.events                 INFO    : bad file: garage-20201224171140.mp4
2020-12-24T16:11:44.005839792Z [mov,mp4,m4a,3gp,3g2,mj2 @ 0x56029c874140] moov atom not found
2020-12-24T16:11:44.005859006Z /tmp/cache/garageptz-20201224171140.mp4: Invalid data found when processing input
2020-12-24T16:11:44.007505598Z frigate.events                 INFO    : bad file: garageptz-20201224171140.mp4
2020-12-24T16:11:44.021574122Z [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55eaeac00140] moov atom not found
2020-12-24T16:11:44.021591957Z /tmp/cache/japonais-20201224171140.mp4: Invalid data found when processing input
2020-12-24T16:11:44.023125343Z frigate.events                 INFO    : bad file: japonais-20201224171140.mp4
2020-12-24T16:11:50.240806347Z watchdog.z5                    INFO    : Terminating the existing ffmpeg process...
2020-12-24T16:11:50.240996310Z watchdog.z5                    INFO    : Waiting for ffmpeg to exit gracefully...
2020-12-24T16:11:50.252682412Z watchdog.garage                INFO    : Terminating the existing ffmpeg process...
2020-12-24T16:11:50.256916837Z watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2020-12-24T16:11:50.274037187Z watchdog.garageptz             INFO    : Terminating the existing ffmpeg process...
2020-12-24T16:11:50.277545350Z watchdog.garageptz             INFO    : Waiting for ffmpeg to exit gracefully...
2020-12-24T16:11:50.280690520Z watchdog.japonais              INFO    : Terminating the existing ffmpeg process...
2020-12-24T16:11:50.281038992Z watchdog.japonais              INFO    : Waiting for ffmpeg to exit gracefully...

blakeblackshear · December 24, 2020, 5:29pm

Try setting the ffmpeg log level to info in the global args for one camera to see what the ffmpeg messages are.

rogerquake · December 24, 2020, 7:59pm

YJB posted an example that makes it so the automation will only execute once every X seconds. The example below only allows the specified automation to run every 30 seconds. You just need to modify the automation name.

- '{{ ( as_timestamp(now()) - as_timestamp(state_attr(''automation.AutomationName'', ''last_triggered'')) |int(0) ) > 30 }}'

Another issue I was experiencing was false positive notifications due to my dock pilings being detected as humans. This is most likely due to the movement of water behind them. I didn’t want to add a mask there, so I created multiple zones to filter out the false positives by setting the min_area value higher. The logic here is that the value being reported was too low to be a human based on my testing.

However, I quickly found out that although the person’s were being filtered out in the zone, this didn’t stop a notification from being sent out. The default examples send out notifications based on the filter values set on the camera level. So to only get notifications if a person enters a zone, we need to add the zone names to the notification conditions.

I’m assuming there is a cleaner way to match any value in the zones array returned by the Events HTTP endpoint, but I’m not sure how to accomplish this. Since the default with conditions is “and” for the statements, I needed to do a mix of “and” & “or” to get the outcome I wanted:

condition:
  - condition: and
    conditions:
    - '{{ trigger.payload_json[''after''][''label''] == ''person'' }}'
    - '{{ ( as_timestamp(now()) - as_timestamp(state_attr(''automation.when_a_person_enters_yard'', ''last_triggered'')) |int(0) ) > 30 }}'
    - condition: or
      conditions:
         # Zone 1
      - '{{ ''backsouth-backdoor-table'' in trigger.payload_json[''after''][''entered_zones'']
        }}'
         # Zone 2
      - '{{ ''backsouth-pole-first'' in trigger.payload_json[''after''][''entered_zones'']
        }}'
         # A different camera Camera with no zones 
      - '{{ trigger.payload_json[''after''][''camera''] == ''back-north'' }}'

biggen1684 · December 24, 2020, 8:04pm

I’m also getting some errors now on beta3 in the console that I don’t recall getting on earlier betas:

Traceback (most recent call last):
frigate    | During handling of the above exception, another exception occurred:                   
frigate    |
frigate    |   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner               
frigate    |     self.run()
frigate    |   File "/opt/frigate/frigate/record.py", line 122, in run                             
frigate    |     self.move_files()
frigate    |   File "/opt/frigate/frigate/record.py", line 48, in move_files                       
frigate    |     if process.name() != 'ffmpeg':
frigate    |   File "/usr/local/lib/python3.8/dist-packages/psutil/__init__.py", line 634, in name 
frigate    |     name = self._proc.name()
frigate    |   File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1515, in wrapper
frigate    |     return fun(self, *args, **kwargs)                                                 
frigate    |   File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1610, in name
frigate    |     name = self._parse_stat_file()['name']                                            
frigate    |   File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1522, in wrapper
frigate    |     raise NoSuchProcess(self.pid, self._name)                                         
frigate    | psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=14254)

Not sure if these are ffmpeg problems or something else.

blakeblackshear · December 24, 2020, 8:20pm

Here is the condition I use to only notify for objects that have entered a zone:

- "{{ trigger.payload_json['after']['entered_zones']|length > 0 }}"

blakeblackshear · December 24, 2020, 8:21pm

You can also throttle the automation with the new run_mode features built into homeassistant: https://www.home-assistant.io/docs/automation/modes/#example-throttled-automation

lawrence-jeff · December 24, 2020, 8:43pm

Great stuff Blake
QQ - Running the latest Beta and the app itself seems to be working as I get clips that properly identify objects and the HTTP test requests all seem to return the correct info. However the home assistant HACS integration does not seem to be working. The stats update correctly and the live video works but the person and car show as Unavailable… They have worked once or twice but then generally stuck with an old image. I have reinstalled the integration, renamed my camera with no luck… I believe this is coming through MQTT so have checked the mosquito logs and they show frigate connecting… no errors on the frigate side either. I set the frigate mqtt log to debug but that doesn’t actually seem to work- just get an info connected event. Any ideas?

blakeblackshear · December 24, 2020, 8:58pm

Since those sensors are mqtt based, they would only be unavailable if something is wrong with frigates mqtt connection. I would start there and use mqtt explorer to see what is being passed around.

lawrence-jeff · December 24, 2020, 9:05pm

That is the direction I was heading. I am using MQTT spy to subscribe to frigate# and am not seeing much come through. I don’t see any issues on the broker side, is there a way to improve logging in frigate related to mqtt? I have

logger:
  logs:
    frigate.mqtt: debug

But this doesn’t seem to log any mqtt related items in the frigate log

blakeblackshear · December 24, 2020, 9:56pm

It would error out if it was unable to connect

lawrence-jeff · December 24, 2020, 10:01pm

Yes -it connects and I can see that in the log of both frigate and mosquito, but I am not seeing anything sent when it creates a clip relevant to a detection event (from mqtt spy)… so if there is anyway I can see confirmation in the frigate log it would help. I do see occasional socket errors on the mosquito side

1608791618: Socket error on client frigate, disconnecting.

Frigate seems to connect back soon after - and doesn’t log anything on the frigate side so I don’t think these are relevant - but maybe

Jafterdark · December 24, 2020, 11:51pm

How do I Access the videos I’m running this in a doctor container and I mapped the folders and there is video files in the folders except for I can’t play them unless I move them out of that folder? Thanks for any help
video https://drive.google.com/file/d/1MxnicOq_64dwBk5dvPBi2eiV7YsxGgyd/view?usp=sharing

Jafterdark · December 25, 2020, 7:10am

flip your resolution that’s what fixed

calypso · December 25, 2020, 2:29pm

So I managed to get away from the Christmas festivities, and I see that my unraid server needs some work. I also noticed that Frigate is now in the app store which is nice. I’ve updated my config file to match the new style, previously I was on version 6 I believe. I just keep getting this issue in logs and I’m not sure why.

Fontconfig error: Cannot load default config file

On connect called
Traceback (most recent call last):
File "detect_objects.py", line 441, in <module>
main()
File "detect_objects.py", line 202, in main
ffmpeg_output_args = ["-r", str(config.get('fps'))] + ffmpeg_output_args
TypeError: can only concatenate list (not "NoneType") to list

The config file definitely is there.

Sometimes I also get this one, but the container still stops at this point.

On connect called
Traceback (most recent call last):
File "detect_objects.py", line 441, in <module>
main()
File "detect_objects.py", line 202, in main
Starting detection process: 23
ffmpeg_output_args = ["-r", str(config.get('fps'))] + ffmpeg_output_args
TypeError: can only concatenate list (not "NoneType") to list
Attempting to load TPU as USB

Any ideas? Thanks

blakeblackshear · December 25, 2020, 3:00pm

Remove all the optional ffmpeg args if you aren’t going to set or customize them. You also may want to look at the 0.8.0 beta as the config is changing again.

S1eepy · December 25, 2020, 4:41pm

thanks. I had an issue with the cache size, the clips could not be saved.

I have not fixed it yet. I’m not an expert in docker-compose, but this part of the example yaml file seems to be the issue for me:

      - type: tmpfs # 1GB of memory, reduces SSD/SD Card wear
        target: /cache
        tmpfs:
          size: 100000000

I see only 8 zeroes: “size: 100 000 000”, means 100MB or 1GB?
Also, looking at the container in portainer, seems the /cache needs to be present on the docker server?
Should I use ram on my docker server for this? (and change /cache with something more appropriate)

thx

blakeblackshear · December 25, 2020, 4:52pm

That volume mount is optional. You can have it write the cache files for clips to tmpfs, but you will need a lot of ram if you have several cameras. I would try omitting that volume entirely at first. Once it’s working, you can look at adding it back.