Local realtime person detection for RTSP cameras

blakeblackshear · December 6, 2020, 11:18pm

Can you post your full config and the output of the new /config endpoint?

Trident29 · December 6, 2020, 11:21pm

Wow. I just want to say you did an amazing job! Installed the new beta pretty easily and quickly. One question. I installed the HA custom component and it easily found everything and imported them into HA. Is there supposed to be a media browser? If so, how do I access it? I assumed a new component or something would pop up in HA but I don’t see any way to view the clips.

blakeblackshear · December 6, 2020, 11:35pm

Are you using a recent version of homeassistant? I thought the media browser was enabled by default in the menu. See the left side of the screenshot on the new Readme.

Trident29 · December 7, 2020, 12:03am

I am but I had to enable media browser by adding ‘media_source:’ in my config file. Thanks!

rayzorben · December 7, 2020, 12:47am

config

detectors:
  coral:
    type: edgetpu
    device: usb

mqtt:
  host: 192.168.0.84
  topic_prefix: motion
  client_id: home
  user: mosquitto
  password: '{FRIGATE_MQTT_PASSWORD}'

save_clips:
  max_seconds: 300
  retain:
    default: 60

objects:
  track:
    - person
  filters:
    person:
      min_area: 5000
      max_area: 100000
      min_score: 0.5
      threshold: 0.80

record:
  enabled: False
  retain_days: 30

cameras:
  porch:
    width: 2592
    height: 1520
    ffmpeg:
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.0.20:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - clips

        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.0.20:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp

output

{
cameras: {
porch: {
best_image_timeout: 60,
ffmpeg_cmds: [
{
cmd: "ffmpeg -hide_banner -loglevel fatal -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=0",
roles: [
"clips"
]
},
{
cmd: "ffmpeg -hide_banner -loglevel fatal -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=1 -c copy -f flv rtmp://127.0.0.1/live/porch -f rawvideo -pix_fmt yuv420p pipe:",
roles: [
"detect",
"rtmp"
]
}
],
fps: null,
frame_shape: [
1520,
2592
],
height: 1520,
name: "porch",
objects: {
filters: {
person: {
max_area: 100000,
min_area: 5000,
min_score: 0.5,
threshold: 0.8
}
},
track: [
"person"
]
},
record: {
enabled: false,
retain_days: 30
},
rtmp: {
enabled: true
},
save_clips: {
enabled: false,
objects: [
"person"
],
pre_capture: 30,
retain: {
default: 10,
objects: { }
}
},
snapshots: {
crop_to_region: true,
draw_bounding_boxes: true,
draw_zones: false,
height: 175,
show_timestamp: true
},
width: 2592,
zones: { }
}
},
detectors: {
coral: {
device: "usb",
type: "edgetpu"
}
},
logger: {
default: "INFO",
logs: { }
},
mqtt: {
client_id: "home",
host: "192.168.0.84",
port: 1883,
topic_prefix: "motion",
user: "mosquitto"
},
save_clips: {
max_seconds: 300,
retain: {
default: 60,
objects: { }
}
}
}

rayzorben · December 7, 2020, 12:48am

I did just notice that width/height applies to the clips, not the detect/rtmp, I assume I should change that to what the detect/rtmp feed is.

rayzorben · December 7, 2020, 12:49am

I tested with the detect/rtmp width/height changed,still same issue.

blakeblackshear · December 7, 2020, 1:08am

I think this is because you have clips assigned as a role to an input, but you don’t have save_clips enabled in the camera settings. Note in your JSON, save_clips is not enabled. Try enabling save_clips in your camera settings. I need to add config validation for that.

rayzorben · December 7, 2020, 1:15am

Thanks, that fixed that error, but still not running.

Just to clarify, what should width and height be set to? The width and height of the clips substream or the width and the height of the detector substream?

* Starting nginx nginx
...done.
frigate.mqtt INFO : MQTT connected
frigate.app INFO : Camera processor started for porch: 31
frigate.app INFO : Capture process started for porch: 32
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
detector.coral INFO : Starting detection process: 28
frigate.edgetpu INFO : Attempting to load TPU as usb
frigate.edgetpu INFO : TPU found
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...

blakeblackshear · December 7, 2020, 1:25am

width and height of the detector stream

rayzorben · December 7, 2020, 1:25am

OK so i’ve got that set correctly then

blakeblackshear · December 7, 2020, 1:29am

Did you have fps set for your cameras before? Try setting that. It should be the fps of the stream with the detect role.

rayzorben · December 7, 2020, 1:31am

I set it, got another error that ‘moov’ atom not found, but not sure if that is related or not

* Starting nginx nginx
...done.
frigate.mqtt INFO : MQTT connected
frigate.app INFO : Camera processor started for porch: 31
frigate.app INFO : Capture process started for porch: 32
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
detector.coral INFO : Starting detection process: 28
frigate.edgetpu INFO : Attempting to load TPU as usb
frigate.edgetpu INFO : TPU found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5639eeb6e140] moov atom not found
/tmp/cache/porch-20201206173031.mp4: Invalid data found when processing input
frigate.events INFO : bad file: porch-20201206173031.mp4
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.

blakeblackshear · December 7, 2020, 1:35am

That’s just cleaning up a broken cache file. No problems there.

Try setting your log level for the detect stream to info.

        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.0.20:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp
          global_args: -hide_banner -loglevel info

[/quote]

yonny24 · December 7, 2020, 2:23am

Hi guys,

blakeblackshear has been helping me on FB so I’d thought I’d add it here now.

Juts getting these errors while trying to start the docker container now:


Traceback (most recent call last):
  File "detect_objects.py", line 457, in <module>
    main()
  File "detect_objects.py", line 206, in main
    ffmpeg_output_args = ["-r", str(config.get('fps'))] + ffmpeg_output_args
TypeError: can only concatenate list (not "NoneType") to list

blakeblackshear · December 7, 2020, 2:48am

That happens when you leave optional args in the config. If you aren’t using the optional args, remove them from your config.

blakeblackshear · December 7, 2020, 2:51am

@mories just pushed up the beta addon in the existing repo. i think you should be able to reload.

rayzorben · December 7, 2020, 2:52am

ErrorWarningSystemArrayLogin


* Starting nginx nginx
...done.
frigate.app INFO : Creating directory: /media/frigate/recordings
frigate.app INFO : Creating directory: /tmp/cache
frigate.mqtt INFO : MQTT connected
frigate.app INFO : Camera processor started for porch: 33
frigate.app INFO : Capture process started for porch: 34
ffmpeg.porch.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono

ffmpeg.porch.detect ERROR : Input #0, rtsp, from 'rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=1':

ffmpeg.porch.detect ERROR : Metadata:

ffmpeg.porch.detect ERROR : title : Media Server

ffmpeg.porch.detect ERROR : Duration: N/A, start: 1607309440.752203, bitrate: N/A

ffmpeg.porch.detect ERROR : Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 704x576, 90k tbr, 90k tbn, 180k tbc

ffmpeg.porch.detect ERROR : Stream #0:1: Audio: pcm_alaw, 64000 Hz, mono, s16, 512 kb/s

ffmpeg.porch.detect ERROR : [flv @ 0x55fb87977c80] FLV does not support sample rate 64000, choose from (44100, 22050, 11025)

ffmpeg.porch.detect ERROR : [flv @ 0x55fb87977c80] Audio codec pcm_alaw not compatible with flv

ffmpeg.porch.detect ERROR : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented

ffmpeg.porch.detect ERROR : Stream mapping:

ffmpeg.porch.detect ERROR : Stream #0:0 -> #0:0 (copy)

ffmpeg.porch.detect ERROR : Stream #0:1 -> #0:1 (copy)

ffmpeg.porch.detect ERROR : Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))

ffmpeg.porch.detect ERROR : Last message repeated 1 times

frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
detector.coral INFO : Starting detection process: 31
frigate.edgetpu INFO : Attempting to load TPU as usb
frigate.edgetpu INFO : TPU found
ffmpeg.porch.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono

ffmpeg.porch.detect ERROR : Input #0, rtsp, from 'rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=1':

ffmpeg.porch.detect ERROR : Metadata:

ffmpeg.porch.detect ERROR : title : Media Server

ffmpeg.porch.detect ERROR : Duration: N/A, start: 1607309450.644203, bitrate: N/A

ffmpeg.porch.detect ERROR : Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 704x576, 90k tbr, 90k tbn, 180k tbc

ffmpeg.porch.detect ERROR : Stream #0:1: Audio: pcm_alaw, 64000 Hz, mono, s16, 512 kb/s

ffmpeg.porch.detect ERROR : [flv @ 0x555d39622580] FLV does not support sample rate 64000, choose from (44100, 22050, 11025)

ffmpeg.porch.detect ERROR : [flv @ 0x555d39622580] Audio codec pcm_alaw not compatible with flv

ffmpeg.porch.detect ERROR : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented

ffmpeg.porch.detect ERROR : Stream mapping:

ffmpeg.porch.detect ERROR : Stream #0:0 -> #0:0 (copy)

ffmpeg.porch.detect ERROR : Stream #0:1 -> #0:1 (copy)

ffmpeg.porch.detect ERROR : Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))

ffmpeg.porch.detect ERROR : Last message repeated 1 times

frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
ffmpeg.porch.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono

ffmpeg.porch.detect ERROR : Input #0, rtsp, from 'rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=1':

ffmpeg.porch.detect ERROR : Metadata:

ffmpeg.porch.detect ERROR : title : Media Server

ffmpeg.porch.detect ERROR : Duration: N/A, start: 1607309450.644203, bitrate: N/A

ffmpeg.porch.detect ERROR : Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 704x576, 90k tbr, 90k tbn, 180k tbc

ffmpeg.porch.detect ERROR : Stream #0:1: Audio: pcm_alaw, 64000 Hz, mono, s16, 512 kb/s

ffmpeg.porch.detect ERROR : [flv @ 0x555d39622580] FLV does not support sample rate 64000, choose from (44100, 22050, 11025)

ffmpeg.porch.detect ERROR : [flv @ 0x555d39622580] Audio codec pcm_alaw not compatible with flv

ffmpeg.porch.detect ERROR : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented

ffmpeg.porch.detect ERROR : Stream mapping:

ffmpeg.porch.detect ERROR : Stream #0:0 -> #0:0 (copy)

ffmpeg.porch.detect ERROR : Stream #0:1 -> #0:1 (copy)

ffmpeg.porch.detect ERROR : Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))

ffmpeg.porch.detect ERROR : Last message repeated 1 times

frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.app INFO : Stopping...
ffmpeg.porch.detect ERROR : Guessed Channel Layout for Input Stream #0.1 : mono

ffmpeg.porch.detect ERROR : Input #0, rtsp, from 'rtsp://[email protected]:554/cam/realmonitor?channel=1&subtype=1':

ffmpeg.porch.detect ERROR : Metadata:

ffmpeg.porch.detect ERROR : title : Media Server

ffmpeg.porch.detect ERROR : Duration: N/A, start: 1607309460.592328, bitrate: N/A

ffmpeg.porch.detect ERROR : Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 704x576, 90k tbr, 90k tbn, 180k tbc

ffmpeg.porch.detect ERROR : Stream #0:1: Audio: pcm_alaw, 64000 Hz, mono, s16, 512 kb/s

ffmpeg.porch.detect ERROR : [flv @ 0x55bb1f189580] FLV does not support sample rate 64000, choose from (44100, 22050, 11025)

ffmpeg.porch.detect ERROR : [flv @ 0x55bb1f189580] Audio codec pcm_alaw not compatible with flv

ffmpeg.porch.detect ERROR : Could not write header for output file #0 (incorrect codec parameters ?): Function not implemented

ffmpeg.porch.detect ERROR : Stream mapping:

ffmpeg.porch.detect ERROR : Stream #0:0 -> #0:0 (copy)

ffmpeg.porch.detect ERROR : Stream #0:1 -> #0:1 (copy)

ffmpeg.porch.detect ERROR : Stream #0:0 -> #1:0 (h264 (native) -> rawvideo (native))

ffmpeg.porch.detect ERROR : Last message repeated 1 times

frigate.video INFO : porch: ffmpeg sent a broken frame. something is wrong.
frigate.video INFO : porch: ffmpeg process is not running. exiting capture thread...
frigate.object_processing INFO : Exiting object processor...
frigate.events INFO : Exiting event cleanup...
frigate.watchdog INFO : Exiting watchdog...
frigate.record INFO : Exiting recording maintenance...
frigate.events INFO : Exiting event processor...

blakeblackshear · December 7, 2020, 3:10am

I see. The audio format from your feed is not compatible with RTMP. If you can, change your audio format on the camera to AAC. You can also try removing the audio from the stream as shown below:

cameras:
  porch:
    width: 2592
    height: 1520
    ffmpeg:
      inputs:
        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.0.20:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - clips

        - path: rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.0.20:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
            - rtmp
      output_args:
        rtmp: -c copy -an -f flv

rayzorben · December 7, 2020, 3:16am

Thanks, switched it to AAC and it is working now.

With info enabled I am getting a lot of errors that I assume I can just ignore?

ffmpeg.porch.detect ERROR : frame= 1620 fps= 15 q=-1.0 q=-0.0 size= 5108kB time=00:01:48.14 bitrate= 387.0kbits/s dup=0 drop=1072 speed=1.01x

ffmpeg.porch.detect ERROR : frame= 1629 fps= 15 q=-1.0 q=-0.0 size= 5148kB time=00:01:48.65 bitrate= 388.1kbits/s dup=0 drop=1079 speed=1.01x