Adding RTSP-Stream from ESP32-CAM into Frigate

Hello,

has anybody successfully integrated a RTSP-stream from an ESP32-CAM into Frigate? I flashed GitHub - geeksville/Micro-RTSP: A RTSP video server intended for very small CPUs (ESP32 etc) on my ESP32 and I am able to watch the video in VLC (via RTSP) and in the webbrowser. Frigate however is not able to receive the stream:

frigate.yml:

cameras:
  back:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -r
        - '3'
        - -use_wallclock_as_timestamps
        - '1'
      inputs:
        - path: rtsp://192.168.188.127:8554/mjpeg/1
          roles:
            - detect
            - rtmp
    width: 800
    height: 600

Log output:

frigate.mqtt INFO : MQTT connected
watchdog.back INFO : No frames received from back in 20 seconds. Exiting ffmpeg…
watchdog.back INFO : Waiting for ffmpeg to exit gracefully…
frigate.video INFO : back: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video INFO : back: ffmpeg process is not running. exiting capture thread…
frigate.watchdog INFO : Detection appears to have stopped. Exiting frigate…
frigate.app INFO : Stopping…
frigate.record INFO : Exiting recording maintenance…
frigate.object_processing INFO : Exiting object processor…
ffmpeg.back.detect ERROR : [udp @ 0x16111c0] attempted to set receive buffer to size 393216 but it only ended up set as 360448
ffmpeg.back.detect ERROR : [udp @ 0x1621510] attempted to set receive buffer to size 393216 but it only ended up set as 360448
ffmpeg.back.detect ERROR : [rtsp @ 0x16095a0] UDP timeout, retrying with TCP
ffmpeg.back.detect ERROR : [rtsp @ 0x16095a0] Could not find codec parameters for stream 0 (Video: mjpeg, none(bt470bg/unknown/unknown)): unspecified size
ffmpeg.back.detect ERROR : Consider increasing the value for the ‘analyzeduration’ and ‘probesize’ options
ffmpeg.back.detect ERROR : Output file #0 does not contain any stream
frigate.events INFO : Exiting event cleanup…
frigate.events INFO : Exiting event processor…
frigate.watchdog INFO : Exiting watchdog…

I think tweaking the ffmpeg parameters is the way to go but I am clueless which parameters propably work.

In the codec information displayed by VLC it says “Decoded format: Planar 4:2:2 YUV full scale”.

Has anybody a suggestion?

I got it working and I must say I am really impressed. Even without a Google Coral Stick it’s fast and usable. Didn’t expected that.

To get it working I flashed the firmware from GitHub - geeksville/TenDollarWebcam: A small example app for ESP32 + Micro-RTSP onto my TTGO T-Camera ESP32 so it is able to output an RTSP stream. The AutoWifi-setup can be a bit annoying but it worked in the end.

After that I use the following configuration (frigate.yml):

cameras:
  ESP_TTGO_Cam:
    ffmpeg:
      global_args: -hide_banner -loglevel warning
      # Hardware acceleration does not work for me as described in the Frigate documentation on my Raspberry Pi 4 32Bit OS.
      hwaccel_args: []
      #  - -c:v
      #  - h264_mmal
      inputs:
        - path: rtsp://192.168.188.139:8554/mjpeg/1
          roles:
            - detect
     #       - rtmp # deactivate rtmp, otherwise you only see a green image
    width: 800
    height: 600
    fps: 5

What still not working is that Frigate offers the stream as RTMP and I think it’s somehow related to the ffmpeg output parameters that need tweaking.

After that I installed the HACS integration and I am seeing a screenshot of the last event in Lovelace. Pretty nice - EXCACTLY what I need.

Maybe my issues that I had are somehow related to the ESP32 board I used. I switched from a M5-Stack ESP32-CAM to the TTGO Cam-Board which has plenty of PSRAM.

Have fun with it.

2 Likes

Hi,

Are you going to dig more with output parameters?

Yes I will revisit the topic if I find some time.

hello @viertausend I guess German ? :slight_smile: I am also thinking to attach some ESP32Cam modules to frigate. But without the RTMP stream it’s useless. :frowning: I need the mac showing in my dasboard. But if I am not wrong… The rtsp stram can be added to homeassistant outside frigate right? That might be ok. I will give it a shot an revert with my results :slight_smile:

You are right with your assumption about my nationatliy.

Some more information on the topic ESP32-CAM in Frigate:
Using the following arguments for the rtmp option let it “work”:

ffmpeg:
  output_args:
    rtmp: -c:v libx264 -f flv

However this slows the whole Raspberry Pi 4 down until the point that the Dashboard isn’t even reacting anymore. It is because the transcoding is done without hardware acceleration (Processor usage is >> 80% with this option). The camera feed can now be seen in the dashboard and it’s technically working but with a huge delay that makes it unusable even for the modest users.

Adding the following configuration to use hardware acceleration leads to a huge amount of warnings and errors in the log file:

  hwaccel_args:
    - -c:v
    - h264_mmal

Just a short part of the log-file:

ffmpeg.ESP_TTGO_Cam.detect     ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
ffmpeg.ESP_TTGO_Cam.detect     ERROR   : mmal: mmal_vc_shm_init: could not initialize vc shared memory service
ffmpeg.ESP_TTGO_Cam.detect     ERROR   : mmal: mmal_vc_component_create: failed to initialise shm for 'vc.ril.video_decode' (7:EIO)
ffmpeg.ESP_TTGO_Cam.detect     ERROR   : mmal: mmal_component_create_core: could not create component 'vc.ril.video_decode' (7)
ffmpeg.ESP_TTGO_Cam.detect     ERROR   : Error while opening decoder for input stream #0:0 : Unknown error occurred

Another information:
The integration of the live streams even works without the RTMP mentioned before but the huge delay is there as well. Processor usage is not as high as with RTMP.

Hello,
please help!
I also flashed the firmware on my ESP32 Cam and I am also able to watch the video in VLC and in the webbrowser but in Frigate I see just a green pictures. Not stream is avalible.

my frigate.yaml configuration:

cameras:

  esp_cam:

    ffmpeg:
      global_args: -hide_banner -loglevel warning
          hwaccel_args: []
      #  - -c:v
      #  - h264_mmal
      inputs:
        - path: rtsp://192.168.178.44:8554/mjpeg/1
          roles:
            - detect
            - clips
     #       - rtmp # deactivate rtmp, otherwise you only see a green image
    detect:
      width: 800
      height: 600
      fps: 5

Here is the Log Output:

[2022-02-17 17:06:21] frigate.mqtt                   INFO    : MQTT connected
[2022-02-17 17:06:21] frigate.app                    INFO    : Output process started: 219
[2022-02-17 17:06:21] detector.cpu                   INFO    : Starting detection process: 218
[2022-02-17 17:06:21] frigate.app                    INFO    : Camera processor started for esp_cam: 222
[2022-02-17 17:06:21] frigate.edgetpu                WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
[2022-02-17 17:06:21] frigate.app                    INFO    : Capture process started for esp_cam: 223
[2022-02-17 17:06:21] ws4py                          INFO    : Using epoll
[2022-02-17 17:06:21] ws4py                          INFO    : Using epoll
[2022-02-17 17:06:26] frigate.video                  INFO    : esp_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-02-17 17:06:26] frigate.video                  INFO    : esp_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-02-17 17:06:26] frigate.video                  INFO    : esp_cam: ffmpeg process is not running. exiting capture thread...
[2022-02-17 17:06:41] watchdog.esp_cam               ERROR   : FFMPEG process crashed unexpectedly for esp_cam.
[2022-02-17 17:06:41] watchdog.esp_cam               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-17 17:06:41] watchdog.esp_cam               ERROR   : You may have invalid args defined for this camera.
[2022-02-17 17:06:41] ffmpeg.esp_cam.detect          ERROR   : [tcp @ 0x55b2438eb0] Connection to tcp://192.168.178.44:8554?timeout=5000000 failed: Connection timed out
[2022-02-17 17:06:41] ffmpeg.esp_cam.detect          ERROR   : rtsp://192.168.178.44:8554/mjpeg/1: Connection timed out
[2022-02-17 17:06:46] frigate.video                  INFO    : esp_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-02-17 17:06:46] frigate.video                  INFO    : esp_cam: ffmpeg process is not running. exiting capture thread...
[2022-02-17 17:06:51] watchdog.esp_cam               ERROR   : FFMPEG process crashed unexpectedly for esp_cam.
[2022-02-17 17:06:51] watchdog.esp_cam               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

What is wrong?
Has anybody an ideas or suggestion?

Hi EKo,

I see that you use my configuration posted a couple of months ago. What I can say about that is that it worked with an earlier release with Frigate but not with the current release anymore.

Using this configuration the current release produces the exact same error “esp_cam: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures” in my setup.

I didn’t have the time to troubleshoot this yet. Depending on how desperate you want to get it running you may downgrade your Frigate version. Another way would be to find out which ffmpeg parameters are required to get it running with the current version.

1 Like

Not sure if you ended up updating the config for the new release of frigate but if not here it is

cameras:
  ESP_32_CAM:
    ffmpeg:
      inputs:
        - path: rtsp://Your_IP_Address:8554/mjpeg/1
          roles:
            - detect
    detect:
      # Optional: enables detection for the camera (default: True)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of frames without a detection before frigate considers an object to be gone. (default: 5x the frame rate)
      # max_disappeared: 25
      width: 640
      height: 360
      fps: 5
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: false
      retain:
        default: 10
        objects:
          person: 15 # Per object retention days
    rtmp:
      # Required: Enable the live stream (default: True)
      enabled: False

1 Like

credits to @viertausend, @VijayS and others
I got mine to work…HA config and frigate.yml are as follows
HA:
Installed on generic x86-64
Home Assistant 2022.11.4
Supervisor 2022.11.2
Operating System 9.3
Frontend 20221108.0

Frigate: v0.11.1

  lounge_tv:
    ffmpeg:
      global_args: -hide_banner -loglevel warning
      inputs:
        - path: rtsp://10.0.1.72:8554/mjpeg/1
          roles:
            - detect
            - record
    detect:
      width: 800
      height: 600
      fps: 5
    rtmp:
      # Required: Enable the live stream (default: True)
      enabled: False
    record: 
        enabled: True
2 Likes

Work for me Thanks.