Go2rtc RTSP relay is very laggy, original stream is quite smooth

Third party integrations
1

I'd like to use Home Assistant with Frigate, which uses go2rtc.

I've set up a dummy RTSP server that plays a video on loop. That stream works smoothly without lag. Now I'm trying to send this into go2rtc and stream its RTSP output.

Config looks like this:

go2rtc:
  streams:
    test_stream: "ffmpeg:rtsp://rtsp:8554/stream"

When I go to http://localhost:1984/stream.html?src=test_stream, the resulting stream is quite choppy. The Intel N200 is barely breaking a sweat and intel_gpu_top barely indicates activity. The go2rtc log doesn't contain anything that looks like a clue.

5:47:46.052 PM	info	go2rtc version=1.9.10 platform=linux/amd64 revision=df95ce3
5:47:46.052 PM	info	config path=/config/go2rtc_homekit.yml
5:47:46.052 PM	info	[rtsp] listen addr=:8554
5:47:46.053 PM	info	[api] listen addr=:1984
5:47:46.053 PM	info	[webrtc] listen addr=:8555
5:51:08.743 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:08.754 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:08.883 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:08.892 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:08.902 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:08.933 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:09.059 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:09.071 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:09.124 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260
5:51:09.135 PM	warn	undefined error=unsupported method: GET caller=github.com/AlexxIT/go2rtc/internal/rtsp/rtsp.go:260

Stream info from the API:

{
  "producers": [
    {
      "id": 533,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:36524 forwarded rtsp:8554",
      "source": "exec:/usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -v error -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://rtsp:8554/stream -c copy -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/d19f680a6b8dc48b3b08bfd114f30cec",
      "url": "rtsp://rtsp:8554/stream",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=Stream\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.1.100\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AH+ygKALdgLUBAQFAAAADAEAAr8gDxgxlgA==,aO+PyA==; profile-level-id=4D401F\r\na=control:streamid=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 MPEG4-GENERIC/48000/2\r\na=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=119056E500\r\na=control:streamid=1\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H264",
        "audio, recvonly, MPEG4-GENERIC/48000/2"
      ],
      "receivers": [
        {
          "id": 534,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 31,
            "profile": "Main"
          },
          "childs": [
            536,
            545
          ],
          "bytes": 7287472,
          "packets": 7292
        },
        {
          "id": 535,
          "codec": {
            "channels": 2,
            "codec_name": "aac",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "childs": [
            540,
            546
          ],
          "bytes": 2429334,
          "packets": 1930
        }
      ],
      "bytes_recv": 9828758
    }
  ],
  "consumers": [
    {
      "id": 530,
      "format_name": "mse/fmp4",
      "protocol": "ws",
      "remote_addr": "172.18.0.1:36032",
      "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:150.0) Gecko/20100101 Firefox/150.0",
      "medias": [
        "video, sendonly, H264",
        "audio, sendonly, MPEG4-GENERIC, PCMA, PCMU, L16, PCML, OPUS"
      ],
      "senders": [
        {
          "id": 536,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 31,
            "profile": "Main"
          },
          "parent": 534,
          "bytes": 7287472,
          "packets": 7292
        },
        {
          "id": 540,
          "codec": {
            "channels": 2,
            "codec_name": "aac",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "parent": 535,
          "bytes": 2429334,
          "packets": 1930
        }
      ],
      "bytes_send": 10449608
    },
    {
      "id": 544,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:58090",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.10\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z01AH+ygKALdgLUBAQFAAAADAEAAr8gDxgxlgA==,aO+PyA==; profile-level-id=4D401F\r\na=recvonly\r\na=control:trackID=0\r\nm=audio 0 RTP/AVP 97\r\na=rtpmap:97 MPEG4-GENERIC/48000/2\r\na=fmtp:97 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=119056E500\r\na=recvonly\r\na=control:trackID=1\r\n",
      "user_agent": "FFmpeg Frigate/0.17.1-416a9b7",
      "medias": [
        "video, sendonly, ANY",
        "audio, sendonly, ANY"
      ],
      "senders": [
        {
          "id": 545,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 31,
            "profile": "Main"
          },
          "parent": 534,
          "bytes": 2652162,
          "packets": 2692
        },
        {
          "id": 546,
          "codec": {
            "channels": 2,
            "codec_name": "aac",
            "codec_type": "audio",
            "sample_rate": 48000
          },
          "parent": 535,
          "bytes": 828873,
          "packets": 661
        }
      ],
      "bytes_send": 3534683
    }
  ]
}

How can I further debug this?

2

You don't need ffmpeg, you can just use rtsp url in config file, and give it a try

3

HA is might be using go2rtc by default here already. It is supposed to try to pick the best way on it’s own.

4

Good point about ffmpeg being redundant here. The Frigate guide mentions it sometimes being needed for increased compatibility, but I can confirm that it isn't needed here. I've removed the ffmpeg: prefix.

When I take the go2rtc RTSP output at rtsp://localhost:8554/test_stream and play it in VLC, I get smooth playback. But in every browser I've tried on my desktop and mobile, however, playing the stream through Frigate's website doesn't work smoothly.

My Frigate camera config:

cameras:
  test_stream:
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: "rtsp://127.0.0.1:8554/test_stream"
          roles:
            - detect

On desktop I've tried playing http://localhost:1984/stream.html?src=test_stream&mode=webrtc and http://localhost:1984/stream.html?src=test_stream&mode=mse, neither of which gave good results.

Any idea why go2rtc's RTSP output is rendering a lot better than its WebRTC/MSE streams?

5

I see that your audio codec is aac, and webrtc is not capable of transferring it, so it fallsbacks to mse.

You can use go2rtc to convert it to opus to use webrtc, but I don't think it is the problem here.

You are saying that playing go2rtc based stream on vlc works fine but not on browsers.

Did you try different computer, phones?

Lastly, start simple and skip hardware acceleration, maybe even handling it with cpu might be good enough as a starting point

6

I really think you're onto something here. WebRTC browser playback is mostly fine (although still a little more choppy than RTSP, but usable), MSE browser playback is unusable, autodetect falls back to MSE.

How can I tell go2rtc to convert the audio to opus? I tried ffmpeg:rtsp://rtsp:8554/stream#audio=opus but get no stream when I do that. Nothing special to be seen in the go2rtc logs. (Edit: fixed it using the info at Configuring go2rtc | Frigate)

7

I do wonder, is there anything I can do to improve MSE playback? It appears to be Frigate's default because it has certain advantages.