Speed of camera feed in HomeKit integration

Configuration
1

Hi,

I currently have a Lorex Doorbell. I have setup a frigate docker outside of HA, and used frigate to pull via RTSP data from the doorbell and do detection. I also enabled hardware acceleration in frigate to get better performance. Everything works in frigate and detection, recording is fine.
I then installed the frigate integration in HA, so i can get the data in HA for automations.

Finally i push the camera device that frigate creates in HA to HomeKit so i can see the video feed with my iOS and other Apple devices. I do see the camera feed in HomeKit, but when i load the live feed it’s slow, it takes about 5 maybe 10 seconds to load, and it’s choppy a little and then it starts to be ok.

Is there something i need to adjust to get this working perfectly?

Because, when i load the video feed from the camera, with the Lorex app on my iOS it’s instant, no delay, no lag and the image is really crips. I would have thought, that i could get the same performance, and image quality, with my Frigate + HA + HomeKit setup. I know, i added some bottlenecks, with all of this, but not that much.

My HA is running in a VM, and it was more power then it needs.

Anything i should change, adjust to get better performance out of this?

Thanks.

2

Frigate’s feed is RTMP so it is presented as HLS and that is why it is so far behind. In the upcoming 0.12 version of frigate (currently in beta) we have deprecated the RTMP restream in favor of an RTSP restream which works much better

3

Thanks. I will look into this to see if it fixes the issue.

4

I updated my frigate and the HA add on, to the latest 0.12 beta 1 and 4.0.0 beta01 of the add on. Everything works fine. I also deleted and recreated my HomeKit accessory integration for the camera.

In HA i see the frigate integration just fine. I also see the camera in HomeKit, but when i click on it to get live stream, i get a timeout or no connection possible, but in HA i can see the camera in live stream.

any idea on what could break, since i can see the live in HA?

Base on this documentation, for 0.12 beta 1 https://deploy-preview-4055–frigate-docs.netlify.app/configuration/restream/#rtsp i have opened port 8554, and i have confirm i can connect to it with VLC. So i don’t think, it’s the problem, but i can’t confirm.

5

Do you see any logs in HA? If VLC and others can connect to the camera then I’d expect it to work in homekit without issue. Difficult to say why it isn’t working without logs

6

Here is the relevant logs, i think.

Also changed restream configuration in frigate, because now frigate sends out H265, so i set the restream parameters to H264, because iOS/HomeKit can’t do H265 (i think).

2023-01-07 11:04:08.994 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): h11 Event: PAUSED

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): h11 Event: Request(method=b’PUT’, headers=<Headers([(b’host’, b’Doorbell\032CC93F0._hap._tcp.local’), (b’content-length’, b’216’), (b’content-type’, b’application/hap+json’)])>, target=b’/characteristics’, http_version=b’1.1’)

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): h11 Event: Data(data=bytearray(b’{“characteristics”:[{“aid”:1,“iid”:24,“value”:“ARUCAQEBEOvo\/IC6EU8dsO1S9VztRZcCNAEBAAIJAQECAgECAwEAAwsBAgAFAgLQAgMBHgQXAQFjAgRxK9ZJAwIrAQQEAAAAPwUCYgUDLAEBAwIMAQEBAgEAAwEBBAEUAxYBAW4CBJw9fMsDAhgABAQAAKBABgENBAEA”}]}'), chunk_start=False, chunk_end=False)

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): h11 Event: EndOfMessage(headers=<Headers()>)

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_handler] (‘192.168.1.81’, 50731): Request PUT for path ‘/characteristics’: {‘host’: ‘Doorbell\032CC93F0._hap._tcp.local’, ‘content-length’: ‘216’, ‘content-type’: ‘application/hap+json’}

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_handler] (‘192.168.1.81’, 50731): Set characteristics content: {‘characteristics’: [{‘aid’: 1, ‘iid’: 24, ‘value’: ‘ARUCAQEBEOvo/IC6EU8dsO1S9VztRZcCNAEBAAIJAQECAgECAwEAAwsBAgAFAgLQAgMBHgQXAQFjAgRxK9ZJAwIrAQQEAAAAPwUCYgUDLAEBAwIMAQEBAgEAAwEBBAEUAxYBAW4CBJw9fMsDAhgABAQAAKBABgENBAEA’}]}

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.characteristic] client_update_value: SelectedRTPStreamConfiguration to ARUCAQEBEOvo/IC6EU8dsO1S9VztRZcCNAEBAAIJAQECAgECAwEAAwsBAgAFAgLQAgMBHgQXAQFjAgRxK9ZJAwIrAQQEAAAAPwUCYgUDLAEBAwIMAQEBAgEAAwEBBAEUAxYBAW4CBJw9fMsDAhgABAQAAKBABgENBAEA (original: ARUCAQEBEOvo/IC6EU8dsO1S9VztRZcCNAEBAAIJAQECAgECAwEAAwsBAgAFAgLQAgMBHgQXAQFjAgRxK9ZJAwIrAQQEAAAAPwUCYgUDLAEBAwIMAQEBAgEAAwEBBAEUAxYBAW4CBJw9fMsDAhgABAQAAKBABgENBAEA) from client: (‘192.168.1.81’, 50731)

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.camera] set_selected_stream_config - value - ARUCAQEBEOvo/IC6EU8dsO1S9VztRZcCNAEBAAIJAQECAgECAwEAAwsBAgAFAgLQAgMBHgQXAQFjAgRxK9ZJAwIrAQQEAAAAPwUCYgUDLAEBAwIMAQEBAgEAAwEBBAEUAxYBAW4CBJw9fMsDAhgABAQAAKBABgENBAEA

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.camera] Set stream config request: 1

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): Send encrypted: b’HTTP/1.1 204 No Content\r\n\r\n’

2023-01-07 11:04:08.995 DEBUG (MainThread) [pyhap.hap_protocol] (‘192.168.1.81’, 50731) (d10849c5-ada5-4ec7-be7b-d5e9d2cb03ed): h11 Event: NEED_DATA

2023-01-07 11:04:08.996 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] [ebe8fc80-ba11-4f1d-b0ed-52f55ced4597] Starting stream with the following parameters: {‘v_profile_id’: b’\x02’, ‘v_level’: b’\x02’, ‘width’: 1280, ‘height’: 720, ‘fps’: 30, ‘v_ssrc’: 12144375, ‘v_payload_type’: b’c’, ‘v_max_bitrate’: 299, ‘v_rtcp_interval’: 0.5, ‘v_max_mtu’: b’b\x05’, ‘a_codec’: b’\x03’, ‘a_comfort_noise’: b’\x00’, ‘a_channel’: 1, ‘a_bitrate’: False, ‘a_sample_rate’: 16, ‘a_packet_time’: 20, ‘a_ssrc’: 2396896, ‘a_payload_type’: b’n’, ‘a_max_bitrate’: 24, ‘a_rtcp_interval’: 5.0, ‘a_comfort_payload_type’: b’\r’, ‘id’: UUID(‘ebe8fc80-ba11-4f1d-b0ed-52f55ced4597’), ‘stream_idx’: 1, ‘address’: ‘192.168.1.81’, ‘v_port’: 61178, ‘v_srtp_key’: ‘+pA0/MMoOwAN2bppqZ8LFXiOz1CHH2qilEstL4Ro’, ‘a_port’: 62075, ‘a_srtp_key’: ‘wAYOOSV6Jjpmt4rV0ZWm4rRCd3CobLZk/1GQd9hN’}

2023-01-07 11:04:08.996 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] FFmpeg output settings: -map 0:v:0 -an -c:v copy -tune zerolatency -pix_fmt yuv420p -r 30 -b:v 299k -bufsize 1196k -maxrate 299k -payload_type 99 -ssrc 12144375 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params +pA0/MMoOwAN2bppqZ8LFXiOz1CHH2qilEstL4Ro srtp://192.168.1.81:61178?rtcpport=61178&localrtcpport=61178&pkt_size=1316

2023-01-07 11:04:08.998 INFO (MainThread) [homeassistant.components.homekit.type_cameras] [ebe8fc80-ba11-4f1d-b0ed-52f55ced4597] Started stream process - PID 142

2023-01-07 11:04:08.998 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: started

2023-01-07 11:04:09.013 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’[rtsp @ 0x7f76df085000] method SETUP failed: 461 Unsupported transport’

2023-01-07 11:04:09.591 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b"Input #0, rtsp, from ‘rtsp://192.168.1.150:8554/doorbell’:"

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Duration: N/A, start: 0.000000, bitrate: N/A’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 100 fps, 100 tbr, 90k tbn’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b"Output #0, rtp, to ‘srtp://192.168.1.81:61178?rtcpport=61178&localrtcpport=61178&pkt_size=1316’:"

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Metadata:’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ encoder : Lavf59.16.100’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, q=2-31, 299 kb/s, 100 fps, 100 tbr, 90k tbn’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’Stream mapping:’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’ Stream #0:0 → #0:0 (copy)’

2023-01-07 11:04:09.592 DEBUG (MainThread) [homeassistant.components.homekit.type_cameras] Doorbell: ffmpeg: b’Press [q] to stop, [?] for help’

i have a unsupported transport error, i will do some research on this error.

8

It’s worth noting that go2rtc restream only supports tcp. It’s possible homekit is trying to use udp for some reason

9

go2rtc can restream RTSP UDP via FFmpeg source and #input=rtsp/udp param.

10

I would then have to find a way to do this in frigate for the restream.

11

You can put it in just like AlexxIt recommended above. Frigate go2rtc config is passed directly to go2rtc

12

Important, your docker container should have network host and camera should be in same network as server.

13

Any luck with this? having the same issue.

14

No, i never found the solution.

15

This seems to have resolved itself, almost as if it was a HomeKit issue with the feed changing and it just needed time for the database to be reset. I just left it alone and now it streams just fine.

16

Ok, my setup must be broken then, because i still got the problem.
Did you do something in Frigate or Home Assistant ?
Are you using Frigate v12 + HA Frigate integration v4 ?

17

Did you do something in Frigate or Home Assistant ?

Here’s my config to check against yours.

mqtt:
  host: !secret mqtt_address
  user: mqtt
  password: !secret mqtt_password
 
detectors:
  coral:
    type: edgetpu
    device: usb

go2rtc:
  streams:
    doorbell:
      - rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0

cameras:
  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/doorbell
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      output_args:
        detect: -f rawvideo -pix_fmt yuv420p
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
    detect:
      width: 1920
      height: 1080
    motion:
      mask:
        - 0,1080,775,1080,779,0,0,0
        - 1137,0,1138,59,1920,72,1920,0
        - 1505,943,1508,0,1920,0,1920,1080
        - 795,737,1515,686,1506,180,798,154
    objects:
      filters:
        person:
          mask:
            - 0,0,0,1080,785,1080,790,740,1533,688,1920,701,1920,161,1920,0
    snapshots:
      enabled: True
      retain:
        default: 30
    record:
      enabled: True
      events:
        retain:
          default: 30

Are you using Frigate v12 + HA Frigate integration v4 ?

Yes, I’m using these versions.

18

Pretty much the same. I’m not using go2rtc, because it makes my Lorex Doorbell unresponsive, for i don’t know what reason. So it’s the only part i don’t have.
But everything works in Frigate, i can see live cam in Home Assistant with the Frigate AddOn.
I can add the camera in HomeKit, and see a thumbnail refresh every 10s but no live stream in HomeKit.

it’s really weird. Before frigate v12, everything was working flawless.

19

it’s because the doorbell has two way talk. You simply need to disable bachchannel audio by adding #backchannel=0 to the end of the stream url in go2rtc config

20

Thanks, that fixed the connection problem.

But i still can’t see my live stream in HomeKit.

21

Now, my understanding is that HomeKit can’t receive H265 i think.
So i’m trying to convert the stream to H264, but that doesn’t seem to work at all.

getting frustrated with something that work before, and now doesn’t work no more.