Yet another two way audio post

Configuration
1

So it seems like Home Assistant doesn’t support two way audio at all, not on the server, the web client, or any of the mobile apps, and all the videos I find of people doing this really have other apps running or have built custom software into lovelace or whatever, and the audio stream is not going through the HA server at all? Of all my issues with HomeKit, I still haven’t moved my doorbells into HA because the intercom works, it works very well. I get a notification when someone rings the doorbell, I click on it, there’s a talk button, and now that I’ve gotten rid of all my HomePods, the audio stream is sent through my hardwired Apple TV and just works. I haven’t been able to test because my doorbell is HomeKit only and I can’t connect it to HA right now but I’m assuming there is no “talk” button in the native app?

So what would I have to do? So far I’ve thought of the following solutions, none of which I really like:
Keep my HomeKit doorbell: I was contemplating creating a HomeKit bridge anyway so I can use the iOS control center and Siri anyway. However, HomeKit is still not as reliable as I had always wished. It doesn’t seem like Apple really cares about it. It isn’t trying to use WiFi Home Hubs now to record cameras, causing 3 times the wifi bandwidth necessary anymore, but the Apple TV is still an Apple TV, not a true server I can monitor. And the recordings are in the cloud and encrypted so it’s slow to review recordings when I need to. I also still can’t view it in Home Assistant, and I need two apps still. I can’t view it on Windows, or any non Apple OS.
Amcrest Doorbell: I like the Amcrest camera I got as a pet camera, it works very well with DW Spectrum, which is currently my top contender for NVR/VMS. However, I don’t even know if it would be possible to get the ringer into HA without the video feed. I want to restream the cameras from DWS to minimize WiFi congestion. And I assume for two way audio I would need to use the amcrest app? So I’m back to two apps and I need to be VPN connected. And I have to be on a platform that supports the VPN and Amcrest app.
Cloud Connected Doorbell: like ring, nest, ecobee, etc. This would still probably need a separate app for two way audio, and I want to be recording locally. The only reason I was fine with HomeKit recording cameras to the cloud was because it was encrypted before it ever leaves the local network which I’ve heard is not the case with any other cloud doorbells.

Am I missing an option? I want to hear what you guys do. I’m willing to do some programming in HA, scripting, making dashboards, etc. But the ultimate requirements is that there’s an onvif stream that can be recorded locally and streamed into HA, and one app that controls the entire home including intercom for my roomate, parents, or whoever I choose, without them needing separate apps, accounts, and permissions.

Obviously my requirements are pretty tight but I’m glad to hear any ideas thrown around even if they aren’t full solutions.

2

2 way audio is enabled using go2rtc

3

Can the Home Assistant app be the microphone though?

4

Yes

I honestly haven’t bothered to setup 2way yet but several users have had success and all functional through HA APP

5

So what exactly would that look like? I’d have my recording server streaming the camera, recording it, restreaming it to HA, streaming it to the app. go2rtc is added to home assistant somehow and the hass app can be configured to send talkback to the go2rtc server which sends it directly to the doorbell?

6

Go2rtc takes camera stream and formats output to match on the fly

So

If using software nvr you can send to go2rtc first then feed streams from go2rtc to HA and NVR

Else

From nvr you can send stream to both go2rtc and HA or use go2rtc to restream to HA

Many HA users use frigate nvr which has go2rtc built in. Usage is dependent on your needs, setup and hardware

7

I want the main stream to go to DWS because I plan to run it in a cluster for high availability. Recording is more important than anything else. Can you clarify what you mean by

is the “feedback audio” stream an actual part of onvif? I want to cut down on the number of links in any chain. The only reason I want to restream from the NVR to other applications is to to prevent the video stream from being sent over wifi twice. Once the stream is on ethernet, I don’t mind, and would prefer it to go to multiple places directly instead of being restreamed again. Although I am confused what role Go2rtc plays here.

I was not aware that the HA app has two way audio support at all however. If I use the amcrest integration on my amcrest camera I have will it just work natively? If the feedback audio is part of onvif would it be remotely possible to just have that tunneled directly through DW Spectrum then?

Thanks

8

I just added the Amcrest camera using the Amcrest integration and noticed two things:
The video stream was delayed by 21 seconds. Not an exaggeration, I timed it.
There was no PTZ or ability to talk back to the camera.

9

That’s entirely different than what the other user was suggesting you do, not sure why you’d expect that to support two way talk when it doesn’t.

10

Two way talk is supported via GitHub - AlexxIT/WebRTC: Home Assistant custom component for real-time viewing of almost any camera stream using WebRTC and other technologies. or GitHub - dermotduffy/frigate-hass-card: A Lovelace card for Frigate in Home Assistant

11

Not sure myself either, thought it was worth a shot.

I’m reading over the docs for go2rtc but I can’t figure out how to get it to send the feedback voice stream one way while the incoming video and audio is coming from the vms. Does that mean I would need to put go2rtc first in the chain and the other components (HA and DWS) behind it?

Thanks

12

You can just install the webrtc component in HA that I linked above

13

I’m not sure I just want to install the integration into my HA. My HA runs on ESXi in a DRS cluster and I was having issues with the camera dropping the connection just to VLC player when a vmotion occured, even though it’s supposed to be “seamless.”

But that still doesn’t cover my concern about camera streams to and from the camera. I only want the video stream to go over wifi once. I don’t want it streaming to the VMS and HA. I don’t think the restream out of the VMS will include the ability to feedback the audio stream to the doorbell’s camera.

Also, if the webrtc integration you linked above is just go2rtc packaged inside it, is the stream authenticated with the rest of HA? Or is the card in HA just telling the client’s browser to load the stream from the go2rtc server which, if discovered could be streamed without logging in through HA?

14

The requirements you’re describing are more than original. In this case you’ll want to install go2rtc addon where you can configure rtsp authentication and use the go2rtc proxy for the VMS

15

Sorry, I probably did a bad job explaining it originally. Here’s a full diagram of what I envisioned. Hass VMS Diagram.pdf - Google Drive

Sounds like I would need to put the go2rtc first in line? Then is HA restreaming again or would clients need to have line of sight to the go2rtc server?

16

I’m impressed! I setup go2rtc and hass in a lab and applied a firewall to the go2rtc server and it appears at first glance that it is streaming through the hass server perfectly. I need to setup a certificate for https to be able to test two way audio and it’s late so that will happen tomorrow. Thanks @crzynik for the info so far!

17

Not sure if this has been said before, just use frigate

Free. :white_check_mark:
Open source :white_check_mark:
Good api✅
Has go2rtc built it :white_check_mark:
Records✅
Object detection. It heavily built around this but really you can disable on per cam basis and all is configurable to your needs :white_check_mark:

DW Spectrum look like something that is subscription based, poorly supported are just not worth it. I didn’t read deeply on their page but the janky website and big “sales” and “support” tabs just scream “hello sir, please hand us your wallet as you browse our 1990s website and select the stock image that applies to your use case. If this is commercial need there are better solutions but definitely not this for residential.

18

Frigate looks cool, but I ultimately decided it was too lightweight for my needs.

Dw spectrum isn’t subscription based, it’s licenced perpetually per camera. Additionally:

  • it has failover and load balancing between recording servers when the amount of cameras you want to record exceeds what a single server can do, or a server fails
  • has a more robust configuration system that has auditing, change management, and api, error handling, and is more accessible. Frigate’s configuration is a single yaml file on a single server
  • load balancing between storage repos
  • a real relational database server backing the system, not sqllite
  • runs on Windows, which isn’t to say Linux is bad but having the option is a plus
  • faster object search after the fact meaning if motion is detected in a motion zone or an object while being recorded, frigate indexes it, but I don’t think it has a way to search months of recordings in parallel after they’re recorded
  • ldap authentication
  • mobile app
  • windows, linux, macos clients for reviewing footage which performed much faster than frigate in my testing
  • and as you mentioned: support. I don’t need to go to volunteers on github when something breaks. Don’t get me wrong, the community is amazing, but you can’t rely on it for instant support at midnight.

I’ll admit their website is a little gross. You can take a look at NX Witness’ website, which is the core of DW Spectrum, but they don’t sell direct to the US. It’s the same software though. They have a free trial and it’s really easy to get started with on Windows. You should give it a try, it has profiles for tons of cameras built in so it’ll do fisheye automatically, etc. It also has PTZ, monitoring layouts, tons of other features.

Frigate is fine, just not as mature.

Also, I use DWS at work. We have more than 2 thousand cameras. It works flawlessly. I’ll probably never trust anything else except maybe onguard or avigilon but those are actually thousands of dollars a year.

19

Have you setup the authentication before? Is it supposed to be processed by HA or is HA authenticating the user and the user/client still needs line of sight to the proxy server? I noticed when I firewall off the proxy server, the codec used switches to MSE, which isn’t listed under supports two way audio. The only two that are listed that seem to apply here are onvif T and WebRTC. The WebRTC stream appears to be unauthenticated and anyone with access to the server can view it if they have the link. And it sounds like it has to be direct, which means if it were to be internet routable anyone on the internet could view and speak back to it. Not ideal.

image
Also, is there a particular stream url I need to use to get the onvif compliant restream? Onvif isn’t listed on the streams page but it is on the github. When I put in any of the other streams into dws there’s no ptz, two way audio, secondary stream, camera metadata, time sync, etc.

Charles

20

That’s not how webrtc works. Webrtc is encrypted and peer to peer so as long as you’re not exposing the webui of HA to the public. I’d suggest reading up more on webrtc.