Definitely. Have a look at the MotionEye add-on.
1 Like
Yes AlexITs card is a really great option for low CPU and real-time, thereâs still nothing else with those 2 features other than the proxy cameras which no longer give me full frame rate. Iâm starting to suspect that I might be having intermittent WiFi problems as my routers are rather old (holding out for WiFi 6e before making the upgrade).
Actually it does, your router just opened the port for you without telling you 
WebRTC uses a (random) UDP port it opens by itself, many consumer level routers with asymmetric cone style NAT will allow this through UDP hole punching, trading off convenience for security. But that wonât work with SSH reverse tunneled connections, as only a single port is forwarded. Your home IP will never be exposed to the remote device, so it canât establish the UDP connection.
In the latest release of HA, the camera component has been moved to the âcoreâ startup component so it changes when and how it starts at the boot up of HA. I have observed some weird behaviors of my camera components, crashing other integrations, most notably cloud ones, when a camera is offline as a result of it. As a reminder, I am have forked my owned version of HA to massively improve the camera stream handling from the ffmpeg component. (by using openCVâs ffmpeg instead of ffmpeg directly, then by using GPU decoding and finally having the camera return a numpy array to the video processing component instead of decoding, reincoding to jpeg and then redecoding to numpy). This has reduced the CPU loading by 80% per camera.
Wow I didnât know that, thanks for the lesson. Itâs less secure but the performance is hard to ignore. Was putting up with 4-5s lags until Alexit card, hard to go back.
Well technically itâs not that much of a security concern. The port is random and only open during streaming. So as long as you donât stream 24/7, itâs not that bad. This is a common technique for other protocols too, like peer-to-peer stuff. But it just doesnât work with certain more secure ways to remotely access HA (or only with a much more complex setup that adds maintenance and lag again). So itâs not a magic bullet, sadly.
Sorry to hear that the stream component isnât working out for many of you. As discussed previously, the latency is a function of the HLS protocol. Glad to see alternatives out there like AlexxITâs WebRTC addon - it looks pretty cool and seems to be working well for some people. The WebRTC protocol is probably your best option if latency is your main concern. Of course as with most protocols there are tradeoffs: Since the transport is not over http, the connection from HA to your client will be different than the rest of HA. Generally this is not an issue, and the transport is probably âfasterâ than going over http in most cases, but since the protocol allows for trying several different connection types, your experiences may differ depending on what network your client is on. Another thing to consider is that WebRTC is driven by Google, so it doesnât and most likely wonât ever support H.265 which is the preferred codec used on todayâs cameras (HLS supports H.265, but support on clients is also limited thanks to ⊠Google). WebRTC also doesnât support AAC audio, but it does support PCM codecs which HLS doesnât support (why is everything so difficult?!). Also, fundamentally WebRTC is generally very secure, but there have been security issues related to the protocol (see e.g. https://www.rtcsec.com), and since there are more different parts involved with the add-on, the attack surface will be larger.
I do have a PR open to add LL-HLS to stream which should improve latency by quite a bit, but it has yet to be tested by anyone but me, so Iâm not sure how well it will work across different hardware. The latency definitely wonât be as good as with WebRTC, but it might be sufficient for most. In addition to the tradeoffs mentioned above, there are actually some other advantages to the HLS protocol - since segments are being transferred instead of packets, problematic connections will probably see buffering instead of visual artifacts. And this is a more technical detail, but also because of segments vs packets, it can probably deal better with specialized codec variations with very low I-frame frequency such as H.264+/H.265+ (those codec variations can offer massive bandwidth savings).
A note (excuse?) about the pace of development - this is an open source project so contributors are really just other users working in their spare time. Itâs hard to juggle our own things sometimes, and to get things done we have to submit PRs and reviews so we have to coordinate with other devs who are also juggling their own things (thatâs why turnaround is generally much faster for custom components). Also, there are always issues that pop up, especially with a component like stream which is used on all types of setups/hardware. One more note - the slower speed of open source is compounded because the libraries we use are also open source, so if we identify an issue (or even if we submit a PR to address the issue) in a library, it may take months or even longer before the issue is addressed. To illustrate, to prepare for the aforementioned LL-HLS pull request I first had to wait for the hls.js and the ExoPlayer projects to be updated with LL-HLS client support. I then submitted this PR along with another one on the ExoPlayer repo back in March, and it has yet to be reviewed and merged. And that is already reasonably quick since the ExoPlayer project is driven by Google.
BTW, Iâm not even a developer by trade (my first GitHub PR was in 2020) - I just had some programming background and decided to start contributing because there were a few things I wanted to fix in my HA setup. In this year and a half Iâve learned quite a bit about video streaming, networking, and programming, and I now maintain a custom component and am a codeowner on two core integrations (although I sorely need to update one of them). Anyway, maybe my experience can inspire someone else to get started contributing to HA - I highly recommend it.
9 Likes
Justin,
I think I speak for most people here when I say that you did a tremendous job on the stream component. The issues are indeed do to the underlying limitations of HLS, which was never meant to be a realtime streaming protocol. I think everybody is well aware of that. Your component is working great considering the technical constraints of trying to squeeze near RT streaming into HLS.
Iâm excited about your work on LL-HLS ! While WebRTC is certainly the best choice in terms of latency, it does come with its own unique issues, as you mentioned. LL-HLS could be at an interesting in-between spot. Iâd love to try out your PR, if you feel it is ready for testing.
If only it wasnât Python 
1 Like
Iâd like to echo the comments above. My intention was never to bag on the HLS component and I hope it never came across that way. I still think the HLS component is by far the most reliable and easy way to start with cameras in hass and should be the starting point for most people. It just didnât happen to fit my use case.
In this way, itâs good that we have options. I just wish they were all as good of quality as yours is, and were as vebose in their communication about what they are doing as AlexeyITâs is. Then we could truly have an option for each situation and could effectively debug them. Hopefully we will get there eventually if people continue to maintain the other camera platforms.
I still wish we could have a âmasterâ camera platform where you just choose some parameters and were certain that it was as up to date as the HLS platform is.
With LL-HLS, WebRTC, and MSE, I think we are now becoming quite modern with our camera handling, and I have great hope that between these options alone we can cover most use cases. But the more I mess with this stuff the more I realize that every option still has itâs plusses and minuses and there are so many factors coming into play in the transport, encoding, server, network, and endpoint layers that it will probably always be a bit of a black art getting everything working perfectly.
PS I would be happy to test your LL-HLS cameras!
@HeyImAlex Thanks for the kind words. I wouldnât call stream my component - @hunterjm is the original author. Iâve just made some contributions to it (@allenporter also did a large refactoring earlier this year in order to better support cameras with expiring tokens).
I agree that having different/overlapping ways to set up cameras can be confusing. I myself am not familiar with anything except generic (and the stream component, but thatâs integrated into camera and not a camera platform). I find FFMPEG and MJPEG inefficient with processing and network resources, so I wouldnât ever use those myself. I am sure some people find them useful for certain cases, like using ffmpeg for reducing lag, changing codecs, or scaling, but I would consider these advanced cases and there may be better ways to achieve the same goal. Anyway, given that they are fundamentally different components that were contributed by different developers and use different parameters (e.g. MJPEG uses MJPEG instead of RTSP streams), I donât see a good way to put them under one umbrella (well, they are essentially under one umbrella now - the camera platform). I think documentation or user guides could be helpful - @scstraus maybe you could transfer some of your camera experiences from this post to a PR on the documentation repo.
On a tangential note, I think particularly with how seamlessly we are able to experience multimedia across our devices today, itâs easy for users to develop unrealistic expectations of video capabilities. Any tablet or phone from the past decade has been able to display smooth, full screen video. But what codecs does the device support? An older device most likely wonât play HEVC videos, for instance. Also, while they might seem similar, displaying full screen video is one thing, but paneling several feeds together is another thing. A device can only decode so many streams at once. Tablets are also misunderstood - Iâve seen some people complain about not being able to view their full resolution 8MP camera feeds on tablets. Guess what - many tablets have less capable GPUs than top end phones and many canât decode video much higher than 1080p (2MP cameras are 1080p and most of todayâs security cameras are way above that). Try opening a 4K video on your Android tablet. Network is another issue - MJPEG uses a lot of bandwidth, so itâs no surprise that a panel of high resolution MJPEG feeds doesnât work reliably over WiFi. The point here is not to rant, but rather to communicate that there are some constraints that HA canât work around, and that these constraints will come into play differently across different hardware/frontend setups/user requirements. Users will have to experiment to see what works for them, and they will need to understand that there are limitations and work around them accordingly. Thereâs no magic bullet from the developer side, Iâm afraid.
As for LL-HLS, yes feel free to give the PR above a shot. You will need to drop the files from the PR into the correct folders on your HA install, and you will need to use the generic component like so:
- platform: generic
stream_source: rtsp://user:[email protected]:554/
ll_hls: true
You will need to be on 2021.5 for the frontend to be updated with support (although Android and iOS shouldnât require this). The Android app will have the wrong starting position until the ExoPlayer PR I mentioned previously is merged, so youâll need to drag the timeline forward a few seconds after starting the feed. Post on the GitHub PR if you have any questions.
1 Like
Iâve considered doing that, but I feel like my experience is a bit to anecdotal to be useful for documentation. What would be excellent if someone who actually understood the workings of these components could verify some of my guesses as to why things are the way they are and those could go into the documentation. For now they are just observations with guesses as to what the causes might be. I havenât been able to concretely prove almost any of my guesses yet⊠I just know that some things work better than others in my environment, no idea how much of that maps to a different environment or why/why not.
Some things you mention would be great candidates⊠What is the bandwidth usage of a stream in different camera components? Which options can be used to increase/decrease CPU usage? Which technologies use more or less resources on the frontend? Which browsers/OSâ have better support for which technologies? None of these things are things I know for sure, but Iâd bet some of you devs do.
If you open a PR, Iâd be happy to provide some input, and Iâm sure others will chime in as well. Actually I think a lot of the people in this thread would be able to provide good input too. I am a bit busy these few weeks so I wonât have that much time (and I just got summoned to troubleshoot an old issue), but it might actually not take too much time if we have a few people working together. 
What would I put in the PR? Just a fill in the blanks structure of things Iâd like others to confirm?
Yeah, you could make a draft PR with something like that. I guess we have two different inputs to better documentation - you know what questions you want the answers to and we (hopefully) know the answers to some of them.
I was thinking of a simple analogy to compare the the mjpeg, ffmpeg, and stream (hls) components and came up with this. The video comes in from the camera and is encoded by hardware using a video codec like h.264. This is like a present that is packed nicely in a box and wrapped up in the box. In this example, demuxing the stream is like unwrapping the box and decoding the codec is like unpacking the box. The unwrapping and rewrapping is pretty easy, the unpacking is harder, and repacking is the hardest.
The box is easiest to transfer when it is still boxed up. We can transfer it when it is unboxed but then itâs an unruly mess that takes up a lot more space.
For the video feed to get from the camera to the client device, we have to transport it from the camera to HA to the client device.
For the mjpeg component, you have to access a mjpeg feed from the camera. mjpeg is much larger than regular video codecs because it doesnât compress anything across time. The analogy is that the camera doesnât give you a nice small box, it gives you a huge box that is not very efficient space wise. Depending on the camera, this might add a lot of processing load to the camera itself if it cannot do this encoding in hardware. So the downsides of this method are that it might load the camera CPU, and it also takes a lot of bandwidth, both from the camera to HA and from HA to the client. The upside of this method, using the analogy, is that since the box is packed so loosely, itâs very easy to unpack, so as long as the transport is OK the client should be able to open it easily.
The ffmpeg component is basically like a repacking station centered at the HA server. Using the analogy, it will unpack the box and repackage the box, but how it repackages everything depends on the options used. It usually repackages everything into an mjpeg stream, so it ends up with a similar result to the mjpeg component, only the work is done on the HA server rather than at the camera. The downside of this is high load on the HA server and high bandwidth from the HA server to the client.
The stream (hls) component is basically like a rewrapping station. It doesnât do any transcoding (unboxing or reboxing in the analogy), so there is not much CPU overhead (although there will be more with ll-hls) and the size is small all the way to the client. The downside of this is that the client needs to be powerful enough/have the right codecs to decode the stream, which isnât generally a problem but can be when devices are old/cheap or there are too many streams to decode (an aside - I donât think it makes sense to have a panel of live streams - with HLS you might get too many streams to decode on the client, and with the other methods there will be high bandwidth usage. Using substreams may help. I donât use a live view on my own setup.) The other downside of HLS is the latency, which has been discussed ad nauseam above.
WebRTC is similar to stream in that thereâs no transcoding, but as discussed earlier the transport is different and the supported codecs are different for whatever reason. Besides the differences in codecs, the upside vs HLS is better latency, but the downside is less robustness to missing packets.
Sorry for the muddled analogy, it was just a shower thought and then it lost traction as I was typing. Anyway, hope this helps.
Okay, letâs give it a try. Rather than trying to open a PR with text I donât know yet, I opened an issue here in order to get the data together so that I can compile the actual text to include in the PR. Anything I can get confirmed well enough to include will be brought into a PR I will write later.
Did you ever have a look at the Amcrest camera platform? Except for camera options this can also add motion/audio sensors to your HA. Iâm using it with 2 Dahua cameraâs with the following setup:
amcrest:
- host: 192.168.0.20
name: Voordeur
username: !secret dahua_username
password: !secret dahua_password
stream_source: rtsp
binary_sensors:
- motion_detected
- online
sensors:
- sdcard
I donât have an Amcrest, so I havenât used it or many of the other branded camera integrations. Iâm guessing that most of them are just reusing the other options here as they are mostly just opening an RTSP, ONVIF, or MJPEG camera stream, but they may have special implementations. Too many of those to test, I have only tried the ones which apply to a Hikvision, which is what I have.
The amcrest component doesnât help for the purpose of real time streaming. I am using it⊠on 16 cameras. It is good for taking snapshot and displaying them on lovelace but it basically relies on the haffmpeg component. It adds a couple of other API endpoints and is very useful but not for streamingâŠ
Any new updates since this was posted? I am looking into options for a security camera now. I have it on a stronger server and there is a GPU in one of the PCIE slots if that can be used for anything.
I try to update the OP when I find new info. The biggest new development is the WebRTC card. Personally I am still using FFMPEG with Live555 proxy as rhat produces the best results for me. Fast loading, high frame rate, low latency, pretty reliable. Main downside is high server CPU usage, but it sounds like it may be a good option for you. Thereâs even a chance you could get it using the GPU if configured right in which case it wouldnât affect your CPU much.