Add support for H.265 in stream component

Feature Requests
#1

Having the stream component support decoding of h.265 rtsp streams would lower the bandwidth requirements of displaying local cctv cameras.

It was raised here as a bug https://github.com/home-assistant/core/issues/22424 but correctly identified as a feature request. But it looks like noone has put it here, so I now I am.

I’d note that also identifying that this was the issue wasn’t easy, as there was no error thrown up in the logs.

#2

Second for that

#3

Wondering no one else has a need for that. :slight_smile: would appreciate it much! More and more cameras support H.265.

#4

I had do dumb mine down to h.264. Luckily, I have that as an option.

#5

Same here. It’s just that I know it’s using more bandwidth - also I think it uses the same compression for storage on the CCTV NVR too, so it’s also taking up more disk space!

#6

I had a look at implementing it, but I didn’t see an easy way to do it using the available libraries (currently the component uses the PyAV library which is a wrapper for ffmpeg). I’m by no means an expert on AV encoding but it seems there are a few issues at play:

  1. Many HA installations run on low powered devices without transcoding hardware eg Raspberry Pis, so the code specifically checks to avoid video transcoding here https://github.com/home-assistant/core/blob/e379bfe38337be1ae8ac609aee1c6fd0a7d372e5/homeassistant/components/stream/worker.py#L118 . If you have adequate hardware you could edit this and transcode from a H.265 input stream to H.264, but you still wouldn’t be able to use H.265/HEVC on the output due to reasons below.
  2. The spec for H.265/HEVC over HLS requires the use of fmp4 segments instead of the MPEG-TS segments currently used by the HA HLS implementation. However, creating fmp4s the way we want (segment by segment into memory which we can serve via http) is not readily doable in PyAV as the PyAV ffmpeg fmp4 muxer outputs all the segments to the file system and there is no easy way to mux individual input segments directly into individual fmp4 segments.
  3. Playback of H.265/HEVC is less supported than playback of H.264 across different devices/browsers, so even if you get the backend working you may run into frontend issues.
1 Like
#7

I ended up exploring this further and was able to implement the remuxing of streams into fmp4 segments by editing the mp4 containers manually.
This takes care of problems 1 and 2 that I noted above, but problem 3 still remains - H265 playback really only works on iOS and Safari (See https://caniuse.com/#feat=hevc or look at H.265 support under html5test.com). Still, this gives the option to use H.265 for those who can use it, and changing from TS streams to fmp4 streams shouldn’t break anything for H.264.
I’ll create a PR with my changes shortly.

2 Likes
#8

Nice work! I understand some of those words.
I checked up on Chromecasts, and it’s only the Ultra that is currently capable of playing H265 too.
https://developers.google.com/cast/docs/media#video_codecs

So it’s fantastic that you found a solution, but clearly it won’t have the broad application that I’d initially imagined. Is it worth an update to the stream docs to describe the compatibility too?

#9

For the docs let’s wait until @hunterjm gets a chance to review the PR. I tested it on my devices and it didn’t break anything with H.264 and H.265 worked on iOS and Safari, but it’s best to have someone else have a look and test it out before we move forward.
The problem with H.265 on Android and Windows is due to H.265 being unsupported in Firefox and Chromium (which underlies most other browsers and Android apps). A lot of Android devices actually do support H.265 at the hardware level, and there is support at the platform level since Android 5.0, but H.265 videos in HTML5 don’t work because of the lack H.265 support in Chromium (the basis for WebView which Android apps use to render HTML). This is also why desktop browsers built on Chromium including Chrome and the new Microsoft Edge don’t support H.265 (the previous non-Chromium version of Edge, EdgeHTML, did support H.265 if the codec add-on was installed). Firefox, although not based on Chromium, also does not support H.265. However, as Chromium itself is open source, and there are community builds available that do support H.265. See https://chromium.woolyss.com/
For more details on H.265 support in Chromium see https://bugs.chromium.org/p/chromium/issues/detail?id=460703#c18 and for H.265 support in Firefox see https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#HEVC

1 Like
#10

I was able to get H.265 working on an Android device by modifying the Frontend and the Android App to use Exoplayer instead of hls.js/native. If the first PR goes through I can create PRs for those as well.

2 Likes
#11

Tell me more about this.

#12

Not sure which part you are asking about.
Both the H.265 PR and the audio PRs have been pulled into dev, so they should be available in 0.115.

1 Like
#13

How can I do it?

#14

The PRs for the frontend and the android app have been merged. You can load the dev versions or just wait a few weeks until 0.115 comes out.

1 Like
#15

Hi
Now that 0.115 is out, and installed, I was hoping my existing setup would just work, but sadly it doesn’t!

I have an rtsp HEVC camera, any pointers on a config that works, both the front and back end parts?

Camera url is :-
rtsp://192.168.1.10:554/user=admin&password=&channel=1&stream=0.sdp?

Thanks in advance to anyone who can help.

Ps I’m running hass.io on rpi 4, I don’t expect the performance to be good!

Kind regards

Jonathan

#16

What device/browser are you using?
It will only work in iOS and Safari for now.
The reason for this is that Google has actively blocked H265 in Chromium. It also got Firefox to get on board with its crusade under the banner of promoting “royalty-free codecs”. (In reality it is a strategic business tactic - as a big content provider Google doesn’t want to be on the hook for big royalties to MPEG-LA, so they keep pushing their own codecs even though they are slightly inferior to the MPEG-LA ones.)
As I’ve mentioned above, there are community builds of Chromium which do support H.265.
We did add a workaround for playing H.265 on Android but temporarily removed it for 2 reasons:

  1. Since the work around involves drawing on top of the webview, it doesn’t work when scrolling or when drawn over, so we had to restrict the usage to exclude live_view picture-entities.
  2. With the addition of audio, some people realized that it would be better to start play muted. This required a change to our Android workaround. We have made the change to the Android side, but we want to wait until the app gets republished before making the corresponding change on the frontend to avoid causing conflicts.
#17

Ah, thanks, that explains it.
I’d misunderstood and thought the Android bit had been enabled.

Thanks for your work on this and for responding.

Time for me to go and play with browsers!

#18

Yeah it’s a bit annoying how we have to work around this. Hope you can get it to work. We’ll try to get the Android part working again soon, but it will most likely always have somewhat limited functionality.

#19

By the way, the browser alternatives I suggested may not work since HEVC is not fully working in hls.js.
For iOS/Safari you have [HLS video source] -> [native Safari HLS player]
For non-Safari, you have [HLS video source] -> [hls.js player] -> [browser]
So for non-Safari, installing a HEVC capable browser is necessary but not sufficient as hls.js HEVC support is also required. HEVC support in hls.js is still lacking, and there isn’t much demand to get it working as most browsers except Safari don’t support HEVC, and Safari users don’t need hls.js.
See https://github.com/video-dev/hls.js/issues/3003#issuecomment-687160523 . I was able to get the Bitmovin stream to play but not the Apple one.

#20

Dang - this is frustrating, for some reason my hikvision cameras keep reverting to h265 even after switching them down to 264.

I can’t get it working with h265 in iOS in the actionable notifications preview, in safari or in Chrome (as mentioned) :confused: