"Matter Casting" ("Matter Casting Client") support in Home Assistant and upcoming Voice Assistants? - new open protocol standard for local video and audio streaming

Please consider adding “Matter Casting” support to Home Assistant Cast (as an alternative to the Google Cast protocol), and more importantly, please look at adding Matter Casting support to any upcoming Home Assistant’s Assist Voice Assistant Smart Speakers and Smart Displays (regardless if they are based on Wyoming Satellite or ESPHome Voice Assistant platforms) as Matter Casting is aimed at democratize local video and audio casting in a universal way that can be supported by all ecosystems and platforms.

Matter Casting” is a new open protocol media streaming standard and the Matter Casting APIs for casting video and audio streams over a local network is only a small part of the currently much-hyped Matter standard suite for IoT which is being led, promoted and developed by the CSA (Connectivity Standards Alliance) and its very impressive list of member companies:

This is simply meant as an informal feature request that can also be as an open discussion about this new open video and audio streaming standard and Matter Casting API/protocol that might eventually become an industry-standard in the future, meant to replace or compete directly with proprietary and closed source media streaming protocols used by Google Cast (Google Chromecast), Apple’s AirPlay (formerly AirTunes for AirPort), Alexa Cast (Amazon Alexa / Amazon Echo), and Miracast. Being an enthusiast of open-source software and non-proprietary standards I am hoping for industry-wide adoption of this new open standard for casting video and audio streams in the future.

Anyway, “Matter Casting” (also wrongly referred to as “Matter Cast”) is a technology standard that uses open APIs to enable you to cast video and/or audio content from your smartphone, tablet, and computer to your TV or smart display and smart speakers / audio-recievers + voice assistant appliances. Matter Casting is primarily meant to let you cast content and control playback from an app on your phone/tablet to an app running on a TV, smart display or smart speaker. From an end-user’s point-of-view Matter Casting is meant to work the same thing as Google Cast, Apple AirPlay, and Miracast or other similar streaming media protocols for sharing multimedia on a local network, with a few major differences: implementing support for this “Matter Casting” is royalty-free and it is based on open-source code, so not proprietary. That means it is designed from the ground up to be a “free and open source” video and audio casting/streaming standard for the best possible cross-platform compatibility.

Matter casting is an open protocol that does not require specific hardware, only that the app on your source device and the app on the target device you are casting to both have the feature enabled. So again, from an end-users point-of-view it is essentially like Apple’s AirPlay or Google Cast but can be available to every app or hardware maker to implement and not limited to specific mobile OS and ecosystems, or partnerships, however under the hood Matter Casting is also superior to those other protocols in many ways.

"In a brief demo at CES, Amazon showed how users would get all the features of its Prime Video app while casting, including its X-Ray feature, and could then back out from the video player to the Prime Video menu. Chris DeCenzo, the senior principal engineer at Amazon who led Matter Casting development, confirmed that with linear apps such as Sling TV, you might be able to flip through channels with the TV remote after launching a channel through Matter Casting.

“It actually is a different architecture from what you see with the other casting protocols,” DeCenzo says. “Matter Casting is really about communicating with apps on the TV.”

Chromecast doesn’t work quite the same way. On Android TV and Google TV devices, Chromecast’s video streams are usually separate from the in-app versions, so you can’t get full controls with your regular remote. While Google does offer a way for apps to connect directly with Chromecast, I haven’t seen any major streaming services use this feature except Netflix. And if you’re using Chromecast on a different TV platform such as Vizio SmartCast, there’s no app connection at all.

Apple’s AirPlay has a similar limitation: Videos you launch via AirPlay are separate from the apps on your TV, and using AirPlay even precludes you from playing other media on the iPhone or iPad from which the video is streaming.

All of which mean that Matter Casting will feel more like an extension of the apps on your TV than an entirely separate way to watch video. That might make you more likely to use it in the first place."

Again, Matter Casting is in its infancy and its current architecture still looks to be missing audio playback features specifically for music, such as audio-only or multi-room and real-time audio synchronization to allow for synchronized multi-room music playback, but at least is it already supporting Android, Darwin (for iOS and Mac), and Linux, allowing any developer to implement it into its products. Hopefully ensuring wider-spread compatibility between devices of different manufacturers and ecosystems.

So the reason why I think this is interesting is that it is the first attempt by the technology consortium governing the overall Matter standard suite to create also an open standard for local video and audio casting that is meant for broader adoption across different ecosystems for interoperability.

https://csa-iot.org/all-solutions/matter/

Because the Matter standard matters if adopted as the foundation for all connected things:

https://csa-iot.org/newsroom/building-a-standard-that-really-matters/

Matter in general already has hundreds of member companies, but understand that not everyone has publicly committed to every part of that standard, as most have only announced support for the IoT parts that control smart devices, however, support for additional device types is being announced regularly. “Matter Casting” is needed in this toolkit for streaming audio and video locally in a unified way.

Matter Casting adoption has so far been announced by Amazon who will first add support to newer Amazon Fire TV and Amazon Echo Show series of smart display devices via an upcoming firmware update, (in which it sounds like they might initially only implement video streaming?).

More interestingly, Amazon has already embraced Matter Casting fully have publicly stated that they do not plan on ever adding support for Google Cast or Apple AirPlay video casting to their newer Fire OS based devices (running Amazon’s Fire OS 7 or higher). This is just hot from the press of the CES 2024 (Consumer Electronics Show yearly tech event):

Amazon is a long-time supporter of using open technology standards to give customers more choice over the devices and services they use in their homes. That’s also why Amazon is a founding member and key contributor to the Matter Standard. At CES, Amazon announced Matter Casting, which enables customers to cast content to Fire TV and Echo Show 15 devices directly from supported streaming apps on iOS and Android. Customers can begin watching a movie or browse for their next favorite show from Prime Video on their phone, and cast it to their compatible Fire TV device or Echo Show 15. This is an industry-first demonstration of implementing Matter Casting.

Chris DeCenzo (a principal engineer at Amazon who works on Alexa and Fire OS devices) says that if all the big companies adopted Matter Casting, it would be better for the industry, as app developers would only have to develop for one protocol — one of the founding principles of the Matter smart home standard. “We believe in open standards,” he says, “because it simplifies things for developers.

Additionally, it would make things easier for customers. Casting from apps would work with every streaming box, TV, and smart display, and you wouldn’t have to think about what hardware you have in order to cast your content. That sounds great in theory, but — as with Matter in general — it looks like it will be a long journey to get there.

Matter Casting upstream development looks to be progressing nicely, it is now used on Amazon Fire OS7 and as such Matter Casting is already available on latest Amazon Fire TV and Amazon Echo Show 15, however support for Matter Casting on the mobile app side it is so far only available from Amazon’s Prime Video app the and the STARZ app, but Amazon wrote that they are working on Plex, Pluto TV, Sling TV, STARZ, and ZDF to add Matter Casting support from those later this year:

Screenshot showning all casting options through Amazon’s Prime Video app on an iPhone to a Pixel Tablet and Chromecast with Google TV device via Chromecast and to Echo Show 15 devices through Matter:

Any feedback on this feature request asking for devs to look at Matter Casting support to ESPHome?

Also, if you are a developer please post your input on these feature request ideas about Matter Casting support for audio-only streaming and multi-room support for syncronized music stream playback in multiple rooms?

FYI, chrisdecenzo (Chris DeCenzo from Amazon who is the lead Matter Casting developer and a member is in the Matter working group, a.k.a. Project CHIP) wrote today that there is now an official effort within the Matter working group (in Project CHIP) right now to define use cases for “Matter Casting” to audio players and smart speakers or voice assistants within the Matter standard (but only Matter members can join a Matter working group and participate with that work as far as known), see his comment here about joining that effort → [Feature] Matter Casting support for "Audio Player Architecture" with a new"Casting Audio Player device type" and "Audio Player endpoint" ("Casting Audio Player" and adding "Basic Audio Player") + look into adding multi-room music streaming? · Issue #31389 · project-chip/connectedhomeip · GitHub

@marcelveldt As “the Matter guy” at Nabu Casa (and Home Assistant) who is a member of Matter and the CSA/CSA-IOT perhaps this is something that you (and maybe also ESPHome developers) at Nabu Casa would be interested in helping architecture and develop for the Matter project?

I would think that epecially audio-only streaming with focus on high-quality audio-streams and music-playback + multi-room support functionality might be very relative to current Home Assistant roadmap now that Home Assistant’s recent announcementabout both your “Music Assistant” add-on and the Open Home Foundation + the Home Assistant’s voice assistant work, which are separate things that I believe all align in spirit with this concept at a high-level?

PS: I also read that Nabu Casa is developing your own ESPHome based smart-speakers (and/or smart-display) hardware, and for such devices to also work as music streaming and audio player (A/V-receiver endpoint) without your “Music Assistant” integration acting as middleware I am guessing you are eventually going to want to add support for some cross-ecosystem support for some kind of standardized audio streaming protocol like Matter Casting with audio player endpoint features?

Matter v1.3 brought new improvements to Matter Casting for media players + media playback control:

Enhanced Entertainment and Smart Home Management Functionality for Media Devices

Matter Casting Media Players / TVs — Matter 1.3 offers improvements to TV functionality, including push messages and dialog support for new ambient experiences, casting initialization enhancements, expanded interactivity options for TV apps, text and track support, and improved search functionality. Interaction with other devices in the home is also enhanced, enabling other Matter devices to send notifications to TVs or other devices with screens (for instance, notifying that a robot vacuum is stuck, the laundry is done, etc.)

FYI, they also updated Matter Casting APIs documentation in tv-casting-app as part of Matter v1.3:

They also updated their tv-app (Matter TV Casting app example):

Matter Casting APIs

Matter Casting consists of three parts:

  • The mobile app: For most content providers, this would be your
    consumer-facing mobile app. By making your mobile app a Matter “Casting
    Client”, you enable the user to discover casting targets, cast content, and
    control casting sessions. The
    example Matter tv-casting-app
    for Android / iOS and Linux builds on top of the Matter SDK to demonstrate
    how a TV Casting mobile app works.
  • The TV content app: For most content providers, this would be your
    consumer-facing app on a Smart TV. By enhancing your TV app to act as a
    Matter “Content app”, you enable Matter Casting Clients to cast content. The
    example Matter content-app
    for Android builds on top of the Matter SDK to demonstrate how a TV Content
    app works.
  • The TV platform app: The TV platform app implements the Casting Video
    Player device type and provides common capabilities around media playback on
    the TV such as play/pause, keypad navigation, input and output control,
    content search, and an implementation of an app platform as described in the
    media chapter of the device library specification. This is generally
    implemented by the TV manufacturer. The
    example Matter tv-app
    for Android builds on top of the Matter SDK to demonstrate how a TV platform
    app works.

This document describes how enable your Android and iOS apps to act as a Matter
“Casting Client”. This documentation is also designed to work with the example
example Matter tv-casting-app
samples so you can see the experience end to end.

Introduction

A Casting Client (e.g. a mobile phone app) is expected to be a Matter
Commissionable Node and a CastingPlayer (i.e. a TV) is expected to be a Matter
Commissioner. In the context of the Matter Video Player architecture, a
CastingPlayer would map to Casting “Video” Player. The CastingPlayer is
expected to be hosting one or more Endpoints (some of which can represent
Content Apps in the Matter Video Player architecture) that support one or more
Matter Media Clusters.