Play nice please.
5 Likes
One tester should be sufficient for the start. Setting up a representative test lab at that persons home would be the real challenge. Without it should be at least the cross product of all supported installation types on the most used hardware platforms, each
- using the most popular integrations + hardware required to actually use the integrations
- and for the sake of it also including HACS and itās most popular custom integrations (itās common to test unsupported but heavily used configurations in SQA). You may not support them but you definitely want to know if things break to prepare for the upcoming support efforts (people WILL come up with that and telling them ānoā is support effort as well, see this thread)
Designing proper automated and manual test cases and keeping them fast and stable to execute is also a big challenge whenever real hardware is involved.
Test effectiveness (by finding real world issues that will affect real world use cases if not fixed until release) would however drastically improve, if you had one person per configuration (by that I mean installation type on hardware platform on typcially mutually exclusive integrations [zigbee2mqtt vs. zha for example]) and have them use the development branch (beta versions in release phases) as daily driver. If you go for one person, that person should run at least one of these configurations as daily driver to uncover this class of bugs.
Trying to achieve that with the open source community only is very unlikely to happen.
The only realistic way I see is Nabu Casa hiring such person(s). If you decide to do so: Please hire people with experience. Most people not actually doing software testing thinks thatās something everyone can do. Itās not. It requires the same amount of practical experience and the same level of background knowledge like any other profession.
If you need help finding someone and setting this up, please donāt hesitate to ask.
12 Likes
Itās not mutually exclusive: you can do both
in-house testing and public testing. It is also very common for open source projects to release public betas. A company where I worked (and was part of this setup), we made nightly beta releases and had a public beta (opt-in) group that would voluntarily test it (just via normal usage). Those users thought it was quite fancy to have this privilege. We regardless did tons of in-house QA.
Also, excellent summary, @nohn.
General comments: What happened here was a rare occurrence for HA and itās not like there werenāt options: users could roll back. The HA team was quick to respond. Iāve been following the GitHub issue. I donāt think it couldāve gone quicker, as there was a risk of making things even worse. Even the biggest of corporations with much larger test teams and rigs have made bigger screw-ups.
Itās certainly not nice to be affected by an issue like this, but just remember that itās not intentional.
5 Likes
Some sensible comments above. Iāll make one final comment; I think the monthly release cycle where there seems to be an attempt to include new functionality every time is not conducive to recruiting beta testers. If we had instead, say, monthly security and bug fix releases (with very limited and carefully reviewed changes, no breaking changes, etc) and then bigger six monthly releases with all the new functionality with a longer testing cycle, I think that would encourage more beta testers. With such an approach I would be more likely to consider becoming a beta tester.
3 Likes
Some really valuable Comments here!
First things first: Thanks to everybody involved to quickly providing a Fix for the DB Issue! Without panic and making things worse.
I think everybody would agree, that the ultimative Goal is to frequently improve and Features, while never breaking anything.
Personally, beeing a User, i would always put the last first.
HA develops clearly towards becoming Mass-Compatible, which i think is the good and right decision (UI over YAML, simplification).
But IMHO it will never really reach the Masses (or even worse, they turn back from Homeassistant after their first āAfter Update nothing worksā).
For a long Time i am not really happy with the Strategy to have HA informing me about an Update, where users can easily hit update, while it takes some effort to read changelogs / breaking changes, and then decide if it might/will/probably not affect the System.
It might be a valid decision to say ācustom addons are not in scopeā.
I wouldnāt agree, as it guess >90% HomeAssistants are using them, as they provide functionality not nativly included in HA.
But i think we all would benefit from first informing about possible breaking changes. And probably modified update/upgrade scopes (less and then bigger features adding, more fixes).
And if i could dream, i would just love having a āUpdate checkerā component, going through all configs, addons, devices and checking against all changes. Most probably a really big Project and not easily doable.
Lets think it from the End: No more hitting update, and then āi canāt control my tv any longer, the card stopped working, the forecast is gone, cant make updatesā.
As of Today, i canāt recommend my Matesā Girlfriend to start with Homeassistant.
All Developers for Homeassistant, of course mainly from NabuCasa, do an awesome Job they can be proud of. And i really appreciate it! 
Yet, the less breaking, the more Mass-Compatible it will become.
3 Likes
Just having something that filters for changes related to the integrations you run (or its dependencies) would already be tremendously helpful. I donāt mind reading. Itās the filtering thatās the hard, unnecessary part. GitHub issues are already labelled with the integration affected, so the underlying data exists.
EDIT: I know thereās a section in the release notes thatās grouped by integration, but thatās typically for breaking changes (aka backwards incompatible changes). Iām saying this for all changes (the full changelog).
Iāve had related comments in another channel recently. I donāt believe there needs to be monthly feature releases, or that there could be more of a balance between features, bug fixes and longer term stability.
That said, Iām not implying at all that the project is suffering. I think itās being managed pretty well. Itās an enormous project, both in size and complexity. My comments are to improve things on all fronts.
1 Like
Back on 2024.7.2 and fixed a few of my issues, but still can not figure out the chromecast issue. Is anyone else having issues with radio-browser casting to devices?
The onkyo integration does not have a unique ID so I can not get logs or do much troubleshooting from the UI. My HA is not that complex, and I have never had to dig very deep to find log info. Any recommendations where to start looking?
These discussions about update philosophy are good to have. And the fact that theyāve been staying (mostly) polite is great.
Thereās always going to be a tension between developers and users. Developers want to run on the latest and greatest hardware. They can imagine all kinds of cool features and want to add them all. They love to tinker and tweak, so when things break itās a fun challenge, not something that ruins their day. They like speaking a secret language of jargon which excludes outsiders. Some even like obfuscating code to make themselves seem smarter.
Users just want things to work, quietly and in the background, without a lot of effort, without having to learn a bunch of jargon, without having to buy new hardware and without having to research how every ābumpā change really impacts them.
Each group needs to understand - and listen to - the other if HA is to grow beyond a niche market of tinkerers.
8 Likes
TBH it became obvious to me once I got well down the rabbit hole of HA that I needed to make sure I had resilience in my system - via backups. If your system is mission critical then you shouldnāt be updating at all without a robust strategy in place. To me that means HA backups on device, in GoogleDrive and finally - importantly - image based backups (in my case via proxmox).
I can trial new releases and find problems and roll back - either to when I have time to deal with the problems or till they are solved in a later release. This way Iām also able to test for the project and raise issues or feedback for the team. I very much value the hard work on this project and I have realistic expectations. That said as others have pointed out we are all going to be using HACS for some stuff and thatās just something the project needs to live with. The ethos here is not to wall us in but allow us to play - and as end users we must accept some bumps on the road.
Iām a software engineer by profession for some decades (OK technically Iām management now) and the project(s) we work on are even more complex and very much hardware linked in terms of our product. This stuff aināt easy and the team here are doing an incredible job.
Remember, you very likely donāt need to upgrade - at least not immediately. Itās handy if you do find problems thoughā¦
7 Likes
Itās timing out. Start looking into the media_player youāre casting to.
1 Like
I like your point of view and thank you for your though, because I started thinking about what are the availability option today to achieve redundancy and high available. Iām happy to learn more if anyone would like to share it
1 Like
Iām having the same issue:
Also I have a 50/50 chance that my Alexa media player integration will load after a reboot. If I manually reload it itās usually fine though.
Alexa Media Player is on Life support, it does not have an active developer. I expect things to get worse as time goes on for that custom integration.
2 Likes
Is there a commercial product that meets those requirements?
Low effort.
No jargon.
No new hardware.
No need to read release notes.
Or are those requirements merely aspirational?
3 Likes
Oh good grief - I love this integration! Thanks, I guess, for the heads upā¦
1 Like
I see your point. But yeah, the successful ones donāt require me to read hundreds of lines of release notes, then follow and understand links to highly technical Github threads. Or figure out whether or not a ābumpā affects me. I generally just let my cell phone update without giving it a second thought. Many people do the same for Windows and the apps on it. I have multiple old machines and old apps which still work just fine with virtually no effort.
We who like fooling around with this stuff find it hard to believe, but most people will think of HA as sort of an appliance. Something you set up once and then it just works, doing what you need it to do, when you need it to. That should be the aspiration.
2 Likes
I have to call shenaniganās on this. I do this for a living. We have to actively turn off all updates so that hardware continues to work because windows updates constantly break things. OSās I manage: Windows NT, Windows XP, Windows 7, Windows 8, Windows 10. No windows 11 yet. Every single one has updates off because something breaks randomly when Microsoft pushes updates.
Recently (Last 12 months), Microsoft made speed improvements to COM communication in Windows 10 which flooded the buffer for some lasers that run on our assembly lines. The lasers simply couldnāt keep up with the buffer speed and thereās no fix for it from the laser vendor. We had to downgrade windows on all machines running the laser.
Dare I even bring up the whole Win10 FireWire (IEEE1394) deprecation?
EDIT: Oh, I had auto-updates on a Windows 10 box running BlueIris. The windows update borked the install and I had to wipe and reload. This happened in March, forgot about that. Needless to say, windows update service is disabled on that machine now.
8 Likes
If weāre being open to apples and oranges comparisons: my old car has required no updates ever and I didnāt need to read the manual to use it.
On a tangential note, people who recently allowed their iPhones to automatically update their Sonos app have learned they should have given it a second thought. (No rollbacks on iOS)
6 Likes
Typical open source software does not have the problems, Home Assistant has. At least not at this scale and frequency. Home Assistant is the only open source product I have a bad feeling with about every upgrade. You need to very carefully read the release notes and still need fear things breaking with every major update. Home Assistant feels more and more like a commercial product: Things only working through the UI, not through config files (YMMV, of course you still can modify config files, but thatās not supported anymore). Things breaking with every nth update because of arbitrary decisions the developers made.
You know what I do with home assistant updates? Wait at least until mid of month before upgrading to the next major release. And before that very carefully reading the community thread about the release and the summaries of all Github bug reports since the release date to know if still something will break and if yes at least get an idea of the blast radius.
7 Likes
How many other open-source home automation projects are being included in your comparison?
Because comparing it to all other open-source projects seems like weāre back to an apples and oranges comparison.
How may developers did you contact who have confirmed they make arbitrary decisions?
2 Likes