I understand testing requires some skill and its not for everyone - I’ve tested software for my employer, and found a major bug in one of our main programs prior to its release where the zip code field only accepted numbers, where some of our work is in Canada and they have letters in their postal codes. They promptly fixed it before the software went live.
I’ve posted issues to Github, including relevant logs with communication to and with developers, and have even submitted a few (admittedly minor) PRs to fix things. I’m not trying to “brag”, but I think I can be a successful tester. I also think the “end users” of Home Assistant are more technical and are a different breed versus the end user of say an Iphone, so I don’t think the importance of beta testing, in live environments with many devices that can’t always be replicated by developers, can be underestimated.
That being said, I also travel for work frequently a week at a time. There is no way I am going to install a beta test environment for my family remotely while I am out of town, so with the short window, I frequently can’t do the testing. I’m just saying at least a longer window would allow me, and likely other users, to test things out more frequently.
Software testing, particularly manual testing like acceptance testing and exploratory testing, goes beyond sporadic bug finding. It’s a structured, systematic process that requires a deep understanding of the software under test. This involves not just identifying what’s currently wrong, but also anticipating potential issues based on the changes made in the software. It’s about understanding the software’s intended behavior in a variety of scenarios and contexts, and verifying that it meets those expectations. At the same time it’s also about developing a deep understanding of the user’s needs and testing the intention vs. that expectation. A correct implementation can be wrong and a good tester files things that adhere 100% to the specification if the specification itself is wrong.
Beta testing is great at uncovering usability issues, compatibility problems, minor functional bugs, real world usage scenarios, l10n & i18n. It’s not so great at uncovering deep technical bugs, broken specification, things happening at scale or edge cases. Beta testing is also great at testing your testing process. Whenever you do find more than expected¹ deep technical bugs, broken specification, things happening at scale or in edge cases, your testing process is flawed, as it is unable to uncover the things before or your development process is flawed, as it is unable to properly fix these things before. Worse is when end users are finding these kind of things. These things are either not uncovered by the testing before the release or worse ignored or worst declared “by design” by the developers. And this is, where I feel we are at the moment with home assistant.
And if that’s where the developers want to go, that’s fine. The vast majority of contributors to home assistant is doing that on their free time. They don’t need to justify. They don’t need to care for their user’s preferences and problems. The need to feel happy, it’s their hobby. It’s just that I don’t feel that this direction and priorities are good, and I wanted to start a discussion if that direction should be changed.
¹ No testing, no matter how thorough it is, will uncover all bugs. But with a mature testing process you can pretty precisely predict, how many bugs you did not find and are expected to be found in beta testing or in production. There are techniques to increase the yield of beta testing leading to less unknown bugs in production.
Thank for raising this topic. I want to subscribe to a few remarks above:
I understand removing weather sensor attributes, but i can’t understand the deprecation of get_forecast. It smells engineering delight causing a lot of rework for end users. The new service call easily can coexist and should be named more differently. I.e: advanced_get_forecasts to express the usage is less easy. I doubt querying multiple weather sensors is the typical use.
moving away from yaml is fine, but deal with configurations in the same way as scripts and automations: support both. Note the UI simply isn’t offering all yaml options ( like the initial value for select).
The art of excellent user experience is to facilitate both a good entry level and advanced views (e.g switch to yaml in automations is great)