I would suggest that nabu casa hire software testers. There are usually several updates made to each months release. To me, this indicates the release wasn’t tested well enough before it was published.
I worked in software verification for a major medical imaging company based in the Netherlands. Ideally, we would write a test suite that was run before every release. When new bugs or regressions were found in testing or the real world, new tests were written and added to the suite.
I know the variability of home assistant installations can make this a difficult task, but it could improve the quality of the product.
Also, such tests are normally written against the written requirements for the product. I don’t know if anything like that exists.
This exists already, and it runs every build and on every single PR that’s created (even on PRs that don’t make it into the software).
Every month there’s a week long beta that occurs where thousands of users test the new build of HA before it’s released.
You are trying to compare open source software written by a community of volunteers trying to find a way to access the devices and data they own which is not always allowed or otherwise clearly documented from the manufacturer, to a highly specialized company who is commissioned and paid millions to write purpose built programming that does not change unless a customer requests changes.
You are more than welcome to contribute to the beta testing and write a PR to fix any bugs you may discover.
I honestly don’t see why everyone thinks it’s perfectly normal for a commercial company to hire devs to write code for open source software, but somehow the idea of the same company hiring professional testers is met with hostile reaction, because this is open source and there are community beta testers.
In regards to traditional software testers, the developers are testers. Traditional software testers write tests for the code that developers wrote. This is already handled because tests are required for every PR.
In regards to hiring people, I really don’t think it’s needed. Beta uncovers a ton of issues because we have thousands people testing the beta. Adding a single person to that pool of people isn’t really going to make a huge difference.
However, if that money went towards adding another developer (who’s salaries are typically similar to testers)… we would have another person fixing issues with HA. Which would be a great benefit because we realistically only have less than 100 dedicated developers in HA.
So, in my mind, it makes way more sense to hire additional developers than testers. Contrary to what people believe, HA has a really good test process.
We must live in vastly different realities.
Where I live, the difference between a software tester and a non-principle software developer is about $10k on an already high salary.
I did not know anything about the process used by nabu casa. I only see a release, then several patch releases each month.
If bugs found in beta and after release prompt new tests to be added to each developer’s test suite, that is a step in the right direction.
Isn’t there a release planned every Friday? The minor releases between main releases are not usually pushed just to patch things that the major release has broken…
Yep, that’s how it currently works.
for actual QA testing this would be improbable there are 1000s of devices that need to be sufficiently tested on each and every release that those components are updated.
one question I have is asides from unit testing what automated frontend tests are there? are tools like testim and webdriverio used at all?
unit testing only tells you so much, end to end automated testing does much more.
I’m enrolled in the beta testing for HA. My brother in law used to be a dedicated software test engineer for a telecom company. From what he described what he did in his job, I think he would easily be worth a 1000 times me when it comes to effectiveness of finding bugs. I don’t actually spend time testing, let alone write code specifically meant to stress the system and expose errors. I just install and check if the lights still go on and off, basically.
I don’t think I’ve ever seen a comment section on a release notes that didn’t include a few bona fide “breaking changes that weren’t mentioned in the notes” if you know what I mean. So I think there is room for improvement in the testing process. That doesn’t mean it should have priority over development of course.
It would be an interesting blog post to describe the testing process sometime. How does that work? Is there a server with a thousand virtual machines that emulate different user setups using the shared data about them? How do you anticipate the behavior of external devices?
Such a blog post might also attract more people to the beta testing program, which would be another way to improve testing.
you are correct that a BonaFide ‘tester’ would do MILES beyond what a ‘beta tester’ would do for the EXACT reason petro states. Testers are like super devs - if you hire good testers… (Theres a reason windows betas are so long - and usually the test results by the end users only find a very few bugs that the dev team didn’t)
Testers aren’t just people who test. The companies doing it ‘correctly’ testers are rather senior devs - (read $$$) because their entire job is to know how to code AND ‘think like a user’ and devise ways to break it AND then translate what they broke into actual actionable bug reports, so the coders have a shot in hell in figuring out what’s broken.
With respect to @dshokouhi and the rest of the team - I think automated testing is table stakes at this point, and if you’re not doing it you’re flat out wrong - good devops is good devops. (glad to hear that HA is using automated testing - every single feature should have a test case assigned as it checks in and it should run against the regression set) But (speaking with MANY years of experience with a few pieces of software MANY of us use every day) There is NO substitute for a good tester on staff. I’ve seen teams try to shed cost by relying entirely on automated testing - eliminating the testers and EVERY SINGLE TIME the code suffers. Sorry - good devs, like surgeons have egos - BUT they need to be popped now and then 
Knowing ‘Joe’ down the hall WILL find your crappy bugs does a lot… So would I support paying for a tester - sure (see next)
But because of the reality of cost, this person is probably also splitting time as a product mgr. who interfaces with the client success team (modern speak for helpdesk) to ensure that bugs from the field are also being addressed. (read: more of Frenk… Who’s hair is probably always on fire. - we see you man - keep it up.)
Beta testers are NECESSARY - because the best test suites can’t account for every single configuration. We as end-users need to realize that for every one of our configurations - there’s at least 100 others who have DIFFERENT configurations. Automated testing CANNOT even begin to address the permutations. It just gets the does it do what it’s supposed to do. Beta testers catch the permutations and combinations. BUT Beta testers - it’s not just the privilege of running it early, it means ACTUALLY testing out the things that are important to YOU and REPORTING them reliably, early and often - and if they’re not addressed, yell before it makes it out. (This is why I DO NOT test btw - I don’t have the time to give quality results, I’d be a looky-loo and 12 years of dealing with it left a bad taste in my mouth.)
I don’t know the answer because Petro is also correct in allocation of funding. (remember testers ain’t cheap) Right now - from what I can see on the outside - the entire HA project could use AT LEAST 5 more fulltime devs and still have more work to keep everyone busy - just killing bugs. But I’m also sure the funds aren’t there else they would have them already.
So if you look at the tools available to dev to squash bugs and had to deal with HIGH amounts of divergent configurations and had these options:
Senior dev / Team lead
Automated Testing
Unit testing
Beta Testing
I’d do pretty much the same EXACT thing the HA team already does, and spend my $ on senior dev as possible, lean heavy in automated testing, skip the Test engineer and limit unit testing (they’re SPENSIVE) and then beta test the heck out of it on as many configs as I can. Yes HA seems to have better quality betas than something like Windows (Canary and Beta for MS in consumerland is pretty much looky-loo land with rare feedback, server is way better) But IMHO the HA betas are WAY too fast for me to believe any good results on the permutations are coming back. I’d be really interested in seeing what kind of coverage they’re getting on the configs. IMHO, a week is barely enough time to know anything happened and god help you if you need to fix anything.
What’s the answer? Well unless someone wants to throw 100K at Paulus for hiring subsidizing a tester (because you wont pay for them 100% at 100K) Im probably going on a crusade to get BETTER beta testers. Actively pursuing testers with odd configurations or weird use cases who will actually report bugs. Unfortunately, good betas take work and slow releases.
But I’d also probably try to slow the feature train. Put a higher quality bar to release.
Probably not what people want to hear - but you can get new, or you can get bugs fixed - you cant have both. It’s this that’s driving the ever-louder yell for an LTS build (and IMHO I’m in that crowd)
It’s a hard problem indeed - but just blindly ‘hire some testers’ isnt the answer.
I do agree that a full time tester would help a lot specifically to find bugs in the frontend (even if currently some frontend devs do seems to work only on this).
But HA uses a lot of third party librairies that are not maintained by Nabu Casa but by third party developers. So even if Nabu Casa hire a QA dev, what if a bug is found in a third party library?
If you check carefully each minor release, there’s little to 0 patch to HA core code (because it is fully tested). It’s nearly always bug patches to third party libraries or integrations (not internal)