[Add testing]Let the community be the automated (e2e) QA team

I am a test engineer at work and a “hacker” at home. I don’t mind breaking some stuff here and there but for many consumers it will not do any good.

Lets add testing from the community, automated.

automated testing is great. It takes time to invest But it will be faster than any human can. The more and more integrations added to home assistant the more regression testing is needed. Sure there are unit tests. But some things shall not fail in a unit test.

Instead of scripts you have a test section where you can add test scripts (just looks like scripts) but with asserts that check the flow of the script and the end result.
a pack of test scripts is a test scenario that covers one (could be more but more isolated the better) integration.

running all the test scenarios should give a sign how good the new software performs.
Sharing this info with the rest of the world could give the opportunity for an given update to compare your local integrations with tests results done by others.
green all ok
yellow needs attention like a parameter is rewritten. (snapclient went from on/off to a lots of states system. breaking all my speaker switch templates )
red: does not work (see bug …)

This will make home assistant easier to maintain with less time spend.
I’m not sure to make a special test release or not. If this system exists i would make a virtual machine that does a smoke test before i (automatically) install it on my main system.

I think it is not that hard to include hacks integrations into the system. So that it will not be a gamble every month if all the hacks integrations work.