Change of authentication credentials should be a requirement for all official integrations

[KennethLavrsen]

Many integrations have been migrated from YAML to graphical user interface the past years and there has been many oppinions about it. I do not want to recreate that debate again. But there is one mistake that has been repeated many times.

When an integration was migrated, the developer did not implement the UI so you can change the configuration settings. Or he/she only implemented some of the configuration.

Many have since fixed this by allowing doing a reconfiguration that takes you through the same steps as when you install the integration the first time. This may not be very clear but at least it is possible.

But there are integrations that do not do this. And the worst of them make the end users skip the most basic security practices. These bad integrations do not allow changing username and password used to authenticate the device that it integrates. Only way to update credentials is to remove the entire integration and start all over from scratch.

A really bad example is the ONVIF integration.

You can give username and password when you install a camera the first time. But when you want to rotate credentials after maybe a year or two you cannot change it. You have to delete the integration and re-add the cameras. You may have changed some entity names. That you need to do again.

My feature proposal is that it should be a requirement for an integration to be accepted or remain part of the official distribution, that as a minimum authentication credentials must be possible to change via the UI if the integration is working with UI and not yaml.

I raised a bug against several integrations that has this flaw back in 2021. Some have been resolved by at least allowing a full re-configuration flow. But ONVIF remains untouched. I just checked. A reconfiguration only shows this

You cannot change the URL of the camera. You cannot change the username or the password.

As a bare minimum - if you consider yourself concerned with security - the UI should allow entering a new password.

I cannot understand how these integrations could be changed from YAML to GUI this way. I am not against making the HA experience more user friendly by moving from geeky YAML to a nice UI. But this is not a nice UI. This encourages poor security behaviour because it has become a painful experience to change a password in a camera.

New YAML to GUI moves should as a minimum provide means to change credentials to be allowed. And for those that have been moved, the lack of this should be considered a bug and not a feature proposal. Asking for the ability to change passwords should not be a feature proposal.

See how the onvif bug was treated here

ONVIF integration does not allow changing original config settings incl changing password

[petro]

Reconfiguration should be a requirement for all integrations.

[KennethLavrsen]

I’d like to give a good example of one of the integrations that were converted from YAML to GUI and does it right.

The generic camera integration. Let us say I want to change credentials on camera 2. We find the integration in the GUI

Select camera 2

Hit configure. And now we can alter the URL if the camera has been changed to a new IP. And notice that you can enter new credentials. The existing are not shown which is good and safe.

If you scroll down there are a few more things you can alter. Super GUI and very user friendly. And secure as it enables us to rotate credentials which became very much a must for many of us after the latest HA security issue.

This flow is exactly the same flow as when you setup the camera the first time so the code to do this should already be there. It is simply a matter of repeating it when you hit configure. Surely it would be even more user friendly if you could just edit the one field you want to change but I am personally 100% ok with going through a complete config flow again as long as my existing values are there for URL etc. and all my device names and entity IDs remain unchanged after the reconfig.

[KennethLavrsen]

One more post on my own proposal in case people find this with a search

The hacker way to change credentials and the one I use since there is no better way is to edit file you should not edit

in config/.storage/core.config_entries you find the credentials and URLs for various integrations

One mistake in the editing and HA will crash and may not restart. Make a backup of HA. Make a backup of the file before you change anything

For onvif a camera looks like this

      {
        "entry_id": "169854b93b6cdb8aa67cda55e3d50b13",
        "version": 1,
        "domain": "onvif",
        "title": "ThreeD - 00:B0:3A:A4:27:32",
        "data": {
          "port": 8080,
          "name": "ThreeD",
          "host": "192.168.1.150",
          "username": "admin",
          "password": "supersecretpassword",
          "snapshot_auth": "basic"
        },
        "options": {
          "extra_arguments": "-pred 1",
          "rtsp_transport": "tcp",
          "use_wallclock_as_timestamps": false
        },
        "pref_disable_new_entities": false,
        "pref_disable_polling": false,
        "source": "user",
        "unique_id": "00:B0:3A:A4:27:32",
        "disabled_by": null
      },

You simply open the file in an editor. Change the password. Save the file. And immediately restart home assistant. Do not do anything else from you open the file and till you save it to avoid that HA wrote something else to the file that will be lost.

There is a significant risk doing this and it should not be needed. These files are not for human editing. But as long as things are implemented as they are - it is the only alternative to removing integrations and setting them up again with the problems that this gives.

I wish someone that can code Python would make a HACS thing that can more safely hack things in the core.* files in .storage. It happens too often that the only way to get rid of orphan devices and entities is hacking these files and missing one comma and it all goes down.

[petro]

@KennethLavrsen

I wish someone that can code Python would make a HACS thing that can more safely hack things in the core.* files in .storage. It happens too often that the only way to get rid of orphan devices and entities is hacking these files and missing one comma and it all goes down.

Well that wouldn’t work because you have to edit the storage files when HA is off, otherwise HA will overwrite information in storage as it restarts or shutdown.

[Olivier1974 (Olivier Toussaint)]

I completely agree with you.
Even partially UI-based integrations should have a way to manage credentials, tokens or API keys.

Let me share a personal story that highlights this issue:

Occasionally, my SmartThings integration malfunctions and repeatedly reports an invalid token for my client. To resolve this issue, I go back to the SmartThings integration page to get the link to the token page, delete my token, then create a new one. However, this is where things get frustrating. I have to remove the SmartThings integration and then recreate it since there is no way to update the token otherwise.

To make matters worse, sometimes it creates all my entities twice, with a _2 appended to each and every one. This is extremely annoying.

In my latest encounter with this issue, the integration was not even restarting after the manipulation. It displayed an error message stating “unique id blablablabla is already in use by sensor.i_can_not_remember_2”. I had to stop Home Assistant, go into the .storage directory, find and remove all SmartThings related entries, and then finally restart Home Assistant without any errors.

This statement is acknowledging that it can be challenging to communicate issues or feature requests to the developer of an integration. I fully recognize that the developers work on integrations in their spare/free time and their efforts are greatly appreciated. However, users may still experience frustration when issues arise or desired features are not available.

[nickrout]

I query whether this should be an architecture discussion?

[kidnor (Per)]

I agree… this is a big weakness and creates a complete mess. Drags the Home Assistant user experience down to a low standard.
Home Assistant architecture should have a structure on how usernames, passwords or authentications are handled in ALL integrations. There should be clear guidelines for for all developer on how this should be implemented when building the integration. Similar to how other rules are setup.
Any user or person setting up a Home Assistent system should have the option to admin, input… add or change any username or password for an integration from the UI. Make it simple.

Gone through the bloggs… No-one seem to be interested in this… !? How to campaign for this to get the opinion to the top of the list?

[RobertD502]

Not quite. You can edit files in the .storage directory (although not recommended) while Home Assistant is running, save the changes, restart Home Assistant, and the changes will persist. I do this if I need to quickly check a config flow change I made for one my custom components. If you take the config entry file, for example, and add an extra letter to a username for one of the config entries that supports a reauth flow, once you restart Home Assistant you’ll see that your changes weren’t overwritten as the reauth flow gets triggered for the integration you just edited.

Although not very user friendly, if you change the password for an integration, you can open the config entry file, locate the entry for the integration in question, replace the password with the new password, restart Home Assistant, and you’re up and running with the new password instead of having to delete the integration and readding it with the new password.

[KennethLavrsen]

That is how I do it too. More often I hack these files to remove orphaned devices though the Spook custom component is also capable of getting rid of them so I try that first.
It should not be needed to do this. The files in .storage are not meant for human editing and if you leave one comma wrong HA will not start. So ALWAYS make a backup of the file and make sure you can access the .storage files even with HA stopped before you attempt this kind of hack.

I wish some Python guru would make a hacking custom component that can edit fields and delete entire orphaned devices and entities with a UI that at least leaves a valid json file. That would be a popular contribution

The main problem with hacking with HA open is that you open the file in an editor and do your edits and then save. This save will overwrite anything HA did meanwhile. Best is that HA is stopped! And if not plan your edit carefully and do the edit and save within a few seconds and always do it when the HA is quiet. Never when you are installing or upgrading or addings things. Ideally this should be done atomic from within the HA API. I just cannot do that. I find Python so different from the C, Pascal, and Perl that I know

[RobertD502]

If something like this was created (with a custom component that exposes a custom service call which will edit a config entry), it would still require the user to know the exact key that they need to edit in order to make the change - at this point, a user that knows this will already know how to manually edit the file.

For example, the username for the ring integration is an email. The ring integration uses the username config key. For an end user of the custom component that you suggested, if they changed their email, would need to know that the email is stored in the username key and not in a key titled email. They would only know this by looking at the config_entries file at which point they are knowledgeable enough to make the change manually.

For what you’re asking, the most straight forward way around this is to have integrations add the option of changing the username/email, password/API key via the configure option in the UI. Another way would be for Home Assistant to build out handling application credentials in a visual manner such as the way some integrations handle OAuth credentials (although editing current client IDs/secrets isn’t an option in the credentials menu).

However, all of this can be avoided by having integrations implement a reauth step for integrations that rely on credentials. If an integration is setup to handle a reauthorization, the integration should already ask the user to enter a new valid username/password in the event that credentials were changed/are no longer valid.

In @Olivier1974’s case, the smartthings integration doesn’t look to have a reauth step (no async def async_step_reauth function in config_flow.py). If this were added/a ConfigEntryAuthFailed exception raised when credentials are deemed invalid, a reauth would allow them to enter the new credentials via the UI without having to delete the integration and readding it.

[KennethLavrsen]

We can all agree that hacking with any tool should never be needed.

But while we wait for that…

… a custom component could do the editing more atomic.

When you manually edit and leave Home Assistant running because you access the UI for an editor through the HA front end, then there is a risk that HA writes to the file while you have the file open in an editor and when you save, those changes are reverted. That could potentially create severe problems. So best is to do it with HA itself stopped.

A good tool would either write to the file through HA or at least do the change by merging any changing that happen concurrently to the new file. This is important

[RobertD502]

If HA is stopped, the custom component wouldn’t be running to do what is asked of it.

[KennethLavrsen]

Correct.

But it can load the file, display the relevant content you search for data you can edit and when you commit the data, you reload the file, change the data, and save the merged result leaving open a time window of a fraction of a second, versus, maybe minutes when you manually edit using a text editor. I do not know the inner engine of HA but there may also be a way to force HA to flush data in ram to the file before. Or the component could hold the altered data in a temp file, restart HA and merge. The two dangers of hacking are human goof up (so backup) and overwriting data when saving. As long as you edit files related to devices, the risk is relatively low as long as you do not try to add or remove anything in parallel. You do not want to start a permit join in your zigbee network while you have the file open in an editor unless you like to punish yourself

[RobertD502]

Makes sense. The config entries file, for example, is not going to be changed by Home Assistant while you’re editing the file manually (unless while editing the file you are adding a new integration), so the risk there is pretty much non-existent (aside from a user messing up formatting). If the desired outcome is to truly have Home Assistant stopped while the files are being edited, then you don’t have to rely on knowing python to build out a solution. You can create a standalone program or a script (with a user interface within a terminal window) that allows for searching for an integration by name and parsing the json file to find the related config entry where a user can then choose to edit specific fields. Seems like a lot more work than is worth when in reality, the only time credentials would need to be changed can already be handled with a reauth flow, but integrations that use credentials need to implement it.

[lonevvolf]

I recently ran into this on the Tado integration, so I started digging into the code, and I see that it’s actually an issue for pretty much every integration. The reason seems to stem from this design decision:
ConfigFlowHandler and OptionsFlowHandler - managing the same parameter - Development - Home Assistant Community (home-assistant.io)

and this:
Ability to edit/update config entries · Issue #377 · home-assistant/architecture (github.com)

Put simply, Home Assistant stores settings which are set on first setup (config) separately from settings which are changed after the first setup (options). This leaves the extra effort of “changing” settings up to the integration developer. Judging by the response on that second link, no one is really looking into improving this. I would bet if the two configuration flows were able to read/write from the same place, this would be pretty straightforward for developers to offer the functionality we’re looking for.
I can’t test it at the moment, but looking at the code for the Generic Camera plugin, it looks like changing the settings after first configuration will leave two entries in the config files. Maybe someone else can confirm this.

[petro]

Typically, you can just re-add the integration with your existing integration in place to replace the current one. And it will.

[bkbartk]

can we then also say that
configuration and reconfiguration should be the same wizard?
so a dev only has to create it once.

I had an integration where I made a mistake in a field which was only there on configuration, not on reconfig so I had to re-add my device.

[petro]

You didn’t have to re-add the device, you could have just re-configured it by going to add integration and add it again with the existing integration in place. This works for most integrations. It will replace the current integration but keep all your settings.

[bkbartk]

I wonder how that would work, I do not own the device I was talking about anymore.
but if I take a look at for instance my printer

Let’s say I used the wrong relative path.
while readding,
I still have to configure everything else.
where a configure button with all options I use while adding would help me out.