Google-Nest Offical Device Access Console Finally Released!

[allenporter]

To clarify, you won’t have to worry about this in 2021.01.

1. The next release handles reauth with a ui flow that won’t add remove the devices. https://github.com/home-assistant/core/pull/44202 so even if you did hit this again and your decide IDs aren’t stable they won’t be added and removed anymore. I didn’t mention this in great detail but this was already merged by the time you reported this.

2. I added debugging to help you diagnose with Google why your access token is getting invalidated https://github.com/home-assistant/core/issues/44584. – see the blog post linked in the issue for the many reasons why this happens and maybe you can understand how it’s difficult to diagnose from here

3. They are deterministic based on the device IDs from the nest api. Did the device IDs from the nest api? Did you in fact see that they changed when you reauthed? If so, I’m happy to investigate. I only said “supposed to” because I didn’t prove it myself yet. I was hoping you could report back with what happened: apologies I couldn’t look closer I’ve been busy making other authentication related improvements and working through ways to simplify the setup flow with the Nabu Casa folks.

[allenporter]

I think most people use the UI to create automations at least initially to get the device IDs based on the device names from he drop down. That is what I do to start, anyway.

You should start a separate thread on device automations so we can get your device automation feedback some proper discussion. I’m not saying this to be dismissive but really because (1) I think it’s a separate issue that will get lost here (2) If you actually want to see it change, it will be more effective to discuss in a more focused way. Fwiw I mildly agree as I think entity names are simpler to work with and all of my devices and entities with automations have a 1:1 mapping.

[dbrunt (Daniel)]

@joesilva101

Just wondering if what I am experiencing has been seen by any of you.
following the instructions in https://www.home-assistant.io/integrations/nest section Device Setup, I can complete all of the OAuth steps but after clicking “Allow” on the final step, the web page closes and reverts back to HA but nothing happens. the pop-up that initiates the entire process is still being displayed.
closing it and checking the logs, no entries at all. my system has all of the updates.
any feedback on what I may be missing is appreciated.

Yes I had that happen to me at one point in my trial and error process. I wasn’t doing something correctly but what that was exactly now I don’t recall as I had done many thing wrong in my numerous attempts…

[dbrunt (Daniel)]

@Giuseppe_Torres

I seem to have reduced the duration of the OFFLINE times (when turning all WiFi devices in the house off, and using a dedicated 2.4GHz WiFi for my Thermostat) to less than a couple of minutes (I had around 30 minutes, worst case, before), by adding the Earth (which was absent before, my bad) connection - [I have a Nest Thermostat Learning (Learning what?), sending 12V DC from T1-T2 to the backplate with the Thermostat - I had previously said I had a C-Wire, well, I was slightly wrong].
Not sure if the Earth connection has made things any better, because, when looking at the DEBUG of the sdm.devices.traits.Connectivity, it seems to go OFFLINE more frequently than before.

Thermostat wiring:
2-wire - contact closes the circuit completing the 24VAC through the furnace to the gas valve, etc. so when contact is open there’s no source of 24VAC for power as there is no “ground” back to the transformer in the furnace without sending it through the gas valve which may activate it. I’ve seen some thermostats which can do this but it’s iffy.
3-wire: Adds a C-Wire which is wired directly back to the other side of the transformer in the furnace so it plus the “hot” wire from the transformer can be used as a 24VAC power source.
Grounding the thermostat C-wire connection to earth may work depending on the furnace wiring. I’ve not had any luck trying this.

Are you 100% sure the Nest has a solid Wi-Fi connection?
Is it able to resolve DNS 100% of the time?
Is your internet connection solid 100% of the time?

2.4 GHz Wi-Fi can be disrupted by… microwave ovens is close proximity, baby-monitors, cordless phones using 2.4, etc., etc.

You mentioned 12VDC…is that a typo?

[joesilva101 (Joe Silva)]

The integration never appears. I waited 30mins and it never appeared in “Configuration > Integrations”. I’ve been reviewing the logs from the front-end (configuration > Logs) and even checking from the terminal just to be sure something isn’t being missed (in /config/home-assistant.log). I also turned the default logger level to debug. In the logs I can see the request come in and it entering a data_entry_flow:

2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.components.http.view] Serving /auth/external/callback to 10.58.1.1 (auth: False)
2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event data_entry_flow_progressed[L]: handler=nest, flow_id=c440740a76934c6eb6c72d03872ea21c, refresh=True>
2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [547865054224] Sending {"id": 11, "type": "event", "event": {"event_type": "data_entry_flow_progressed", "data": {"handler": "nest", "flow_id": "c440740a76934c6eb6c72d03872ea21c", "refresh": true}, "origin": "LOCAL", "time_fired": "2020-12-31T21:05:51.239313+00:00", "context": {"id": "f1d14269929ace1a49f9fb5a538af855", "parent_id": null, "user_id": null}}}

There are a couple more lines after that about the data flow progressing but then that’s it, nothing saying “error thing broken” or a similar sentiment. When I search the logs for anything from homeassistant.components.nest or any of its subcomponents I get zero results. I have no logs coming from any homeassistant.components.nest components.

Is there something specific I can search the log files for? Further debugging options I can change?
My logger config looks like this:

logger:
  default: debug
  logs:
    homeassistant.components.nest: debug
    homeassistant.components.nest.climate_sdm: debug
    homeassistant.components.nest.camera_sdm: debug
    homeassistant.components.nest.sensor_sdm: debug
    google_nest_sdm: debug
    google_nest_sdm.device: debug
    google_nest_sdm.device_manager: debug
    google_nest_sdm.google_nest_subscriber: debug
    google_nest_sdm.event: debug
    google.cloud.pubsub_v1: debug
    google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager: debug

I’ve gone over my nest config in /config/configuration.yaml multiple times and verified it is being picked up correctly by putting bad/incorrect values in. Even got a second pair of eyes to go over all the configuration steps (thank you son) to ensure I didn’t typo/copy the wrong thing somewhere.

Any tips or advice for further troubleshooting would be greatly appreciated.

[allenporter]

If it is not on “Configuration > Integrations” then it never succeed the “config flow” for OAuth and didn’t create a “Config Entry”. (No waiting needed, it will either show up or it won’t)

There is something going wrong in the OAuth setup.

• As for additional debugging info, you may want to add homeassistant.helpers.config_entry_flow and homeassistant.helpers.config_entry_oauth2_flow to the list of debug log components and go through the setup flow again
• I would suggest double checking all the oauth creds/consent related setup again.

Let us know how it goes.

[dbrunt (Daniel)]

@joesilva101

The integration never appears…In the logs I can see the request come in and it entering a data_entry_flow :

Copy to clipboard

2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.components.http.view] Serving /auth/external/callback to 10.58.1.1 (auth: False)
2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event data_entry_flow_progressed[L]: handler=nest, flow_id=c440740a76934c6eb6c72d03872ea21c, refresh=True>
2020-12-31 16:05:51 DEBUG (MainThread) [homeassistant.components.websocket_api.http.conne

What IP is this?
“Serving /auth/external/callback to 10.58.1.1 (auth: False)”

Is your browser connected to HA UI via your public IP or private IP?

[joesilva101 (Joe Silva)]

Thanks so much! I will give those a try.

@dbrunt That IP address is the router sitting between my HA server and the internet

[dbrunt (Daniel)]

That does not seem right to me…
@allenporter ?

[allenporter]

I assume this is a port forwarding kind of thing happening, but unclear to me.

[Giuseppe_Torres]

Thanks for looking at my problem

@dbrunt

You mentioned 12VDC…is that a typo?

No, this is what I have:

The installation guide states I should connect the Earth. In the initial installation I did not connect any Earth. I never had any problems until a few months ago (I changed Internet supplier, modem/router, in the middle).
I also had an “accident” of dropping the thermostat, though I think the OFFLINE issue was present already before then.

@dbrunt

Are you 100% sure the Nest has a solid Wi-Fi connection?
Is it able to resolve DNS 100% of the time?
Is your internet connection solid 100% of the time?

My Wifi is pretty solid where the Thermostat is. I am now using a dedicated WiFi point exclusively for the Thermostat.
As for Internet, as I said, I changed provider around 12 months ago. Not sure this is related to it, though I must say I am not totally happy with the ISP. Speed is pretty variable, so are the pings, but mainly in the evening I get dropouts in speeds and pings can get closer to 150msec.
The frequency of the OFFLINE events, though, is much higher, and I get these events rather randomly.

As I said before, after adding the Earth this morning, the frequency of OFFLINE events reduced greatly.
But still not enough to remove the events completely.

HAPPY NEW YEAR!!!

[dbrunt (Daniel)]

Is your browser connected to HA via your public IP while adding the integration?

[dbrunt (Daniel)]

See this: https://www.google.com/search?q=nest+thermostat+supply+voltage&oq=nest+thermostat+supply+voltage&aqs=chrome..69i57.6716j0j7&sourceid=chrome&ie=UTF-8
12VDC might be your problem?

[dbrunt (Daniel)]

Also this… https://support.google.com/googlenest/answer/9241211?hl=en

In the Nest app, go to settings, Technical Info and see what your battery voitage currently is.

[joesilva101 (Joe Silva)]

Correct it is a port forwarding setup. I have the router routing traffic from port 443 externally to port 8123 on the HA server. It’s a Linksys router. Do you think it’s not routing properly? I didn’t think it could manipulate the traffic when it’s singed by an SSL cert. I’ve never had issues with port forwarding. I also have a VPN that I port forward to and that works fine.

Is your browser connected to HA via your public IP while adding the integration?

Yup, I have it setup with a DuckDNS domain and I’m accessing HA through that domain. It’s also setup with a Let’s Encrypt certificate.

@allenporter I added those debugging lines but I’m getting zero results for homeassistant.helpers.config_entry_flow and homeassistant.helpers.config_entry_oauth2_flow in home-assistant.log. I can see them initializing with the correct debug level:

2020-12-31 17:17:56 DEBUG (MainThread) 
[homeassistant.components.upnp] 
async_setup, config: 
OrderedDict([('default_config', {}), ('logger', OrderedDict([('default', 'debug'), ('logs', OrderedDict([('homeassistant.helpers.config_entry_flow', 'debug'), ('homeassistant.helpers.config_entry_oauth2_flow', 'debug')]....<omitting the rest>

I can also see nest picking up all the config data from configuration.yaml later in the same log line. I’ve checked the credentials again (and a second time by my son) to ensure we’re using the correct credentials. Famous last words but I’m VERY confident that the credentials are correct. We’ve CTRL+F’d all credentials to ensure they’re the same everywhere.

Again, the callback log lines, adding some more lines from the /api/config/config_entries/flow/ call which I didn’t include last time.

2020-12-31 17:19:33 DEBUG (MainThread) [homeassistant.components.http.view] Serving /auth/external/callback to 10.58.1.1 (auth: False)
2020-12-31 17:19:33 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event data_entry_flow_progressed[L]: handler=nest, flow_id=e3a3cfbad2cb43b899935d02331d2be2, refresh=True>
2020-12-31 17:19:33 DEBUG (MainThread) [homeassistant.components.websocket_api.http.connection] [547492612992] Sending {"id": 31, "type": "event", "event": {"event_type": "data_entry_flow_progressed", "data": {"handler": "nest", "flow_id": "e3a3cfbad2cb43b899935d02331d2be2", "refresh": true}, "origin": "LOCAL", "time_fired": "2020-12-31T22:19:33.748596+00:00", "context": {"id": "4b2b21bc203146dc5dfbfbe7dfc6cab9", "parent_id": null, "user_id": null}}}
2020-12-31 17:19:33 DEBUG (MainThread) [homeassistant.components.http.auth] Authenticated 10.58.1.1 for /api/config/config_entries/flow/e3a3cfbad2cb43b899935d02331d2be2 using bearer token
2020-12-31 17:19:33 DEBUG (MainThread) [homeassistant.components.http.view] Serving /api/config/config_entries/flow/e3a3cfbad2cb43b899935d02331d2be2 to 10.58.1.1 (auth: True)
...<farther down, same flow ID>
2020-12-31 17:19:38 DEBUG (MainThread) [homeassistant.components.http.auth] Authenticated 10.58.1.1 for /api/config/config_entries/flow/e3a3cfbad2cb43b899935d02331d2be2 using bearer token
2020-12-31 17:19:38 DEBUG (MainThread) [homeassistant.components.http.view] Serving /api/config/config_entries/flow/e3a3cfbad2cb43b899935d02331d2be2 to 10.58.1.1 (auth: True)
2020-12-31 17:19:38 DEBUG (MainThread) [homeassistant.components.http.auth] Authenticated 10.58.1.1 for /api/config/config_entries/entry using bearer token
2020-12-31 17:19:38 DEBUG (MainThread) [homeassistant.components.http.view] Serving /api/config/config_entries/entry to 10.58.1.1 (auth: True)

and then it’s the end of the log file after a few more unrelated lines.

I guess the main thing I’m wondering now is why am I not seeing any log lines for the OAuth flow??

Thanks so much for the suggestions so far, from both of you!

[jrlhass (James)]

I agree that there should be a separate thread on this, as these design decisions sit far above the level of the Nest integration. I just have no faith that there will be traction, and I get a little exhausted trying to interpret the economic and political motivations of the founders.

I find HA useful as a platform for accessing the drivers and interfaces written by many talented devs (such as yourself). The logical layer above that is very frustrating, and I find myself reproducing a lot of the functionality locally (thank god for AppDaemon).

I have see 100 threads asking about why HA has to be restarted to, say, add a single new binary_sensor. And there is no substantive discussion of why this is the case, whether there is a principled reason behind it, and whether the entire ecosystem has passed the point of no return.

The front end should be a logically independent entity, so the fact that even making use of integrations is being tied to the front end is maddening, and it means that the most valuable part of HA for me (the integrations) are going to become increasingly more frustrating. If I go into the front end to try to create an automation based on a device trigger, I have over 200 devices, many with the same name (b/c they are derived from the “Friendly Name” which is often auto-populated based on metadata). I also have 6 Nest cameras and 2 thermostats, so now I need to create 8 template automations and keep making up fields until I’ve input enough data so that I can make a fake automation and copy-paste the device_id (which is not guaranteed to remain fixed). This is by design…

[dbrunt (Daniel)]

I’m not sure now why I have 443:443 but it may have been for an Alexa custom skill I setup for HA (TTS?)

I think I only had 8123:8123 when I set up Nest.
If you can, try 8123:8123 on your firewall and re-do the nest setup for the n-teenth time!

[allenporter]

OK but again @jrlhass if your device ids are not stable let me know and we’ll figure out why. again (1) they “should” be (i just read the code and i can see where they looking deleted device ids and reuse them), and (2) the reauth fix has been submitted so you won’t be adding/removing devices anyway (3) i’ve added debugging to help you understand your auth issues. I worry there is a separate issue to tackle here… Best of luck and let us know if we can help you. (also check out blueprints :))

[allenporter]

My last suggestion is to try to capture the HTTP redirects / request flow via chrome developer tools or something. You might see a useful error message in an error response somewhere.

[dbrunt (Daniel)]

@jrlhass

I have see 100 threads asking about why HA has to be restarted to, say, add a single new binary_sensor.

HA has made some progress in addressing this just in the 7 months I’ve been using HA. There are a lot more reload options to pick from and integrations seem to slowly be incorporating a reload option. Custom non-templated sensors, binary_sensors, lights, etc. would be a welcome addition!