Configuring Home Assistant, Mosquito broker, DuckDNS, and Node-Red to control lights

keanha · December 4, 2018, 8:53pm

Hello all,

I saw this video last month about using Home Assistant and Node-Red to control led light strips. I have installed home assistant. Installed Mosquito broker, DuckDNS, and Node-Red. I think I have their respective configs setup properly, especially for SSL, but I am not sure. I do want to be able to control these lights remotely using my iPhone and from my computer at home. There has been a lot of trial and error but I am still not there yet. I thought I had it figured out but the light controls are not showing up in home assistant. I am not sure where to start to explain the problems. I guess it would be best to make sure home assitant and the respective add-ons are properly configured.

In the Developer Tools in Home Assistant I am getting these errors.

Unable to load /config/known_devices.yaml: Config file not found: /config/known_devices.yaml

3:22 PM components/device_tracker/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Error doing job: Task exception was never retrieved

3:22 PM components/mqtt/__init__.py (ERROR)

Unable to connect to the MQTT broker: Connection Refused: not authorised.

3:22 PM components/mqtt/__init__.py (ERROR)

In Node-Red config i am getting these two lines over and over

4 Dec 15:22:38 - [info] [server:Home Assistant] WebSocket Closed http://hassio/homeassistant
4 Dec 15:22:38 - [info] [server:Home Assistant] WebSocket Connected to http://hassio/homeassistant

Aephir · December 4, 2018, 9:17pm

I think the best approach is to be reductionist about it. Test one thing (Home Assistant), then add the next (Mosquitto), then the next, etc.

I can’t open the video link; it doesn’t even show a link address. So not sure exactly how you did this.

I’m guessing Home Assistant boots up normally (just doesn’t show the light)?
a. How did you install Home Assistant? I.e. is it hass.io or “regular” Home Assistant? *
For mosquitto, you can test the broker. If you install mosquitto-clients, you can have two terminal, the use mosquitto_sub and mosquitto_pub to test that it works.
Do you need/want to control from your phone while you are not on your home wifi? If not, you can skip the DuckDNS (and let’s encrypt, I assume?) altogether, and just not expose HASS to the internet.

Other than that, could you leave the link to the video? Then it might be easier to try to troubleshoot.

keanha · December 4, 2018, 9:37pm

Hello Aephir

Thank you for your response. Here is the link to the video. https://www.youtube.com/watch?v=Q8w56sOnadU. Sorry about that. I am using the additions to the configuration yaml, group yaml and the node red flow from their github.

I downloaded the hassio image and installed on a raspberry pi.
I will try that with mosquito.
I don’t have to have access off home wifi initially. I didn’t use let’s encrypt b/c I thought it wasn’t required when you use the duck dns addon with home assistant.

Aephir · December 4, 2018, 10:44pm

I’m not very familiar with hassio, so maybe I’ll say something stupid But I actually don’t think the mosquitto addon for hassio includes mosquitto-clients, so testing is going to be slightly more involved. But given the mqtt errors, this is probably something that should be checked.

Let me go through a few of the first things to check, and forgive me if any of them are obvious to you, better safe than sorry!

Did you remember to edit the Holyday_LEDS file? There’s the whole section with WIFI AND MQTT SETUP you need to configure to match your setup.

Do you see your ESP8266 showing up in your router configuration under “connected devices” (or similar, depending on your router model)?

Did you assign static IP’s (easiest via your routers DHCP)?

If your ESP8266 is not assigned static IP, that’s something you can live with (though I’d still recommend giving everything static IP), but if your hassio (Raspberry pi?) doesn’t have a static IP, then the ESP8266 can’t connect to the mosquitto broker (mqtt_server, line 32 here).

Could you post your entire Home Assistant configuration.yaml? Just to make sure it’s formatted correctly?
You remembered to to both configure and “start” the mosquitto addon, right? Go to the left side panel -> hassio -> addons and find mosquitto. If it looks like this, then it has not been started:

06%201222×1080 113 KB
To see your lights and switches, go to states (in the left panel, there is a small <>symbol at the bottom. Click that, and you’ll see the states part. Look for your lights/switches here to start with).

Do you have anything else available you can use to test mosquitto? Another raspberry pi? Or another computer? Otherwise you could use this, but I think you need to enable websockets on the addon, and I’m not really sure how to do (this thread talks about it, but I haven’t used mosquitto broker on hassio).

A side note, this was made for the “old” frontend in Home Assistant. You should really set this up in Lovelace, since this is now fully functional, and will soon (two weeks, I think?) be the official way.

And yes, the hassio DuckDNS includes let’s encrypt, so you should be good to go!

Well, let’s start with this, and see how far we get!

keanha · December 4, 2018, 11:47pm

Hello Aephir

Don’t worry about saying anything stupid. I greatly appreciate the help. I will go through your list and answer.

Yes I did edit the Holiday_LEDs file with the proper settings and the ESP shows up on the router.
I did assign a static IP to the ESP. And the Pi is assigned a static IP as well.
Below is my configuration.yaml file.
I did configure and start the Mosquitto broker
I do see the switches and lights in the states section of home assistant. I don’t mind trying lovelace if you think it is a better way to go.

Thanks again for your help.

I also set port fowarding on my router

And here is the configuration.yaml file.

homeassistant:
  # Name of the location where Home Assistant is running
  name: Home
  # Location required to calculate the time the sun rises and sets
  latitude: 37.9924
  longitude: -84.3752
  # Impacts weather/sunrise data (altitude above sea level in meters)
  elevation: 0
  # metric for Metric, imperial for Imperial
  unit_system: imperial
  # Pick yours from here: http://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  time_zone: America/New_York
  # Customization file
  customize: !include customize.yaml

# Show links to resources in log and frontend
introduction:

# Enables the frontend
frontend:

# Enables configuration UI
config:

#MQTT Boker
mqtt:
  broker: 192.168.1.10
  
# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
http:
  base_url: blah.blah:8123
  api_password: !secret http password
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

# Checks for available updates
# Note: This component will send some information about your system to
# the developers to assist with development of Home Assistant.
# For more information, please see:
# https://home-assistant.io/blog/2016/10/25/explaining-the-updater/
updater:
  # Optional, allows Home Assistant developers to focus on popular components.
  # include_used_components: true

# Discover some devices automatically
discovery:

# Allows you to issue voice commands from the frontend in enabled browsers
conversation:

# Enables support for tracking state changes over time
history:

# View all events in a logbook
logbook:

# Enables a map showing the location of tracked devices
map:

# Track the sun
sun:

# Sensors
sensor:
  # Weather prediction
  - platform: yr

# Text to speech
tts:
  - platform: google

# Cloud
cloud:

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml

# Extra config for holiday lights
switch:
  - platform: mqtt
name: "Audio Effects"
state_topic: "holidayLights/audio/state"
command_topic: "holidayLights/addEffects"
payload_on: "Audio On"
payload_off: "Audio Off"
retain: false
  - platform: mqtt
name: "Add Glitter"
state_topic: "holidayLights/glitter/state"
command_topic: "holidayLights/addEffects"
payload_on: "Glitter On"
payload_off: "Glitter Off"
retain: false
  - platform: mqtt
name: "Add Lightning"
state_topic: "holidayLights/lightning/state"
command_topic: "holidayLights/addEffects"
payload_on: "Lightning On"
payload_off: "Lightning Off"
retain: false
light:
  - platform: mqtt_json
name: "Holiday Lights"
state_topic: "holidayLights/state"
command_topic: "holidayLights/commands"
effect: true
brightness: true
effect_list:
  - Color_Chase
  - Color_Glitter
  - Single_Race
  - Double_Crash
  - Rainbow
  - Blocked_Colors
  - BPM
  - Twinkle
  - Fire
  - Fill_Solid
  - Spooky_Eyes
  - Halloween
  - LED_Locator
qos: 1
  - platform: mqtt_json
name: "Holiday Lights Color 1"
state_topic: "holidayLights/state1"
command_topic: "holidayLights/commands1"
rgb: true
qos: 1
  - platform: mqtt_json
name: "Holiday Lights Color 2"
state_topic: "holidayLights/state2"
command_topic: "holidayLights/commands2" 
rgb: true
qos: 1
  - platform: mqtt_json
name: "Holiday Lights Color 3"
state_topic: "holidayLights/state3"
command_topic: "holidayLights/commands3"  
rgb: true
qos: 1
input_number:
  led_identification:
name: LED identification
initial: 0
min: 0
max: 500
step: 1

Aephir · December 5, 2018, 8:59am

There are some indentation errors in the config though (probably the most common mistake in Home Assistant) from the “switch” part and below. I use Atom editor, it catches some of these errors automatically. yaml files are extremely strict about correct indentation. Try:

switch:
  - platform: mqtt
    name: "Audio Effects"
    state_topic: "holidayLights/audio/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Audio On"
    payload_off: "Audio Off"
    retain: false
  - platform: mqtt
    name: "Add Glitter"
    state_topic: "holidayLights/glitter/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Glitter On"
    payload_off: "Glitter Off"
    retain: false
  - platform: mqtt
    name: "Add Lightning"
    state_topic: "holidayLights/lightning/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Lightning On"
    payload_off: "Lightning Off"
    retain: false
light:
  - platform: mqtt_json
    name: "Holiday Lights"
    state_topic: "holidayLights/state"
    command_topic: "holidayLights/commands"
    effect: true
    brightness: true
    effect_list:
      - Color_Chase
      - Color_Glitter
      - Single_Race
      - Double_Crash
      - Rainbow
      - Blocked_Colors
      - BPM
      - Twinkle
      - Fire
      - Fill_Solid
      - Spooky_Eyes
      - Halloween
      - LED_Locator
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 1"
    state_topic: "holidayLights/state1"
    command_topic: "holidayLights/commands1"
    rgb: true
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 2"
    state_topic: "holidayLights/state2"
    command_topic: "holidayLights/commands2" 
    rgb: true
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 3"
    state_topic: "holidayLights/state3"
    command_topic: "holidayLights/commands3"  
    rgb: true
    qos: 1
input_number:
  led_identification:
    name: LED identification
    initial: 0
    min: 0
    max: 500
    step: 1

Look for the same in your !includes (groups.yaml, automations.yaml, and scripts.yaml). Or share the configs, if you are in doubt.

You definitely should use Lovelace; I’m not sure how long the “old way” will even be supported. So no sense in making it all, just to redo it in a few months. But this is only relevant for how you see it in your browser/app, for troubleshooting, it’s not too important.

And just a few other notes, not related to your issues.

Just for good measure, I’d probably put base_url in secrets.yaml, just in (the paranoid) case anyone wants to target you and your home, specifically. No need to give the world the address for your home (assistant).
No need to expose most of those ports. Only the things that need to connect or receive connections from the internet.
- 8123 is not needed, you access hassio via duckdns (port 443) when outside your network. Inside your network, you use port 8123, but there you don’t need to expose it in the router config (do you need to forward port 80 though? Haven’t used the addon, but I remember I did that when I was using duckDNS/letsencrypt?).
- 1883 is the unsecure mosqiotto. As is, you are probably not connecting anything externally to your broker. The ESP8266 (NodeMCU) is on the same network as you hassio. If at some pint you want to send mqtt from outside you network, then secure and expose a secure port (usually 8883).
- I think the same goes for NodeRED? Hassio accesses NodeRED, that is running on the same computer (as an addon?). If so, there’s probably no need to expose port 1880 to the big bad internet. If you do, make sure it is secured, at least.

keanha · December 6, 2018, 2:47am

Aephir

Thank you for the corrected configuration file. The “check config” in Home Assistant didn’t report any problems. So thank you. I checked the other files and the only one i have is the groups.yaml. I don’t have anything in the others.

I went ahead and deleted my duckdns domain and stopped the addon. Removed any remote connection references…i think…I am going to keep this simple and only have the functionality on my local network like you suggested. Maybe I will add it later. I removed the port forwarding as you suggested but kept 3218 and 443. I am still getting the following errors:

In the developer tools info I get the following:

Unable to load /config/known_devices.yaml: Config file not found: /config/known_devices.yaml
9:08 PM components/device_tracker/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Error doing job: Task exception was never retrieved
9:07 PM components/mqtt/__init__.py (ERROR)
Failed to connect due to exception: [Errno -3] Try again
9:07 PM components/mqtt/__init__.py (ERROR)

In the node-red log i get the following no matter if I use my pi ip or the address in the error

5 Dec 21:34:44 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:34:59 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:35:14 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:35:29 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:35:44 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:35:59 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:36:14 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant
5 Dec 21:36:29 - [info] [mqtt-broker:57cf052.173a5fc] Connection failed to broker: http://hassio/homeassistant

On a positive note the ESP connects to the mqtt as reported by the serial out in the arduinio ide and in mosquitto logs it reports that the device is connected.

I am a bit perplexed.

Aephir · December 6, 2018, 2:50pm

Progress, that’s good

You might have more luck on the discord chat under hassio. There are more people, and replies are usually faster! And not really using hassio, I might be reaching the limits of my knowledge.

Do you see your lights and switches in Home Assistant now?

Did you set up your device (iPhone, you said?)? Otherwise, you will not have a known_devices.yaml (see here). The easiest way for an iPhone is probably using the official app, you need to install on your phone. After you log on in the app and enable tracking (I think it’s called, but there are a few things you can enable), you should have a known_devices.yaml.

If you don’t need to access from outside your network, then no port forwarding is needed. Without DuckDNS/let’s encrypt, nothing uses port 443. It shouldn’t hurt your config to open, but the more you open outward, the more security concerns you’ll have.

But for the rest, I would only have guesses, so I’d really recommend the hassio discord channel. If you don’t get anything there, I’ll be happy to give my guesses, but I doubt that they are any good…

keanha · December 6, 2018, 10:12pm

Thank you Aephir. I don’t see the switches but the led locator slider has always showed up in the web interface but the switches do not. I can confirm that the slider does send to the ESP based on the serial port feed. No matter what I do I can’t see the switches. Puzzling.

Aephir · December 6, 2018, 10:42pm

That is weird. Only think I can think of is a type somewhere (unless you didn’t restart after editing the config files?).

When I copied the text from my previous post into my switch.yaml, I get the switch showing up after a reboot.

It doesn’t do anything, since I have nothing listening for that mqtt topic/payload, but the switch does show up. If you’ve copy/pasted the exact code, then I can’t really think of anything… Just in case, could you post your new configuration.yaml? And make sure you restart after changing (preferably ssh into the hassio, and type hassio host reboot, though in theory a restart from the web interface configuraion should also do the trick).

keanha · December 7, 2018, 12:33am

here is my latest configuration.yaml file. I will answer the other after trying. Thanks.

homeassistant:
  # Name of the location where Home Assistant is running
  name: Home
  # Location required to calculate the time the sun rises and sets
  latitude: 37.9924
  longitude: -84.3752
  # Impacts weather/sunrise data (altitude above sea level in meters)
  elevation: 0
  # metric for Metric, imperial for Imperial
  unit_system: imperial
  # Pick yours from here: http://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  time_zone: America/New_York
  # Customization file
  customize: !include customize.yaml

# Show links to resources in log and frontend
introduction:

# Enables the frontend
frontend:

# Enables configuration UI
config:

#MQTT Boker
mqtt:
  broker: 192.168.1.10
  port: 1883

# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
http:
  api_password: !secret http password
#  ssl_certificate: /ssl/fullchain.pem
#  ssl_key: /ssl/privkey.pem

# Checks for available updates
# Note: This component will send some information about your system to
# the developers to assist with development of Home Assistant.
# For more information, please see:
# https://home-assistant.io/blog/2016/10/25/explaining-the-updater/
updater:
  # Optional, allows Home Assistant developers to focus on popular components.
  # include_used_components: true

# Discover some devices automatically
discovery:

# Allows you to issue voice commands from the frontend in enabled browsers
conversation:

# Enables support for tracking state changes over time
history:

# View all events in a logbook
logbook:

# Enables a map showing the location of tracked devices
map:

# Track the sun
sun:

# Sensors
sensor:
  # Weather prediction
  - platform: yr

# Text to speech
tts:
  - platform: google

# Cloud
cloud:

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml

# Extra config for holiday lights
switch:
  - platform: mqtt
    name: "Audio Effects"
    state_topic: "holidayLights/audio/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Audio On"
    payload_off: "Audio Off"
    retain: false
  - platform: mqtt
    name: "Add Glitter"
    state_topic: "holidayLights/glitter/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Glitter On"
    payload_off: "Glitter Off"
    retain: false
  - platform: mqtt
    name: "Add Lightning"
    state_topic: "holidayLights/lightning/state"
    command_topic: "holidayLights/addEffects"
    payload_on: "Lightning On"
    payload_off: "Lightning Off"
    retain: false
light:
  - platform: mqtt_json
    name: "Holiday Lights"
    state_topic: "holidayLights/state"
    command_topic: "holidayLights/commands"
    effect: true
    brightness: true
    effect_list:
      - Color_Chase
      - Color_Glitter
      - Single_Race
      - Double_Crash
      - Rainbow
      - Blocked_Colors
      - BPM
      - Twinkle
      - Fire
      - Fill_Solid
      - Spooky_Eyes
      - Halloween
      - LED_Locator
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 1"
    state_topic: "holidayLights/state1"
    command_topic: "holidayLights/commands1"
    rgb: true
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 2"
    state_topic: "holidayLights/state2"
    command_topic: "holidayLights/commands2" 
    rgb: true
    qos: 1
  - platform: mqtt_json
    name: "Holiday Lights Color 3"
    state_topic: "holidayLights/state3"
    command_topic: "holidayLights/commands3"  
    rgb: true
    qos: 1
input_number:
  led_identification:
    name: LED identification
    initial: 0
    min: 0
    max: 500
    step: 1

keanha · December 7, 2018, 1:25am

This is what I get.

keanha · December 7, 2018, 2:31am

here is the groups.yaml file which maybe causing an issue too

  houseLights:
    name: Holiday Lights
    control: hidden
    entities:     
      - light.holiday_lights
      - light.holiday_lights_color_1
      - light.holiday_lights_color_2
      - light.holiday_lights_color_3
      - switch.add_glitter
      - switch.add_lightning
      - switch.audio_effects
      - input_number.led_identification

Aephir · December 7, 2018, 1:38pm

I just noticed something, everything you have that relates to mqtt doesn’t work (lights and switches), while everything you have that relates to anything else (input_number) does work. So I’m pretty sure the issue is with mosquitto, or maybe “just” with home assistant being able to connect to the broker.

A few more things to try, focusing on MQTT:

Did you set a username/password in the mosquitto addon? If so, you have to put that in your home assistant config (put it in the secrets.yaml, and refer to it).
If you did, and can’t get it to work, try just removing the whole logins part from the addon config. Just for testing.
If none of that works, I’ve seen something about the broker IP, but I think we should’t look at that yet, I’m pretty sure the way you did is the right way (now).

To test mosquitto from home assistant, use MQTT publishing in home assistant developer tools. There should be an MQTT icon (#5 from the left below). Try publishing something from home assistant, and see if it appears in mosquitto. Either in logs, or using mosquitto_sub command; the simplest being mosquitto_sub -t "#", then you can publish under any topic to test.

18%20

keanha · December 7, 2018, 2:55pm

Aephir,

I will confirm all this when I get home later but I am sure I set a username/password in the mosquitto addon but I didn’t put it in the secrets.yaml. I followed the instructions on this page. https://www.home-assistant.io/addons/mosquitto/. I did not disable the ports yets in the mosquitto addon yet b/c I thought it was working and needed when I was trying to access remotely from outside my home network. When I get home I will try and test mosquitto. Would the mosquitto addon also cause the switches not to show properly in the HA UI?

Thanks

Aephir · December 7, 2018, 6:05pm

Well, my guess is that because Home Assistant can’t connect to the mosquitto broker (because the username/password is not set in home assistant), it won’t make the entities that rely on mosquitto.

I think something like this would work.

mqtt:
  broker: 192.168.1.10
  port: 1883
  username: !secret mosquitto_user
  password: !secret mosquitto_pass

But maybe for testing, just try disabling the username/password in the mosquitto addon configuration. Keep it minimal until the basics work

keanha · December 8, 2018, 1:49pm

Aephir,

Adding the username and password to the configuration.yaml and changing the addon configuration as well to reflect this did the trick and all the switches show up. Thank you so much. I really appreciate it. After testing all the actions from the UI showed up in the serial monitor of the Arduino IDE but they didn’t always correspond to lights turning on…I will have to look into the nodemcu code and see what the problem is.

keanha · December 9, 2018, 1:53am

Aephir,

It all works now. Again thank you so much for your help/advice/suggestion/everything. I really do appreciate it.

Aephir · December 9, 2018, 11:44am

Sure, great to hear it worked!