MQTT Broker with certificates (AWS IoT)

Hi folks, I’m looking for some help with mqtt integration. I have to use certificates to auth against an external mqtt broker I’ve setup in configuration.yaml. But I’m getting errors when homeassistant tries to connect.

I have my own MQTT broker (an AWS IoT endpoint). The goal is to eventually use the Statesteam configuration to push all entity event changes up to AWS where I can persist and graph for longer. So, I’ve started work, adding the lines below to my configuration.yaml. See below:

mqtt:  
  broker: !secret awsiot-url
  port: 8883
  certificate: /config/certs/AmazonRootCA1.pem
  client_key: /config/certs/b5dd525b0f-certificate.pem.crt
  client_cert: /config/certs/b5dd525b0f-private.pem.key
  tls_version: '1.2'
  tls_insecure: false
  protocol: 3.1.1

I’ve jumped onto the console of my homeassistant container and verified the cert paths are ok. I’ve also checked these certificates and broker address from a local MQTT client. The local client connects and can publish and subscribe to topics. But with the config above in place, HomeAssistant Config check reported ok, I reboot HomeAssistant, and in my logs I see the below (and I cannot call the mqtt/publish service):

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 216, in async_setup hass, self
File "/usr/src/homeassistant/homeassistant/components/mqtt/__init__.py", line 649, in async_setup_entry tls_version=tls_version,
File "/usr/src/homeassistant/homeassistant/components/mqtt/__init__.py", line 799, in __init__ tls_version=tls_version,
File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 819, in tls_set context.load_cert_chain(certfile, keyfile) 
ssl.SSLError: [SSL] PEM lib (_ssl.c:3880)

I’ve been working away at this for the last couple of days, and not been able to get this to work. I have a suspicion that maybe my problem relates to the chain of the root cert, rather than anything else. Any ideas?

Have you been able to get this working? I’m trying to connect to my AWS IoT instance from Home Assistant.

However unlike you I’m not able to verify the file attributes as I can’t find the /config directory on the console. I have installed Home Assistant via the official .vmdk file onto VirtualBox running on a Fedora machine.

I’m gaining access to configuration.yaml via the VSC add-on in the home assistant UI and have created a directory within config to hold the 3 key/cert files.

Questions/Points:

  • I still don’t really understand the install via .vmdk (other than following the guide). Is this a docker installation? If so how do I find the config directory from the command line (I know how to get access to the host system)?

  • I get an error ERROR (MainThread) [homeassistant.components.mqtt] Failed to connect to MQTT server due to exception: [Errno -3] Try again but can’t find any info on this. First I’d like to ensure that file access is ok

  • I can successfully connect via MQTTBox to AWS IoT so the issue is on Home Assistant

  • I have removed tls_version: '1.2' as it’s being removed

mqtt:
  certificate: AWS_IoT/xxxxxxx.pem
  client_key: AWS_IoT/xxxxxxx-private.pem.key
  client_cert: AWS_IoT/xxxxx-certificate.pem.crt
  broker: xxxxxxxxx-1.amazonaws.com
  port: 8883
  tls_insecure: false

So I’m stuck on next steps to establish the connection to AWS IoT

I’m afraid I’m still struggling with this. I’m the same in that I can connect to AWS directly from my MQTT client, but not from HA. I’m no further and now not clear on where I need to configure this MQTT bridge information. Should I configure it in:

  1. “configuration.yaml” - as a new mqtt key (as attempted above).

  2. Use the MQTT add-in from the supervisor, and add an additional configuration for AWS-IOT into …/hassio/share/mosquitto which will be loaded alongside the core MQTT configuration.

Can anyone help us work past this?

I get the feeling most home automation setups are local only for security and availability reasons and hence there may be limited focus/experience. Having said that the official documentation supports public brokers (https://www.home-assistant.io/docs/mqtt/broker) so getting AWS IoT to work should be feasible.

There’s also this post on a Home Assistant user utilising AWS IoT to monitor plants (https://medium.com/@crhuber/monitoring-plants-with-aws-iot-and-home-assistant-e5aaae42a582) and I also found an article on the AWS IoT forum (https://aws.amazon.com/blogs/iot/how-to-bridge-mosquitto-mqtt-broker-to-aws-iot/).

Upon comparing these two it looks like the plant setup is using the concept of an “MQTT bridge” whereas I have tried to go down the path of only having the AWS IoT’s broker without a local mosquitto setup.

I might try this bridge setup and see how far I get. Reason I’m doing is to get the best of both worlds: Local monitoring and control via Home Assistant while having the ability to securely monitor and manage the home remotely via the AWS IoT Core via MQTT.

On your specific setup you mention “I’ve jumped onto the console of my homeassistant container and verified the cert paths are ok”. How did you install Home Assistant? I struggle to find my config folder via the console so I can’t check the certs can actually accessed as I used Windows (via Samba) and VSC to edit configuration.yaml

When I access the host system from the VM running on VirtualBox the default directory is/root and all there is is a .docker and .ssh directory

I have to take a couple of steps back as my head is spinning.

What I would like to do is to setup an MQTT bridge to AWS IoT to mimic the setup https://medium.com/@crhuber/monitoring-plants-with-aws-iot-and-home-assistant-e5aaae42a582. However I have a different starting point hence the confusion.

I have setup Home Assistant as a VM on VirtualBox (installed the official .vmdk file) running on a Fedora 25 system. HA comes up fine and I have a GUI for configuration. Firstly is this known as a docker install or does docker not come into play at all for this installation method?

Why the question? To configure this bridge, I have enabled the official Mosquitto add-on and when looking at the log there’s a mention that

1600258818: Config loaded from /etc/mosquitto.conf.

but when I go to the console (from within the VM; i.e. ‘root’ followed by ‘login’) I can’t find /etc/mosquitto.conf. When going to the host system outside the VM again I can’t find /etc/mosquitto.conf. So where is mosquitto on my system?

As part of the MQTT bridge setup I need to install AWS CLI and I need to first understand the overall setup before I can attempt to make any installations.

Anyone able to shed some light into this?

Ok, received some info that I should do a Linux (or Docker) install for AWS-CLI. Have done the Linux install - I believe successfully.

As I couldn’t find /etc/mosquitto.conf (as indicated in my earlier post) I have uninstalled Mosquitto from the Home Assistant UI and done a Linux install on the command line sudo dnf mosquitto. This has created /etc/mosquitto including the mosquitto.conf. So I still have no idea why mosquitto installed via Home Assistant doesn’t create this - still very interested.

However another challenge now is when I edit configuration.yaml to add the mqtt section; i.e.

mqtt:
  certificate: /etc/mosquitto/certs/rootCA.pem
  client_key: /etc/mosquitto/certs/private.key
  client_cert: /etc/mosquitto/certs/cert.crt
  broker: xxxxxxxxxxxxxxxx.iot.us-east-1.amazonaws.com
  port: 8883
  tls_insecure: false

and check the validity of configuration.yaml in the HA Server Controls I get the following error

Invalid config for [mqtt]: not a file for dictionary value @ data['mqtt']['client_cert']. Got '/etc/mosquitto/certs/cert.crt'
not a file for dictionary value @ data['mqtt']['client_key']. Got '/etc/mosquitto/certs/private.key'
not a valid value for dictionary value @ data['mqtt']['certificate']. Got '/etc/mosquitto/certs/rootCA.pem'. (See /config/configuration.yaml, line 14). 

I should have access to these files as per the following

[root@vortexbox mosquitto]# ls /etc/mosquitto/certs -al
total 24
drwxr-xr-x 2 root root 4096 Sep 18 09:29 .
drwxr-xr-x 4 root root 4096 Sep 18 09:53 ..
-rw-r--r-- 1 root root 1220 Sep 18 09:24 cert.crt
-rw-r--r-- 1 root root 1675 Sep 18 09:24 private.key
-rw------- 1 root root  451 Sep 18 09:24 public.key
-rw-r--r-- 1 root root 1188 Apr 15 16:24 rootCA.pem
[root@vortexbox mosquitto]#

To further complicate things I have installed the official Terminal & SSH add-on. When I start that and check for the certs via ls /etc/mosquitto/certs I get a No such file or directory error. In fact the whole /etc folder looks very different to when I login to the system from outside Home Assistant.

So why is this not working? I do apologize if I’m missing something totally obvious but I just figure it out :slight_smile:

Hi Thomas,

Sorry for the long delay. Between work and other things I’ve managed to take more of a look at this. Below is a walk through of my current setup, I believe I am almost there, but if anyone here can help, the ultimate question is “how do I modify my Mosquitto AddOn configuration to not have it post messages with the retain flag enabled?

Setup Walkthrough:-

I also have the Supervised installation of Home Assistant (running from Docker). To this I have added the Mosquitto Add-On. Here is my Mosquitto AddOn Configuration:

Mosquitto Add-On configuration:

anonymous: false
customize:
  active: true
  folder: mosquitto
certfile: fullchain.pem
keyfile: privkey.pem
require_certificate: false

With that configuration in place, then - using my choice of File Editor for Home Assistant (I use the Visual Studio Code), I browse to the following folder … /share/mosquitto. In that folder, I have created a file called aws-iot.conf. The configuration I use is below:

# ============================================================
# Bridge to AWS IOT
# ============================================================

# Define the connection
connection HomeAssistant
address (put your AWS IOT endpoint here, without the https:// and :8883 at the end)

# Bridge all local topics (not including $SYS) to AWS IOT, with QoS Flag 1
topic # out 1

# Setting protocol version
bridge_protocol_version mqttv311

# Connection Settings
bridge_insecure false
cleansession true 
try_private true
local_clientid homeassistant-mosquitto 
remote_clientid mosquittobroker
start_type automatic
notifications false
log_type all

# ============================================================
# Certificate based SSL/TLS support
# ============================================================

#Path to the rootCA
bridge_cafile /ssl/rootCA.pem

# Path to the PEM encoded client certificate
bridge_certfile /ssl/cert.crt

# Path to the PEM encoded client private key
bridge_keyfile /ssl/private.key

#END of bridge.conf

I followed the steps on the link you shared (here). I then copied the certificates I had generated following those steps into a specific home assistant folder. For me, that is back to the same File Editor in Home Assistant, browsing to this folder “/ssl”.

All of these folder references should make sense if you are using the Mosquitto AddOn for Home Assistant, with the Supervised installation. If not, then you will need to find these folders in your own Home Assistant configuration.

So - with that all in place we are almost there!!! My AWS IoT Cloudwatch logs show that Mosquitto is making successful connections. But it fails when it tries to to publish into AWS IoT. The issue looks to be because Mosquitto is publishing with the Retain=true flag enabled. This will error out, and AWS IoT will drop the publish event.

So. If you are still with me by this point - my question is does anyone know how to modify my Mosquitto AddOn configuration to not have it post messages with the retain flag enabled?

For anyone else who might be able to help. I have removed and re-installed the Mosquitto addon tonight (that has cleared out existing retained messages), I also removed the MQTT integration and reinstalled it after I had re-installed the Mosquitto addon.

I’ve checked the logs on the Mosquitto AddOn, and I can see messages continue to be published to the broker with the retain flag set. See below:

1602012428: Received PUBLISH from auto-B3C597D9-194A-56B0-5112-0DFD78322B93 (d0, q1, r1, m1787, 'homeassistant/from-home/light/downstairs_toilet_light_level/state', ... (3 bytes))
1602012428: Sending PUBACK to auto-B3C597D9-194A-56B0-5112-0DFD78322B93 (m1787, rc0)
1602012428: Received PUBLISH from auto-B3C597D9-194A-56B0-5112-0DFD78322B93 (d0, q1, r1, m1788, 'homeassistant/from-home/light/downstairs_toilet_light_level/last_updated', ... (32 bytes))
1602012428: Sending PUBACK to auto-B3C597D9-194A-56B0-5112-0DFD78322B93 (m1788, rc0)
1602012428: Received PUBLISH from auto-B3C597D9-194A-56B0-5112-0DFD78322B93 (d0, q1, r1, m1789, 'homeassistant/from-home/light/downstairs_toilet_light_level/last_changed', ... (32 bytes))

The value of “r1” indicates the retain flag is true right? If so, how can I ask HA to not set this value on the MQTT messages?

Thanks for getting back. We all have our “day jobs” so things get put on the back-burner at times.

I’m trying to sync with you here but stumble on Mosquitto already :slight_smile:

From the Home Assistant UI I have added the Mosquitto add-on

Re the config I had to add the first line; i.e. logins. Threw errors without it. NOTE: I’m running VirtualBox as a normal user (not root) but have added this user to the Linux group vboxusers as per the installation instructions

> logins: []
> anonymous: false
> customize:
>   active: true
>   folder: mosquitto
> certfile: fullchain.pem
> keyfile: privkey.pem
> require_certificate: false

Looking at the Mosquitto log (from the Home Assistant add-on screen)

> 1602640443: Error: Unable to open include_dir '/share/mosquitto'.
> 1602640443: Error found at /etc/mosquitto.conf:29.

I can’t see either

> /share/mosquitto

or

> /etc/mosquitto.conf

Did you have to create these? How are you running VirtualBox?

With regards to retain messages have you checked the mosquitto.conf options (https://mosquitto.org/man/mosquitto-conf-5.html)? A quick search for retain showed a number of hits so this may be an option to add to the config file…

Apologies, you are correct with your AddOn configuration, it should have the logins: [] line at the top of the configuration, I had missed it off when I copied in the text block.

For the Mosquito folder. I did not create these, they were created for me. If I look at the real location of this folder, it is here for me … “/usr/share/hassio/share/mosquitto/”. Inside that folder I created a file called aws-iot.conf, and put the values I shared.

For the retain flag, I spoke to AWS Support and they helped me identify that I can add the following to the aws-iot.conf file. This is retain_available false. I have that set now, and can see new events being logged to the Mosquitto AddOn log, see below:

1602660725: Will message specified (7 bytes) (r0, q0).
1602660725: 	homeassistant/status
1602660725: Sending CONNACK to auto-248A7305-6EC4-6D48-55EB-3DEFAECDD853 (0, 0)
1602660725: Socket error on client auto-248A7305-6EC4-6D48-55EB-3DEFAECDD853, disconnecting.
1602660726: Sending PINGREQ to homeassistant-mosquitto 
1602660726: New connection from 172.30.32.1 on port 1883.
1602660726: New client connected from 172.30.32.1 as auto-DDECD7C7-64DD-5C72-298B-149E08B5C31B (p2, c1, k60, u'mqtt-user').

Looking at the Mosquitto configuration file info, adding the retain_available false flag means that any client making connections to Mosquitto specifying a retain flag on the message will be disconnected. So the logs now seem to show me that Home Assistant continues to post messages with the retain flag enabled, but not Mosquitto drops the connection attempts.

I need to find a way for either Home Assistant to support MQTT messages without a retain flag being set, or AWS IoT supporting the retain flag. I think this is the core of the problem. I cannot find a way for an MQTT message with the regain flag set to have it removed before it is sent to another MQTT broker.

Thanks for the confirmation.

Tried to look for /usr/share/hassio/share/mosquitto/ but this folder doesn’t exist, hassio doesn’t exist. To clarify I installed Home Assistant from the appliance file (.ova), imported to VirtualBox and started the VM. Home Assistant then becomes available at <IP:8123>

While challenges like this are exciting it can also be very frustrating. Other than the Mosquitto log is there anywhere else I would see the Mosquitto installation log/status to see where I went wrong?

No problems. I have created another home assistant installation to work out how you do this with your install. (I imported the .ova and ran it). So …

  1. Go to Supervisor and Add-On Store, make sure you have added the “File editor” AddOn as well as Mosquito. For File Editor, you need to change its configuration to allow you to move outside of the Home Assistant configuration directory. This allows you to get to the /share folder which Mosquito wants to use for custom configuration files. To do this, change the File Editor configuration to the below (I modified only the enforce_basepath item to false):
dirsfirst: false
enforce_basepath: false
ignore_pattern:
  - __pycache__
  - .cloud
  - .storage
  - deps
ssh_keys: []
  1. With that done, run it and launch the File Editor UI. Then:
    a. Click the Folder icon in the top left.
    b. Click the Left Arrow to move you up a directory.
    c. Find the folder called ‘Share’, and create a directory in there called “mosquitto”
    d. In there place your mosquitto bridge configuration.

That should be it, and you should be at the same point as me. Retained MQTT messages are still being dropped by AWS IoT, and I cannot find a way to make Home Assistant not sent MQTT statestream events as retained messages.

Thanks for taking that effort. Will try this over the weekend.

Out of curiosity is there a similar config step needed for Visual Studio Code as this is my editor of choice as well and from memory it only exposes /config? Edit: I have found the answer to VSC already so all good here.

Also is the mosquito.conf in the same folder, i.e. /share?

Great - glad you found out how to do it (re: VSCode).

For the mosquito.conf file. I leave that alone and consider it managed by the addon. I am only enabling the Addon Customisation folder, that is where I drop in the bridge to AWS configuration file. The Home Assistant addon is configured to load any mosquito .conf files I drop into there.

Chris

Ok, I’m cathching up :slight_smile:

  • Installed AWS-CLI and created the certificates
  • Created the bridge file
  • Can’t connect :slight_smile:

Looking at the log file in the Mosquitto Home Assistant add-on first of all it looks like the bridge file (bridge.conf) is read (row 7) but it looks like there’s an issue with the CA cert - see last row

> [17:44:25] INFO: Setup mosquitto configuration
> [17:44:25] WARNING: SSL not enabled - No valid certs found!
> [17:44:25] INFO: No local user available
> [17:44:26] INFO: Initialize Hass.io Add-on services
> [17:44:26] INFO: Initialize Home Assistant discovery
> [17:44:26] INFO: Start Mosquitto daemon
> 1603003466: Loading config file /share/mosquitto/bridge.conf
> 1603003466: mosquitto version 1.6.3 starting
> 1603003466: Config loaded from /etc/mosquitto.conf.
> 1603003466: Loading plugin: /usr/share/mosquitto/auth-plug.so
> 1603003466: |-- *** auth-plug: startup
> 1603003466:  ├── Username/password checking enabled.
> 1603003466:  ├── TLS-PSK checking enabled.
> 1603003466:  └── Extended authentication not enabled.
> 1603003466: Opening ipv4 listen socket on port 1883.
> 1603003466: Opening ipv6 listen socket on port 1883.
> 1603003466: Opening websockets listen socket on port 1884.
> 1603003466: Warning: Mosquitto should not be run as root/administrator.
> 1603003466: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic localgateway_to_awsiot
> 1603003466: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic both_directions
> 1603003466: Connecting bridge awsiot (xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com:8883)
> 1603003466: Error: Unable to load CA certificates, check bridge_cafile "/ssl/rootCA.pem".

What’s weird is I couldn’t find the rootCA.pem (or the other files) in the Linux file system so it’s entirely possible that I have the wrong file attributes and hence the error. Given I couldn’t access /share/ssl from Linux I tried to be smart by doing a backdoor; I sent the cert files to myself and copying them to the /ssl folder I created within /share via Windows/Samba.

So I’d like to check the file attributes but a find - name rootCA.pem didn’t yield any results. Why are these files not visible from within the Linux file system?

hmm, looks like I have had some success…

I have given up in trying to find the cert files on the local filesystem outside Home Assistant and have created the folder /ssl via Windows (Samba) and copied the certs into this folder. Then I configured the bridge file (/share/mosquitto/bridge.conf) accordingly. Now Mosquitto successfully establishes a connection as seen in the Mosquitto Home Assistant log - see below. If you have any idea why I can’t see this in the Linux file system I’d be interested to hear. However given the success below this is probably secondary now.

If “rx” is indeed the indication of the retain flag mine shows r0 as you can see in the log. However mine is a brand new default Home Assistant installation on a new “test system” I’m building (so that I don’t mess up the production server once in full swing) and I haven’t made any changes. But the good news then is, it must be a configuration somewhere that changes the what looks like to be default of retain off to on. Logically this can only be in configuration.yaml, mosquitto.conf or the MQTT bridge file.

Testing

  • Local to AWS-IoT On the AWS-IoT console I subscribed to the topic localgateway_to_awsiot and use Configuration | Integration | MQTT-configure menu to publish a message to topic localgateway_to_awsiot >> It’s successfully received on the AWS-IoT console

  • AWS-IoT to local I subscribed to the topic awsiot_to_localgateway on Mosquitto and sent a message from AWS_IoT to Mosquitto >> It was received successfully on the Mosquitto end

So from this initial test it looks like this is working.

I have produced the complete log from Mosquitto restart as comparison. So for a starter maybe you can have a glance through your log and verify whether you spot any differences in order to narrow this down. Also do you use Tasmota at all? There’s this guy called Stephan Hadinger who is a real guru on AWS/MQTT and he may have some further insight if nothing obvious jumps out. He is active on the Tasmota forums (Discord).

Lastly why is the retain flag on an issue in your case? From the little research I have done retain on may be beneficial for clients that connect in order to get the last message on the topic immediately without having to wait for the next message update.

[23:50:32] INFO: Setup mosquitto configuration
[23:50:32] WARNING: SSL not enabled - No valid certs found!
[23:50:32] INFO: No local user available
[23:50:32] INFO: Initialize Hass.io Add-on services
[23:50:32] INFO: Initialize Home Assistant discovery
[23:50:32] INFO: Start Mosquitto daemon
1603025432: Loading config file /share/mosquitto/bridge.conf
1603025432: mosquitto version 1.6.3 starting
1603025432: Config loaded from /etc/mosquitto.conf.
1603025432: Loading plugin: /usr/share/mosquitto/auth-plug.so
1603025432: |-- *** auth-plug: startup
1603025432:  ├── Username/password checking enabled.
1603025432:  ├── TLS-PSK checking enabled.
1603025432:  └── Extended authentication not enabled.
1603025432: Opening ipv4 listen socket on port 1883.
1603025432: Opening ipv6 listen socket on port 1883.
1603025432: Opening websockets listen socket on port 1884.
1603025433: Warning: Mosquitto should not be run as root/administrator.
1603025433: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic localgateway_to_awsiot
1603025433: Bridge local.bridgeawsiot doing local SUBSCRIBE on topic both_directions
1603025433: Connecting bridge awsiot (xxxxxxxxxxxxxxx.iot.us-east-1.amazonaws.com:8883)
1603025433: Bridge bridgeawsiot sending CONNECT
1603025433: Received CONNACK on connection local.bridgeawsiot.
1603025433: Bridge local.bridgeawsiot sending SUBSCRIBE (Mid: 1, Topic: awsiot_to_localgateway, QoS: 1, Options: 0x00)
1603025433: Bridge local.bridgeawsiot sending UNSUBSCRIBE (Mid: 2, Topic: localgateway_to_awsiot)
1603025433: Bridge local.bridgeawsiot sending SUBSCRIBE (Mid: 3, Topic: both_directions, QoS: 1, Options: 0x00)
1603025434: Received SUBACK from local.bridgeawsiot
1603025434: Received UNSUBACK from local.bridgeawsiot
1603025434: Received SUBACK from local.bridgeawsiot
1603025434: New connection from 172.30.32.1 on port 1883.
[INFO] found homeassistant on local database
1603025434: New client connected from 172.30.32.1 as auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (p2, c1, k60, u'homeassistant').
1603025434: Will message specified (7 bytes) (r0, q0).
1603025434: 	homeassistant/status
1603025434: Sending CONNACK to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (0, 0)
1603025434: Received SUBSCRIBE from auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1
1603025434: 	homeassistant/# (QoS 0)
1603025434: auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 0 homeassistant/#
1603025434: Sending SUBACK to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1
1603025436: Received PUBLISH from auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (d0, q0, r0, m0, 'homeassistant/status', ... (6 bytes))
1603025436: Sending PUBLISH to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (d0, q0, r0, m0, 'homeassistant/status', ... (6 bytes))
1603025490: Received SUBSCRIBE from auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1
1603025490: 	awsiot_to_localgateway (QoS 0)
1603025490: auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 0 awsiot_to_localgateway
1603025490: Sending SUBACK to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1
1603025493: Sending PINGREQ to local.bridgeawsiot
1603025493: Received PINGRESP from local.bridgeawsiot
1603025503: Received PUBLISH from auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (d0, q0, r0, m0, 'localgateway_to_awsiot', ... (21 bytes))
1603025503: Sending PUBLISH to local.bridgeawsiot (d0, q0, r0, m0, 'localgateway_to_awsiot', ... (21 bytes))
1603025521: Received PUBLISH from local.bridgeawsiot (d0, q0, r0, m0, 'awsiot_to_localgateway', ... (45 bytes))
1603025521: Sending PUBLISH to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 (d0, q0, r0, m0, 'awsiot_to_localgateway', ... (45 bytes))
1603025525: Received UNSUBSCRIBE from auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1
1603025525: 	awsiot_to_localgateway
1603025525: auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1 awsiot_to_localgateway
1603025525: Sending UNSUBACK to auto-F6C5B5B6-0630-007D-218C-CD97AC2979D1

@chrisjcbt, did some more testing on this setup.

With the Mosquitto bridge configured, I setup a Tasmota device (ESP8266) and pointed the MQTT broker to Moquitto running on Home Assistant. I subscribed to topic localgateway_to_awsiot on both AWS IoT and MQTTBox on Windows.

Then I published a message from the Tasmota console to topic localgateway_to_awsiot and it’s successfully received by both MQTT clients. So the Tasmota device only sees the local (Mosquitto) broker which in turn forwards the message given the topic as configured in the bridge. That’s pretty damn sweet!

The message attributes on MQTTBox are as follows

"Hello From Tasmota :)"
qos : 0, retain : false, cmd : publish, dup : false, topic : localgateway_to_awsiot, messageId : , length : 47, Raw payload : 347210110810811132701141111093284971151091111169732584134

I don’t know MQTT well enough, but saw that Tasmota’s MQTT client has the ability to turn on/off the retain flag (https://tasmota.github.io/docs/Commands/#mqtt). Thinking about it this probably makes sense as messages always originate from MQTT clients, the broker is merely a relay if I understand this correctly. So are you able to modify the retain flag from your MQTT client?

Thanks for the updates Thomas.

I can replicate the same as you, pushing messages from the UI, or using Tasmota plugs (I have a few) are working ok. But what I’m hoping to do is stream all of the Home Assistant events to AWS. From there, I can play with visualisation and AWS IoT rules for the storage of those events into DynamoDB and S3 for example.

From the looks of it, my messages being sent with Retain=true are the messages related to the StateStream configuration. This gives me the MQTT events for all domains in HomeAssistant (Light/Sensor/Switch…). Those messages appear to be the ones being posted with the retain flag enabled.

Thanks for the updates. I will spin up another HomeAssistant install (on a Raspberry Pi 4B), I’ll test using your updates and see how the messages look.

Cheers,
Chris

Sounds good. Maybe I can try to replicate the StateStream config on my environment. What’s your configuration.yaml entry for that? Also remember the contact I mentioned S Hadinger who is very knowledgeable around MQTT.

Pardon the question but why is retain=on not wanted in this case?

Visualisation as in Grafana and the like?

Below is what I have added to my configuration.yaml:


mqtt_statestream:
  base_topic: homeassistant
  publish_attributes: true
  publish_timestamps: true
  exclude:
    domains:
      - media_player
      - zwave
      - sensor

Currently, AWS IoT will drop MQTT messages where the retail flag is set to true. I can see successful connections to AWS IoT, but messages coming from StateStream have the retain flag set.