MQTT Broker with certificates (AWS IoT)

Had some feedback from the Mosquitto forum and also here on another thread as I have experienced the same behavior you describe (i.e. disconnects). Here’s the (hopefully accurate) summary:

  • The retain flag is set at the client, the broker merely acts on it (if supported)
  • Given the client (originator) is responsible to handle retain, there is no global setting to switch this off. There may be MQTT brokers that allow you to switch this off but Mosquitto is not one of them
  • For Home Assistant that means any instance; e.g. entity, automation etc., device is responsible to handle retain; for instance Tasmota has PowerRetain, SwitchRetain, ButtonRetain
  • I have a few Tasmota devices directly connecting to AWS IoT’s broker and they have no issue as retain is switched off. However I really like the idea of the bridge concept I recently learned to allow for quick local execution and with the bridge (and possibly statestream) gaining the ability to “mirror” the local environment for the purpose of remote control, data storage, analysis

So I don’t think there’s an easy way from my (still) limited understanding to deal with this. It’s probably a way for AWS to push their device shadow concept which looks like something similar to retain.

Maybe I have a look at the IBM Watson IoT platform. There is a (forever) free tier and while it also doesn’t support retain it seems to have a mechanism to strip off retain in case a message containing it arrives but I have no idea whether you would still be able to utilise AWS services like SNS, DynamoDB etc.

… it’s all part of the journey :slight_smile:

Hello,

I have configured my broker to be a bridge to AWS IoT…does that prevent me from sending local messages to my tasmota rfbridge? I can see the mqtt payload in my aws test when subscribed to the topic but the tasmota rfbridge sees no messages. My mosquitto docker logs show the device connecting on the 1883 port as well.

Morning @snapping,

Adding the bridge to AWS won’t impact your local MQTT calls. I can still connect to my local mosquitto broker - publishing and subscribing to topics (I have tasmota sockets which are publishing events). You might need to check the topic your rfbridge is subscribed to? A good place to start is download an MQTT client for your OS, and then use that to connect to your mosquitto instance. From there you will be able to see what messages are flowing through your broker, and on what topics. I have a mac here, and I use MQTT Explorer.

Cheers,
Chris

Thank you both so much for this! Quite the journey - but this worked for me too!

Hi guys Im new to home assistant. what would be the best way to get my sensor values to my AWS IOT core account using MQTT

Morning @devilian,

I have this up and running. Since August 24th, AWS IoT Core added support for retained messages. So this all runs much more smoothly now. (see here for details on the release AWS IoT Core now supports MQTT retained messages).

Below is the configuration and setup I have:

  1. I have edited my configuration.yaml to use the stateStream configuration. See below for that I am using. This means all state changes will be sent to my Mosquitto broker on the topic “ha”
mqtt_statestream:
  base_topic: ha
  publish_attributes: true
  publish_timestamps: true
  1. I have the “Mosquitto broker” AddOn installed. Into this, on the Configuration Tab I have the following:
logins: []
customize:
  active: true
  folder: mosquitto
certfile: fullchain.pem
keyfile: privkey.pem
require_certificate: false
anonymous: false
  1. I have the “Samba share” AddOn installed. This allows me to easily access the “share” folder. In this folder I have a folder called “mosquitto”, into which I’ve created a file called aws-iot.conf

In that file, I have entered the following. Sections in [] you need to replace with your own values, removing the [] themselves

# ============================================================
# Bridge to AWS IOT
# ============================================================

connection HomeAssistant

address [enter your AWS IoT Endpoint - found in AWS IoT, on the left hand side under settings]:8883

# Specifying which topics are bridged and in what fashion
topic [enter your statestream topic configured in configuration.yaml]/# out 1

# Setting protocol version explicitly
bridge_protocol_version mqttv311

bridge_insecure false

# Bridge connection name and MQTT client Id, enabling the connection automatically when the broker starts.
cleansession true
clientid [declare what clientID you want your homeassistant to present itself as, anything will do]
start_type automatic
notifications false
log_type all


# ============================================================
# Certificate based SSL/TLS support
# ============================================================

#Path to the rootCA
bridge_cafile /ssl/rootCA.pem

# Path to the PEM encoded client certificate
bridge_certfile /ssl/homeAssistant_StateStream_device_cert.crt

# Path to the PEM encoded client private key
bridge_keyfile /ssl/homeAssistant_StateStream_private_key.crt.key

#END of bridge.conf
  1. In AWS, I have gone to AWS IoT Core, and created my Thing using the “Onboard” and “Get Started”. I’ve created my Device, and downloaded all the certificate files. The important ones are the Device Certificate, the Private Key and the Root CA. Using the “Samba share” addon, I uploaded those three files into the ssl folder.

Then I restart HA, logon to AWS IoT Core, choose “Test” and “MQTT test client”. Using that I can subscribe the topics AWS IoT Core is seeing. I use # (a wildcard in MQTT) to see all topics coming in. From there I can see all of my state changes coming in from Home Assistant on the topic ha

Hopefully this helps - message back if any of this isn’t clear.

1 Like

Hi, @chrisjcbt
Could you please share the aws-iot.conf file of yours.
Thanks in advance!

here you go. Obviously replace “my-aws-endpoint” with your own AWS IoT Endpoint address. I’m using their Ireland AWS Region. To find this see this link - Connecting devices to AWS IoT - AWS IoT Core.

# ============================================================
# Bridge to AWS IOT
# ============================================================

connection HomeAssistant

address my-aws-endpoint.iot.eu-west-1.amazonaws.com:8883

# Specifying which topics are bridged and in what fashion
topic ha/# out 1

# Setting protocol version explicitly
bridge_protocol_version mqttv311

bridge_insecure false

# Bridge connection name and MQTT client Id, enabling the connection automatically when the broker starts.
cleansession true
clientid homeassistant-mosquitto
start_type automatic
notifications false
log_type all


# ============================================================
# Certificate based SSL/TLS support
# ============================================================

#Path to the rootCA
bridge_cafile /ssl/rootCA.pem

# Path to the PEM encoded client certificate
bridge_certfile /ssl/homeAssistant_StateStream_device_cert.crt

# Path to the PEM encoded client private key
bridge_keyfile /ssl/homeAssistant_StateStream_private_key.crt.key

#END of bridge.conf

@chrisjcbt
Thanks for the tutorial. I have followed along and looking at the Mosquitto logs, I see the following:

1641163139: Bridge local.homeassistant-mosquitto doing local SUBSCRIBE on topic ha/#
1641163139: Connecting bridge HomeAssistant ([my-endpoint]-ats.iot.us-east-1.amazonaws.com:8883)
1641163139: Bridge homeassistant-mosquitto sending CONNECT
1641163140: Socket error on client local.homeassistant-mosquitto, disconnecting.

This process repeats indefinitely. From further research, it seems this a completely normal process and occurs when a connection is established, yet the IoT console is subscribed to ‘#’ and is not receiving messages…

The only thing I can think of is that the certificates may be wrong. To be clear, is the rootCA the one listed here by amazon? https://www.amazontrust.com/repository/AmazonRootCA1.pem

Would appreciate help with this. Thanks.

Hi @Jixster07,

The rootCA you’re linking to looks to be the right one. Let me take a look at my own config again to see if there are any differences. Have you enabled logging on the AWSA side? To do this:

  1. logon to AWS, and head over to IoT Core, using the Region you are using.
  2. Choose Settings on the bottom-left hand side.
  3. Scroll down to Logs, and enable logging for Debug logging (you’ll need to allow it to create a role to do the log output to AWS Cloudwatch)
  4. Go to AWS Cloudwatch, choose Logs and Log Groups.
  5. Find the Log Group called AWSIotLogsV2 and where it says Never expire, change this to say 1 Day, or whatever value makes sense for you. If you leave it never expiring logs, you will just spend money you don’t need to.

This is where the logs on the AWS Side will present your connection attempts from HA. Pull some logs from that and let me know what you are seeing.

C.

Hello,

After spending a considerable amount of time trying to find the right solution for connecting my local Home Assistant broker with AWS IoT Core, I have written a tutorial on how to bridge a local MQTT Mosquitto broker with AWS IoT Core. It has been working well for me for several months now. You can find the tutorial at GitHub - vhuynen/Helium-Network-AWS-IoT-Core-Home-Assistant: This post explains how to retreive data from Helium Console to on-premises Home Assistant throught AWS IoT Core infrastructure. We will explain how to configure your local MQTT broker in bridge in order to replicate locally your data from AWS Iot Core.

1 Like

Hello everyone & future me,

First off, I apologize for resurrecting an old thread. This particular discussion popped up during my search, and I thought it would be beneficial for others who might stumble upon it in the future. I’m not asserting this as a definitive guide, but I wanted to share my observations and steps that helped me set up Home Assistant with AWS IoT Core’s MQTT Broker in 2023. I hope it can be of help to someone else.

1. Introduction:

Before diving in, it’s essential to note that most of these settings need to be made through the Home Assistant UI, not directly in the configuration.yaml (except for the specific instruction mentioned). Also, always consider enabling CloudWatch in AWS as it’s immensely helpful for debugging.

2. Prerequisites:

  • A working Home Assistant setup.
  • An AWS account with AWS IoT Core set up.
  • MQTT installed in Home Assistant. You can do this via Settings -> Devices & Services -> Add Integration.

3. AWS IoT Core Settings:

3.1. Certificate and Thing Association:

Instead of using the “Connect a device” wizard in AWS, navigate to the IoT service → Security -> Certificates. Here, opt for “Auto-generate” for the certificate creation. Once created, activate the certificate and download the “Device Certificate” and “Private key file”. You’ll later upload these files to Home Assistant during the MQTT client setup.

3.2. Security Policy Settings:

Set your policy to: IoTSecurityPolicy_TLS13_1_2_2022_10.

3.3. Defining the Security Policy:

The accuracy of this policy is pivotal for the setup’s functionality. If you observe connection errors in CloudWatch, likely, the iot:Connect action isn’t configured correctly. Authorization errors typically indicate issues with other parts of the policy. Ensure your policy paths are precise. Here’s a sample policy that’s quite permissive but functional:

jsonCopy code

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:[your-region]:[your-aws-account-id]:client/[your-client-id]"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive",
        "iot:GetRetainedMessage",
        "iot:RetainPublish"
      ],
      "Resource": "arn:aws:iot:[your-region]:[your-aws-account-id]:topic/*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": "arn:aws:iot:[your-region]:[your-aws-account-id]:topicfilter/*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:ListRetainedMessages",
      "Resource": "*"
    }
  ]
}

4. Setting Up MQTT in Home Assistant:

With the necessary details and files from AWS in hand, proceed to set up the MQTT integration in Home Assistant:

  1. Navigate to Settings -> Devices & Services -> Add Integration and select MQTT.
  2. You’ll be presented with various fields to complete:
  • Broker: [your-aws-broker-url] (e.g., xxxxx-ats.iot.[your-region].amazonaws.com)
  • Port: 8883
  • Username + Password: Leave these empty.
  • Client ID: Designate a unique ID for your client (e.g., clientXXXX).
  • Broker certificate validation: Select Auto.
  • Ignore broker certificate validation: Deactivate this option.
  • MQTT protocol (version): Set to 5.
  • MQTT Transport: Choose TCP. (WebSocket employs port 443, but this wasn’t successful in tests.)
  • Discovery prefix: Decide on a prefix (e.g., myprefix).
  • Enable birth message: Deactivate this setting.
  • Enable will message: Also deactivate this.

For the certificate files:

  • First, enable the “Use a client certificate” option.
  • Proceeding to the next step will display buttons for uploading the certificate files:
    • Click “Upload client certificate file” to upload the “Device Certificate” from AWS.
    • Use “Upload private key file” for the “Private key file” from AWS.

5. Configuring configuration.yaml in Home Assistant:

After the MQTT UI setup, incorporate this minor addition to your configuration.yaml:

yamlCopy code

mqtt_statestream:
  base_topic: [your-discovery-prefix]

Post-change, execute a full restart of Home Assistant — a settings reload won’t suffice.

6. Testing the Setup:

For verification:

  1. In Home Assistant, go to Settings -> Devices & Services -> MQTT -> Configure. Under “Listen to a topic”, insert # as the “Topic to subscribe to” and initiate “Start Listening”.
  2. Within AWS, access the “MQTT test client” and similarly subscribe to #.
  3. Messages can now be dispatched from either platform. Messages sent from Home Assistant should be visible in the AWS MQTT test client and vice versa.

7. Conclusion:

I trust that these insights might aid someone embarking on a similar integration journey. As technology continuously advances, it’s always wise to consult the most recent documentation when uncertain. If obstacles arise, AWS’s CloudWatch logs and the available testing tools in both AWS and Home Assistant are indispensable for troubleshooting.

Disclaimer: For live environments, it’s paramount to draft a more stringent security policy in AWS. The shared policy is illustrative and errs on the side of permissiveness to ensure operability. Always prioritize best practices when deploying in a real-world scenario.

Best wishes on your integration journey!

My dear thanks to ChatGPT who helped immensely with writing this forum post. :slight_smile:

Hmm, @user83838 - one thing I would add, you need to enable advanced mode for the user performing this configuration.

I’m getting this error after following your instructions, but seeing successful connections on the IOTCore Monitoring side.

Broker options
Please enter the connection information of your MQTT broker. 
Failed to connect

Edit: I chose v3.1 on the home assistant side and it proceeded to the Discovery_prefix/ birth_message/ will_message section.