MQTT noise when starting and issues sending commands over 3/4G

1

Interested to get some advice on the following please.

Working on a project for some time now and had this issue with a serial console showing weird characters when sending MQTT commands as below.

Device is a LILYGO® TTGO T-SIM7600 ESP32 (ESP32-WROVER and SIM7600).

Everything is the code works but have some minor issues that I need to resolve.

I have read about baud rates and slowing the code down to communicate to the SIM modem. Baud rate is 115200 played baud rates to 9600 - 115200. 115200 seems stable and is the example speed in the Tiny_GSM library examples.

Q. Where do these extra characters come from when sending local to a cloud MQTT then to the device? (using Node Red flow to pub/sub)
Q. How do I debug them?
Q. Both MQTT and modem AT commands seem to generate them?
Q. MQTT pub topics seem to have more
Q. MQTT sub topics are ok

Using MQTT explorer to the cloud MQTT broker and pub/sub messages are as expected.

In serial console added to the code looking for blanks etc…

    Serial.print("<");
    SerialMon.print((char*)payload);
    Serial.println(">");

image
image417×518 5.26 KB

and connecting to a cloud MQTT broker

image

and sending MQTT commands I see in the serial console extra characters

image

1 Like
2

That looks like RF interference with the serial signaling lines. This is pretty common when you have an RF power amplifier (such as in a cell modem) close to the data lines. I’ve had problems like this in pretty much every prototype design involving a SIMxxxx based cell modem and when things are packed close together without grounded shielding planes. This usually happens when the modem connects to the network and when transmitting data, especially if the cell connection is weak (the modem then uses higher tx power). The RF will induce noise into the serial lines and that creates garbage. It usually does not happen when receiving data (only short ACKs are sent then).

It’s especially bad in class-4 2G modems when on a 2G network, because of the high tx power. What band are you on ? Note that your modem does not support 5G, so you’re not on that. You’re either on 2G, 4G LTE or 4G Cat 1. All of these have different tx power outputs are generate different levels of RF interference.

Getting rid of this can be difficult. Make sure the antenna is not located over the PCB, but is pointing outwards. Try to use an external antenna with a good shielded coax. Make sure your modem uses the lowest tx power possible and make sure you use the right antenna for your frequency band. You may have to add shielding or ground planes. Or physically separate the modem from the MCU by a few cm / inches.

3

Thanks and yes you are correct re not 5G its 3/4G.

 ret = modem.setNetworkMode(2); //Automatic

        2 – Automatic
        13 – GSM Only
        14 – WCDMA Only
        38 – LTE Only
        59 – TDS-CDMA Only
        9 – CDMA Only
        10 – EVDO Only
        19 – GSM+WCDMA Only
        22 – CDMA+EVDO Only
        48 – Any but LTE
        60 – GSM+TDSCDMA Only
        63 – GSM+WCDMA+TDSCDMA Only
        67 – CDMA+EVDO+GSM+WCDMA+TDSCDMA Only
        39 – GSM+WCDMA+LTE Only
        51 – GSM+LTE Only
        54 – WCDMA+LTE Only

Not sure of the actual connection 2G, 4G LTE or 4G Cat 1 need to figure out how to check this and reduce power based on the connection type. Assuming 4G LTE or 4G Cat 1 as 2G networks here is decommissioned.

Signal connection is 21 using a pigtail external antenna as well.
Added the external antenna as its sitting inside on a bench getting - 2…30 – - 109… - 53 dBm

4G LTE Antenna Wifi NB IoT High Gain Router Antenna for Huawei B315 593S B880 B310 SMA Male IPEX Wireless Module Aerial

int csq = modem.getSignalQuality();

Setup isnt ideal but dont want to move/solder until the code is working as expected which is where I am now.

  // 0 – - 113 dBm or less
  // 1 – - 111 dBm
  // 2...30 – - 109... - 53 dBm
  // 31 – - 51 dBm or greater
  // 99 – not known or not detectable
  // 100 – - 116 dBm or less
  // 101 – - 115 dBm
  // 102…191 – - 114... - 26dBm
  // 191 – - 25 dBm or greater
  // 199 – not known or not detectable
  // 100…199 – expand to TDSCDMA, indicate RSCP received
  //Bit Error Rate %
  // 0 – <0.01%
  // 1 – 0.01% --- 0.1%
  // 2 – 0.1% --- 0.5%
  // 3 – 0.5% --- 1.0%
  // 4 – 1.0% --- 2.0%
  // 5 – 2.0% --- 4.0%
  // 6 – 4.0% --- 8.0%
  // 7 – >=8.0%
  // 99 – not known or not detectable
4

Oh it’s an all-in-one board. There’s not much you can do then I’m afraid.

Most of the interference issues appear on 2G boards, due to their 2W tx power. You might remember the click-click noises early cell phones induced into nearby audio equipment. LTE uses maximum 23dBm (200mW), so interference is much less common (but not impossible, especially if the board is badly routed, which is not uncommon on cheap boards). You can only change tx power on 2G, 4G has fully automatic power control that you can’t change. What you can do is reduce the baud rate to the modem. By slowing it down, interference induced errors will be less common.

There’s still the possibility that this is due to some buffer overrun or unterminated string bug in the MQTT implementation. To rule this out, redirect the modem output (from the modem UART) to the debug console directly, bypassing the MQTT entirely, and see if there are strange chars appearing there too. If yes, it’s interference. If not, it’s a software bug.

5

Thanks Tiny_GSM does that by default once remarked.

//See all AT commands, if wanted Stream Debugger
#define DUMP_AT_COMMANDS

//Debug
#ifdef DUMP_AT_COMMANDS
#include <StreamDebugger.h>                              //version 1.0.1
StreamDebugger debugger(SerialAT, SerialMon);
TinyGsm modem(debugger);
#else
TinyGsm modem(SerialAT);
#endif

#define UART_BAUD    57600

  //Set console baud rate
  Serial.begin(57600);                                                         
  delay(10);
  SerialAT.begin(UART_BAUD, SERIAL_8N1, MODEM_RX, MODEM_TX);

Trying down from 115200 to 57600 baud.

10:55:10.105 → AT+CGREG?
10:55:11.325 → AT+CGREG?
10:55:12.590 → fail
10:55:22.599 → AT+CGREG?
10:55:23.583 → Network disconnected

Baud 74880

11:00:14.666 → AT
11:00:14.947 → AT
11:00:15.274 → AT+CNMP=2
11:00:16.258 → AT+CGMM
11:00:17.276 → ATI
11:00:20.322 → AT+CPIN?
11:00:22.305 → AT+CPIN?
11:00:24.331 → AT+CPIN?
11:00:26.316 → AT+CPIN?
11:00:28.296 → AT+CPIN?
11:00:30.345 → Waiting for network, AT+CGREG?
11:00:31.561 → AT+CGREG?
11:00:32.828 → AT+CGREG?
11:00:34.089 → AT+CGREG?
11:00:35.354 → AT+CGREG?
11:00:36.580 → AT+CGREG?
11:01:29.220 → AT+CGREG?
11:01:30.445 → fail
11:01:40.463 → AT+CGREG?
11:01:41.484 → Network disconnected
11:01:41.484 → AT+CSQ
11:01:42.474 → AT+CGREG?
11:01:43.705 → AT+CSQ
11:01:44.749 → AT+CGREG?

Tested 38400 same result as above

11:06:09.808 → AT+CGREG?
11:06:11.067 → AT+CGREG?
11:06:12.343 → fail
11:06:22.307 → AT+CGREG?
11:06:23.338 → Network disconnected
11:06:23.338 → AT+CSQ
11:06:24.329 → AT+CGREG?

Test 9600 same result as above

Even forcing the modem with AT+IPR to 57600 will not register

void setup()
{
  //Set console baud rate
  Serial.begin(57600);                                                         //115200
  delay(10);

  //Set Baud to 
  modem.sendAT("+IPR=57600\r\n"); //command sets the baud rate of module’s serial interface permanently, even after reboot
  modem.waitResponse(1000);
  Serial.println("IPR Baud set to 57600");  
  SerialAT.begin(UART_BAUD, SERIAL_8N1, MODEM_RX, MODEM_TX);
6

SerialAT console shows in yellow extras.

image
image809×985 40.2 KB

MQTT explorer looking at the Cloud MQTT broker no extras for pub/sub

I use pub to remotely see the state of the T-SIM/sensors

Technically all is working been writing this now for a year and it does what it should. But in saying that SerialAT / Serial has extra characters and I wanted to find the source out hence where you are now highlighting.

This sensor is a remote location hence no wireless so 3/4G then reports and is controlled by HA.

The other issue is the sensor (TSIM7600, ESP32CAM, PIR, Shock Sensor) runs for 23.5-24hrs then disconnects from MQTT or the Network and will not reconnect. Mainly based on the Tiny_GSM libary and Knolly Pub/Sub client.

I was looking further into the MQTT noise in the serial console thinking this is a possible reason for the MQTT to then disconnect after working for 23.5-24hrs? The LILYGO® SIM7600 has some status lights - netlight which when it enters the disconnect state looks to be off the network?

Network goes 1 to 0

+CIPCLOSE: 1,0,0,0,0,0,0,0,0,0

+CIPCLOSE: 0,0,0,0,0,0,0,0,0,0

Integer type, which indicates the state of connection identified by
<link_num>. Range is 0-1.
0 – disconnected
1 – connected

image

7

That’s not normal. These modules work fine with all baud rates from 9600 and up. It looks like one side (MCU or modem) is not configured correctly. Mismatched baud rates can also lead to the corruption effects you experienced earlier.

You don’t have to change the baud rate of the console. You only have to set the baud rate of the UART the modem is connected to. SIMCOM modules will auto-sync to the host baud rate, unless forced into a specific rate. You shouldn’t do the latter. I don’t know how that TinyGSM thing does this internally, I never used it.

You need to get the MQTT part completely out of the equation in order to debug this. Look at the raw output of the modem UART. If possible use a hardware probe, connecting to the rx/tx of the modem UART to an extrernal FTDI. If the corrupted chars appear there too, it’s either RF or mismatched baud rates (slight timing mismatches can create this).

Disconnecting from MQTT and disconnecting from the cell network are two very different things. The former is usually due to inadequate ping heartbeats, leading to the MQTT broker to disconnect the client. Network disconnections can have multiple reasons. A cell network will typically auto-disconnect an open TCP socket after some timeout, unless you are using a SIM card with an M2M profile. My modem with a non-M2M card (normal cheap phone plan) will typically kick my MQTT link once a day and I have to auto-reconnect. My other modem with an M2M card will keep the link open indefinitely. It also depends on how your local network provider has the timeouts configured on their link control layer.

8

Looking at your ‘noise’ a bit closer, it doesn’t look like RF noise. It’s a little too repetitive and it contains strings. Interference noise would be entirely random. It might be a bug in the way TinyGSM processes the incoming data and these may be remnants of the MQTT server response packets appearing on the console.

9

Cant add another FTDI to SerialAT (Modem) as it doesnt expose these physically.

image
image933×737 90.5 KB

Blockquote
Looking at your ‘noise’ a bit closer, it doesn’t look like RF noise. It’s a little too repetitive and it contains strings. Interference noise would be entirely random. It might be a bug in the way TinyGSM processes the incoming data and these may be remnants of the MQTT server response packets appearing on the console.

Think you are correct as it works as expected MQTT wise and I can control with pub/sub.

I guess now I just need to confirm the best way forward for this. Suspect the MQTT cant reconnect as the network is disconnected.

Cell provide is Skinny.co.nz on a pre paid (cheap) SIM so guessing its not a M2M (machine-to-machine).

So non-M2M card (normal cheap phone plan) will typically kick my MQTT link once a day and I have to auto-reconnect.

Blockquote
Disconnecting from MQTT and disconnecting from the cell network are two very different things. The former is usually due to inadequate ping heartbeats, leading to the MQTT broker to disconnect the client. Network disconnections can have multiple reasons. A cell network will typically auto-disconnect an open TCP socket after some timeout, unless you are using a SIM card with an M2M profile. My modem with a non-M2M card (normal cheap phone plan) will typically kick my MQTT link once a day and I have to auto-reconnect. My other modem with an M2M card will keep the link open indefinitely. It also depends on how your local network provider has the timeouts configured on their link control layer.

In Tiny_GSM

#if TINY_GSM_USE_GPRS && defined TINY_GSM_MODEM_XBEE
  modem.gprsConnect(apn, gprsUser, gprsPass);
#endif

  SerialMon.print("Waiting for network, ");
  if (!modem.waitForNetwork()) {
    SerialMon.println(" fail");
    delay(10000);
    return;
  }
  SerialMon.print("success ");

  if (modem.isNetworkConnected()) {
    SerialMon.println("network connected");
  }

then this section from Tiny_GSM is meant to reconnect the network then MQTT.

    // Make sure we're still registered on the network
    if (!modem.isNetworkConnected()) {
      SerialMon.println("Network disconnected");
      if (!modem.waitForNetwork(180000L, true)) { //180000L
        SerialMon.println(" fail");
        delay(10000); //10000
        return;
      }
      if (modem.isNetworkConnected()) {
        SerialMon.println("Network re-connected");
      }
      //}

#if TINY_GSM_USE_GPRS
      // and make sure GPRS/EPS is still connected
      if (!modem.isGprsConnected()) {
        SerialMon.println("GPRS disconnected!");
        SerialMon.print(F("Connecting to "));
        SerialMon.print(apn);
        if (!modem.gprsConnect(apn, gprsUser, gprsPass)) {
          SerialMon.println(" fail");
          delay(10000); //10000
          return;
        }
        if (modem.isGprsConnected()) {
          SerialMon.println("GPRS reconnected");
        }
      }
#endif
    }

    if (!mqtt.connected()) {
      SerialMon.println("MQTT NOT CONNECTED!");
      // Reconnect every 10 seconds
      uint32_t t = millis();
      if (t - lastReconnectAttempt > 10000L) {
        lastReconnectAttempt = t;
        if (mqttConnect()) {
          lastReconnectAttempt = 0;
        }
      }
      //delay(10000);                                       //delay for 10 seconds
      delay(1000);
      //delay(100);                                         //delay for 100 milliseconds 0.1 of a second
      //delay(TINY_GSM_YIELD_MS);                           // Add this line to yield to the TinyGSM library 2ms
      return;
    }
    mqtt.loop();                                            //Calls mqtt.loop in the pubsubclient library
10

Also

Modded PubSubClient

from 60 - 15 and remarking out the _client->stop(); line.

// MQTT_KEEPALIVE : keepAlive interval in Seconds. Override with setKeepAlive()
#ifndef MQTT_KEEPALIVE
#define MQTT_KEEPALIVE 15 //60 
#endif

Rather than that added and still to be tested. But if its a network disconnect this wont fix it.

  // Set the keep-alive interval to 30 seconds
  mqttClient.setKeepAlive(30);
11

As mentioned, I don’t use these two libraries. I can’t help you with those. Sorry.

I wrote my connection code to the SIMCOM modules from scratch. I also use a different MQTT client library (embedded paho).

Note that the issue you mentioned above is about GPRS disconnections. Those are only relevant for 2G networks, they don’t exist on 4G.

12

all good you have been super informative in getting me further. Will look to see ifs its MQTT or Network and then try to move forward from there :slight_smile:

Blockquote
Note that the issue you mentioned above is about GPRS disconnections. Those are only relevant for 2G networks, they don’t exist on 4G.

Can you explain a bit more please?

13

GPRS is an old cell network data protocol that is not used on 4G / LTE anymore. The github issue you linked to talked about an issue involving GPRS, so it may not be relevant to your issue, as you don’t use GPRS.

14

Thanks and good point. Need to read up on 4G / LTE.

15

Mentions payload[length] = 0;

To ensure there is a 0 immediately after the valid bytes of the payload

Assume this is then correct

void mqttCallback(char* topic, byte* payload, unsigned int len) {
  SerialMon.print("Message arrived [");
  SerialMon.print(topic);
  SerialMon.print("]: ");
  
  // Add null terminator after valid payload bytes
  payload[len] = 0;
  
  SerialMon.write(payload, len);
  SerialMon.println();
}

This will help a few of my issues but after testing still get those random characters at the start not the end of the message/

16

Right managed to arrange a M2M SIM but working on getting this activated benefits of working for a telco.

In the mean time

http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Ref363645900

If the Keep Alive value is non-zero and the Server does not receive a Control Packet from the Client within one and a half times the Keep Alive time period, it MUST disconnect the Network Connection to the Client as if the network had failed [MQTT-3.1.2-24].

Going to test to see if it will reconnect.

mqtt.disconnect(); // Disconnect from MQTT
17

After a fair bit of time checking it appeared that it was the network operator closing the consumer SIM connection and not MQTT. Once the network closed MQTT started to loop so added this modified block to reconnect and its been over 6 days now running and reconnecting when needed.

if (!mqtt.connected()) {
    SerialMon.println("MQTT NOT CONNECTED! ");
    SerialMon.print("Disconnecting from: ");
    SerialMon.println(broker);
    mqtt.disconnect(); // Disconnect from MQTT  //added 08/04/2023
    delay(500);
    SerialMon.print("Network State is: ");
    SerialMon.println(modem.isGprsConnected()); //added 08/04/2023

    // Reconnect every 10 seconds
    uint32_t t = millis();
    if (t - lastReconnectAttempt > 10000L) {
        lastReconnectAttempt = t;
        if (!modem.isGprsConnected()) {
            // Reconnect to GPRS network if not connected
            SerialMon.println("GPRS not connected, reconnecting...");
            if (!modem.gprsConnect(apn, gprsUser, gprsPass)) {
                SerialMon.println("GPRS reconnect failed");
                delay(10000);
                return;
            }
        }

        if (mqttConnect()) {
            lastReconnectAttempt = 0;
        }
    }
    delay(100);
    return;
}