Weather station and Weather Underground -> Work around

This afternoon PostNL delivered my very first weather station, a " Bresser Prol 5-in-1 Weatherstation incl. Wifi with Wind and Barometer". I chose this one because I really don’t need another dongle to read 868Mhz or whatever these things use.A nice post to some third party site I can scrape will do just fine.

I installed it, verified it was reporting sensible data and then tried to hook it up to the Weather Underground and right there is where things went to crap.

The API in homeassistant for WU doesn’t do anything and the other custom component I found didn’t work either. Right, … Damn it, maybe I should’ve gotten another crap dongle. My server looks like a porcupine already, what’s one more.

However, this thing is reporting to the WU site, so I just need to trap the data.I fired up wireshark and sniffed all the packets from the device ip to the internet. (Mikrotik/Tools/Packet Sniffer --> Stream to server and cap everything coming from the weather station)

Sure enough, it posted to a php file :

HTTP	450	GET 169.47.111.60/weatherstation/updateweatherstation.php?ID=BABABOEY&PASSWORD=FAFAFOHI&action=updateraww&realtime=1&rtfreq=5&dateutc=now&baromin=29.91&tempf=68.9&dewptf=55.0&humidity=61&windspeedmph=4.6&windgustmph=4.6&winddir=112&rainin=0.0&dailyrainin=0.36&indoortempf=72.6&indoorhumidity=54 HTTP/1.1

And it does that often too, like every 20 seconds, that near enough real time, exactly what I wanted in HA.

Let’s tell the router to route that sweet data somewhere else.

[drbytes@MikroTik] /ip firewall nat> add chain=dstnat action=dst-nat to-addresses=10.0.0.100 protocol=tcp src-address=10.0.0.34

This NAT rule will route all traffic from the weather station (with ip .34) to an internal webserver with ip .100 which is where my Nginx reverse proxy lives.

Next I replicated the directory and php script /weatherstation/updateweatherstation.php .

In the php (I hate php, it sucks I can not get used to that style) I have this code :

<?php

require __DIR__ . '/vendor/autoload.php';

use PhpMqtt\Client\Exceptions\ConnectingToBrokerFailedException;
use PhpMqtt\Client\Exceptions\DataTransferException;
use PhpMqtt\Client\Exceptions\UnexpectedAcknowledgementException;
use PhpMqtt\Client\MQTTClient;

$server = "10.0.0.100";
$port = 1883;
$client_id = "weather-data-publisher";
$mqttRoot = "pws/sensors/";

// OVERLY VERBOSE HELPER STUFF
function RoundIt($ee){
  return round($ee, 2);
}
function toKM( $a) {
  return  RoundIt( floatval($a)*1.60934);
}
function toC( $a) {
  return RoundIt(  (floatval($a)-32) * (5/9) );
}
function toMM( $a) {
    return RoundIt( floatval($a)*25.4);
}
  
function toHPA( $a) {
  return RoundIt((floatval($a)*33.8639);
}
// FOUND THIS ON THE NET IN SOME PASTEBIN
function wind_cardinal( $degree ) { 
  switch( $degree ) {
      case ( $degree >= 348.75 && $degree <= 360 ):
          $cardinal = "N";
      break;
      case ( $degree >= 0 && $degree <= 11.249 ):
          $cardinal = "N";
      break;
      case ( $degree >= 11.25 && $degree <= 33.749 ):
          $cardinal = "N NO";
      break;
      case ( $degree >= 33.75 && $degree <= 56.249 ):
          $cardinal = "NO";
      break;
      case ( $degree >= 56.25 && $degree <= 78.749 ):
          $cardinal = "O NO";
      break;
      case ( $degree >= 78.75 && $degree <= 101.249 ):
          $cardinal = "O";
      break;
      case ( $degree >= 101.25 && $degree <= 123.749 ):
          $cardinal = "O ZO";
      break;
      case ( $degree >= 123.75 && $degree <= 146.249 ):
          $cardinal = "ZO";
      break;
      case ( $degree >= 146.25 && $degree <= 168.749 ):
          $cardinal = "N";
      break;
      case ( $degree >= 168.75 && $degree <= 191.249 ):
          $cardinal = "Z";
      break;
      case ( $degree >= 191.25 && $degree <= 213.749 ):
          $cardinal = "Z ZW";
      break;
      case ( $degree >= 213.75 && $degree <= 236.249 ):
          $cardinal = "ZW";
      break;
      case ( $degree >= 236.25 && $degree <= 258.749 ):
          $cardinal = "W ZW";
      break;
      case ( $degree >= 258.75 && $degree <= 281.249 ):
          $cardinal = "W";
      break;
      case ( $degree >= 281.25 && $degree <= 303.749 ):
          $cardinal = "W NW";
      break;
      case ( $degree >= 303.75 && $degree <= 326.249 ):
          $cardinal = "NW";
      break;
      case ( $degree >= 326.25 && $degree <= 348.749 ):
          $cardinal = "N NW";
      break;
      default:
          $cardinal = null;
  }
 return $cardinal;
}

// SHUFF IT TO MQTT
$mqtt = new MQTTClient($server, $port, $client_id);
$mqtt->connect();

$mqtt->publish($mqttRoot .'baromin', toHPA($_GET["baromin"]), 0);
$mqtt->publish($mqttRoot .'temp', toC($_GET["tempf"]), 0);
$mqtt->publish($mqttRoot .'dewpt', toC($_GET["dewptf"]), 0);
$mqtt->publish($mqttRoot .'humidity', $_GET["humidity"], 0);
$mqtt->publish($mqttRoot .'windspeedkph', toKM($_GET["windspeedmph"]), 0);
$mqtt->publish($mqttRoot .'windgustkph', toKM($_GET["windgustmph"]), 0);
$mqtt->publish($mqttRoot .'winddir',wind_cardinal( $_GET["winddir"]), 0);
$mqtt->publish($mqttRoot .'rainmm', toMM($_GET["rainin"]), 0);
$mqtt->publish($mqttRoot .'dailyrainmm', toMM($_GET["dailyrainin"]), 0);
$mqtt->publish($mqttRoot .'indoortemp', toC($_GET["indoortempf"]), 0);
$mqtt->publish($mqttRoot .'indoorhumidity', $_GET["indoorhumidity"], 0);

$mqtt->close();

// POST TO WU .. OPTIONAL, I SHOULD JUST NOT BECAUSE THEY PULLED A FREE SERVICE AND STILL LEECH DATA OF OF MY INVESTMENT WITHOUT ANY REAL RETURN.
$xml = file_get_contents("http://169.47.111.60/weatherstation/updateweatherstation.php?".$_SERVER['QUERY_STRING']);

?>
success

You need the success so the weather station knows it went trough OK.

It’s pretty self explanatory, I just parse the query string, perform some localization on it, translate the wind direction and convert to metric where appropriate and shoot it off to MQTT.

Lastly I update the WU, for shizz n giggles.

For the PHP to work you need to install php-mqtt with composer.

After that I deployed a mqtt docker, added it to the HA configuration and finally added the sensors :

- platform: mqtt
  name: "PWS Barometer"
  state_topic: "pws/sensors/baromin"
  unit_of_measurement: 'HPa'

- platform: mqtt
  name: "PWS Outside Temperature"
  state_topic: "pws/sensors/temp"
  unit_of_measurement: 'C'

- platform: mqtt
  name: "PWS Dew Point"
  state_topic: "pws/sensors/dewpt"
  unit_of_measurement: 'C'

- platform: mqtt
  name: "PWS Outside Humidity"
  state_topic: "pws/sensors/humidity"
  unit_of_measurement: '%'

- platform: mqtt
  name: "PWS Wind speed"
  state_topic: "pws/sensors/windspeedkph"
  unit_of_measurement: 'KMh'

- platform: mqtt
  name: "PWS Wind Gust speed"
  state_topic: "pws/sensors/windgustkph"
  unit_of_measurement: 'KMh'

- platform: mqtt
  name: "PWS Wind direction"
  state_topic: "pws/sensors/winddir"
  unit_of_measurement: ''

- platform: mqtt
  name: "PWS Rain"
  state_topic: "pws/sensors/rainmm"
  unit_of_measurement: 'MM'

- platform: mqtt
  name: "PWS Daily Rain"
  state_topic: "pws/sensors/dailyrainmm"
  unit_of_measurement: 'MM'

- platform: mqtt
  name: "PWS Indoor Temperature"
  state_topic: "pws/sensors/indoortemp"
  unit_of_measurement: 'C'

- platform: mqtt
  name: "PWS Indoor Humidity"
  state_topic: "pws/sensors/indoorhumidity"
  unit_of_measurement: '%'

And we’re done.

Near real time updates in HA and no dependencies other than the ones I already had, locally processed by trapping the outbound sensor data and publishing them via mqtt for easy pickup by HA.

I thought I’d share it here although it’s a bit of a hack, it might help someone.

If it’s been done before I didn’t find it and it would’ve saved me a couple of hours.

14 Likes

I can’t seem to get the script to work. If I strip down the PHP file to the basics, then I can get it to post the data to WU, but when I re-insert the MQTT part, the script completely fails.

This works:

<?php

$xml = file_get_contents("http://169.63.130.180/weatherstation/updateweatherstation.php?".$_SERVER['QUERY_STRING']);

?>
success

Any ideas? Has the code changed?

Still works here to this day, no changes have been made.
Are you sure you have all the relevant libraries (mqtt etc) installed ?
Try to get the actual error message.

I found a missing bracket ) on Line 30 but other than that, no error.

Which PHP>MQTT client are you using? I’m using this: GitHub - php-mqtt/client: An MQTT client written in and for PHP.

I think the problem is with autoload.php. Removing this seems to allow the script to auto update WU again, but without any MQTT:

require __DIR__ . 'vendor/autoload.php';

My weather station PHP is located here:

/var/www/weatherstation/updateweatherstation.php

Hi, thanks for the nice post. But one question.

  1. Your station tries to connect (SYN) to 169.47.111.60 (WU).
  2. Your router then redirects (NAT) this request to your server at *.100
  3. Your server responds (SYN, ACK) to this connection request with the ip *.100

But at this stage my weather station rejects (RST, ACK) because it expects a response from 169.47.111.60 (WU) and not *.100. For me this behavior make sense because the socket is build with IP+Port and the IP in this case changed.

So what have you done to circumvent this?

KR, kr0lly

I guess my station doesn’t care about that because it’s not throwing any errors.
It keeps reporting values via that way. I guess I got lucky otherwise I would have to forge a reply using a reverse proxy or spoof a packet.

Also I don’t actually post to the WU any more, I send all my stats to a local database.

Im still confused how it is working for you. Due to the nature of TCP using IP and port. But it obviously works for you. As it did unfortunately not work for me i needed to find a another way. As there are may be some one else facing the same problem i just wanted to leave my solution her also.

So to get the data i used this nice firewall rule:

iptables -t mangle -A PREROUTING -s 10.10.10.80 -j TEE --gateway 10.10.10.20

This make all the traffic originating at my weather station (.80) so my internal server (.20). This allows the weather station to connect to WU and therefore sends the data there. Which is duplicated and also send to my server.

Here i used the following python script to sniff the traffic and filter out the interesting GET request of the weather station and shove it into mqtt.

from scapy.all import *
from scapy.layers.http import HTTPRequest # import HTTP packet

import urllib.parse as urlparse
from urllib.parse import parse_qs

import paho.mqtt.client as mqtt

MQTT_IP = "10.10.10.20"

BASE_TOPIC = "sensor/home/garden/weather_station"
WIND_DIR_SHORT_NAME = ["N", "NNO", "NO", "ONO", "O", "OSO", "SO", "SSO", "S", "SSW", "SW", "WSW", "W", "WNW", "NW", "NNW", "N"]


def toC(temp_f):
    return round((temp_f - 32) * 5/9, 2)

def toHPA(baromin):
    return round(baromin*33.86388, 2)

def toKMH(speed):
    return round(speed*1.609344, 2)

def toWindDirStr(dir):
    return WIND_DIR_SHORT_NAME[int(round(dir/22.5, 0))]

def toMM(inch):
    return round(inch*25.4, 2)

def pkt_callback(pkt):
    if pkt.haslayer(HTTPRequest):
        if pkt[HTTPRequest].Method.decode() == 'GET':
            path = pkt[HTTPRequest].Path.decode()

            dic = parse_qs(urlparse.urlparse(path).query)

            client.publish(BASE_TOPIC + "/barom_hpa", toHPA(float(dic['baromin'][0])))
            client.publish(BASE_TOPIC + "/temp_c", toC(float(dic['tempf'][0])))
            client.publish(BASE_TOPIC + "/dewpt_c", toC(float(dic['dewptf'][0])))
            client.publish(BASE_TOPIC + "/humidity", float(dic['humidity'][0]))
            client.publish(BASE_TOPIC + "/windspeed_kmh", toKMH(float(dic['windspeedmph'][0])))
            client.publish(BASE_TOPIC + "/windgust_kmh", toKMH(float(dic['windgustmph'][0])))
            client.publish(BASE_TOPIC + "/winddir", toWindDirStr(float(dic['winddir'][0])))
            client.publish(BASE_TOPIC + "/rain_mm", toMM(float(dic['rainin'][0])))
            client.publish(BASE_TOPIC + "/dailyrain_mm", toMM(float(dic['dailyrainin'][0])))
            client.publish(BASE_TOPIC + "/indoortemp_c", toC(float(dic['indoortempf'][0])))
            client.publish(BASE_TOPIC + "/indoorhumidity", float(dic['indoorhumidity'][0]))

if __name__ == '__main__':
    
    client = mqtt.Client()
    client.username_pw_set(username="XXX",password="XXX")
    client.connect(MQTT_IP)

    sniff(filter="tcp and tcp[tcpflags] & (tcp-push) != 0 and src host 10.10.10.80 and dst port 80", prn=pkt_callback)

Not the prettiest code but works. Hope it can help someone.

@drbytes I’m very interested by your post :ok_hand:. Unfortunately beeing a numb in software :worried: I was not able to track my Bresser data in Wireshark. Hope you can detail a bit more how you proceed.
Thank you for supporting me.

I talked with works colleagues who indicated me that I may need to connect the station on some other wifi router and to connect the RJ45 of this router thru hub to my box. Then connect my computer to hub and spy lan connection.
I’ll let know the result of my investigations and if I also see discussion between PWS and weather cloud.
Best regards

hi,
i am very intersted in your solution, but had a hard time recrate it. can you tell me how you actually start the scritp ? i use HA OS with pyscript integration, but think im far off of using this correclty, as in the pyscirpt exampels, there is nothing like your if __name__ == '__main__': mentioned.

many thanks!

hi,

i don’t know your complete setup. I just start the script with systemd as boot time. So the script is written to be executed like so. python3 script.py . I’m not familiar with pyscript but it seems that you can just create a python script and set it up so it is stated automatically. On a quick look you only have to change the if __name__ == '__main__': to a normal function like def start_script(): and then configure your pyscript correctly.
Keep in mind that the script is currently designed to run the whole time. (It will not stop by itself).

Does this help or do you need something else?

Did you figure it out? Key is that you have full control over your router and are able to set rules that forward the traffic to another destination.

thanks for your reply.

actually i already have changed he if __name__ == '__main__': to

@service
def weather():

for a testint purpose. as you metioned the script to run the whole time. indeed, when executing the script comes to sniff(...), HA somehow freezes and becomes unresponsive. do you have a clue why this is the case, and how to aviod this ?

ok, so if i use sniff like this:

sniffedpack = sniff(count=1, filter="tcp and tcp[tcpflags] & (tcp-push) != 0")

, there is no freeze. but as soon as i add this:

sniffedpack = sniff(count=1, filter="tcp and tcp[tcpflags] & (tcp-push) != 0 and src host 192.168.1.221")

home assistant freezes. thils lets me conclude, that as long a sniff() is waiting for packets, home assinstant somewaht freezes. and second, there is not even one packet from my weather sation (192.168.1.221) detected while sniffing ( e.g. while freeze persists, which it was up to now always long enough, that i reseted ha after a couple of minutes), right ?

Just tell your router to forward all requests coming from the weather station to another host where you can parse the values and inject them into homeassistant.

Hi
Yes i figure out to spy it with wireshark. I can see the data.
Now the question is how to pull all data from weather station. I mean data from all other sensors connected. On my PWS I can connect up to 7 more sensors than the main 5 in 1.

Beside that I got also in in the spy informations from weather cloud which pull information only every 10 minutes where weather underground pulls every 5 seconds! Huge amount of data pulled
Can we setup this pulling in WU?

Try to first check if your forwarding is working. This can be done by forwarding it to your pc and use wireshark to check if you can see the packages.

I don’t actually know what you trying to do. If you see the data, just update the script to also filter it out and sent it to mqtt.

We both @drbytes and i don’t use the information on weather underground. We sniff all the available date and store it in our HA so no need to pull anything form WU.

@Kr0llx Sorry if you do not understand what am I doing :confused:. As explained I’m numb in software so I do not know what are the next step and I was just trying to capture what my PWS was sending either to WU or WC. I still do not know how other data are pulled and if I need to order 1 month free test to WC to understand how all the other data from my 7 other sensors are pulled from PWS.

I have no Home Assistant. I have NAS Synology and I’m trying to find out how to use it as HA or use web server to collect data (without know how).
Hope you can share information about your HA or web server type, hardware and software and if you have some more detailed guideline.
thanks in advance

Honestlty I fear that this is beyond what this post is trying to convey.