Push sensor vales to luftdaten

fversteegen · July 4, 2020, 4:39am

I have been looking into a way to push sensor values to luftdaten.info. Up until now the only option I have found is to install the luftdaten firmware on my wemos or ESP32 devices and then get the data back into Homeassistant via the Luftdaten sensor. This however conflicts a little bit with my, and the homeassistant, philosophy that Homeassistant should not have to rely on cloud services and operate locally as much as possible. I would therefore ideally like to rely on the robust esphome integration and afterwards push values to Luftdaten from within Homeassistant.

There is an API available for pushing data to Luftdaten http://api.luftdaten.info/v1/push-sensor-data/ but I have next to zero scripting experience.

Does anyone have a script or something available to achieve this in an easy fashion?

exxamalte · July 4, 2020, 8:18am

I was thinking about making the same change to my current setup just today. I am currently pushing the data from my Luftdaten device into a local Influxdb and from there pull the information into Home Assistant. That approach avoids using the detour through the cloud.

When moving to ESPHome to drive the device the high level approach would be:

Configure device to present sensors of PM2.5 and PM10 to Home Assistant (standard ESPHome functionality).
Create rest_command that points to the Luftdaten API URL you have already found. The payload of the command will be configured as a template and contains some fixed data (like your sensor ID) as well as some variable data (like the current sensor values).
Create automation that gets triggered every time the PM2.5 and PM10 are both updated, and in the action call the rest_command.

I just captured the payload that my sensor has sent:

{
  "software_version": "NRZ-2020-129", 
  "sensordatavalues":[
    {"value_type":"P1","value":"2.93"},
    {"value_type":"P2","value":"1.83"}
  ]
}

My sensor ID was sent as part of the User-Agent HTTP header, but this could also be added to the payload.

fversteegen · July 6, 2020, 4:47am

Apparently someone already did most of the hard work now we just need to transform it into something usable for Homeassistant…

github.com

corny/luftdaten-python/blob/236fbf88978626bc81f356e7487ed8d0e07bc691/main.py

#!/usr/bin/env python3

import sys
import os
import yaml

# Import-Pfade setzen
sys.path.append(os.path.join(sys.path[0],"sds011"))
sys.path.append(os.path.join(sys.path[0],"bme280"))

import time
import json
import requests
import numpy as np
from sds011 import SDS011
from Adafruit_BME280 import *

# Config
with open("config.yml", 'r') as ymlfile:
    config = yaml.load(ymlfile)

This file has been truncated. show original

Hmmbob · August 21, 2020, 2:29pm

Did any of you get around this? I just built a nice sensor with my oldest son, found the luftdaten ‘cloud detour’ to get data into HA too cumbersome, so I set up a ESPHome device and am able to get all data in HA. I’d still like to push the data to Luftdaten though.

fversteegen · August 22, 2020, 5:50am

I haven’t due to vacation and extremely rusty coding skills…

fversteegen · August 22, 2020, 6:06am

Found an even simpler script, but now need some help in getting the sensor values from Homeassistant into it…

EDIT: some progress…

Find your sensebox ID (go to: https://opensensemap.org/account and copy the ID)
Find the ID of the sensor you would like to update (go to https://opensensemap.org/account click under the right sensebox “EDIT”, Sensors and find the long sensor ID)
Create a rest_command like below:

rest_command:
  post_opensensebox:
    url: https://api.opensensemap.org/boxes/SENSBOXID/SENSORID
    method: post
    headers:
      Content-Type: application/json
    payload: '{"value": "{{states.sensor.PM10.state }}" }'

Use this in an automation to push the values.

Hmmbob · August 23, 2020, 1:13pm

Thanks, you put me on the right track. I’ve got it all implemented right now.

So, assuming that you already have the data available in Home Assistant, one could use the following automation to push:

github.com

hmmbob/HomeAssistantConfig/blob/master/automation/air_quality.yaml

##################################
## Push values to Air Quality APIs
##################################
# Sensor uses a NodeMCUv2 and SDS011+BME280 sensors. NodeMCU is flashed with ESPHome
# software. SDS011 updates every 30 mins, the BME280 every 5 minutes. As all BME280
# sensors update at the same time, we only need to monitor one sensor to trigger
# the automation. # Temperature is the sensor that changes most often.
- alias: 'Update Air Quality Sensors'
  initial_state: true
  mode: queued
  trigger: # Trigger on every update of the temperature
    platform: state
    entity_id: sensor.bme280_temperature 
  action:
    ## Send to Luftdaten / Sensor.Community API
    ## Split over two calls as the headers differ
    - service: rest_command.send_luftdaten_tph
    - service: rest_command.send_luftdaten_pm
    ## Send updates to Madavi.de too for statistics.
    - service: rest_command.send_madavi

This file has been truncated. show original

As you can see, the automation calls for 4 rest_commands. These are defined as follows:

github.com

hmmbob/HomeAssistantConfig/blob/master/includes/rest_command.yaml

##################################
## Push values to Air Quality APIs
##################################
# Push to Luftdaten API. Luftdaten uses headers to distinguish between different sensor
# types, so we need to push twice.
send_luftdaten_pm:
  url: https://api.sensor.community/v1/push-sensor-data/
  method: POST
  content_type: 'application/json'
  headers:
    X-Pin: 1  ## This tells Luftdaten that it is SDS011 data
    X-Sensor: !secret luftdaten_x_sensor
  payload: >
    {
      "software_version": "HomeAssistant-{{ states('sensor.current_version') }}",
      "sensordatavalues":[
        {"value_type":"P1","value":"{{ states('sensor.particulate_matter_10_0um_concentration') }}"},
        {"value_type":"P2","value":"{{ states('sensor.particulate_matter_2_5um_concentration') }}"}
      ]
    }

This file has been truncated. show original

As the rest_commands cannot process !secrets, I used a workaround via input_text:

github.com

hmmbob/HomeAssistantConfig/blob/master/includes/input_text.yaml

##################################
## Input text configuration
##################################

# I am using this workarond to hide sensor-IDs needed in the rest_command 
# payloads. These templates cannot handle secrets, but you can set these 
# input_text items with !secret, so in this way the IDs are obfuscated.
# I don't mind having these in my HA instance, I just don't want them on 
# GitHub to prevent copy-paste errors.

opensensebox_sensorid_temp:
  initial: !secret opensensebox_sensorid_temp

opensensebox_sensorid_press:
  initial: !secret opensensebox_sensorid_press

opensensebox_sensorid_hum:
  initial: !secret opensensebox_sensorid_hum

opensensebox_sensorid_pm25:

This file has been truncated. show original

Secrets explained:

!secret luftdaten_x_sensor: contains Luftdaten sensor ID (typically something like esp8266-123456)
!secret opensensebox_url: contains API endpoint for this sensor box (something like https://api.opensensemap.org/boxes/<<yourSenseBoxID>>/data

fversteegen · August 23, 2020, 6:24pm

I don’t know how my minor brain waves caused you to create this whole script, but judging by reading it it entirely fulfills my needs too

Hmmbob · August 23, 2020, 6:41pm

It was a learning opportunity, I built this sensor together with my son (10) and he wanted to upload the data too. So we made it happen.

And kudos to Frenck for helping me tweaking the scripts too. Had an initial proof-of-concept, but that had a lot of duplicate code. Thanks to his input i was able to simplify the script considerably.

And yes - your brainwave put me on the right track

fversteegen · August 24, 2020, 10:10am

Aha: you got help from the master himself

It’s now up and running fine at my end! There were 2 things I had to change in your script:

You multiplied the pressure with 100 before sending it out; I removed that as a pressure of 101540 Pa seems a bit high
I edited the payload for the opensensemap so it reflected my sensors (the part in bold below)

{
“5f2XXXXXXXXXXXX”: “{{ states(‘sensor.bme280_temperature’) }}”,
“5f2XXXXXXXXXXXXXXXXXX”: “{{ states(‘sensor.bme280_pressure’) | float * 100 }}”,
“5f2XXXXXXXXXXXXXXXXXX”: “{{ states(‘sensor.bme280_humidity’) }}”,
“5f2XXXXXXXXXXXXXXXXXX”: “{{ states(‘sensor.particulate_matter_2_5um_concentration’) }}”,
“5f2XXXXXXXXXXXXXXXXXX”: “{{ states(‘sensor.particulate_matter_10_0um_concentration’) }}”
}

EDIT: still figuring out point 1 as your multiplication appears to be correct, but isn’t correctly reflected in the Opensensemap dashboard…

Hmmbob · August 24, 2020, 11:40am

Yeah, I had to do the multiplication as the value in HA is in hPa, while the APIs are expecting Pa. And you are right, you need to change those values in 2) to your own sensor IDs. I’ll update the file on Github with a note (seems I can’t use !secret there?)

edit: fixed it in a new upload, using input_text with !secret now
Updated my result above.

Hmmbob · August 24, 2020, 5:28pm

Made a small write-up: https://hmmbob.tweakblogs.net/blog/18950/push-data-to-luftdaten-and-opensensemap-with-home-assistant

edjeraket · April 6, 2021, 8:13pm

Hi Bob,
I followed your instructions on Tweakblogs. I get all the information in Home Assistant but i can’t push it to opensensemap. Here is my error in HA:

Can you please help me?

Regards, Edwin.

Hmmbob · April 6, 2021, 8:36pm

Is it persistant? I sometimes have these errors too, and either of these two options is happening:

the data I send is invalid (i.e. one of the sensors is listing “unavailable” instead of a normal value)
the API endpoint times out (but this typically doesn’t give you the 400 error.

So, have a good look at your sensor values and see if anything is wrong with them.

Edit: if you want you can shoot me the full error in DM - using the same nickname all over the place (here, discord, Tweakers)

RT1080 · January 10, 2022, 3:06pm

Hi, i built the sensor last year and just migrated it from the standard luftdaten firmware to esphome - followed the blog and it’s working; however, my SDS updates every couple of seconds as opposed to the 30min I put in the script; any thoughts what might going wrong here?

The dht does update every 5minutes.

sensor:
  - platform: sds011
    pm_2_5:
      name: "Particulate Matter <2.5µm Concentration"
      accuracy_decimals: 2
    pm_10_0:
      name: "Particulate Matter <10.0µm Concentration"
      accuracy_decimals: 2
    update_interval: 30min
  - platform: dht
    pin: D7
    temperature:
      name: "Temperature"
      accuracy_decimals: 2
    humidity:
      name: "Humidity"
      accuracy_decimals: 2
    update_interval: 5min

RT1080 · January 10, 2022, 3:37pm

apologies, found the solution elsewhere:

https://community.home-assistant.io/t/sds011-not-updating-at-specified-interval/338892/2?u=rt1080

Hmmbob · January 10, 2022, 3:44pm

Great, I was just about to reply that I couldn’t see anything wrong with your code

joem · July 26, 2022, 7:56am

Can you confirm this is still working?
I’m getting the following error:
2022-07-26 19:51:36 WARNING (MainThread) [homeassistant.components.rest_command] Error. Url: https://api.opensensemap.org/boxes/xxxxxxxxxxxxx/data. Status code 422. Payload: b'{\n "unknown": "2.2",\n "unknown": "9.2",\n "unknown": "6.91",\n "unknown": "100527",\n "unknown": "80.79"\n}'

Hmmbob · July 26, 2022, 8:36am

Verify your sensor ID’s (basically, anything that saus unknown here)

joem · July 26, 2022, 9:47am

Cheers!
I just realized, that my input_text: !include_dir_merge_named was misconfigured, and that caused a chain of errors.