After nights researching I hope to find help here.
I want to scrape some values from a JavaScript-generated Website
https://main.beehivemonitoring.com/47c9f648af3b43d5ba07256925080639
Like “WEIGHT CHANGE 24H” from the tab “WEIGHT”.
I got no clue how to handle this, I tried so much…
Any hints?
You have all the data in json in https://main.beehivemonitoring.com/api/hives?culture=en-EN
.
You have to add a X-Auth-Token
header, though.
❯ curl -H "X-Auth-Token: 47c9f648af3b43d5ba07256925080639" "https://main.beehivemonitoring.com/api/hives?culture=en-EN"
[{"text":["Everything is all right",null,null,null],"id":6131,"name":"Volk 10","simExpirationDates":[],"emptyWeightKg":0.0,"vbatGw":4.202,"rssiGw":-87,"gsmSignal":-87,"weightToday":-0.10,"weightWeek":2.28,"weightMonth":6.05,"activeBees":6181,"bees":255,"motherStatus":255,"swarming":255,"status":255,"vbatOut":4.171,"tempOut":16.50,"humidityOut":62.0,"weight":45.61,"rssiOut":-80,"maxTempOut":16.50,"minTempOut":16.50,"lastWeight":45.61,"pressureGw":942.8,"pollen":false,"closed":false,"open":false,"eggs":false,"mother":false,"fetusOpened":false,"fetusClosed":false,"toCheck":false,"projectName":"Malte Jaurich","credit":0.0,"validTo":"9999-12-31T23:59:59.999999","upFreq":0.5,"canUseMyBees":false,"notificationTime":75600,"showOnlySmartCity":false}]
So from there you need to use a Rest sensor:
Couple of examples, assuming the data structure above without the :%
on the end (that may cause some trouble if it’s real)
rest:
- resource: https://main.beehivemonitoring.com/api/hives?culture=en-EN
headers:
X-Auth-Token: "47c9f648af3b43d5ba07256925080639"
scan_interval: 600
sensor:
- name: "Weight today"
value_template: "{{ value_json[0]['weightToday'] }}"
unit_of_measurement: "kg"
- name: "Active bees"
value_template: "{{ value_json[0]['activeBees'] }}"
unit_of_measurement: "bees"
Hey, thats really really cool, it works! Thank you both so much.
How did you find out that the data exists in https://main.beehivemonitoring.com/api/hives?culture=en-EN ?
Fire up DevTools on your browser, usually F12. Go to the Network tab and reload the page. Look for resources of type xhr
and find one that contains a response with the data in:
How did you get to know the URL with /api ?
Now I try to read wind speed from a weather-station:
→ Holfuy: EINKORN, Wind und Wettermesswerte
So I find wind speed in mjso.php?k=737 in value “speed”.
But what’s the correct URL?
I do not succeed with:
![image](https://community-assets.home-assistant.io/original/4X/7/d/4/7d49e107f4bb284ee842f72edce2238e5092534b.png)
OK, I get closer to it.
https://holfuy.com/puget/mjso.php?k=737
is the link.
Got it by clicking right mousebutton and:
I tried:
But state is still unknown:
For future requests, please:
- never post code as screenshots
- don’t re-use threads for unrelated requests
TIA
Exact same works for me
rest:
- resource: https://holfuy.com/puget/mjso.php?k=737
scan_interval: 60
sensor:
- name: Test speed
value_template: "{{ value_json['speed'] }}"
unit_of_measurement: "km/h"
Sorry, yes I will
sensor.test_speed
is always zero. I tried a lot but I don’t have a clue why the rest-request doesn’t fetch the right value from the json for “speed”.
The URL is returning zero for speed. Perhaps it’s not very windy. Barometric pressure is high:
{
"valid":"ok",
"updated":"<b>34<\/b> sec. ago",
"speed":0,
"dir":314,
"dir_str":"NW",
"gust":0,
"temperature":29.4,
"wind_chill":29.4,
"humidity":40.3,
"dew_point":14.5,
"cloud_base":"<b>2099<\/b><span style=\"font-size:0.6em;\">m AMSL<\/span>",
"pressure":1012.2,
"s":5,
"avg_speed":0,
"max_gust":0,
"avg_dir":303,
"avg_dir_str":"WNW",
"dir_rate":0,
"dir_tendency":"stable",
"speed_rate":0,
"speed_tendency":"calm",
"daily_avg_speed":0,
"daily_max_wind":6,
"daily_max_temp":31.9,
"daily_min_temp":15,
"rain":"Last rain at Jun-09 Fri 12h"
}
I don’t know what the k
parameter in the URL is. The full API is shown here:
EDIT: I think there’s something needed in the headers of the request. Having looked at the weather site, reloading the php
resource now gives 13 for the wind speed.
Best bet is to get yourself an account on the API rather than trying to pull numbers by guesswork.
Ok, thx, I sent a request for an API key for this weather-station. Thanks a lot for your efforts!!!