Curl api calls, switch entity and ESPHome combo not quite right? [SOLVED - bug in API or error in Help files]

Sprinkler device is ESP Home based, running on WEMOS DI R2 arduino type board. The board has a quad relay shield set up as four “switches”. This all works fine from the automation scripts in HA.

So I thought I would look at the REST API and play with CURL commands from my remote console.

Search with “Turn the light on:” to find the example on the help page.

Problem starts when I try a POST, using the curl based example from the REST API “help” page, which is the following:

$ curl -X POST -H "Authorization: Bearer ABCDEFGH" \
       -H "Content-Type: application/json" \
       -d '{"entity_id": "switch.christmas_lights"}' \
       http://localhost:8123/api/services/switch/turn_on

Note I drop the use of “\” when typing in the command onto one line in my console, so I am only using “\” in this post for formatting, if I use it at all.

With “switch.christmas_lights” set to my “switch.zone_1” and ABCDEFGH switched to my permanent token and “localhost” set to the IP of my HA server; I get the following when I use resulting the curl command:

curl: (3) [globbing] unmatched close brace/bracket in column 14
{"message": "Data should be valid JSON."}

So the “switch” in HA is not getting turned on. The relay on the physical device, representing the virtual switch in HA, is certainly not coming on.

The switch and associated relay both still work happily from the HA automations whenever the physical is powered on.

Note the domain of the device in the automations is “switch”, so I am assuming it need be “switch” in the “entity_id” in the curl POST command.

So, is the example POST in the HA help correct?

I have been curling using the same token and IP to GET various data responses based upon the GET examples on the same page. So changing GET to POST and adding the -d clause should not have been a problem.

To save hassles the full call (excluding my actual token) is:

$ curl -X POST -H "Authorization: Bearer ABCDEFGH" \
       -H "Content-Type: application/json" \
       -d '{"entity_id": "switch.zone_1"}' \
       http://192.168.0.100:8123/api/services/switch/turn_on

Moreover, if it helps, the following works fine:

curl -X GET -H "Authorization: Bearer ABCDEFGH"    \    
   -H "Content-Type: application/json"  \
   http://192.168.0.100:8123/api/states/switch.zone_1

Happily returning:

{"attributes": {"friendly_name": "Zone 1", "icon": ""}, "context": {"id": "ad87fc27c3304a5fab6dcf27a8e5868c", "parent_id": null, "user_id": null}, "entity_id": "switch.zone_1", "last_changed": "2020-04-18T14:05:49.638539+00:00", "last_updated": "2020-04-18T14:05:49.638539+00:00", "state": "unavailable"}

Yep, and for the discerning audience, you’ll note the device “state”: “unavailable”. Before you get exited, if I power the device up that then makes the state change from “state”: “unavailable” to “state”: “off” (the default on powerup of the device).

If I then try the curl POST command again, to turn the switch on, the same error response behaviour ensues, that is:

curl: (3) [globbing] unmatched close brace/bracket in column 14
{"message": "Data should be valid JSON."}

So, no it does not depend on switch available state.

Depressing also, there appears nothing in the developers log relating to any of the API based transactions, including the errors.

Any suggestions please?

Cheers,
B

First of all, I can confirm that posting valid JSON to that end point does indeed work. I just tried a similar thing on my install, and was able to turn a switch on and off.

It bothers me that curl is saying

unmatched close brace/bracket in column 14

and column 14 in your JSON is a space. Makes me suspect the quotes around your JSON are being swallowed somewhere. The quick and dirty way to confirm that would be to remove the space from your JSON, ie try sending something like

'{"entity_id":"switch.zone_1"}'

Something else you can try to figure out exactly what’s going on, is with the trace-ascii flag, which tells curl to dump all incoming and outgoing data to a file. Try something like this to dump all of that to stdout:

$ curl -X POST --trace-ascii - \
       -H "Authorization: Bearer ABCDEFGH" \
       -H "Content-Type: application/json" \
       -d '{"entity_id": "switch.zone_1"}' \
       http://192.168.0.100:8123/api/services/switch/turn_on

@stibbons thanks. I will have a poke at it. Thought 1) I used the formatting of the example so still begs the question if the example is right and 2) I may have already tried removing spaces while trying to debug so who knows and 3) if it is formatting is it really a problem with parsing.

The column 14 might be a furphy since I reformatted with “\” for the post, so I will try again and post without reformatting.

Voila!

C:\Users\pierc>curl -X POST --trace-ascii - -H "Authorization: Bearer ABCDEFGH"    -H "Content-Type: application/json" -d '{"entity_id":"switch.zone_1"}'  http://192.168.0.100:8123/api/services/switch/turn_on
Note: Unnecessary use of -X or --request, POST is already inferred.
== Info:   Trying 192.168.0.100...
== Info: TCP_NODELAY set
== Info: Connected to 192.168.0.100 (192.168.0.100) port 8123 (#0)
=> Send header, 369 bytes (0x171)
0000: POST /api/services/switch/turn_on HTTP/1.1
002c: Host: 192.168.0.100:8123
0046: User-Agent: curl/7.55.1
005f: Accept: */*
006c: Authorization: Bearer ABCDEFGH
00ac: ... token continues
00ec: ... token continues
012c: ... token continues
013b: Content-Type: application/json
015b: Content-Length: 27
016f:
=> Send data, 27 bytes (0x1b)
0000: '{entity_id:switch.zone_1}'
== Info: upload completely sent off: 27 out of 27 bytes
<= Recv header, 26 bytes (0x1a)
0000: HTTP/1.1 400 Bad Request
<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
<= Recv header, 20 bytes (0x14)
0000: Content-Length: 41
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 30 Apr 2020 11:42:06 GMT
<= Recv header, 34 bytes (0x22)
0000: Server: Python/3.7 aiohttp/3.6.1
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 41 bytes (0x29)
0000: {"message": "Data should be valid JSON."}
{"message": "Data should be valid JSON."}== Info: Connection #0 to host 192.168.0.100 left intact

Or without the ascii switch:

curl -X POST  -H "Authorization: Bearer ABCDEFGH"    -H "Content-Type: application/json" -d '{"entity_id":"switch.zone_1"}'  http://192.168.0.100:8123/api/services/switch/turn_on 

Note there was one internal space after the “:” so deleting that causes the complaint about the unmatched bracket to drop but the invalid JSON result persists, that is:

{"message": "Data should be valid JSON."}

So that space was at column 14 right? That gives me a better sense of where the error report was coming from thanks.

All GET examples I am running with same token and IP are still working.

Ah, actually, I think I have to go away from using ESPHome and homeassistant combination in any event. So no longer interested chasing down this bug.

My raspingdoodleburry pi has crashed. Yes, HA would come up on restart. But, I found the ESPHome based sprinkler system has 3 of 4 relays on.

Having had a $3k water bill some time back due to sprinkler system leak I gauge it makes no sense keeping this combination of HA and ESPHome for a “mission critical” application, since I am not sure when the p p p p p Pi went d d d d down.

The evidence of a problematic interaction between HA and ESPHome in server crash condition is the relays in this HA/ESPHome version are set up in automations, to come on and turn off in sequence, with a gap in time between off and on, to avoid having to deal with sharing the water pressure. This works fine with HA up and running, so having 3 of 4 on will be an artefact of Pi+HA crashing and likely some interaction between HA and ESPHome. That is, there are no automations in HA that would set 3 of 4 relays on at the same time and so no two relays should be on at the same based on the design of the automations. The automations have been running on a test rig and come on sunrise and sunset without a problem and the logs did, till the crash of Pi, reflect that expected behaviour.

That likely means the usual ping ponging between two user groups blaming the “other” software and, frankly, not my bag. I want to water my gardens and not solve a problem with use cases not being tested.

Even worse, as bringing the Pi and therefore HA back online does not automatically reset ESPHome device. Sure, you could set up something on HA start/restart but that does nothing for the span of time the Pi/HA is down and relays on ESPHome device on. Way too fiddly to sort all likely mishap cases, especially if the actual cause of crash and the breakdown of HA/ESPHome communication causing the relays to come on is unknown. So, even if the Pi/HA came back up by itself, there’d still likely be $K dollar water bills in the mail.

Not a problem, I have previously hand written code to run on the WEMOS UNO format board for the sprinkler system. That code is MQTT driven and was designed from scratch for a sprinkler system with ample safe guards. For example it wont turn on water for less than 5 mins, won’t accept more that a 15 minute request and will turn off a sprinkler after 20 minutes. It will allow a new time setting over top the old so if you want a hour of watering you just ping it with 4x15 minute requests every 15 minutes. I have a node-red setup that also will send an sms if a sprinkler node (either a 4x or 1x wemos setup) raises an LWT. I’ve even designed my own boards to support the 24VAC to WEMOS voltage conversion.

I also have a peg based parser to all setting of sprinkler times, across multiple single and quad sprinklers, via titles of google calendar events. Though, I want to decouple from google and try the same over a local mechanism for both principled and practical reasons.

So, I am also looking at how to hook my parser code into other calendar widgets. I am still to look at HA calendar widgetry, if it exists, since automations are fine but the natural thing is to just want to use even just a weekly calendar and not a full year calendar, or so I imagine I think I want to believe. Although, then node-red calendar widgetry or HA calendar widgetry? It will depend upon ease of integration.

In the end, I actually suspected my hand coded sprinkler option was likely the better approach for exactly what the crashing doodle woodle pi and the interaction between HA and ESPHome has revealed. I would even suspect the ESPHome web site, and the HA one to, have caveats and warnings about not using either for mission critical applications. Especially, if its the emergent properties of the integrations that will get you and no one is really going to arbitrate that except for the end users.

I poked around a bit with this, the problem is a little more insidious.

I thought the server had crashed as I could not log in with MobaXTerm. In fact, I have my server with Ethernet wire connection to a wifi extender and it turned out that the wifi extender has been dropping connection with the ASDL router.

So, I found that re-connecting the wifi extender to the ASDL router saw an interesting picture. The ESPHome device must have reconnected to the HA server since the switches on the HA HMI, representing the relays in the ESPHome device, had the same state as the device. 1, 2 and 4 on, 3 off.

Not a problem you say. Well, it is since the way the system is set up is that you turn the switch on at HA HMI, and automation runs and turns the switch off after 15 minutes.

On top of that, that behaviour was also required when driving the relays by sunrise/sunset since a delayed trigger to turn the switch on, on sunrise/sunset, was used and the trigger to turn the switch off after 15 mins was expected then to trigger on the switch being turned on. Worked a treat while connection between HA and ESPHome device stayed up.

The problem still remained that three relays on ESPHome device were on (1,2,4) and one was off (3). Three switches on HA HMI were on (1,2,4) and one was off (3). HA log reported all switches off. So, there was a disconnect between states internally to HA somewhere.

The automations for all four switches are all triggered on sunrise/sunset. They have delays that set the individual switches on 20 minutes apart. The second set of automations, triggered by individual switches going on, turns the triggered switch off after 15 mins. So, the system works:

  • sunrise->all four switches triggered with delay x mins.
  • 0 mins->switch 1 on trigger switch 1 off automation
  • 15 mins->switch 1 off by switch 1 off automation
  • 20 mins->switch 2 on trigger switch 2 off automation
  • 35 mins->switch 2 off by switch 2 off automation
  • 40 mins->switch 3 on trigger switch 3 off automation
  • 55 mins->switch 3 off by switch 3 off automation
  • 60 mins->switch 4 on trigger switch 3 off automation
  • 75 mins->switch 4 off by switch 4 off automation

Nothing really then to account for why 3 of 4 relays on, why HMI reflected ESPHome device state and then why if relays are on in ESPHome device the associated switch on in HA did not result in a switch off after 15 mins. That is, I have had the ESPHome device powered up and 3 of 4 relays have been on for two days, where they should have off after 15 mins if triggered on by HA. If I manually switch the switches at HA HMI to off today, they are then off at device. I haven’t rebooted device because I want to see if the states coherence is re-established now that all switch states are off. I assume that states will be coherent again if the automations run tonight and the relays behave properly again.

It might have also been useful to just watch the behaviour at sunset since I would almost bet 1,2 and 4 would not behave and 3 would come on 40 minutes after sunset and turn off 55 mins after sunset. Won’t know now since I have polluted the setup by setting everything back to off.

Noting that that won’t redeem the setup for use in a critical applications, since it would take too much energy to understand the interplay and this appears to be a state coherency problem in either or both the design of HA or ESPHome or both.