Pass double quotes as part of the variable in RESTful command

Configuration
1

Hello everybody,

I need to send a curl command like this:

curl -i -H "Content-Type: application/json" -X POST -d '{ "days_to_retrieve": 21, "model_type": "KNN", "var_model": "sensor.consumption_w", "sklearn_model": "KNeighborsRegressor", "num_lags": 48, "split_date_delta": "48h", "perform_backtest": "False"}' http://localhost:5000/action/forecast-model-fit

Please note the double quotes, which are correct (both for variable names and variable values, when not a number).

In order to execute the command I’m using the rest_command integration and I plan to invoke it using an automation, so that I can pass the variable values dynamically if needed (and I do not have to restart the system every time I make a change in the command values).

This is the rest command I wrote in config.yaml:

emhass_forecast_model_fit:
  url: http://localhost:5000/action/forecast-model-fit
  method: POST
  content_type: 'application/json'
  payload: '{ "days_to_retrieve": {{ days_to_retrieve }}, "model_type": {{ model_type }}, "var_model": {{ var_model }}, "sklearn_model": {{ sklearn_model }}, "num_lags": {{ num_lags }}, "split_date_delta": {{ split_date_delta }}, "perform_backtest": {{ perform_backtest }} }'

and this is my test automation:

- id: 'forecast_model_fit'
  alias: "forecast_model_fit"
  description: "forecast_model_fit test"
  trigger: []
  condition: []
  action:
    - service: rest_command.emhass_forecast_model_fit
      data:
        days_to_retrieve: 21
        model_type: "KNN"
        var_model: "sensor.consumption_w"
        sklearn_model: "KNeighborsRegressor"
        num_lags: 48
        split_date_delta: "48h"
        perform_backtest: "True"
  mode: single

The problem is with the double quotes that are not passed to the command.
I already tried a lot to escape the quotes but with no success.

Here a couple of logs with my failures:
Using the automation above as it is with simple double quotes:

2023-10-13 01:36:11.247 WARNING (MainThread) [homeassistant.components.rest_command] Error. Url: http://localhost:5000/action/forecast-model-fit. Status code 400. Payload: b'{ "days_to_retrieve": 21, "model_type": KNN, "var_model": sensor.consumption_w, "sklearn_model": KNeighborsRegressor, "num_lags": 48, "split_date_delta": 48h, "perform_backtest": True }'

When I tried to escape \":

2023-10-13 01:37:23.259 WARNING (MainThread) [homeassistant.components.rest_command] Error. Url: http://localhost:5000/action/forecast-model-fit. Status code 400. Payload: b'{ "days_to_retrieve": 21, "model_type": \\"KNN\\", "var_model": \\"sensor.consumption_w\\", "sklearn_model": \\"KNeighborsRegressor\\", "num_lags": 48, "split_date_delta": \\"48h\\", "perform_backtest": \\"True\\" }'

Any idea on how I can pass the variable values together with the double quotes?
Thank you!

2

Did you try escaping the double quotes with a single backslash? Can you see the server logs at what is listening on 5000 to see what the server is seeing? I would have thought it would work without any escaping, so I wonder why the :5000 server is unhappy.

3

I thimk what you’re looking for is how to do string concatenation. The operator is ~.

"days_to_retrieve": {{ '"' ~ days_to_retrieve ~ '"' }}
4

I tried \"KNN\" but as you can see in the log this is what is passed \\"KNN\\".
So the backslash seems to be duplicated and the server still doesn’t like it.

5

let me give it a try

6

I think the double \ may be something related to the log visualization, actually. Anyway the server still doesn’t like it.

8

It properly concatenates the strings but I also had to update the rest command like this, otherwise HA was not accepting the syntax:

emhass_forecast_model_fit:
  url: http://localhost:5000/action/forecast-model-fit
  method: POST
  content_type: 'application/json'
  payload: >-
    '{ "days_to_retrieve": {{ days_to_retrieve }}, "model_type": {{ '"' ~ model_type ~ '"' }}, "var_model": {{ '"' ~ var_model ~ '"' }}, "sklearn_model": {{ '"' ~ sklearn_model ~ '"' }}, "num_lags": {{ num_lags }}, "split_date_delta": {{ '"' ~ split_date_delta ~ '"' }}, "perform_backtest": {{ '"' ~ perform_backtest ~ '"' }} }'

So I would say the question is answered, thank you!

Still the sever doesn’t like something. I’ll have to investigate and see if as suggested by ThomDyson I can find something in the server logs, which is quite poor from this perspective. :frowning:

Error. Url: http://localhost:5000/action/forecast-model-fit. Status code 400. Payload: b'\'{ "days_to_retrieve": 21, "model_type": "KNN", "var_model": "sensor.consumption_w", "sklearn_model": "KNeighborsRegressor", "num_lags": 48, "split_date_delta": "48h", "perform_backtest": "True" }\''
1 Like
9

Where did that b '\' come from? What’s with the trailing \ ?

10

Great!

Right, that would be because you need it to be a JSON string. If you have a dict, you can use the to_json templating function too. Just giving that as another option. What you have works, so that’s fine.

If you wanted to stick with a one-liner, it would be:

payload: '\'{ "days_to_retrieve": {{ '"' ~ days_to_retrieve ~ '"' }}, ... }\''

The outermost quotes are for HA, so that it will read it as a string and allow the templating inside it. The second level quotes are to make it all a JSON string, and then the innermost ones are for the fields. It can get confusing for many.

Lots of quotes, so like you I’d prefer the multiline indicator, but just for learning’s sake.

1 Like
11

No idea. That’s for the same reason I think the double \ are related to log visualization.
But I need to further investigate and compare with a similar command I know it works and check the logs for comparison.

12

I think that single quotes are automatically escaped in the log messages. At this point I need to understand if this is only in visualization or if it’s passed as part of the payload.

In fact:

'{ "days_to_retrieve": {{ days_to_retrieve }}, "model_type": {{ '"' ~ model_type ~ '"' }}, "var_model": {{ '"' ~ var_model ~ '"' }}, "sklearn_model": {{ '"' ~ sklearn_model ~ '"' }}, "num_lags": {{ num_lags }}, "split_date_delta": {{ '"' ~ split_date_delta ~ '"' }}, "perform_backtest": {{ '"' ~ perform_backtest ~ '"' }} }'

becomes:

\'{ "days_to_retrieve": 21, "model_type": "KNN", "var_model": "sensor.consumption_w", "sklearn_model": "KNeighborsRegressor", "num_lags": 48, "split_date_delta": "48h", "perform_backtest": "True" }\'

plus additional outermost single quotes (which really are part of the logging visualization, maybe together with the trailing “b”):

Error. Url: http://localhost:5000/action/forecast-model-fit. Status code 400. Payload: b'\'{ "days_to_retrieve": 21, "model_type": "KNN", "var_model": "sensor.consumption_w", "sklearn_model": "KNeighborsRegressor", "num_lags": 48, "split_date_delta": "48h", "perform_backtest": "True" }\''

This would also be supported by the fact I tried to move '"' ~ outside the template and I got the following:

Error. Url: http://localhost:5000/action/forecast-model-fit. Status code 400. Payload: b'\'{ "days_to_retrieve": 21, "model_type": \'"\' ~ KNN ~ \'"\' , "var_model": \'"\' ~ sensor.consumption_w ~ \'"\' , "sklearn_model": \'"\' ~ KNeighborsRegressor ~ \'"\', "num_lags": 48, "split_date_delta": \'"\' ~ 48h ~ \'"\', "perform_backtest": \'"\' ~ True ~ \'"\' }\''

which clearly shows automatically escaped single quotes.

13

Yep, that was the problem. It seems that in this case additional single quotes are automatically added somehow.
I modified the RESTful command to this (multiliner, no external single quotes) and it worked (because it seems the single quotes are automatically added).

emhass_forecast_model_fit:
  url: http://localhost:5000/action/forecast-model-fit
  method: POST
  content_type: 'application/json'
  payload: >-
    { "days_to_retrieve": {{ days_to_retrieve }}, "model_type": {{ '"' ~ model_type ~ '"' }}, "var_model": {{ '"' ~ var_model ~ '"' }}, "sklearn_model": {{ '"' ~ sklearn_model ~ '"' }}, "num_lags": {{ num_lags }}, "split_date_delta": {{ '"' ~ split_date_delta ~ '"' }}, "perform_backtest": {{ '"' ~ perform_backtest ~ '"' }} }

Now I’m really curious to understand why this different command is and was already working.

emhass_naive_mpc_optim:
  url: http://localhost:5000/action/naive-mpc-optim
  method: POST
  content_type: 'application/json'
  payload: '{ "prediction_horizon": {{ prediction_horizon }}, "num_def_loads": {{ num_def_loads }}, "P_deferrable_nom": {{ P_deferrable_nom }}, "def_total_hours": {{ def_total_hours }}, "treat_def_as_semi_cont": {{ treat_def_as_semi_cont }}, "set_def_constant": {{ set_def_constant }}, "soc_init": {{ soc_init }}, "soc_final": {{ soc_final }}, "load_cost_forecast": {{ load_cost_forecast }}, "prod_price_forecast": {{ prod_price_forecast }}, "alpha": {{ alpha }}, "beta": {{ beta }}  }'

Maybe it’s related to the explanation provided by @parautenbach

and I thought I was providing the single quotes as required per command

curl -i -H "Content-Type: application/json" -X POST -d '{ "days_to_retrieve": 21, "model_type": "KNN", "var_model": "sensor.consumption_w", "sklearn_model": "KNeighborsRegressor", "num_lags": 48, "split_date_delta": "48h", "perform_backtest": "False"}' http://localhost:5000/action/forecast-model-fit

while I was/am actually just making it HA acceptable.