Running command_line sensor in Hassio and getting inconsistent results

I’ve just been migrating my HA over to the “dark side” running hassio :wink:.

In my old setup I had some command_line sensors that polled my IP camera’s for the status of motion detection and it worked great.

But now I’m trying to get the same functionality in Hassio and I’m hitting a wall.

I know that Hassio has a different (limited) version of the “grep” command so it won’t work the same way that it did on my old set up so I’ve had to switch to using the “regex_findall_index” filter and I’m pretty positive I can get that to work correctly if I can figure out the command_line flakiness.

The problem is that when I run the command via the command_line sensor I don’t get the correct value returned.

The normal command to return the values I need that I can run in a browser is:

http://192.168.1.52:8002/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=my_user&pwd=my_pass

And it results in the following:

ex

But when I run the following command in a command_line sensor in hassio:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.52:8002/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=my_user&pwd=my_pass" '
  name: "Garage Camera Motion test"

I end up with a sensor with the following state:

ex2

Which doesn’t even come close to matching the expected result.

And to add another layer to the mystery if I console into my hassio container via portainer and run exactly the same command I get the expected results:

There is some other strangeness going on with another command_line sensor but I figured if I can see what the fix is for this one it might help sorting out the other issue too.

the new install of hassio is on a NUC installed in Docker running Debian.

Any thoughts on why there is a different result between the two ways of running the same command?

1 Like

Alright, here is an update to this…

I created another sensor in my currently working non-hassio HA in docker and strangely I get that same result.

Here is the sensor code:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyyy" '
  name: "livingroom Camera Motion test"

And here is the resulting sensor which is in the same format as the one I thought was not working correctly:

ex

I still don’t know why the returned value is different running the same command as a command_line sensor compared to running the exact same command in the console.

And the “grep” portion of the existing command_line sensor works with that command too.

here is that sensor that is working exactly as expected:

- platform: command_line
  name: "Livingroom Camera Motion"
  command: 'curl -k --silent "http://192.168.1.53:8003/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyy" | grep -oP "(?<=motionDetectAlarm>).*?(?=</motionDetectAlarm>)"'
  value_template: >-
    {%- if value == "0" -%}
      Disabled
    {%- elif value == "1" -%}
      None
    {%- elif value == "2" -%}
      Detected
    {%- endif -%}
  scan_interval: 3 
  #friendly_name: 'Livingroom'

So, now I’m guessing it really is the way I’m setting up the regex_findall_index filter in the template that is causing the problem.

Here is the new non-working sensor:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyy" '
  name: "livingroom Camera Motion test template"
  value_template: >
    {% set status = value | regex_findall_index('(.*)</motionDetectAlarm>') %}
    {% if status == "0" %}
      Disabled
    {%- elif status == "1" -%}
      None
    {%- elif status == "2" -%}
      Detected
    {% else %}
      Not Determined
    {%- endif -%}
  scan_interval: 3

and here is the error generated:

2019-08-14 14:09:34 ERROR (MainThread) [homeassistant.components.sensor] command_line: Error on device update!
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/helpers/entity_platform.py", line 291, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/app/homeassistant/helpers/entity.py", line 419, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/app/homeassistant/components/command_line/sensor.py", line 126, in update
    value, STATE_UNKNOWN
  File "/usr/src/app/homeassistant/helpers/template.py", line 245, in render_with_possible_json_value
    error_value,
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/src/app/homeassistant/util/async_.py", line 208, in run_callback
    future.set_result(callback(*args))
  File "/usr/src/app/homeassistant/helpers/template.py", line 270, in async_render_with_possible_json_value
    return self._compiled.render(variables).strip()
  File "/usr/local/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/src/app/homeassistant/helpers/template.py", line 820, in regex_findall_index
    return re.findall(find, value, flags)[index]
IndexError: list index out of range

So i’ll tag @123 & @petro to see if they can shed some light on this confusion.

Please? :slightly_smiling_face:

Try:

regex_findall_index('Alarm>(\d+)</motion')

I noticed something odd about regex_findall_index. If the regex pattern doesn’t find a match, it fails with an error: Unknown error rendering template. That seems rather extreme. I would expect it to simply return None. :man_shrugging:

Because one runs in the Home Assistant container and the other is running in a ssh addon container and they have different commands available.

The easiest way to troubleshoot this is to run a console in the container (Portainer makes this simple to do) and you can then run the command and work out the exact command format needed.

I may not be understanding what you’re saying (likely).

Let me try to make this more clear because i’m sure that my two posts above are really long-winded and probably confusing.

First, I created a command-line sensor:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyy" '
  name: "livingroom Camera Motion test"

And its results are:

ex

Then if I go into the container in Portainer and open the console and run the same command from the sensor above I get the following:

This is in a non-hassio HA running in Docker so I’m not using any add-ons.

I have the exact same result doing my initial testing when trying to switch to hassio in Docker.

The thing that I didn’t realize was that the results in both hassio & non-hassio ended up being the same. I never had to try to get any results from just the command portion and had always filtered those results using “grep” in my non-hassio set up and it had always worked so I had no reason to dig in any further.

Once I dug in trying to switch from “grep” to “regex…” is when I noticed the strangeness in both systems.

I just tried it and I end up with the same result.

Here is the new sensor:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyyy" '
  name: "livingroom Camera Motion test template"
  value_template: >
    {% set status = value | regex_findall_index('Alarm>(\d+)</motion') %}
    {% if status == "0" %}
      Disabled
    {%- elif status == "1" -%}
      None
    {%- elif status == "2" -%}
      Detected
    {% else %}
      Not Determined
    {%- endif -%}
  scan_interval: 3

and the I get the same error:

2019-08-15 05:39:39 ERROR (MainThread) [homeassistant.components.sensor] command_line: Error on device update!
Traceback (most recent call last):
  File "/usr/src/app/homeassistant/helpers/entity_platform.py", line 291, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/app/homeassistant/helpers/entity.py", line 419, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/app/homeassistant/components/command_line/sensor.py", line 126, in update
    value, STATE_UNKNOWN
  File "/usr/src/app/homeassistant/helpers/template.py", line 245, in render_with_possible_json_value
    error_value,
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/src/app/homeassistant/util/async_.py", line 208, in run_callback
    future.set_result(callback(*args))
  File "/usr/src/app/homeassistant/helpers/template.py", line 270, in async_render_with_possible_json_value
    return self._compiled.render(variables).strip()
  File "/usr/local/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/src/app/homeassistant/helpers/template.py", line 820, in regex_findall_index
    return re.findall(find, value, flags)[index]
IndexError: list index out of range

The first thing I would want to do is examine what is contained in value. We know what it is supposed to contain (the CGI_Result data in XML format) but what does it actually contain when retrieved by the command_line sensor. Unless CGI_Result exceeds 255 characters, remove the value_template (so that the entire result becomes visible in the sensor’s state).

I did that already. See my post above replying to David.

That’s the “strangeness” i was referring to.

If I use the command_line sensor in the non-hassio HA that works with “grep” the value_template for that sensor works so it has to be seeing the entire correct value that the command in the browser sees or it couldn’t extract the correct state for the template.

but if I remove the “grep” part of the command the sensor returns something that doesn’t look like what it’s supposed to be.

Sorry, I even read that part but it just didn’t take hold between the ears …

That is strange. It’s not like it fails outright, it gets three lines worth of the XML payload. Everything other than the first item, <result>2</result> is missing. Weird.

It’s even more strange…

I missed it at first too but if you look closely, that isn’t a 2, it’s a -2. Which doesn’t exist anywhere in the payload at all.

but like I said, if the same command_line command is run with the “grep” filter added on to the end as in my original sensor then everything works and I get “detected” & “none” just as I would expect so on some level the curl command has to be working and returning the correct payload for “grep” to filter.

And I’ve got four other camera’s that the old grep method works correctly with, too. So it’s not any kind of coincidence.

And to add a bit more to this the “other strangeness going on with another command_line sensor” that I referred to at the end of post #1 is this sensor for a different camera model that uses a different cgi:

- platform: command_line  
  command: 'curl -k --silent "http://192.168.1.51:8001/get_status.cgi?user=xxxx&pwd=yyyyy" '
  name: "Kitchen Camera Motion test"

When that runs it returns this as the state:

But returns this when the same command is run in the console of the container:

And using grep to pull the “alarm_status=” value it also works as expected.

:confused:

Maybe it’s time for a bug report?

And another update…

It’s working!! :tada:

But it still doesn’t make any sense why it didn’t work before so a bug report may still be in order.

What ultimately fixed it was simply removing the outer quotes from the command portion.

Here is what ended up working:

- platform: command_line  
  command: curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyyy"
  name: "Livingroom Camera Motion"
  value_template: >
    {% set status = value | regex_findall_index('Alarm>(\d+)</motion') %}
    {% if status == "0" %}
      Disabled
    {%- elif status == "1" -%}
      None
    {%- elif status == "2" -%}
      Detected
    {% else %}
      Not Determined
    {%- endif -%}
  scan_interval: 3

But again, the “funny” part was that the command wrapped in quotes worked with the grep filter. And every example listed in the docs for a command_line sensor has the “command:” line written with quotes enclosing the entire command.

It looks like the two ways of retrieving the return payload (from the sensor and from the console) did actually end up working the same because when I tried the command from the console I had removed the outer quotes.

But the other thing that is still strange is the value’s of those two sensors I made with just the unfiltered return values. I don’t know where the "<CGI_Result><result>-2</result></CGI_Result>" state even came from. It looks like the command kind of partially worked but I tried playing around with different iterations of the command and I couldn’t get those same state results in the console.

But anyway…it’s working now.

Must be some sort of string within string confusion. :man_shrugging:

FWIW, I couldn’t contribute much toward the solution but I can offer you this streamlined template:

- platform: command_line  
  command: curl -k --silent "http://192.168.1.56:8006/cgi-bin/CGIProxy.fcgi?cmd=getDevState&usr=xxxx&pwd=yyyyy"
  name: "Livingroom Camera Motion"
  value_template: >
    {% set status = value | regex_findall_index('Alarm>(\d+)</motion') %}
    {% set values = {'0':'Disabled', '1':'None', '2':'Detected'} %}
    {{ values[status] if status in values.keys() else 'Not Determined'}}
  scan_interval: 3

I might agree except that it worked in the previous version of the sensor. So, yeah, I have no idea.

Thanks for the updated template. I’ll give it a try.

@finity: seems like you manage to solve your issue. I have tried to get something similar working but failed with all my attempts. Do you have any input for me on my issue below? Thanks!

Hass.io 0.105.2 on a Raspberry Pi 4.

I want to retrieve the installed HassOS version (currently 3.9) to be used in automations and in other places.

Running the command ‘hassio hassos info’ at the command prompt through SSH gives

core-ssh:~# hassio hassos info
board: rpi4
boot: B
version: "3.9"
version_cli: "16"
version_cli_latest: "16"
version_latest: "3.9"
core-ssh:~#

which looks very promising as I can see both ‘version’ and ‘version_latest’

To get it as a sensor into hass.io I was hoping to use the command line sensor:

In my configuration.yaml I added

sensor:
  - platform: command_line
    name: hassos
    command: "hassio hassos info"

but in developer tools > states I find nothing useful after reboot.

Sorry but unfortunately (for you :wink:) I decided to stick with using my non-hassio install in docker.

I just saw too many potential downsides at the time to switch away from what was already working well for me. The only reason I even considered hassio was the availability of the add-ons. But many of those I already created in my own docker containers (or could create if I really needed them) so the benefit was minor for me at the time.

That said I’m not sure how the command_line sensor commands interact with the hassos OS.

I do remember tho that when I was installing my HA in docker that I needed to create a way for HA itself to run commands outside of the container directly on the host.

Here is the link to the basic instructions I used at the time:

https://hastebin.com/sojasolite.sql

I’m not sure if they are still relevant or correct but they might be a good starting point.

1 Like