Understood — just wanted to rule out something like a disk error on the base Python installation, which as I understand it, RestrictedPython just provides a filter to.
1 Like
It’s all good and I appreciate the suggestion to confirm the container’s python3 isn’t the culprit.
It’s no fun being the only user to encounter this weirdness so any help is appreciated. 
I created this script as test1.py and ran it as the python_script.test1 service:
logger.info(datetime.datetime.strptime)
logger.info(datetime.datetime.strftime)
logger.info(datetime.datetime.timetz)
and got this in the logs:
2021-05-21 14:52:27 INFO (SyncWorker_3) [homeassistant.components.python_script] Executing test1.py: {}
2021-05-21 14:52:27 INFO (SyncWorker_3) [homeassistant.components.python_script.test1.py] <built-in method strptime of type object at 0x7fa6ebebb900>
2021-05-21 14:52:27 INFO (SyncWorker_3) [homeassistant.components.python_script.test1.py] <method 'strftime' of 'datetime.date' objects>
2021-05-21 14:52:27 INFO (SyncWorker_3) [homeassistant.components.python_script.test1.py] <method 'timetz' of 'datetime.datetime' objects>
I don’t understand why strptime and strftime show up differently or if that has any relevance (instance method vs class method, I guess); and interesting that it reports on timetz despite that method not appearing in the allowed lists.
Those are just the __repr__'s or __str__'s, that’s normal and those are all datetime.<method> methods. Nothing to be concerned about.
I’d wager that your RestrictedPython’s compiled files are corrupted in some way. Upgrading, downgrading, or deleting python_scripts will not fix that (unless the version for RestrictedPython changes). The files need to be replaced or recompiled.
How would I go about doing that?
It’s my understanding that the python_script integration gets the RestrictedPython module from PyPi. Does it download and cache it somewhere? That may explain why upgrading/downgrading doesn’t shake free from the error because the RestrictedPython module is cached. Or do I have this all wrong?
Yes, it should be in your python Libs/site-packages (I think) when you look in your container. I don’t know the real path, you’ll have to search for it. It might be a wheel file too. It really depends. Either way, if you get inside the container you can just manually uninstall it and HA should add it back. But this is a do at your own risk kind of thing.
Nope, that’s how it works. Home assistant leaves the packages alone unless it needs a specific version and then it updates it. IIRC it does not uninstall things it doesn’t need, i.e. removing the integration doesn’t do anything with the package.
Based on what you described, that it would be located “in your container”, wouldn’t upgrading/downgrading Home Assistant employ a fresh container? Meaning the new container would also have a fresh copy of RestrictedPython.
I guess my point is that RestrictedPython would need to reside outside of the Home Assistant Core container in order to survive unchanged for Core upgrades/downgrades. No?
The files are definitely on the container. I don’t think a new container is deployed on upgrades and downgrades. I could be wrong, I’m basing my info off my dev env.
You could force the update without going into the container. Copy the python script integration as a custom integration, and change the version required in the manifest file (lower it), then restart. Then rinse and repeat but set it back to ‘normal’ or remove the custom integration.
What I’ve seen (via Portainer) is that each change of major/minor version represents an entirely new docker image file from which the Core container is instantiated. This was evident when I used to run Home Assistant Container (instead of Home Assistant Supervised). The Supervised edition appears to do some housekeeping and only keeps the one in use and discards the others:
In the Home Assistant Container version, it would show older images in that list (and you would manually discard them when no longer needed).
Just finished doing that; I changed RestrictedPython’s version from 5.1 to 5.0.
Executing the two-line test script produces the same error message (caused by the first line’s use of strptime) with the only notable difference being it indicates the error was caused by a custom integration:
I deleted the custom integration instance of python_script, restarted, executed the test script and it reported the same error message. The exercise didn’t fix anything but it demonstrated how stubbornly entrenched this thing is.
Completely speculating here, is it possible it depends on what other components you have installed? I know part of the reason for the componentized architecture of HA is to keep the number of core dependencies small and only load dependencies based on what users need/configure.
So I’m wondering if there’s some super common component we all have listed that Taras doesn’t. And because of that we all have some dependency loaded already whereas Taras’s python script is trying to load it for the first time and running into the restricted python’s limitation on imports.
I’m really not enough of a python expert to know if what I’m saying is even possible, feel free to tell me to shut up if it makes no sense. I just can’t fathom why some HA systems have this error and some don’t especially if Taras is using the HA container.
That’s an interesting theory but the machine afflicted with this issue has been running the python_script integration for many months (i.e. over the span of many versions of Home Assistant).
This weirdness only came to light after I upgraded from 2021.5.1 to 2021.5.5. Perhaps the upgrade itself isn’t what instigated it but something else (still unknown) that manifested itself after a restart (which would be a normal activity during an upgrade). I say that because now it makes no difference which version I revert to (2021.5.1, 2021.5.4, etc; I’ve even tried downgrading all the way back to 2021.4.6) the error message won’t go away.
FWIW, the use of strptime occurs just once in a python_script (over 450 lines long) that produces a sensor containing birthdays, anniversaries, tasks, etc. The absence of this sensor left a hole in the Lovelace UI and that’s what initially caught my attention after the upgrade to 2021.5.5. A quick glance at the Logs showed that my python_script was in trouble and I eventually narrowed it down to its one use of strptime. It was difficult to believe because it had worked in the past (and continues to work for everyone else who has participated in this thread).
In addition, I have uninstalled/re-installed the python_integration and it hasn’t eliminated the error message.
Whatever is causing this is keeping itself hidden very well …
Dependencies are handled through manifests, conflictions would get caught unless using a custom integration. It would be displayed in the logs as well.
Not sure if this is related. I am running HA 2021.5.1 in a container (homeassistant/home-assistant:2021.5.1). Tonight I notice some errors in my logs and was surprised to see a previously working template throwing errors. I narrowed it down to the strptime function is returning a string, instead of a datetime as documented here (Templating - Home Assistant).
I can confirm this with a one line example template
{{ strptime(state_attr("sun.sun", "next_setting"), "%Y-%m-%dT%H:%M:%S%z") is string }}
returns “true”
This is breaking a bunch of my templates which depend on strptime returning a datetime so I can futz around with dates and times:
{% set sun_set_utc = strptime(state_attr("sun.sun", "next_setting"), "%Y-%m-%dT%H:%M:%S%z") %}
{% set sun_set_local = sun_set_utc.replace(tzinfo=utcnow().tzinfo).astimezone(now.tzinfo) %}
I can confirm that the two machines I have running 2021.5.5 both fail to convert the attribute’s date string into a datetime object.
Buried in one of my files is this commented out code. It served as a reminder, explaining how to convert next_rising to a datetime object. In other words, it must have worked at one time in the past and I felt it was important enough to make a note of it in case I needed to use it in the future.
## Convert datetime string to a datetime object
# {% set dt = strptime(state_attr('sun.sun', 'next_rising'), '%Y-%m-%dT%H:%M:%S%z') %}
# Time object: {{ dt.time() }}
# Time string: {{ dt.time() | string }}
The fact it no longer works implies something changed in a later version of Home Assistant. I don’t think it’s directly related to the problem I described in the first post but it does suggest some things quietly change from one version to another and no one is aware until they stumble across it.
@123 @acraigbray Your format is wrong. You’re both missing .%f which grabs the microseconds. FYI when strptime fails, it returns the input string. This is how you know it fails to format.
## Convert datetime string to a datetime object
# {% set dt = strptime(state_attr('sun.sun', 'next_rising'), '%Y-%m-%dT%H:%M:%S.%f%z') %}
# Time object: {{ dt.time() }}
# Time string: {{ dt.time() | string }}
1 Like
Also, it should be noted that there was a recent library change to time in home assistant that removed all the ‘goofyness’ that people were reporting.
However, I don’t think this is related to any issue’s you guys are running into. (i.e. formatting and datetime imports)
1 Like
@petro Great catch! Thanks for that.
Seems like maybe the format of the sun integration’s attributes changed in a recent release. According to my git commit history I wrote the templates using that format string last summer and I know it worked then.