Time probelms

Two things happened recently,

  1. the Hassio supervisor was updated to v162
  2. I moved my system from a pi to a PC running the nuc image (Hassio v0.91.4)

Now none of my time templates are working.
I didn’t change any of my config so the timezone setting is the same (UTC + 10).

now().timestamp() seems to be evaluating incorrectly as a UTC timestamp:

{{ utcnow().timestamp() }}
{{ now().timestamp() }}
{{ utcnow() }}
{{ now() }}

1556243287.686556
1556243287.686613
2019-04-26 01:48:07.686663+00:00
2019-04-26 11:48:07.686674+10:00

Can anyone reproduce this in the template editor or tell me what the heck is going on?

Timestamps are always measured in UTC, which is why you’re getting the same value whether you start with now or utcnow.

Try these:

{{ utcnow() }}
{{ now() }}
{{ utcnow().astimezone() }}
{{ now().astimezone() }}
{{ utcnow().tzinfo }}
{{ now().tzinfo }}
{{ now().astimezone().tzinfo }}

The first line should show the current time in UTC.
The next three lines should all show the current time in the local time zone.
The fifth line should say UTC .
And the six and seventh lines should show the local time zone, probably in different formats (but they should be equivalent.)

I have an issue then:

2019-04-26 02:06:37.972355+00:00
2019-04-26 12:06:37.972409+10:00
2019-04-26 02:06:37.972478+00:00
2019-04-26 02:06:37.972594+00:00
UTC
Australia/Hobart
UTC

That’s saying that HA’s time zone configuration is correct, but the time zone setting in the environment in which HA is running is not correct. Note that the astimezone datetime method uses the time zone setting in the “OS” to determine local time.

So the docker container timezone is incorrect?
Do you know if it is possible to correct this (hassio)?

If HA is running in a docker, then yes, I’d say the time zone setting in the docker is incorrect. I have no idea how to fix that. I don’t use hassio or docker. Just a good old venv! :wink:

I’ll have a bit of a search. Thanks for the assistance.

1 Like

cogneato reckons that the container is supposed to be in UTC.

I still think there should be a difference between:

{{ utcnow().timestamp() }}
{{ now().timestamp() }}

As a workaround I just added 36000 seconds to the templates that needed it.

No idea who/what cogneato is. And I seriously doubt the container should be in UTC. That doesn’t make any sense, and it breaks one of the HA install requirements, AFAIK.

If you want to redefine what a “Unix timestamp” is, by all means! :wink: See also datetime.datetime.timestamp.

I installed the SSH web addon and in unprotected mode was able to see the Hassio docker container time. It’s set to my local time. So no idea why the tzinfo’s are different.

As for definitions of timestamps, why did now().timestamp() have my local offset originally? When I wrote my templates this was was how it worked. Now it does not. I have a workaround (I just added 36000 seconds to my templates) so it’s not a huge deal. I just find it odd that there was no mention in any breaking changes and apparently the files haven’t been changed since October last year. Very dam odd.

Is the time zone (that was displayed via SSH) your local time zone? If the time is your local time, but the time zone is UTC, that’s a problem.

Are you sure when HA is launched that something isn’t changing its environment’s time zone setting? Again, not familiar with dockers or hassio, but I’ve seen enough on this forum to indicate things can get set incorrectly.

I’m not sure I understand the question. At any given moment, a Unix timestamp is the same across the globe. It is not affected by time zone. If you’re wondering if it’s right, try comparing what you get in HA (for both now().timestamp() and utcnow().timestamp()) to the “Unix timestamp” website I provided a link to above. If there was, or is, any disagreement, then something in your system was/is misconfigured.

I guess sometimes two wrongs do make it right. :smile:

At one point in time many moons ago I wrote a template that used now().timestamp (actually you and petro helped a great deal with this). This template relied on the fact that now().timestamp included the the local offset. i.e number of seconds from my local time to the epoch.
Today I noticed the templates weren’t working because this is no longer the case. The timestamp now().timestamp is now a measure of the number of seconds between UTC time and the epoch.

This is the old template that did work until recently:

      value_template: >-
        {% set d = now().strftime("%Y-%m-%d ") %}
        {% set t = now().timestamp() %}
        {% set am_start = strptime(d + states('input_datetime.lounge_ac_am_on_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% set am_end = strptime(d + states('input_datetime.lounge_ac_am_off_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {{ (am_start <= t <= am_end) and is_state('input_boolean.lounge_ac_am_automation', 'on') }}

This is what I had to amend it to today:

      value_template: >-
        {% set d = now().strftime("%Y-%m-%d ") %}
        {% set t = now().timestamp() + 36000 %}   ##### <---------- Had to add this to get my "local timestamp" #####
        {% set am_start = strptime(d + states('input_datetime.lounge_ac_am_on_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% set am_end = strptime(d + states('input_datetime.lounge_ac_am_off_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {{ (am_start <= t <= am_end) and is_state('input_boolean.lounge_ac_am_automation', 'on') }}

I have opened an issue: now().timestamp() not returning correct value · Issue #23422 · home-assistant/core · GitHub

This is all because the timestamps on datetimes, without dates included, end up a bit weird. If I set a time only input datetime to 06:20 the timestamp (with today’s date added for comparison) is in local time, not UTC. i.e. number of seconds from 6:20 my local time to the epoch. So I need t in the template above to be in number of seconds local time to the epoch too.

You have to be careful when you talk about, or use, datetimes. There are basically two types: naive and aware. From the Python docs:

There are two kinds of date and time objects: “naive” and “aware”.

An aware object has sufficient knowledge of applicable algorithmic and political time adjustments, such as time zone and daylight saving time information, to locate itself relative to other aware objects. An aware object is used to represent a specific moment in time that is not open to interpretation.

A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it is up to the program whether a particular number represents metres, miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality.

now() and utcnow() both return aware datetime objects. However, when you use the strptime() function, whether the datetime object created is naive or aware depends on the format. E.g.:

{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S') }}
{{ strptime('2019-04-26 10:00:00 +1000', '%Y-%m-%d %H:%M:%S %z') }}
{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S').tzinfo }}
{{ strptime('2019-04-26 10:00:00 +1000', '%Y-%m-%d %H:%M:%S %z').tzinfo }}
{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S').timestamp() }}
{{ strptime('2019-04-26 10:00:00 +1000', '%Y-%m-%d %H:%M:%S %z').timestamp() }}

in my case results in:

2019-04-26 10:00:00
2019-04-26 10:00:00+10:00
None
UTC+10:00
1556290800.0
1556236800.0

The first does not specify a time zone, and the second does. (I’m using your time zone on purpose here instead of mine to show what happens if the OS’s time zone is different than what it should be.) So you can see the first is a naive object, whereas the second is an aware object.

The next two lines show the tzinfo for the first two. Again, no time zone info for the first, and the second is your time zone.

The last two get the Unix timestamp for the first two. Notice how they are different? But they should be the same, right? The reason they are different is because the first is a naive datetime object, so when the timestamp method is called, it gets the time zone from the “OS”, which, of course, in my case is “wrong” because it is not your time zone, but rather -0500 (or CDT.) And using the Unix timestamp website (which takes a UTC time, so to be equivalent to 10:00am in your time zone, I had to use 12:00am UTC):

image

Notice how the returned timestamp from the templates is only correct when an aware datetime object is used. (It would also have been correct if the OS’s time zone was the “correct” time zone.)

So, bottom line, if you create naive datetime objects in your templates, and you want to get a timestamp from them, or want to compare them to aware datetime objects, then the time zone in the “OS” must be correct.

EDIT: Actually, you can’t compare aware and naive datetimes:

>>> datetime.strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S') > datetime.strptime('2019-04-26 10:00:00 +1000', '%Y-%m-%d %H:%M:%S %z')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't compare offset-naive and offset-aware datetimes
2 Likes

Oh, and to show what happens when the OS’s time zone is correct, I’ve adjusted the examples to use my time zone:

{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S') }}
{{ strptime('2019-04-26 10:00:00 -0500', '%Y-%m-%d %H:%M:%S %z') }}
{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S').tzinfo }}
{{ strptime('2019-04-26 10:00:00 -0500', '%Y-%m-%d %H:%M:%S %z').tzinfo }}
{{ strptime('2019-04-26 10:00:00', '%Y-%m-%d %H:%M:%S').timestamp() }}
{{ strptime('2019-04-26 10:00:00 -0500', '%Y-%m-%d %H:%M:%S %z').timestamp() }}

which results in:

2019-04-26 10:00:00
2019-04-26 10:00:00-05:00
None
UTC-05:00
1556290800.0
1556290800.0

And the correct timestamp is (for the corresponding UTC time):

image

You are correct with your assessment of naive datetime objects. I updated to 0.92.0 last night . Now I have to change all my templates back again!
Note that for this

{{ utcnow() }}
{{ now() }}
{{ utcnow().astimezone() }}
{{ now().astimezone() }}
{{ utcnow().tzinfo }}
{{ now().tzinfo }}
{{ now().astimezone().tzinfo }}

The bottom two tzs now agree (in different formats):

2019-04-26 23:17:17.043223+00:00
2019-04-27 09:17:17.043243+10:00
2019-04-27 09:17:17.043268+10:00
2019-04-27 09:17:17.043317+10:00
UTC
Australia/Hobart
AEST

So it was a system tz setting problem. Specifically HassOS (in the docker container) had UTC instead of AEST set.

I would mark your post as the solution but the check box for this seems to be missing. Edit: fixed, the original post was uncategorised.

1 Like

Apparently not an arcane topic because searching for ‘docker container timezone’ produces many results. This article describes several ways to set the container’s TZ:
https://bobcares.com/blog/change-time-in-docker-container/

I discovered I was using the fourth technique. I’ll admit that when I created the docker-compose.yaml file I didn’t think much about it and included the relevant lines because many examples had them. Now I have better appreciation of what these two lines do:

volumes:
- "/etc/timezone:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
1 Like

I transferred from the raspi image to the NUC image. The container tz info was obviously not transferred when I restored my snapshot from the old to the new system as this is HA information, not HassOS info. This makes sense as a guess. Fortunately updating versions seems to have fixed it.