Yes, that would be possible with an additional rejectattr-Filter. The third rejectattr filter in the following template filters out all entities that changed their state less than 7200 secs (2 hours) ago. As this filter only receives “on” lights because of the previous filter rejectattr('state','in','off'), all remaining entities must have changed from “off” to “on”. Thus, their last_changed attibute represents the last change to “on”. The statement seems to be a bit clunky, but essentially
strptime(((now().timestamp() - 7200) | timestamp_utc) ~ '+00:00', '%Y-%m-%d %H:%M:%S%z')
just creates a datetime value from 2 hours ago, because rejectattr needs two values with the same datatype for comparsion:
all_out:
sequence:
- variables:
list_light: >
{% set exclude_light = [
'light.house_number',
'light.aquarium',
'light.entrance'
] %}
{%- for device in states.light
|rejectattr('entity_id','in',exclude_light)
|rejectattr('state','in','off')
|rejectattr('last_changed', '>', strptime(((now().timestamp() - 7200) | timestamp_utc) ~ '+00:00', '%Y-%m-%d %H:%M:%S%z')) %}
{%- if loop.first %}
{%- else %}
,
{%- endif -%}
{{device.entity_id}}
{%- if loop.last %}
{% endif %}
{%- endfor %}
- choose:
- conditions:
- condition: template
value_template: "{{ list_light|length > 0 }}"
sequence:
- service: light.turn_off
data:
entity_id: "{{ list_light }}"
Please be aware that this doesn’t work very well, i.e. not at all, after HA restarts, because the last_changed attributes of all HA entities update on every restart. If you don’t expect this, it may drive you crazy when trying to find out wth your template isn’t working as expected… 
Oh, and yes, the same can of course be achieved with an appropriate selectattr filter instead of rejectattr. This may even be a bit more straight forward, but in the end it’s just a matter of personal preference.