Can't get the new IMAP template to work to check my email

Configuration
,
35

hi im still having issue that IMAP intergration can not detect the emails… and sort them in the 3 states i set in my orginal post above… i dunno if its a hotmail issue and you need gmail instead… the fix in the latest HA update 5.4 didnt fix the issue…i got it to work but not really on its own had used another email coming in to detect… and i have no idea why something is blocking i posted before to the github but never a reply … the info doesnt say whats blocking HA

Home Assistant Core
2023-05-28 09:12:45.658 INFO (MainThread) [homeassistant.bootstrap] Setting up stage 2: {'hacs', 'application_credentials', 'input_datetime', 'trace', 'file_upload', 'template', 'speedtestdotnet', 'group', 'automation', 'conversation', 'stream', 'device_automation', 'image_upload', 'onboarding', 'input_boolean', 'energy', 'homeassistant_alerts', 'auth', 'input_select', 'shelly', 'reolink_discovery', 'auto_backup', 'persistent_notification', 'search', 'browser_mod', 'climate', 'blueprint', 'mqtt', 'notify', 'repairs', 'device_tracker', 'remote_homeassistant', 'sun', 'my', 'tts', 'local_calendar', 'assist_pipeline', 'uptime', 'analytics', 'switch', 'config', 'timer', 'counter', 'input_text', 'lovelace', 'scene', 'tag', 'sensor', 'diagnostics', 'media_source', 'schedule', 'person', 'binary_sensor', 'imap', 'default_config', 'dlna_dmr', 'logbook', 'map', 'vlc_telnet', 'zone', 'dlna_dms', 'python_script', 'panel_custom', 'input_number', 'stt', 'cast', 'history', 'script', 'adguard', 'hardware', 'tplink', 'met', 'mobile_app', 'volumio', 'input_button', 'system_health', 'esphome', 'environment_canada'}
2023-05-28 09:12:47.394 INFO (MainThread) [homeassistant.setup] Setting up imap
2023-05-28 09:12:47.394 INFO (MainThread) [homeassistant.setup] Setup of domain imap took 0.0 seconds
2023-05-28 09:12:57.640 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Finished fetching imap data in 0.000 seconds (success: True)
2023-05-28 09:12:57.642 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.imap
2023-05-28 09:13:04.244 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Message processed, sender: [email protected], subject: How to Make an Epoxy Clock
2023-05-28 09:13:04.244 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Manually updated imap data
2023-05-28 09:18:25.084 WARNING (MainThread) [homeassistant.core] Something is blocking Home Assistant from wrapping up the start up phase. We're going to continue anyway. Please report the following info at https://github.com/home-assistant/core/issues: sensor.systemmonitor, application_credentials, sensor.mqtt, hacs, cover.shelly, frontend, file_upload, ping, media_player.browser_mod, system_log, sensor.shelly, notify.smtp, sensor.time_date, button.mqtt, binary_sensor.shelly, sensor.speedtestdotnet, humidifier.mqtt, image_upload, cover, sensor.remote_homeassistant, energy, switch.esphome, auth, dhcp, binary_sensor.esphome, reolink_discovery, webhook, auto_backup, browser_mod, blueprint, mqtt, ssdp, device_tracker, remote_homeassistant, sun, my, binary_sensor.mqtt, light, vacuum, local_calendar, media_player.dlna_dmr, sensor.sun, assist_pipeline, uptime, camera.mqtt, weather.environment_canada, update, api, config, counter, input_text, update.mqtt, scene, tag, diagnostics, sensor, media_source, person, binary_sensor, imap, light.browser_mod, update.shelly, vlc_telnet, lock, weather.met, dlna_dms, light.shelly, panel_custom, scene.mqtt, input_number, network, sensor.mobile_app, sensor.hacs, history, sensor.adguard, adguard, binary_sensor.mobile_app, sensor.tplink, sensor.browser_mod, humidifier, tplink, met, number.mqtt, mobile_app, input_button, system_health, climate.generic_thermostat, text, fan.mqtt, switch.adguard, switch.tplink, environment_canada, input_datetime, binary_sensor.hassio, trace, reolink, template, speedtestdotnet, binary_sensor.command_line, sensor.template, tts.google_translate, switch.mqtt, group, sensor.uptime, automation, select.mqtt, conversation, stream, device_automation, onboarding, input_boolean, media_player, homeassistant_alerts, light.mqtt, select, input_select, button.shelly, shelly, zeroconf, climate.mqtt, scene.homeassistant, fan, siren.mqtt, button, logger, device_tracker.bluetooth_tracker, persistent_notification, search, device_tracker.mqtt, climate, camera.environment_canada, hassio, alarm_control_panel, notify, repairs, tts, usb, homeassistant, lock.mqtt, calendar.local_calendar, switch.shelly, light.tplink, analytics, text.mqtt, binary_sensor.ping, switch, notify.mobile_app, timer, siren, lovelace, schedule, camera, dlna_dmr, map, logbook, default_config, alarm_control_panel.mqtt, bluetooth, sensor.imap, vacuum.mqtt, zone, media_player.cast, calendar, cover.mqtt, python_script, media_player.volumio, sensor.auto_backup, sensor.energy, update.hassio, media_player.vlc_telnet, stt, http, cast, weather, script, binary_sensor.browser_mod, hardware, number, cloud, sensor.environment_canada, recorder, sensor.hassio, volumio, websocket_api, sensor.esphome, esphome, camera.browser_mod, device_tracker.mobile_app
2023-05-28 09:23:43.605 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Message processed, sender: [email protected], subject: Your Epic Games Receipt F2305281318320451
2023-05-28 09:23:43.605 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Manually updated imap data
2023-05-28 09:23:51.286 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Message processed, sender: [email protected], subject: Your package has been received!
2023-05-28 09:23:51.286 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Manually updated imap data
2023-05-28 09:23:52.351 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Manually updated imap data
2023-05-28 09:25:00.911 ERROR (MainThread) [homeassistant.components.imap.coordinator] Error requesting imap data:
2023-05-28 09:26:16.866 DEBUG (MainThread) [homeassistant.components.imap.coordinator] Manually updated imap data
2023-05-28 09:27:10.397 INFO (MainThread) [aioimaplib.aioimaplib] ignored untagged response : b'BYE Connection is closed. 13'
2023-05-28 09:27:19.839 INFO (MainThread) [aioimaplib.aioimaplib] ignored untagged response : b'BYE Connection is closed. 13'
2023-05-28 09:27:23.308 INFO (MainThread) [aioimaplib.aioimaplib] ignored untagged response : b'BYE Connection is closed. 13
36

To overcome issues with event sizes, you could consider using the new custom template setting. This templated is evaluated before the event is sent, and will also work on large emails. The result of the custom template will be added to the event data.
See: IMAP - Home Assistant

Note that the setting is an advanced setting.

37

I could not figure out how to use the custom template from the documentation. From what I could tell it did not provide a example you could copy/paste and make it work. It referenced some custom template code but didn’t say where to put it. It just says Custom: True in the yaml.

38

You decide what the output of your custom template is. The custom template is not limited to the message size. The result of the template will be available in the event data.

39

Hi,

I have a similar problem, the IMAP integration works fine, it reads the number of emails I have and I can create a sensor with the email data.

My problem comes when I want to create sensors with the information that comes in the body of the message.

The “main” sensor shows the message text, but when I try to create a new one with part of it I get Unavailable.

template:
  - trigger:
      - platform: event
        event_type: "imap_content"
        id: "mail_eventid"
    sensor:
      - name: "mail_sensor"
        state: "{{ trigger.event.data['subject'] }}"
        attributes:
          Message: "{{ trigger.event.data['text'] }}"

image

As you can see, in the sensor attributes I can see the message data.

I do go to templates, and I put this:

{{ ((state_attr('sensor.mail_sensor', 'Message') | regex_findall_index(find='([0-9].*C)', index=0))[:-1]) }}

The correct result appears, 24.7

But if I enter the code for a new sensor I get the message UNAVAILABLE.

image
image1034×75 6.72 KB

40

What does this mean? Show the actual sensor configuration.

41

I got it. Thanks

42

Where are you putting the template and sensor setup?
is it in the configuration.yaml file?

I have followed the imap example, but not getting any new sensors except the imap count of unread emails.
any working examples with specific places to put them would be really helpful.
something simple like the subject in a sensor or text regex to populate another sensor.

43

The trigger-based template sensor used to get the content of incoming emails can go directly in configuration.yaml or any properly split file that uses template as it’s top-level configuration key.

One easy and flexible method for splitting the config is to use a Packages folder. Here is a screenshot of the basic “subject” sensor.

image

For reference, my Packages folder is split in configuration.yaml as follows:

homeassistant:
  packages: !include_dir_named packages
44

Hi, thanks for the pointers on the IMAP integration;
just one issue here and couldn’t find an answer so far

Everything works as expected, I can trigger on subject but not on body (“text”)

  - trigger:
      - platform: event
        event_type: "imap_content"        
    binary_sensor:
      - name: gmail_hydro_test
        auto_off: "00:02:00"
        state: >
          {% if trigger.event.data['text'] is match
          "test|Test|Zzz|zzz"
          %}
            true
          {% endif %}

To troubleshoot I’ve added a sensor to extract all data and the “text” is in the wrong format;
(the body of the message here is simply “test” but here’s the result)

  - trigger:
      - platform: event
        event_type: "imap_content"
        id: "custom_event"
    sensor:
      - name: imap_content
        unique_id: 2604a5fa-d31e-4ab1-bced-084b0cfe81a2
        state: "{{ trigger.event.data['subject'] }}"
        attributes:
          Message: "{{ trigger.event.data['text'] }}"
          Server: "{{ trigger.event.data['server'] }}"
          Username: "{{ trigger.event.data['username'] }}"
          Search: "{{ trigger.event.data['search'] }}"
          Folder: "{{ trigger.event.data['folder'] }}"
          Sender: "{{ trigger.event.data['sender'] }}"
          Date: "{{ trigger.event.data['date'] }}"
          Subject: "{{ trigger.event.data['subject'] }}"
          To: "{{ trigger.event.data['headers']['Delivered-To'][0] }}"
          Return_Path: "{{ trigger.event.data['headers']['Return-Path'][0] }}"
          Received-first: "{{ trigger.event.data['headers']['Received'][0] }}"
          Received-last: "{{ trigger.event.data['headers']['Received'][-1] }}"

image

I’ve tried changing the charset to US-ASCII instead of utf-8 but still doesn’t work

Any idea what could be wrong ?

45

It’s base64 encoded.

- trigger:
      - platform: event
        event_type: "imap_content"        
    binary_sensor:
      - name: gmail_hydro_test
        auto_off: "00:02:00"
        state: >
          {{ trigger.event.data['text'] | base64_decode  is match("test|Test|Zzz|zzz") }}

FWIW, match() will only match the find expression at the beginning of the string. If the criteria could be anywhere other than the beginning you’ll need to use search().

46

ok will test and report back;
what’s odd though is the fact that using the integration template does work

image

image
image784×132 9.06 KB

47

Every single one of my dozen tests now working … :sunglasses:
Thank you :+1:

Does the base64 text comes from Gmail or the sender ?
surprised this hasn’t surfaced elsewhere on the forum if it is Gmail related…

48

As I understand it, the actual search is performed by the email provider/server so they must decode it at some point… I don’t know why they don’t forward the decoded version.

The sender.

There have been a couple questions about it in the past month or two.

49

For future reference, here’s a tested version to ensure it handles both decodes.

First Sensor is simply to get a clear text version of status/attributes (including base64 message)
Second one to be adapted vs use case.

Thanks for your assistance @Didgeridrew

- trigger:
  - platform: event
    event_type: "imap_content"
    #event_data:
    #  sender: "xxxxxxxxxxxx@xxxxxxxxx"
  sensor:
  - name: gmail_imap_sensor_data
    state: "{{ trigger.event.data['subject'] }}"
    attributes:
      Message: "{{ trigger.event.data['text'] }}"
      Message-Base64: "{{ trigger.event.data['text'] | base64_decode }}"
      Server: "{{ trigger.event.data['server'] }}"
      Username: "{{ trigger.event.data['username'] }}"
      Search: "{{ trigger.event.data['search'] }}"
      Folder: "{{ trigger.event.data['folder'] }}"
      Sender: "{{ trigger.event.data['sender'] }}"
      Date: "{{ trigger.event.data['date'] }}"
      Subject: "{{ trigger.event.data['subject'] }}"
      To: "{{ trigger.event.data['headers'].get('Delivered-To', ['n/a'])[0] }}"
      Return-Path: "{{ trigger.event.data['headers'].get('Return-Path',['n/a'])[0] }}"
      Received-first: "{{ trigger.event.data['headers'].get('Received',['n/a'])[0] }}"
      Received-last: "{{ trigger.event.data['headers'].get('Received',['n/a'])[-1] }}"

- trigger:
    - platform: event
      event_type: "imap_content"
      #event_data:
      #  sender: "xxxxxxxxxxxxxx@xxxxxxxxxxxxxxxx"      
  binary_sensor:
  - name: gmail_imap_hydroqc_am_peak_period
    auto_off: "12:00:00"
    state: >-
      {% if 'Am' in trigger.event.data["text"] or 'Am' in trigger.event.data["text"] | base64_decode %}
        true
      {% endif %}
50

Hello @comet424,

Have you fix your UNKNOWN problem? IF, yes, how?
Other question, Where did you put the code that you submit previously (### Intelcom Deliveries Template ###)? (in configuration.yaml?, In automations.yaml? Or directly in the configuration of IMAP integration?).

Thanks for your help!
Richard

51

@rikray02 no i never i use the uknown in my esphome code but i have a double problem it seems to work but it re pops up the on my screen that delivery has a arrived even though the automation hasnt ran… so i not sure whats in the code why its doing that its either the esp device is rebooting and and it is blank and then it updates th ands why it displays multiple times thats my guess for my current issue

as for my code i did

an automation here is my code… it does different sensors as i couldnt change the template on a nextion screen right or so… so it sends it to a screen and on my cell and emails

alias: Amazon Delivery Gifts (Intelcom)
description: Intelcom Amazon Delivery Gifts
trigger:
  - platform: state
    entity_id:
      - sensor.intelcom_amazon_delivery
condition: []
action:
  - choose:
      - conditions:
          - condition: state
            entity_id: sensor.intelcom_amazon_delivery
            state: received
        sequence:
          - service: notify.text_cell_notification
            data:
              message: >-
                Gifts Have Been Recieved At Intel Drop Off Location {{
                as_timestamp(states('sensor.date_time_iso')) |
                timestamp_custom('%I:%M %p') }}
          - service: python_script.set_state
            data:
              entity_id: sensor.deliveries_alert_message
              state: >-
                Gifts Have Been    Recieved At Intel   Drop Off Location   {{
                now().strftime('%B %-d, %I:%M %p')  }}
      - conditions:
          - condition: state
            entity_id: sensor.intelcom_amazon_delivery
            state: delivered
        sequence:
          - service: notify.text_cell_notification
            data:
              message: >-
                Gifts Have Been Delivered '{{
                as_timestamp(states('sensor.date_time_iso')) |
                timestamp_custom('%I:%M %p') }}'
          - service: python_script.set_state
            data:
              entity_id: sensor.deliveries_alert_message
              state: >-
                Amazon Gifts Been Delivered                Intelcom  {{
                now().strftime('%B %-d, %I:%M %p')  }}
          - delay:
              hours: 10
              minutes: 0
              seconds: 0
              milliseconds: 0
            enabled: false
          - service: python_script.set_state
            data:
              entity_id: sensor.intelcom_amazon_delivery
              state: unknown
            enabled: false
      - conditions:
          - condition: state
            entity_id: sensor.intelcom_amazon_delivery
            state: on_the_way
        sequence:
          - service: notify.text_cell_notification
            data:
              message: >-
                Gifts Are On There Way '{{
                as_timestamp(states('sensor.date_time_iso')) |
                timestamp_custom('%I:%M %p') }}'
          - service: python_script.set_state
            data:
              entity_id: sensor.deliveries_alert_message
              state: >-
                Amazon Gifts On   There Way              (Intelcom)  {{
                now().strftime('%B %-d, %I:%M %p')  }}
mode: single

the unknown issue i still having cuz in esphome it has issue with Unknown and unknown

here is my code for the ESP device using a Nextion screen it works but its not 100% fool proof and has issue i think it reboots on me or or something that it re toggles the display on its own when the automation hasnt ran to update the sensor so the screen should not be updating

###########################################################################################
### Deliveries Alert Message Page..  Display a Warning  Information Press Screen To Get Off It ###
###########################################################################################
  - platform: homeassistant
    id: deliveries_alert_message
    entity_id: sensor.deliveries_alert_message
    on_value:
      then:
        - if:
            condition:
              not:
                # Check When Home Assistant Boots Up. and prints "unknown" or "Unknow" not to send it to the screen
                - lambda: |-
                    return (id(deliveries_alert_message).state == "unknown"); // || "Unknown");

                # Prints the Red Alert Message when it doesnt equal unknown as the message    
            then:
              lambda: |-
                id(nextion1).goto_page("2");
                           
                id(nextion1).send_command_printf("title.pco=2016"); 
                id(nextion1).set_component_text("title","Deliveries"); 
                id(nextion1).set_component_text("wmessage",id(deliveries_alert_message).state.c_str()); 
                id(warningc) = 1;
52

Hi my friend, I Was with a similar issue since imap_email_content stop working in new version HA core, and I tryed rewrite only using ui to create the sensor and in the configuration.yaml using event as the documentation show in this example using regex:

template:

  • trigger:
    • platform: event
      event_type: “imap_content”
      id: “custom_event”
      event_data:
      sender: “[email protected]”
      sensor:
    • name: “Previous Day Energy Use”
      unit_of_measurement: “kWh”
      state: >
      {{ trigger.event.data[“text”]
      | regex_findall_index(“*Yesterday’s Energy Use:* ([0-9]+) kWh”) }}
    • name: “Previous Day Cost”
      unit_of_measurement: “$”
      state: >
      {{ trigger.event.data[“text”]
      | regex_findall_index(“*Yesterday’s estimated energy cost:* $([0-9.]+)”) }}
    • name: “Billing Cycle Total”
      unit_of_measurement: “$”
      state: >
      {{ trigger.event.data[“text”]
      | regex_findall_index(“\ days:* $([0-9.]+)”) }}

but this doesn´t works for me i this code below:

state: >
{{ trigger.event.data[“text”]
| regex_findall_index(“Valor da corrida:\D* ([0-9,]+)”) }}

finnaly I did a little modification using state with my old template, and bellow the complete solution, in this example I save in the sensor the value that I spent in one uber travel when I receive the email, and the sensor update correctely:

template:

  • trigger:

    • platform: event
      event_type: “imap_content”
      id: “custom_event”
      event_data:
      sender: “[email protected]”
      sensor:

    • name: “valor_recibo_uber”
      unit_of_measurement: “R$”

      state: >
      {% set email_content = trigger.event.data[“text”] %}
      {% set regex_match = email_content | regex_findall(‘Valor da corrida:(\d+,\d+)’) %}
      {% if regex_match %}
      {{ regex_match[0] | regex_replace(‘,’, ‘.’) }}
      {% else %}
      0
      {% endif %}

53

Good to know with 2024.2.0 an annoying bug was fixed for users connection to an Microsoft IMAP mailbox. As the encoding needs to be set to us-ascii this also was used to decode the encoded mailparts. As this failed, the content wasn’t decoded correctly. The fix no ensures the correct decoding of the message parts.

54

Did you ever find out where to put the template code to create a sensor based on say email subject?