Rest JSON File format Assistant please

Configuration
1

Hi all

I have tried many options to get the REST sensor to read my EMail Headers. HA keeps giving me “REST result could not be parsed as JSON”
This is the JSON File:

{
	"email": [
		{
			 "data": [
				
				{
					"Date: Thu, 18 Jan 2024 08:00:27 +0000, From: =?utf-8?Q?Communica=20Marketing?= <[email protected]>, Subject: =?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
				},
				{
					"Date: Sun, 21 Jan 2024 16:16:36 +0200 (SAST), From: <[email protected]>, Subject: SOM"
				},
				{
					"Date: Sun, 21 Jan 2024 16:18:14 +0200 (SAST), From: Gia <[email protected]>, Subject: Follow up"
				},

			]
		}
	]
}

  - authentication: basic
    resource: http://x.x.x.x:xxx/htdocs/email.txt
    headers:
      content-type: "application/json"
    scan_interval: 15
    timeout: 5
    sensor:
      - name: my_email_headers
        json_attributes_path: $.data
        json_attributes:
          - data
        #value_template: "{{ value_json.data }}"
        value_template: NA
2

Your data list contains dictionaries according to the bracketing, but the content of those brackets are simple strings.

Are you producing that (non-)JSON or just trying to process it?

This should be valid, for example:

{
  "email": [
    {
      "data": [	
        "Date: Thu, 18 Jan 2024 08:00:27 +0000, From: =?utf-8?Q?Communica=20Marketing?= <[email protected]>, Subject: =?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?=",
        "Date: Sun, 21 Jan 2024 16:16:36 +0200 (SAST), From: <[email protected]>, Subject: SOM",
        "Date: Sun, 21 Jan 2024 16:18:14 +0200 (SAST), From: Gia <[email protected]>, Subject: Follow up"
      ]
    }
  ]
}

Alternatively, using dictionaries (first one only shown as I’m doing this on a phone):

{
  "email": [
    {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        INCLUDE OTHERS HERE
      ]
    }
  ]
}

Best to use spaces not tab characters.

3

Thanks for you reply Troon. Unfortunately not one of your suggestions are working :frowning:

Failed to set state for sensor.my_email_headers, fall back to unknown
19:33:31 – (ERROR) helpers/entity.py - message first occurred at 19:33:01 and shows up 9 times
REST result could not be parsed as JSON
19:33:31 – (WARNING) RESTful - message first occurred at 19:33:01 and shows up 3 times
Unexpected error fetching multiscrape data:
19:33:30 – (ERROR) multiscrape (custom integration) - message first occurred at 19:33:10 and shows up 3 times
4

Unfortunately you didn’t answer my question. Where is this pseudo-JSON coming from? What are you trying to achieve?

There are simpler ways to structure the data you have presented. Explain the inputs and desired outcomes.

With the current structure, the sensor json_attributes_path should be $.email.

Both of my examples validate at jsonlint.com as it happens.

5

Hi Troon
My apologies. I have written a Perl Script on an Ubuntu PC that generates a JSON text file - all done manually. The HA Rest Addon then retrieves the text file every 30 minutes (15 seconds for testing). I mainly want HA display my unread Messages if there are new EMails in my POP3 EMail account.

6

So first step is to get a JSON file that validates.

As a test that the file is being read, comment out the json bits and set value_template to value[:200]. The sensor state should then contain the first 200 characters of the file without trying to interpret it.

7 
{{ states['sensor.my_email_headers'].state }}

Outputs value[:200] only - No characters

I suspect there is a problem with my scripting in Configuration.yaml. I changed my Rest to:

 - authentication: basic
    resource: http://xxx.xxx.xxx.xxx:xxxx/htdocs/email.txt
    headers:
      content-type: "application/json"
    scan_interval: 15
    timeout: 5
    sensor:
      - name: my_email_headers
        #json_attributes_path: $.email
        #json_attributes:
        #  - data
        value_template: value[:200]
8

You should not have the final comma. That breaks parsing.

{
  "email": [
    {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        INCLUDE OTHERS HERE
      ]
    }
  ]
}

Where Troon has posted the example of how the JSON should be formatted - where INCLUDE OTHERS HERE is - the LAST item in that list MUST NOT be followed by a comma, before the end of the list ]

Eg:

{
  "email": [
    {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
{
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
      ]
    }
  ]
}

That will NOT parse.

{
  "email": [
    {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
{
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        }
      ]
    }
  ]
}

but that will.

9

I meant:

value_template: "{{ value[:200] }}"

Apologies for assuming more familiarity than you have.

Use jsonlint.com to check your files for validity.

10

Thanks Andrew
Also did not work - I’m sure there is something else wrong. What about the headers? If I type that URL into a browser, I get the correct JSON file response. REST will not parse it. I now get the error: “Unexpected error fetching multiscrape data”

This is a copy of the browser output:

{
  "email": [
    {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        }
      ]
    }
  ]
}
11

That error is nothing to do with this rest sensor. You must have a multiscrape sensor set up somewhere else.

12

OK. So with the correct JSON file above, I get this error

JSON result was not a dictionary or list with 0th element a dictionary
13

Ah yes: your email is a list not a dictionary. Try:

{
  "email": {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        }
      ]
    }
}

Does this help?

14

This is the result that I get when

value_template: "{{ value[:255] }}"

If I use >255 then I get “unknown”; So I presume the buffer is only 255?

I also looked at IMAP - I could not figure out how to access my POP3 Server. So I wrote the Perl script - which works fine - I just cannot seem to retrieve the data from the JSON File.

What would my rest sensor look like to get the data from the JSON File?

{{ states['sensor.my_email_headers'].state }}

{
  "email": {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our
15

With a file like:

{
  "email": {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        }
      ]
    }
}

you can do things like this:

- authentication: basic
    resource: http://x.x.x.x:xxx/htdocs/email.txt
    headers:
      content-type: "application/json"
    scan_interval: 15
    timeout: 5
    sensor:
      - name: Top email date
        value_template: "{{ value_json['email']['data'][0]['Date'] }}"
      - name: Email count
        value_template: "{{ value_json['email']['data']|count }}"
        json_attributes:
          - "email"

Suggest you paste some example data into Developer Tools / Template and have a play. Use this as a starting point:

{% set value_json = {
  "email": {
      "data": [	
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        },
        {
          "Date": "Thu, 18 Jan 2024 08:00:27 +0000",
          "From": "=?utf-8?Q?Communica=20Marketing?= <[email protected]>",
          "Subject": "=?utf-8?Q?Empower=20Your=20Industry=3A=20Discover=20Our=20Latest=20Gateway=20Solutions=21?="
        }
      ]
    }
}%}
First email date: {{ value_json['email']['data'][0]['Date'] }}
Number of emails: {{ value_json['email']['data']|count }}
Last email from:  {{ value_json['email']['data'][-1]['From'] }}

If you decide what output you want, we can advise. As you’ve found out, sensor states are limited to 255 characters. Attributes can hold more but are fiddly to set up on rest sensors and then trickier to access. My example above should load the entire file into the attributes of the “count” sensor.

16

Thank you Troon
I will experiment today and let you know of the outcome.

17

Thank you Troon for all you assistance. This worked perfectly.

1 Like