Rest Sensor with multiple json results returned, how to craft value template

I am trying to gather info from an API endpoint that returns a bunch of web-page structure parameters, but has some kernels of useful information I need to automate against. I currently have a working python script that outputs just the useful info in json

However I’m struggling to find an integration that will work with this, and the issue is that each request to the API shows the latest 50 messages, so if I run it at 10:00 and again at 10:30, but only 1 new message has come in, there will be 49 duplicates.

Can someone point me in the right direction to achieve this short of writing a custom component or processing/deduplicating the output external from HA.

I think I’m stumped here

Show us the data and explain what you want.

If you have freedom in how the JSON is structured, it should be easy to populate a rest sensor’s attributes as needed.

Thanks for looking

The data is private. But heres an example

{
conversations {
  conversation {
    id: "38e93ieid"
    subject: "example"
    Body: "body"
  }
  conversation {
    id: "37d73uejd"
    subject: "example2"
    Body: "body"
  }
 }
}

The ids are unique, so i just cant figure out a valid integration to set the status as ID and deduplicate the responce elements across scans.

For example, if a new message is in this system, the next respose will look like below with a max messages of 50. In the next scan, only conversations with subject example3 and example4 but using the id to determine uniqueness should be added to the sensor state.
Im ok if each state is only instantaneously displayed, i will just be passing it thru processing the attributes as they come in.

Im thinking i can just use the set_state python script to just force a dummy sensor to be updated with each message.

I need to use this format because its the only way i can get the data i need into home assistant so i can process each message in an automation that will send certain high priority messages to a satellite communicator.
I have existing logic that will only send these high priority alerts if my phone is not responding due to being out of cell range.

{
conversations {
  conversation {
    id: "zjdi3ieidj"
    subject: "example3"
    Body: "body"
  }
  conversation {
    id: "d883ejdj"
    subject: "eexampl4"
    Body: "body"
  }
  conversation {
    id: "38e93ieid"
    subject: "example"
    Body: "body"
  }
  conversation {
    id: "37d73uejd"
    subject: "example2"
    Body: "body"
  }
 }
}

Are you trying to say your sensor will grow forever as you add only the unique id’s and not remove the old ones? Or you just want a sensor that only has the new unique id’s?

I mean i only need momentary state of each message because they only are useful the first time they re seen after each scan, so the state should just flash (thus triggering automations) the new id states with other properties as attributes.
I dont mean that the state will combine all new messages in to a chain of strings

The only reason any history is needed is just as a likely reference point for deduplication. But id only need 50 states stored in history

And the data is not returning any form of time stamp? (if not the system is poorly designed).

So can you say this:

I make a call to the this service and it returns 50 things.
The next call I make returns 50 things, and I only want is different from the previous call (not from many past calls, just the last one).

If so, that should be doable.

Yes, looking at the total raw json pre parsing there are timestamps associated with each conversation object

Im tinkering to see if there are parameters to request only messages since now() - 30 minutes, but i think the most stable method would be to just deal with the default call I captured

What you described is exactly what i want to achieve