[Under New Management] Interactive history explorer custom card

HeyImAlex · December 21, 2021, 4:55am

The version built by @HeyImAlex has been deprecated, but a new verison has been taken over by @SpangleLabs

Original post:

Hey all

I created a new custom history card for HA. HA’s native history panel is very nice, but it does have a number of serious limitations. Most notably the slow speed, the somewhat awkward UI and the lack of configurability. So I tried to approach this differently. The history explorer card uses asynchronous streaming to retrieve the database data in the background while you scroll and slide around your history. While this can’t entirely get rid of the database latency, it makes it much less noticeable. The card also uses adaptive data decimation and various other little tricks to make it fast and smooth.

New: From V1.0.30 on, the card also supports seamless integration of long term statistics into the graphs, combining your history database and your statistics into a single explorable history. See here.

There’s a video showing the interactive use of the card on the github readme

Just select the day and your time range and drag around the charts with your mouse (or finger) to move through time. You can also zoom in and out, all in realtime by using the time range selector, the interactive zoom button or by using the scrollwheel while pressing Ctrl.

It’s also entirely configurable ! Put any number of graphs with any combination of entities in any order. The colors for line graphs, the state timelines and bar graphs are all configurable. And because this is a card, you can have multiple independent histories at the same time. The card can be configured entirely over the UI without changing the Lovelace config.

New: From V1.0.27 on, total increasing sensors, like energy data, are now also fully integrated. The time interval used to display these values (hourly, daily, etc) can be changed on the fly and you can compare multiple intervals in realtime. More info here.

I originally made this for my own personal use. But I thought that some others may be interested too. I’d love to hear some feedback from you guys !

Let me know what you think !

sophof · December 21, 2021, 4:35pm

I came upon this thread via the feature request Prevent History page from preloading values

I am very interested in this, but am most interested in the features mentioned there, do you plan to add the possibility to not preload anything and allow multiple entities to be selected without changing the configuration?

HeyImAlex · December 21, 2021, 7:00pm

Yes to this. I’m planning to add this next.

Not really. Could you explain why you would need such a feature ? I understand that in the other thread the reasoning was the slowness of opening the history panel. This doesn’t really apply to this card. If you keep the amount of entities reasonable, opening the history for the last 24 hours is pretty much instant.

HeyImAlex · December 21, 2021, 9:52pm

Fixed a major issue that would prevent the card from opening on some browsers. Sorry for that. If it didn’t work for you before, please try again.

sophof · December 22, 2021, 9:43am

Well, I probably wasn’t entirely clear. It isn’t just the slowness - although that is what makes the problem more annoying - but the way I use the history tab. IMO it is useless to have fixed entities there since that is what we have the normal views for. The history tab is there for spontaneous questions of every form (debugging, very specific information, setting up new automations, etc.).

I’ve tried discussing this before on this forum, the history integration right now is a weird combination of 2 functions. First it has the very important task of saving the state history. Second it has the task of allowing you to visualise this history. But since the View and storage are combined, removing something from the cluttered view, also removes it from storage. I think this creates confusion and is the reason the history tab still isn’t very flexible. It would make more sense to keep these things apart, just as entities are no longer strictly tied to their visual controls. In a sense that is what your explorer does.

Therefore it just doesn’t make sense at the very least to show everything, but really it doesn’t make sense to hard configure any default entities at all. I imagine we can have a much cleaner interface if it is more focused on the task it is used for.

HeyImAlex · December 23, 2021, 6:19am

I disagree that it doesn’t make sense to not be able to hard code entities. There are several use cases where you specifically want to observe a certain well defined set of entities. Case in point, I do need this feature myself, so it’s going to stay in

That said, I added the possibility to add and remove entities on the fly. And that’s definitely a very nice and useful feature. If you don’t want to hard code any entities, you can just supply on empty YAML. There’s an example on the github. The card will then just start with an empty view.

VDRainer · December 23, 2021, 9:50am

I like it!
Is there a chance to make it more readable on dark themes?
Here default dark:

EDIT: Seems like this is only if the card is defined empty.

type: custom:history-explorer-card
graphs:

If entities are added, all looks good.
Auswahl_054

EDIT2: I can’t reproduce it anymore.
Maybe a browser cache problem.

Sorry for the false report.

sophof · December 23, 2021, 10:52am

Well obviously that’s ok I was just trying to explain my point of view. Just out of curiosity, why don’t you use the standard Views for that though? I guess that is the part I don’t understand. All histories I want to regularly view I have added to a View (for instance all temperatures over the past few days).

But I will give your card a whirl, it looks very promising!

HeyImAlex · December 23, 2021, 5:19pm

Hey, glad you like it ! There’s indeed some problem somewhere with theme switches, some UI refresh issue I can’t always reproduce myself. I’ll look into this.

How’s performance for you when you swipe / scroll horizontally through the graphs, let’s say with a 1 Day time range ? Is it smooth or does the graph play lots of catching up with the DB background loading ? Writing this card really made me aware of just how incredibly slow the HA recorder / database component is. It’s unreal, there’s so much redundant useless information stored and retrieved. I’m seriously thinking about adding a local persistent cache with a highly optimized and compressed binary representation of the history for viewing purposes.

HeyImAlex · December 23, 2021, 5:25pm

Pretty much the same use case as yours. Debugging sensors, optimizing automations, situational awareness of some data over varying time scales. Like for example, I’m showing the history of temperature, humidity, window open/close states and heaters for a certain badly insulated room, so I can see if there are patterns (short term and longer term) that I can use to optimize heater control. I want the entire history easily available for these sensors, at all time scales and I want all of them sync’ed at the same time. But I don’t want to add them everytime I open Lovelace, because it’s a longer term observation of the same sensor set.

The ideal solution would be to keep the dynamically added entities in a local persistent storage and optionally keep them through Lovelace reloads. That way I could entirely get rid of the preconfigured YAML (maybe only for global options, like custom colors). But the way HA manages custom cards is very much pre-configured YAML centric. That doesn’t mean there’s no way around this.

HeyImAlex · December 27, 2021, 6:34am

I improved the card a lot over the weekend. It now supports fully interactive configuration (without changing the Lovelace config) and retains it in your browser storage. No need to mess around with YAML anymore (but it still works for those who like YAML). There’s also a new region zoom function that works over the entire history database, making it even easier to dive into your sensor history.

I uploaded a video showing the use of the card in realtime on the github.

finity · December 28, 2021, 4:56am

Maybe I’m confused about how to use the card.

I put the UI configurable example in a view with other existing cards as below:

title: Test
type: custom:vertical-layout
layout:
  max_cols: 4

cards:

  - type: custom:history-explorer-card
    graphs:

  - type: history-graph
    title: actual Lights History 96
    entities:
      - light.basement_bath_light
...

and when I do a refresh of the view the history graph card doesn’t show up and all of the other cards all disappear so I’m left with a blank page.

is this supposed to be in a view all by itself (panel_mode: true) or can it be used in a view along with other cards?

HeyImAlex · December 28, 2021, 5:13am

You can just add it like any other card. It works along with other cards, but it’s best used on its own in a panel_mode view or dashboard (because it can get pretty large if you add entities).

Make sure you download the latest release if you haven’t already, I made the configuration much easier (clear the browser cache if you had a previous one installed). You don’t need to use the YAML at all anymore now, you can do it all through the card UI.

Just hit add card and select history explorer:

And then just leave everything at default and hit save:

Now add your entities by searching for them in the search field (1) and adding them with the plus button (2):

The added entities will be stored in your brower local data. If you want to add and customize them in the YAML instead, you can do that later on.

finity · December 28, 2021, 5:28am

I just added the github repo as a custom repositoy in HACS and downloaded it from there. and updated my lovelace resources file.

then restarted.

I don’t use UI mode at all for lovelace. I configure it all thru yaml.

The yaml I used for testing it in a view is above.

tried that already too.

I just noticed in the UI screenshot that there is no graphs: line after type:.

is that correct?

should I remove that line from my yaml config?

EDIT: Nope, just tried it and it still shows a blank page.

I’ll try that next.

I was just wondering if I was missing something…and kind of still am. it seems you are saying it should work in a view along with other cards but I don’t seem to be successful with that.

HeyImAlex · December 28, 2021, 5:36am

Hmm, doesn’t seem to be anything wrong from a first glance. If all cards disappear, then this usually means that something is wrong with the YAML syntax. Maybe try in a panel_mode only dashboard to see if it works there ?

Doesn’t really make a difference. The empty graph: section is optional, you can just leave it out.

Let me know if it works as a standalone. If not we can investigate further. I don’t use HACS myself, so maybe something went wrong with the installation.

For reference, my test YAML looks like this (for a panel):

title: Home
views:
  - title: History test
    path: card-test
    type: panel
    badges: []
    cards:
      - type: vertical-stack
        cards:
          - type: custom:history-explorer-card

finity · December 28, 2021, 5:42am

I just tried that too and it still just shows as a blank page.

I even tried switching back to the default theme just in case that was an issue but no change.

I’ll dig into it and try that next.

I originally downloaded the files from github manually but before I went any further I figured I’d try HACS and when it didn’t complain and it downloaded everything I figured it was good.

EDIT:

I didn’t try it as another dashboard. Just as a view in my main (only) dashboard.

I don’t have any real experience with multiple dashboards so it may take me a bit to try it like that.

EDIT 2:

I tried it as a standalone dashboard and it seemed to work that way. At least some of the card elents are showing up now.

I’ll have play at it and see how it goes from here.

EDIT 3:

Nope, I can’t get it to work.

I see the following:

but nothing on the page actually does anything even if I start typing an entity id.

and opening the console in Chrome I see a flood of these errors:

I had the page open for about 30 seconds and the error count got all the way over 1000.

I’ve removed the card from HACS and manually installed it. Restarted HA, cleared browser cache and nothing seems to be working for me.

HeyImAlex · December 28, 2021, 7:32am

Huh. I can’t really reproduce this. What version of Chrome are you using ?

I pushed a new build you may want to try. I fixed another Chrome related bug in there, but it’s probably not related to your issue. Your problem seems to come from locale related getters. I added a YAML switch to skip the locale checks and replace them with a default locale. That should avoid the problematic code for you. Just add this with the card YAML:

type: custom:history-explorer-card
skiplocale: true

Let me know if that helps.

finity · December 28, 2021, 8:56am

ex3

I updated the card with the latest and it seems to be working now.

I’ll let you know if I see anything else as I start to use it more.

Thanks for your help.

Let me know if there is anything else I can do to help you with the locale bug fix testing.

finity · December 31, 2021, 1:39am

playing with the card a bit more and seeing a couple of things…

where does it get the theme from?

it looks OK in the default theme but using my Slate theme the grid lines/legend is pretty light and the items in drop down for the time doesn’t show up until I mouse over it.

second the charts don’t always expand to fill the screen as you can see in the screenshot.

and I believe you said that once you configure a chart it should be remembered by the browser so if you navigate away and back it should bring up the charts that were configured previously. It’s not doing that. If I navigate away and back it comes back blank again.

I really don’t know if it’s my browser/computer tho because i’m not seeing any errors in the browser console.

HeyImAlex · December 31, 2021, 2:28am

From nowhere. It’s just dark vs. light right now. If you’re using a dark custom theme but the HA dark flag is not set, the card thinks it’s a light one and uses the light colorset. All that stuff is extremely badly documented (or the documentation is really well hidden), so it’s mostly reverse engineering the HA source code. I personally don’t use themes. I’ll try a couple and look into getting it work with themes.

Uhm, isn’t that simply because you reached the end of the history database on the right ? The card can’t look into the future

Could you please open a github issue on that ?

Yeah you do have a few interesting issues lol. I did end up finding the locale one you had though.