Why is some of my data disappearing

I use curl to scrape data from a web site about my propane tank levels.

Every now and then older data disappears.

I have never understand long term stats. Is that what the problem is? Is there an easy/quick change I can make to ensure I have this data?

Here is the code:

command_line:
  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 371 Gallons"
      unique_id: propane_tank_371_gallons
      value_template: "{{ value_json[0].Level|float(0) * 2.4 if value_json[0].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  # 371 Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 629 Gallons"
      unique_id: propane_tank_629_gallons
      value_template: "{{ value_json[1].Level|float(0) * 1.2 if value_json[1].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  # 629 Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 255 Gallons"
      unique_id: propane_tank_255_gallons
      value_template: "{{ value_json[3].Level|float(0) * 1.2 if value_json[2].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  # 255 Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 76 Gallons"
      unique_id: propane_tank_76_gallons
      value_template: "{{ value_json[4].Level|float(0) * 1.2 if value_json[3].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  #76 Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 125 Gallons"
      unique_id: propane_tank_125_gallons
      value_template: "{{ value_json[5].Level|float(0) * 1.2 if value_json[4].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  # 125 Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 630 Gallons"
      unique_id: propane_tank_630_gallons
      value_template: "{{ value_json[2].Level|float(0) * 2.4 if value_json[5].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
  # ??? Propane

  - sensor:
      command: "curl -k -u <user>:<pass> https://ws.otodatanetwork.com/neevoapp/v1/DataService.svc/GetAllDisplayPropaneDevices --header Content-Type:application/json"
      name: "Propane Tank Neevo 6 Gallons"
      unique_id: propane_tank_6_gallons
      value_template: "{{ value_json[6].Level|float(0) * 2.4 if value_json[6].Level is defined else this.state }}"
      scan_interval: 14400
      state_class: measurement
      unit_of_measurement: "Gallons"
# ??? Propane

And this is what my current screen looks like. Note missing data from late November, as well as the past couple of day (but only for a few sources):

Are there any errors in Settings → System → Logs related to your recorder database?

Go to Developer Tools → Statistics, are there any FIX ISSUE buttons?

EDIT: what graph card is that (it’s not the core card as that only supports one Y asis)?

It may only support displaying state data, not LTS. Or you may have to configure it to display LTS (like apex charts).

I am using plotly.

Do you think the solution lies with the card?

Thank you!

Yes. See GitHub - dbuezas/lovelace-plotly-graph-card: Highly customisable Lovelace card to plot interactive graphs. Brings scrolling, zooming, and much more!

I’m sure it’s me, and I apologize, but I don’t see anything in that link that explains what I’m experiencing.

Your graph is currently plotting state data which only goes back 10 days by default. To plot statistics data instead (which goes back to whenever the sensor was defined) use the statistic configuration option shown in the link.

1 Like

Wow!

You did it!

It was all a matter of plotly understanding the type of data.

BTW, I read the plotly link and if only it would have included language such as this I might have understood what to do:

“To plot statistics date instead (which goes back to whener the sensor was defined) use the statistic configuration option.”

If only the developer had you write the read.me (:wink:

Thank you so very much for your generous help.

I still have one data source that isn’t playing nice: tank_utility.

This sensor (sensor.tank_utility_003200223638383015473830) is created by the tank_utility integration.

DEVELOPER | STATES show it’s details as follows:

And here is the full card showing my propane levels going back as far I’d like:

Thank you again!

EDIT: I checked the entity directly and the data goes back only as far as plotly is showing it. So, maybe this is an LTS issue?

That sensor has no state_class and thus is not generating statistics. Which integration did you use to add it to home assistant?

The integration is TANKUTILITY

So this one?

If so that does not seem to support the state_class option. You can open an issue for that here: Issues · home-assistant/core · GitHub

However in the meantime you can add the state class using manual customisation, see: Home Assistant Core Integration - Home Assistant

e.g.

# configuration.yaml

homeassistant:
  customize:
    sensor.tank_utility12345677: # change this to your sensor entity id
      state_class: measurement

Then restart HA.

Perfect!

I made the change and it seems that it works:

I added to the plotly card:

    statistics: state
    period: 5minute

If I use a longer period, the value disappears and in its place is “n/a”

I assume because as a sensor with state_class: measurement, it simply hasn’t existed long enough for an hour or day period to be populated.

Interestingly, if I remove the statistics: and period: from the card, the 7 or so days of data show up.

This is such a great solution for rogue intrgrations.

Thank you again!

Correct.

Yeah that’s the state data, not statistics. It only goes back 10 days by default.

The developers don’t like people relying on it. You should definitely still open the issue and get the integration fixed.

The problem, as I understand it, with TankUtility, is that Generac bought the small company (TankUtility) that developed the product and changed it substantially and now does not support it.

That means that the HA integration is no longer being worked on.

There are stories out there of people trying to simply change the SSID that the device connects to and winding up with a non-functioning and non-supported device.

One would think that this means there’s an opportunity in the market for a wifi-connected propane tank level sensor. If I was 30 years younger, I might just jump on that.

Thank you again so much for your help, now, and in the past, and for the entire “Cause” (HA, that is).

1 Like

Does tank utility actually still work? I am struggling to get it working with my setup and I have a older wifi unit, but my dashboard automatically defaults to app.generacfuel.com when I login.

Sorry for the delay – just seeing this.

Yes, tank utility is working for me.

Config.yaml simply has:

sensors:
  - platform: tank_utility
    email: "<EMAIL_ADDRESS)"
    password: "<PASSWORD>"
    devices:
      - "<DEVICE_NUMBER>"

I am far from an expert at anything HA, but I hope this helps.