Does anyone know if there’s a way to migrate old data from InfluxDB to VictoriaMetrics? I was using InfluxDB on HA OS with Frenc’s Addon.
Yes, but it crashes (in the same way) even before I use the influxdb integration inside HA.
Also, I cannot find a way to get the token, etc. that is in his influxdb integration.
Just a quick search I found this that may help… github - Git for windows version 2.5.3 not able to push changes - Stack Overflow
…but you may need to reach out to the vm community.
Not sure on the influx part either. I took a different approach using prometheus integration + vm agent.
That doesn’t seem to be relevant for my case (my disk is far from full, and using linux). Could you point to a way to set it up the way that you did?
This entry above pretty much sums up how I’m running it. I’ve made some tweaks based on some discussion within this thread so for clarity my current configs are below.
BUT it seems like something is up with your disk and vm can’t write to it for some reason… what is at /media/virtual/victoriametrics …a usb drive?
my docker-compose.yml (just the relevant entries)
version: "3"
services:
vmagent:
container_name: vmagent
image: victoriametrics/vmagent:v1.80.0
depends_on:
- "victoriametrics"
volumes:
- $HOME/docker/vmagentdata:/vmagentdata
- $HOME/docker/vmprometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- "--promscrape.config=/etc/prometheus/prometheus.yml"
- "--remoteWrite.url=http://127.0.0.1:8428/api/v1/write"
network_mode: host
restart: always
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:v1.80.0
volumes:
- $HOME/docker/vmdata:/storage
- /etc/localtime:/etc/localtime:ro
command:
- "--storageDataPath=/storage"
- "--httpListenAddr=:8428"
- "--retentionPeriod=3y"
- "--selfScrapeInterval=60s"
#- "--downsampling.period=30d:5m,360d:1h"
network_mode: host
restart: always
Then in ~/docker/vmprometheus/ I have prometheus.yml
global:
scrape_interval: 60s
scrape_timeout: 20s
scrape_configs:
- job_name: "hass"
scrape_interval: 60s
metrics_path: /api/prometheus
bearer_token: !!!see HA promethues integration on how to generate!!!
scheme: https
static_configs:
- targets: ['localhost:8123']
…the instructions for generating bearer_token is on home assistant’s prometheus intergration page (see link below).
finally in home assistant’s configuration.yaml I’ve setup the prometheus integration like this…
prometheus:
namespace: hass
filter:
include_domains:
- sensor
- climate
- binary_sensor
1 Like
Thanks for your configuration. No, it is not USB, but rather a RAID with several free terabyte. When starting the container, it writes a configuration just fine to the directory. It is only when connecting HA to the container that it fails.
I’m using VictoriaMetrics in combination with Home Assistant for a couple of months now, and I am pretty pleased with it. It runs very stable, and the database size remains within limits. I run both HA and VM in docker containers, and wanted to share my configuration for those who would like to run the same, or a similar setup.
Here is my docker compose file to start the VM service, and to have the VM data stored in an external data container:
services:
victoriametrics:
container_name: victoriametrics
image: victoriametrics/victoria-metrics:latest
ports:
- 8428:8428
volumes:
- vm-data:/victoria-metrics-data
command:
- "--retentionPeriod=99y"
restart: unless-stopped
volumes:
vm-data:
external: true
I use the influxDB integration of HA to write to the VM database, so I don’t need a VM agent and prometheus configuration. Note that I do not use an Influx database, I only use the HA integration. In my configuration.yaml file I have added:
influxdb:
api_version: 1
host: <host_ip_adres>
port: 8428
measurement_attr: entity_id
tags_attributes:
- friendly_name
- unit_of_measurement
include:
domains:
- binary_sensor
- climate
- counter
- cover
- light
- sensor
- switch
That’s it. In Grafana I can now choose to use the VM database as InfluxDB Timeseries Database, or as Prometheus Timeseries Database, or both. Both work, but use a different query language.
I have about 1000 HA entities, and my VM database grows about 1 MB per day.
2 Likes
@CreebleHoost , thanks for sharing your configs! They look great except one thing - it is better to specify a particular release tag for victoriametrics instead of latest, since the latest tag may point to some testing release. For example, it is much better to use v1.90.0 instead of latest. It is also recommended periodically checking VictoriaMetrics changelog for new releases.
1 Like
This is a very important point and actually one of the inspirations of this thread.
1 Like
Thanks, that is a good suggestion. Ideally there would also be a stable release tag. Then I would get the release with the latest fixes, without running the risk of getting experimental code.
Typically I don’t check manually if there are new stable releases available. Not only for VM, but for any docker.
1 Like
VictoriaMetrics already provides stable Docker image tag for new releases - see this comment.
Note that it is recommended to do periodic docker pull for upgrading VictoriaMetrics Docker image to the latest stable release.
1 Like
For migrating your influxdb data you should use the vmctl tool from victoria metrics. You can use it on your host or any other linux machine.
In this thread you might find some hints using it:
The bug i posted is sorted out and it runs without any errors even if you have empty measurements.
Using the config suggested above, I’m getting ..._value and a ..._state metric for each binary_sensor when looking at the data via Grafana (using the VictoriaMetrics plugin for connection). And a light goes between 1 and 0 for on vs off. Is this by design or a misconfiguration on my part? Is there a way to have the actual states (occupied, on, etc) tracked? I also don’t see the UX to do things like aggregate (e.g., average for each hour, etc). Perhaps user error here?
measurement_attr: entity_id
tags_attributes:
- friendly_name
- unit_of_measurement
I am currently using InfluxDB, and my DB is growing in size very fast, so I am looking for a long-term solution to this issue.
Can anyone maybe provide some comparison info on the size of the DB, or the expected size reduction when using VictoriaMetrics instead of InfluxDB?
Also, is there a downsampling feature in VictoriaMetrics for HA? Their official gitHub states it is available in VM Enterprise only…
Thanks!
1 Like
I have a fundamental question concerning the feeding of measurements to VM if using the influxdb integration:
Is there a fix schedule when all metrics are pushed to VM (say 1min or 1sec or…) or is every metric only pushed if it has changed?
And if the former: is there a way to adjust the interval?
If I use prometheus integration I can set the interval for scraping myself.
A time-based push would lead to much redundancy (or does VM suppress doublettes internally?) but with pushing on-change only, I’d imagine future downsampling would be tricky?
How are people viewing the data in VM? Are you using Grafana + Prometheus data source or Grafana + Influx v1 data source? Or something else?
Grafana + prometheus data source. Grafana dashboard in HA (How to seamlessly add Grafana graphs to Home Assistant » The smarthome journey)
2 Likes
We have created an own datasource with some additional features for VictoriaMetrics GitHub - VictoriaMetrics/grafana-datasource: Grafana Plugin for VictoriaMetrics so you can use it with Grafana instead of Prometheus datasource.
1 Like
Prefer using Home Assistant’s Prometheus integration rather than its InfluxDB one. VictoriaMetrics supports InfluxDB data sources but it really seems like it’s more designed for Prometheus-compatible ones.
I’m just using VictoriaMetrics’ web UI (VMUI) at the moment. Haven’t gotten around to setting up Grafana yet. VMUI is pretty good for ad-hoc queries. I was just using it today to check the daily peak output reached by my solar power system for the past six months:
max_over_time(homeassistant_sensor_unit_w{entity="sensor.envoy_current_power_production"}[24h])
I haven’t used InfluxDB for a long time, but I reduced the size of one of my DBs from 43GB to 15GB by switching from Prometheus to VictoriaMetrics. This particular DB was for server monitoring. 2 years of data, ~70k time series, with most metrics logged once per minute.
VictoriaMetrics (and Prometheus) are time series databases, which means they can only store numbers and how those numbers change over time, with tags per time series (entity name, etc). Boolean values (true/false, on/off) are always logged as 0 and 1. Do you have a use case where this isn’t sufficient?
1 Like