Victron VRM Portal API data integration

pblgomez · June 13, 2019, 11:01am

I have written a script to make this work for the problem of the changes in id numbers.
Right now I execute this every boot, as we are modifying sensors.yaml and has to be rebooted to make the changes appear on lovelace.

You can modify it and make it look for different values.

#!/usr/bin/env bash
id=$(curl -H "X-Authorization: Token XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" "https://vrmapi.victronenergy.com/v2/installations/XXXXX/diagnostics?count=1000" | jq '.records[] | select(.description=="Voltage")' | jq .id | head -n 1)
echo "The id is: $id"

echo "We are going to susbtract 1 because the array starts with 0: $x "
echo "I think, but maybe we have to subtract 2 ???"
x=$((id-1))


sed -ibackup -e "/^    battery/,/value/ s/\[[0-9][0-9]\]/[$x]/1" sensors.yaml

lsgv · June 13, 2019, 11:22am

Hi. On the road today so will take a closer look later, but I can already see you have a typo in x-authorization field.
Can you correct it and try again?
I also found a way to parse the json response for the VRM variables I want to display rather than by position number, which solves the problem of having to change it when Victor adapts it’s API.
I’ll post it later.

pblgomez · June 13, 2019, 11:38am

No typo that I can see, script still works But you are welcome to improve it.

pblgomez · June 14, 2019, 12:27pm

Still really curious about your method about getting the VRM variables.

Please share

lsgv · June 15, 2019, 2:43pm

The typo thing was for Denzel, not for you. Sorry

lsgv · June 15, 2019, 2:55pm

This is the way I’m doing it. Probably not the most elegant or efficient, but it works. I don’t know python so I’m doing it all using templating in HA yaml.

First, the API call

# Victron call for current installation data
 - platform: rest
   name: vrm data
   json_attributes:
    - success
    - records
    - num_records
   resource: https://vrmapi.victronenergy.com/v2/installations/10986/diagnostics?count=1000
   headers:
      X-Authorization: Token mytoken
   value_template: '{{ value_json.records[0].timestamp }}'

Then the parsing in a set of convoluted for and if loops.
I first loop through the “device”. 258 for the battery monitor, 256 for the MPPT charge, etc. using Victron’s own device numbering.
Second I look for the target string in the “description” field of the JSON output.
I run these loops for every sensor and sensor value I want to capture.
Important to check the formatting as the API output sometimes comes with units, and other times no units, etc.

Here is the example for Voltage and Current from the BMV (battery monitor).

battery_voltage:
  friendly_name: 'Battery Voltage'
  unit_of_measurement: 'V'
  value_template: '{%- set object = "formattedValue" -%}
                    {%- set vrm_records =  states.sensor.vrm_data.attributes.records -%}
                    {%- for device in ["258"] -%}
                      {%- for target in ["Voltage"] -%}
                        {%- for record in vrm_records -%}
                          {%- for key, value in record.items() -%}
                            {%- if value|string() == target|string() -%}
                              {%- for key, value in record.items() -%}
                                {%- if value|string() == device|string() -%}
                                  {%- for key , value in record.items() -%}
                                    {%- if key|string() == object|string() -%}
                                     {{ value [:-2] | float() }}
                                    {% endif -%}
                                  {%- endfor -%}
                                {%- endif -%}
                              {%- endfor -%}
                            {%- endif -%}
                          {%- endfor -%}
                        {%- endfor -%}
                      {%- endfor -%}
                    {%- endfor -%}'

battery_current:
  friendly_name: 'Battery Current'
  unit_of_measurement: 'A'
  value_template: '{%- set object = "formattedValue" -%}
                    {%- set vrm_records =  states.sensor.vrm_data.attributes.records -%}
                    {%- for device in ["258"] -%}
                      {%- for target in ["Current"] -%}
                        {%- for record in vrm_records -%}
                          {%- for key, value in record.items() -%}
                            {%- if value|string() == target|string() -%}
                              {%- for key, value in record.items() -%}
                                {%- if value|string() == device|string() -%}
                                  {%- for key , value in record.items() -%}
                                    {%- if key|string() == object|string() -%}
                                     {{ value [:-2] | float() }}
                                    {% endif -%}
                                  {%- endfor -%}
                                {%- endif -%}
                              {%- endfor -%}
                            {%- endif -%}
                          {%- endfor -%}
                        {%- endfor -%}
                      {%- endfor -%}
                    {%- endfor -%}'

And the result

Hey, it ain’t pretty (the code), but it works!
One day I’ll try to do it in python as an HA module…

Dan264 · June 18, 2019, 10:22pm

Isgv, thanks for the original post, it was very informative and got me started.
My son has devised a slightly different way to extract the values by their code, rather than position. I have added an and to include instance as depending on your setup you may have more than one occurrence of the same code.

- platform: rest
name: vrm_data
json_attributes:
 - success
 - records
 - num_records
resource: https://vrmapi.victronenergy.com/v2/installations/<siteID>/diagnostics?count=1000
headers:
  X-Authorization: Token <your token here>
value_template: '{{ value_json.records[0].timestamp }}'

- platform: template
sensors:
    battery:
      friendly_name: 'Battery'
      value_template: '{% for item in states.sensor.vrm_data.attributes.records if item["code"] == "SOC" and item["instance"] == 288 %}{{ item["formattedValue"][:-2] | float() | round(1) }}{% endfor %}' unit_of_measurement: '%'

lsgv · June 19, 2019, 6:43am

Excellent! Thanks a lot, this seems much better and cleaner than what I had.
I had already taken into account the fact that the same value might occur more than once by looking for the device number. But as said, this code is much better.
I noticed that from time to time Victron implements breaking changes. For instance, up to a month or so ago, battery monitor battery value was “Battery voltage” and “Battery current”. But then those values stopped showing up on the front end. I checked using Postman and they had changed them to simply “Voltage” and “Current”. Easily fixed but I keep an eye for these changes.
Thanks for your contribution.

ansopa · June 28, 2019, 9:48am

Hello Luis, I have managed to obtain information from the Api through the temporary token, but I have tried to follow the last part of your tutorial with the permanent token and it returns an error
{
“success”: false,
“errors”: “Access denied for this object”,
“error_code”: null
}

I’ve been reviewing everything for a while and I can not find the error

Could you help me, please?

lsgv · June 28, 2019, 10:24am

Hi.
I can recreate the error message (403) you show by making an error in the installation number which I obviously cannot access.
An error with the token is type 401 Unauthorized.
So please check you are using the correct installation number and retry?

ansopa · June 28, 2019, 11:18am

I’m using my idUser as an identifier, is that correct?

If I use the idSite it does not work either, I get this
{
“success”: false,
“errors”: “Path not found.”,
“error_code”: null
}

lsgv · June 28, 2019, 11:43am

First you need to get your credentials using POST to https://vrmapi.victronenergy.com/v2/auth/login with your VRM user ID and password (those you would use to connect to VRM web)
This gives you back a token (short-lived) and your idUser which is typically 5 digits long

Second you use GET to https://vrmapi.victronenergy.com/v2/users/your_user_id_here/installations to get the list of installations associated with this user id. idSite is what you are looking for and it is usually also a 5 digit number

Then you run the GET https://vrmapi.victronenergy.com/v2/installations/your_installation_id_here/diagnostics?count=1000 to get to the data.

If you are using the short-lived token use Bearer insert_your_token_here in the header of the request
If you are using long-lived tokens, use Token insert_your_token_here in the header of the request

Hope you can make it work.

ansopa · July 1, 2019, 8:33am

Than you very much, i dont know the reason i can´t get the data, but i found a way to get the data i need with
https://vrmapi.victronenergy.com/v2/users/XXXXX/installations?extended=1
i don´t need more, this is enought for me

Thanks very much, you are great

ansopa · July 1, 2019, 4:31pm

I found the error
" This endpoint can be used to retrieve all most recent logged data for a given installation. It can only be accessed if the current logged in user has Full access to the specified installation. Endpoint:

/v2/installations/{idSite}/diagnostics?count=1000
```"

Im not a profesional installer, i´m only a user, i think i don´t have a full access acount

lsgv · July 1, 2019, 5:47pm

I’m not a professional installer myself and it works well. How did you create your user in VRM? Maybe that’s the issue.

alnavasa · December 14, 2020, 3:11pm

Up to here All working, Ill keep going. (up until POSTMAN finishes, now going to HA)
Only thing was missing is that on the last step you need to input the installation ID instead of the user ID.
To get the Installation ID you need to do another “Get” to: https://vrmapi.victronenergy.com/v2/users/USERID/installations

Using header tab
key: X-Authorization
Value: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

And you will receive the “idSite” value. that you need to use in the last step.

SO many thanks for this guide, I have ZERO knowledge about coding/programing, and so far so good!

(if you could edit your comment and include it, others will appreciate it)

alnavasa · December 14, 2020, 6:34pm

Systems up and running, Last thing I need to create is a traker entity from the info gathered from the VRM portal

Current state:

denzel · January 7, 2021, 10:06am

This is great,

I got this working using MODBUS:

I got it running this way for about 1 year now
Here is my code if anyone wants to use it:

###############################################################

Victron Sensor Configurations

com.victronenergy.system

###############################################################

“V”= scale: 0.1

“A”= scale: 0.1

###############################################################
sensor:

platform: modbus
scan_interval: 10
registers:
- name: “Battery Power”
  hub: victron
  unit_of_measurement: “W”
  slave: 100
  register: 842
- name: “Battery SOC” # // Best battery state of charge, determined from different measurements. read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “%”
  scale: 1
  precision: 1
  slave: 100
  register: 843
- name: “Battery Voltage” # // Battery Voltage determined from different measurements. In order of preference: BMV-voltage (V), Multi-DC-Voltage (CV), MPPT-DC-Voltage (ScV), Charger voltage //com.victronenergy.system
  hub: victron
  unit_of_measurement: “V”
  scale: 0.1
  precision: 1
  slave: 100
  register: 840
- name: “Battery Current” # // Postive: battery begin charged. Negative: battery being discharged read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “A”
  scale: 0.1
  precision: 1
  slave: 100
  register: 841
- name: “Battery Power” # // Postive: battery begin charged. Negative: battery being discharged read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “W”
  scale: 1
  precision: 1
  slave: 100
  register: 842
- name: “Battery State”
  hub: victron
  scale: 1
  precision: 1
  slave: 100
  register: 844
- name: “Battery Consumed Amphours”
  hub: victron
  unit_of_measurement: “Ah”
  scale: -10
  precision: 1
  slave: 100
  register: 845
- name: “Battery Time to Go” #\ Special value: 0 = charging read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “s”
  scale: 0.01
  precision: 1
  slave: 100
  register: 846
- name: “Eskom Grid” #\ Power supplied by Grid to system. read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “W”
  scale: 1
  precision: 1
  slave: 100
  register: 820
- name: “AC Loads” #\ Power supplied by Inverter to loads. read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “W”
  scale: 1
  precision: 1
  slave: 100
  register: 817
- name: “Active input source” #\ Power supplied by Grid to system. read from //com.victronenergy.system
  hub: victron
  unit_of_measurement: “W” # \ 0=Not available;1=Grid;2=Generator;3=Eskom power;240=Not connected
  scale: 1
  precision: 1
  slave: 100
  register: 826

- name: “Grid lost alarm” # \ 0=Ok;2=Alarm

hub: victron

scale: 1

precision: 1

slave: 261

register: 64

- name: “VE.Bus state” # \ 0=Off;1=Low Power;2=Fault;3=Bulk;4=Absorption;5=Float;6=Storage;7=Equalize;8=Passthru;9=Inverting;10=Power assist;11=Power supply;252=Bulk protection

hub: victron

scale: 1

precision: 1

slave: 261

register: 31

###############################################################

Sensor Configurations - com.victronenergy.solarcharger

###############################################################

  - name: " Solar Charge state" # \\ 0=Off;2=Fault;3=Bulk;4=Absorption;5=Float;6=Storage;7=Equalize;11=Other (Hub-1);252=External control
    hub: victron
    scale: 1
    precision: 1
    slave: 243
    register: 775

  - name: " PV voltage" # \\ Not available if multiple VE.Can chargers are combined
    hub: victron
    unit_of_measurement: "V"
    scale: 0.01
    slave: 243
    register: 776

  - name: " PV Current" # \\ Not available if multiple VE.Can chargers are combined
    hub: victron
    unit_of_measurement: "A"
    scale: 0.01
    precision: 1
    slave: 243
    register: 776

  - name: " PV Power" # \\ Not available if multiple VE.Can chargers are combined
    hub: victron
    unit_of_measurement: "W"
    scale: 0.1
    precision: 1
    slave: 243
    register: 789
    
  - name: " Yield today" # \\ Today's yield
    hub: victron
    unit_of_measurement: "kWh"
    scale: 10
    precision: 1
    slave: 243
    register: 784

  - name: " Maximum charge power today" # \\ Today's maximum power
    hub: victron
    unit_of_measurement: "kWh"
    scale: 1
    precision: 1
    slave: 243
    register: 785

  - name: " Yield Yesterday" # \\ Yesterdays's yield
    hub: victron
    unit_of_measurement: "kWh"
    scale: 10
    precision: 1
    slave: 243
    register: 786

  - name: " Maximum charge power yesterday" # \\ Yesterdays's maximum power
    hub: victron
    unit_of_measurement: "kWh"
    scale: 1
    precision: 1
    slave: 243
    register: 785

  - name: "Battery Temperature" # \\ Degrees celsius
    hub: victron
    scale: 10
    precision: 1
    slave: 243
    register: 773

###############################################################

Sensor Configurations - com.victronenergy.tank

###############################################################
- name: “Tank Capacity” # \ Tank Informaiton Capacity
hub: victron
scale: 10000
precision: 1
slave: 22
register: 3001

  - name: "Tank Levels" # \\ Tank Informaiton Levels
    hub: victron
    scale: 10000
    precision: 1
    slave: 22
    register: 3004

lsgv · January 7, 2021, 3:41pm

Good to see you got it to work with modbus. It is a very powerful protocol which I use for my swimming pool equipment (automatic filters and saline chlorinator) but it takes some time to learn.

When I started this thread 3 years ago I was new to HA and most of the possible integrations as well as to Victron but I did have some experience with APIs and remote calls so I naturally went that way.

In the meantime, I’ve learned a few things and have moved on with my installation. When I started out I had a small 6 panel 1.8kWatt installation running a pump and irrigation system. Since then I built my off-grid house with 13kW installed power, 2 Fronius PV inverters and 3 Victron Multiplus II, 18kWh Lithium batteries and a 10kW gas generator as back-up just in case. This system is powering a rather complex set of installations and I wanted more robust and resilient solution which needed to be 100% local and not dependent on availability of internet connection.

Lo and behold I “discovered” the fact the Victron CCGX or any implementation of their VenusOS software has an embedded MQTT server publishing all the data. I don’t know if this feature was already present in 2017 but at that time I didn’t look for it as I wasn’t aware or knew how to use MQTT.

The advantage of using MQTT integration with HA is that, assuming your HA and VenuOS instances are on the same network, it is not dependant on internet connection, which is what I was looking for. Furthermore, it is quite easy and user friendly to set-up.

1 - Enable MQTT on the Victron VenusOS device by going into Settings/Services/Enable MQTT on LAN (SSL)

2 - Test the connection using for instance MQTT Explorer pointing to the IP of the VenusOS device and port 1883 (standard port). You will see the N-topic with the device mac-address. At this stage you will probably not see any sensor data because the VenusOS device MPTT server needs a keepalive signal every 30-so seconds, otherwise it just goes silent

3 - Install a Mosquito broker on your HA instance (I use the official add-on which works perfectly) and configure it to your taste. In my case, as I have 2 VenusOS devices (one for the well-house PV installation, another one for the mani house) I needed to set-up “Customize = true” and set-up 2 connections in /share/mosquitto/mosquitto.conf specifying the top topic (but you can specify lower level topics if you want to)

connection casa
address 192.168.X.XXX:1883
topic N/b827ebbbwxyz/# in
topic R/b827ebbbwxyz/# out

connection caseta
address 192.168.X.XXX:1883
topic N/04a316c4wxyz/# in
topic R/04a316c4wxyz/# out

This setting allows the HA MQTT broker to connect to both VenusOS instances. You can go further and customise the topic command with prefix and so forth as explained in Mosquito Broker documentation but in my case I didn’t think that was worth the trouble

4 - Enable the integration in HA in Configuration/Integrations adding MQTT and pointing it to your HA instance IP

5 - Create an automation for the keepalive of each of the VenusOS MQTT server

- id: '1604576260450'
  alias: mqtt keep alive pour caseta
  description: ''
  trigger:
  - platform: time_pattern
    seconds: /30
  condition: []
  action:
  - data:
      topic: R/04a316c4wxyz/system/0/Serial
    service: mqtt.publish
  mode: single

6 - Using MQTT Explorer browse through all the data exposed by your VenusOS device and select what you want/need to create HA sensors

7 - Enjoy

CountParadox · January 11, 2021, 12:47am

Thank you !

I was able to easily adapt this to the data I get out from VRM.

Anybody know how to set a static icon for these template sensors?
Or maybe even a dynamic one that changes color on voltage / current? hmm