Sensor data insert into existing table in database

hi, I can’t seem to find a simple answer to this.

I have an existing table in a secondary database in MariaDB, that I would like to insert sensor data into on a 5 minutely basis.

I have searched and found appdaemon, nodered, or command line solutions. They seem like an outdated way to achieve this.
I feel like I should be able to insert the data directly via a script or restful command or SQL integration even directly via the recorder. I can’t find an example of that being done however.

Is there a mysql insert documented somewhere for HA?
Also can you reset a restful command from yaml configuration reloading?

It is core-mariadb install and not_homeassistant_database. port is 3306

If someone could point me to an example that would be great!

The url’s ive tried are the same as what would similar as from the recorder
mysql://user:password@SERVER_IP:{tried with and without port here}/DB_NAME?charset=utf8mb4

Logger: homeassistant.components.rest_command
Source: components/rest_command/__init__.py:148
Integration: RESTful Command (documentation, issues)
First occurred: 14:21:16 (1 occurrences)
Last logged: 14:21:16

Client error mysql://user:passwd@core-mariadb:3306/DB_name?charset=utf8mb4
Source: components/rest_command/__init__.py:121
Integration: Automation (documentation, issues)
First occurred: 14:07:12 (2 occurrences)
Last logged: 14:07:12

insert sbfspot: Error executing script. Unexpected error for call_service at pos 1:
While executing automation automation.insert_sbfspot
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 367, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 570, in _async_call_service_step
    await service_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1636, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1673, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/components/rest_command/__init__.py", line 121, in async_service_handler
    async with getattr(websession, method)(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1146, in _create_direct_connection
    assert port is not None
AssertionError

I have also tried the insert link from the tables page in myPHPadmin, which gives the 401 Unauthorised below

rest_command:
  powerpal_insert_sbfspot: 
    url: 'http://url_sanitised/index.php?route=/table/change&db=DB_name&table=Consumption'
    headers:
      accept: "application/json, text/html"
    method: post
    payload: "{{now() | as_timestamp | int }},{{states('sensor.power_total_consumption')| float * 1000 }},{{states('sensor.power_live_consumption')| float * 1000 }},"
Logger: homeassistant.components.rest_command
Source: components/rest_command/__init__.py:137
Integration: RESTful Command (documentation, issues)
First occurred: 13:50:00 (1 occurrences)
Last logged: 13:50:00

Error. Url: http://url_sanitised/index.php?route=/table/change&db=DB_name&table=Consumption. Status code 401. Payload: b'1648608600,4870547.0,0.0,'
1 Like

I don’t have an out-of-the-box solution for you, but can explain why these attempts won’t work:

That won’t work because the rest_command integration only works with http/https based end-points which MySQL itself does not provide.

That also won’t work because phpMyAdmin is just a UI, not a rest end-point. And the reason it’s responding with 401 Unauthorised is because phpMyAdmin requires you to login via form which the rest_command integration does not support.

If you have a webserver that can run PHP you could deploy something like GitHub - mevdschee/php-crud-api: Single file PHP script that adds a REST API to a SQL database in between the rest_command and your MySQL database.

1 Like

Thanks, that is useful information. I was starting to gather that after trying some in the command line and being able to see the responses. You have definitely put it succinctly.

Webserver idea makes more sense now.

Seems a shame recorder isn’t capable of this function. At least it doesn’t look like it is.
Could I point recorder at the DB in question and do something in the DB to get the info in the right table?

Well, recorder has a very specific purpose and is integrated both ways, i.e. sensor data is automatically stored and then retrieved to show graphs and other historic data.
I don’t think you can have multiple recorder configurations, so if you wanted to use it then you would sacrifice all the goodness that this integration otherwise provides.

A few more options:

  • Many people, including myself, use InfluxDB to store data of specific sensors for other applications or to create history graphs. If you are not bound to MySQL, then this may be an options.
  • If you don’t have to have this data in a separate MySQL database, but would be OK storing it in a table in the Home Assistant database, you could implement a MySQL trigger that copies state changes into a different table.
1 Like

It actually only came about because I just got around to adding consumption to PVoutput.

The rest(solar) of my PVoutput data is stored by SBFspot into mariadb and uploaded from there(I think that’s how it flows anyway).

So I was trying to reduce an upload by 1, while adding the data to the existing database for backup and archival uploads.

So I was trying to merge a few things, that seemed like might make sense.

Technically the data already exists in HA DB, is it possible to shift it from there? I suppose I could add the SBFspot tables to the HA DB…

I’ve not tried it, however perhaps you could craft a sql INSERT statement or stored procedure call and put it in the ‘query:’ of the SQL Sensor, the content of the ‘query:’ value seems to allow any SQL statement:

1 Like

That was my initial thinking. Although the docs don’t list an insert as being possible or a syntax. I will perhaps fiddle with it later.

I have it solved for now I believe with a nodered flow. not sure if the debounce requirement is my fault or an NR error. It pops triplets message after the first message. As in first inject posts 1 message, but subsequent Injects post triple.

##Join payloads and insert function node.

var TimeStamp = msg.payload.TimeStamp
var EnergyUsed = msg.payload.EnergyUsed
var PowerUsed = msg.payload.PowerUsed
msg.topic = "INSERT INTO Consumption(`TimeStamp`,`EnergyUsed`, `PowerUsed`) VALUES ('"+TimeStamp+"','"+EnergyUsed+"','"+PowerUsed+"')";


return msg;

flow for future beings

[{"id":"cdb5bad0b80fb740","type":"tab","label":"SBFspot insert mysql","disabled":false,"info":"","env":[]},{"id":"55a241febe3a5755","type":"mysql","z":"cdb5bad0b80fb740","mydb":"df28645b3bc41a9c","name":"insert SBFspot","x":1020,"y":200,"wires":[["bc21762ed999437d"]]},{"id":"5507e1a717da4c80","type":"inject","z":"cdb5bad0b80fb740","name":"TS Trigger","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"300","crontab":"","once":true,"onceDelay":"100","topic":"TimeStamp","payload":"","payloadType":"date","x":150,"y":140,"wires":[["51ea2a50706d1f4e","146ede6a00449568","cbb9399c767b3de3"]]},{"id":"146ede6a00449568","type":"api-current-state","z":"cdb5bad0b80fb740","name":"live consumption","server":"df1a2a65.adcab8","version":3,"outputs":1,"halt_if":"","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.powerpal_live_consumption","state_type":"str","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"topic","propertyType":"msg","value":"PowerUsed","valueType":"str"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":430,"y":100,"wires":[["e3fb4e247c0895a7"]]},{"id":"c3ebfe900c57e242","type":"debug","z":"cdb5bad0b80fb740","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":910,"y":40,"wires":[]},{"id":"51ea2a50706d1f4e","type":"api-current-state","z":"cdb5bad0b80fb740","name":"total consumption","server":"df1a2a65.adcab8","version":3,"outputs":1,"halt_if":"","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.powerpal_total_consumption","state_type":"num","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"topic","propertyType":"msg","value":"EnergyUsed","valueType":"str"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":390,"y":40,"wires":[["e3fb4e247c0895a7"]]},{"id":"e3fb4e247c0895a7","type":"function","z":"cdb5bad0b80fb740","name":"Unit convert *1000","func":"msg.payload = Number(msg.payload);\nmsg.payload = (msg.payload)*1000;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":650,"y":40,"wires":[["50c3dd4c2005358b"]]},{"id":"50c3dd4c2005358b","type":"join","z":"cdb5bad0b80fb740","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"3","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":650,"y":160,"wires":[["7e966461178798df"]]},{"id":"7e966461178798df","type":"function","z":"cdb5bad0b80fb740","name":"join payloads and insert","func":"var TimeStamp = msg.payload.TimeStamp\nvar EnergyUsed = msg.payload.EnergyUsed\nvar PowerUsed = msg.payload.PowerUsed\nmsg.topic = \"INSERT INTO Consumption(`TimeStamp`,`EnergyUsed`, `PowerUsed`) VALUES ('\"+TimeStamp+\"','\"+EnergyUsed+\"','\"+PowerUsed+\"')\";\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":810,"y":120,"wires":[["c3ebfe900c57e242","92878e2fc6feb5b1"]]},{"id":"bc21762ed999437d","type":"debug","z":"cdb5bad0b80fb740","name":"response","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1000,"y":280,"wires":[]},{"id":"cbb9399c767b3de3","type":"function","z":"cdb5bad0b80fb740","name":"convert TS milli to unix > /1000","func":"msg.payload = Number(msg.payload);\nmsg.payload = (msg.payload)/1000;\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":410,"y":180,"wires":[["50c3dd4c2005358b"]]},{"id":"92878e2fc6feb5b1","type":"debounce","z":"cdb5bad0b80fb740","time":"2000","name":"","x":800,"y":220,"wires":[["55a241febe3a5755"]]},{"id":"df28645b3bc41a9c","type":"MySQLdatabase","name":"","host":"127.0.0.1","port":"3306","db":"DB-name","tz":"","charset":"UTF8"},{"id":"df1a2a65.adcab8","type":"server","name":"Home Assistant","version":2,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30}]

Hi!

I know this is an older topic, but if someone is still looking for an uncomplicated way to insert data from HA into a MySQL database (internal or external), you might want to check out the custom component I just built:

2 Likes

Hi. Just trying your code.

I’m very new to HA. Struggling with basic functions now.

I managed to download files, edit configuration,yaml, check configuration and restart HA.

so far all ok, but now I have a few questions.

  • Where can I test if the connection to the DB is working?
  • How to trigger the service? Using developer tools/services doesn’t seem to work.
  • are tables in DB generated automatically or do I need to do it manually?

Thx
Br,
kd