2023.2: How can I Assist?

MichaelR · February 11, 2023, 11:12am

Unfortunately, I went back to version 2023.1.7 because the Modbus module was constantly delivering errors:

Modbus Error: [Input/Output] Modbus Error: [Invalid Message] No response received, expected at least 8 bytes (0 received)

Sensors were suddenly not available, reported back after a few seconds, but other sensors were not available.

No changes were made in my modbus.yaml; After restoring to the old version, everything is stable again.

I think there is a problem with pymodbus here.

Best regards
Michael

nickrout · February 11, 2023, 7:39pm

Github issue time then

ThePapaMaan · February 11, 2023, 9:02pm

Agree. Now I’m not an average user (i use mariadb) but what we can learn from Steve’s question that sometimes there is some ambiguity that could be prevented with a short clarification such as “does not affect default installation” or similar.

stevemann · February 12, 2023, 3:01am

It wasn’t a question. Just an observation that the average user does not know what database they are using which could lead to confusion.

agathon55 · February 12, 2023, 4:23am

Has anyone had any issues with Smart IR after updating to 2023.2.3? I use it with my Broadlink RM3 Pro to control both my AC and tower fan. After updating, I can no longer control anything via my Broadlink. Rolled back to 1.7 and the issue goes away.

vajdum · February 12, 2023, 1:40pm

I can’t really think of any uses for anything but the “Occupancy” sensor but maybe someone more creative than me can think of something with the energy/distances!

I think this are the raw values used if you config sensitivity using energie/distance (gate)
I can imagine to feed them into an AI to find patterns and deviations of usual routines to act in such cases, fe. in elderly care (fall detection and such things).

mirekmal · February 12, 2023, 3:19pm

Finally got some time and followed your advise. So I installed 10.9.5 in docker and used HeidiSQL to migrate data. Since docker version is running on native to MariaDB port 3306 it was even possible to copy data directly between databases, without intermediate step of creating SQL dump. Some observations:

it is possible to create data folder outside of container, so this gives full controll over placement and allows to directly observe size of database.
initially I placed database on spinning drives (10 x WD Reds on SHR2) and performance of DB was terrible! Filling in history view for ~50 entities for 1 day took about 20 seconds (from ~2s previously) and displaying chart for 1 month of history took over 3 minutes (30 seconds previously). After that I moved all data files (was easy due to data being exposed outside docker) to 2 disks RAID 1 volume and timings are now maybe 50% slower than originally (1 day history ~3 seconds, 1 month history ~45 seconds). This shows how big influence is on IOPS of underlying disks. Small performance hit might be due to version of MariaDB, docker installation, keeping data files outside of container or any combination of these. But this decrease is acceptable.
data migration itself took ~1 hour for ~18M rows (~4GB database size). Probably if I would do migration not to spinnig disks but to SSD instantly, it would be way faster.
and one more side effect (can’t really explain); data write rate to disk reduced from ~6mbps I had previously to ~800kbps (measured on volume level). So additional benefit of extending life expectancy of SSD

stevemann · February 12, 2023, 5:41pm

I have two here. Works fine on 2023.2.3
Have you rebooted the Broadlinks?
Can you ping them?

bdraco · February 12, 2023, 5:46pm

It’s probably this change Increase default recorder commit interval to 5 seconds by bdraco · Pull Request #86115 · home-assistant/core · GitHub

You could probably adjust it higher but it’s diminishing returns past 5s

Expect it to go down a bit more in 2023.3.x anyways since statistics will be more efficient

giovanne · February 12, 2023, 8:39pm

Hi mirek,
what is your Hardware and OS for using Mariadb 10.9.5 in Docker? (RPi…?)

mirekmal · February 12, 2023, 9:18pm

I’m running it on Synology NAS RS2418+ (Intel Atom 2.1 GHz, 32 GB RAM) and DSM 7.1.1… so pretty decent HW, but I use it for whole my home lab (some iSCSI LUNs for home computers, backups, datastores for VMware ESXI server, media servers, Roon, AdGuard, some web pages…

daywalker03 · February 14, 2023, 7:48pm

I see it on my Home Assistant Yellow running 2023.2.4.

ljmerza · February 15, 2023, 3:44pm

started to get db errors thats crashing my instance. something about "Incorrect string value: '\\xF0\\x93\\x86\\x9A\\xF0\\x93...' for column hass.statistics_meta.name` - my db is set to utf16

2023-02-15T15:29:58.863656782Z 2023-02-15 10:29:58.862 ERROR (Recorder) [homeassistant.components.recorder.util] Error executing query: (pymysql.err.DataError) (1366, "Incorrect string value: '\\xF0\\x93\\x86\\x9A\\xF0\\x93...' for column `hass`.`statistics_meta`.`name` at row 1") 2023-02-15T15:29:58.863690852Z [SQL: INSERT INTO statistics_meta (statistic_id, source, unit_of_measurement, has_mean, has_sum, name) VALUES (%(statistic_id)s, %(source)s, %(unit_of_measurement)s, %(has_mean)s, %(has_sum)s, %(name)s)] 2023-02-15T15:29:58.863694247Z [parameters: {'statistic_id': 'recorder.db_test', 'source': 'recorder', 'unit_of_measurement': None, 'has_mean': 1, 'has_sum': 1, 'name': '𓆚𓃗'}] 2023-02-15T15:29:58.863697159Z (Background on this error at: https://sqlalche.me/e/14/9h9h) 2023-02-15T15:29:58.871358343Z 2023-02-15 10:29:58.866 ERROR (Recorder) [homeassistant.components.recorder.statistics] Error when validating DB schema: (pymysql.err.DataError) (1366, "Incorrect string value: '\\xF0\\x93\\x86\\x9A\\xF0\\x93...' for column `hass`.`statistics_meta`.`name` at row 1") 2023-02-15T15:29:58.871385685Z [SQL: INSERT INTO statistics_meta (statistic_id, source, unit_of_measurement, has_mean, has_sum, name) VALUES (%(statistic_id)s, %(source)s, %(unit_of_measurement)s, %(has_mean)s, %(has_sum)s, %(name)s)] 2023-02-15T15:29:58.871389338Z [parameters: {'statistic_id': 'recorder.db_test', 'source': 'recorder', 'unit_of_measurement': None, 'has_mean': 1, 'has_sum': 1, 'name': '𓆚𓃗'}] 2023-02-15T15:29:58.871392168Z (Background on this error at: https://sqlalche.me/e/14/9h9h) 2023-02-15T15:29:58.871394484Z Traceback (most recent call last): 2023-02-15T15:29:58.871396673Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context 2023-02-15T15:29:58.871399005Z self.dialect.do_execute( 2023-02-15T15:29:58.871401139Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute 2023-02-15T15:29:58.871403397Z cursor.execute(statement, parameters) 2023-02-15T15:29:58.871405549Z File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute 2023-02-15T15:29:58.871407765Z result = self._query(query) 2023-02-15T15:29:58.871409870Z File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query 2023-02-15T15:29:58.871412119Z conn.query(q) 2023-02-15T15:29:58.871414224Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query 2023-02-15T15:29:58.871416419Z self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2023-02-15T15:29:58.871418585Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result 2023-02-15T15:29:58.871420818Z result.read() 2023-02-15T15:29:58.871422904Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read 2023-02-15T15:29:58.871425089Z first_packet = self.connection._read_packet() 2023-02-15T15:29:58.871427206Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet 2023-02-15T15:29:58.871429404Z packet.raise_for_error() 2023-02-15T15:29:58.871431511Z File "/usr/local/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error 2023-02-15T15:29:58.871441631Z err.raise_mysql_exception(self._data) 2023-02-15T15:29:58.871443997Z File "/usr/local/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception 2023-02-15T15:29:58.871447287Z raise errorclass(errno, errval) 2023-02-15T15:29:58.871449457Z pymysql.err.DataError: (1366, "Incorrect string value: '\\xF0\\x93\\x86\\x9A\\xF0\\x93...' for column `hass`.`statistics_meta`.`name` at row 1") 2023-02-15T15:29:58.871451917Z 2023-02-15T15:29:58.871454281Z The above exception was the direct cause of the following exception: 2023-02-15T15:29:58.871456486Z 2023-02-15T15:29:58.871458568Z Traceback (most recent call last): 2023-02-15T15:29:58.871460752Z File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 2276, in _validate_db_schema_utf8 2023-02-15T15:29:58.871463098Z _update_or_add_metadata(session, metadata, old_metadata_dict) 2023-02-15T15:29:58.871465270Z File "/usr/src/homeassistant/homeassistant/components/recorder/statistics.py", line 369, in _update_or_add_metadata 2023-02-15T15:29:58.871467564Z session.flush() # Flush to get the metadata id assigned 2023-02-15T15:29:58.871469713Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3444, in flush 2023-02-15T15:29:58.871471920Z self._flush(objects) 2023-02-15T15:29:58.871474086Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3583, in _flush 2023-02-15T15:29:58.871476301Z with util.safe_reraise(): 2023-02-15T15:29:58.871478389Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ 2023-02-15T15:29:58.871480599Z compat.raise_( 2023-02-15T15:29:58.871482719Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ 2023-02-15T15:29:58.871484924Z raise exception 2023-02-15T15:29:58.871487045Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush 2023-02-15T15:29:58.871489242Z flush_context.execute() 2023-02-15T15:29:58.871491357Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute 2023-02-15T15:29:58.871493557Z rec.execute(self) 2023-02-15T15:29:58.871495669Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute 2023-02-15T15:29:58.871497858Z util.preloaded.orm_persistence.save_obj( 2023-02-15T15:29:58.871499965Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj 2023-02-15T15:29:58.871502200Z _emit_insert_statements( 2023-02-15T15:29:58.871504318Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements 2023-02-15T15:29:58.871508720Z result = connection._execute_20( 2023-02-15T15:29:58.871510979Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 2023-02-15T15:29:58.871513211Z return meth(self, args_10style, kwargs_10style, execution_options) 2023-02-15T15:29:58.871515648Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection 2023-02-15T15:29:58.871517912Z return connection._execute_clauseelement( 2023-02-15T15:29:58.871520035Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement 2023-02-15T15:29:58.871522260Z ret = self._execute_context( 2023-02-15T15:29:58.871524391Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context 2023-02-15T15:29:58.871526612Z self._handle_dbapi_exception( 2023-02-15T15:29:58.871528708Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception 2023-02-15T15:29:58.871530963Z util.raise_( 2023-02-15T15:29:58.871533123Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ 2023-02-15T15:29:58.871535340Z raise exception 2023-02-15T15:29:58.871537468Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context 2023-02-15T15:29:58.871539686Z self.dialect.do_execute( 2023-02-15T15:29:58.871541847Z File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute 2023-02-15T15:29:58.871544073Z cursor.execute(statement, parameters) 2023-02-15T15:29:58.871546204Z File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute 2023-02-15T15:29:58.871548404Z result = self._query(query) 2023-02-15T15:29:58.871550502Z File "/usr/local/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query 2023-02-15T15:29:58.871552718Z conn.query(q) 2023-02-15T15:29:58.871554828Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query 2023-02-15T15:29:58.871557042Z self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2023-02-15T15:29:58.871559210Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result 2023-02-15T15:29:58.871561441Z result.read() 2023-02-15T15:29:58.871563557Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read 2023-02-15T15:29:58.871565765Z first_packet = self.connection._read_packet() 2023-02-15T15:29:58.871567876Z File "/usr/local/lib/python3.10/site-packages/pymysql/connections.py", line 725, in _read_packet 2023-02-15T15:29:58.871570086Z packet.raise_for_error() 2023-02-15T15:29:58.871572184Z File "/usr/local/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error 2023-02-15T15:29:58.871576261Z err.raise_mysql_exception(self._data) 2023-02-15T15:29:58.871578450Z File "/usr/local/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception 2023-02-15T15:29:58.871580690Z raise errorclass(errno, errval) 2023-02-15T15:29:58.871583283Z sqlalchemy.exc.DataError: (pymysql.err.DataError) (1366, "Incorrect string value: '\\xF0\\x93\\x86\\x9A\\xF0\\x93...' for column `hass`.`statistics_meta`.`name` at row 1") 2023-02-15T15:29:58.871585753Z [SQL: INSERT INTO statistics_meta (statistic_id, source, unit_of_measurement, has_mean, has_sum, name) VALUES (%(statistic_id)s, %(source)s, %(unit_of_measurement)s, %(has_mean)s, %(has_sum)s, %(name)s)] 2023-02-15T15:29:58.871588118Z [parameters: {'statistic_id': 'recorder.db_test', 'source': 'recorder', 'unit_of_measurement': None, 'has_mean': 1, 'has_sum': 1, 'name': '𓆚𓃗'}] 2023-02-15T15:29:58.871590482Z (Background on this error at: https://sqlalche.me/e/14/9h9h)

boheme61 · February 15, 2023, 4:35pm

What is 'recorder.db_test' ?

Also, the

All seems relates to some kind of disconnects

rypi17 · February 15, 2023, 5:47pm

Has anyone had issues with the energy dashboard not updateing?, all sensors are showing values direct from the sensor but the energy dashboard has been blank for the last few days

KennethLavrsen · February 15, 2023, 8:09pm

Try and go to Developer Tools → Statistics.
Look for thing needing fixing.
A lot of integrations and 3rd party MQTT sensors have had problems and fixes related to units which halts the long term statics. It may be such an issue you have and the fixing may solve the problem

boheme61 · February 15, 2023, 10:08pm

Nice one, haven’t used this , about 20 needed to be fixed

bdraco · February 15, 2023, 10:44pm

Only utf8mb4 is supported. utf16 won’t work.

SLiK420 · February 15, 2023, 11:31pm

I set up a separate VM to do a fresh install of deConz cause testing. I reloaded my backup there and it’s holding now, but barely.

leopster · February 15, 2023, 11:42pm

Ok, good for you, but first of all, you do not represent 100% of HA’s users. There are disabled and elderly people in the world who would really enjoy being able to control devices with their voice and have an assistant like that. You should really try to have empathy and see the world around you instead of assuming it is always easier to reach for a device and click on some buttons to accomplish a task, just because you think so.

I’m not a native english speaker but still prefer to ask google to turn off my home devices instead of finding my phone, opening an app and clicking on a button to do one thing. So, next time you guys stumble upon a feature like this and think it is useless, maybe it’s just not for you