For begin, it will be nice to have auto db drop parameter, for example, every 30 days
1 Like
you mean auto_purge?
If the DB was a proper relational database with all repeated data stored in dedicated tables, with those tables linked to the equivalent of a new โstatesโ table, then:
-
Database size would be reduced by 100x
-
Queries would be 1000 - 10,000 times faster
-
Amount of bytes written would be reduced 100x (big deal for SD cards
-
Amount of processing needed in Python would be reduced considerably because efficient database queries could be written
-
The need to actively manage the database would go away.
As opposed to how it treated now which is basically like a flat log file with entire entity names written as varchars with all attributes in a big varchar field on every state update. It is, and I say this with no malice intended, terrible. The absolute worst possible way to set up a database.
2 Likes
Given the infinite diversity of data sources and their volatile data structure in continuous change I would say that the optimal would be to go for a non-SQL database as MongoDB for instance.
That could absolutely work as well. Just anything but how the DB is used today where it is an inefficient, bloated, ticking time bomb.
Sound like ideal for using non SQL like MongoDB instead
How large is your setup (how many entities) and how much memory would it consume. Furthermore; (how) do you save data to disk for persistancy (dump every x time?)
Itโs been more than two years since I last ran that setup and a lot has changed since then, so I canโt recall the details from that far back.
It is mostly fixed with the new database structure where attributes and other data are now stored as related tables.