Database purge/repack problem

Running HA latest OS on RPI4, database dates back to several years of long term statistic data (yeah I know should use some other solution for longterm data).
Now every purge/repack ends with error (but database works OK by itself):

Logija: homeassistant.components.recorder.core
Allikas: components/recorder/core.py:887
sidumine: Recorder (dokumentatsioon, probleemi)
Esmalt esines: 09:39:16 (2 esinemist)
Viimati logitud: 10:11:08

Error while processing event PurgeTask(purge_before=datetime.datetime(2025, 4, 11, 6, 38, 54, 994200, tzinfo=datetime.timezone.utc), repack=True, apply_filter=True)
Error while processing event PurgeEntitiesTask(entity_filter=<functools._lru_cache_wrapper object at 0x7f320d9430>, purge_before=datetime.datetime(2025, 4, 11, 7, 11, 8, 256072, tzinfo=datetime.timezone.utc))
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 887, in _guarded_process_one_task_or_event_or_recover
    self._process_one_task_or_event_or_recover(task)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 907, in _process_one_task_or_event_or_recover
    task.run(self)
    ~~~~~~~~^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 120, in run
    if purge.purge_old_data(
       ~~~~~~~~~~~~~~~~~~~~^
        instance, self.purge_before, self.repack, self.apply_filter
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 642, in wrapper
    return job(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/recorder/purge.py", line 112, in purge_old_data
    if apply_filter and not _purge_filtered_data(instance, session):
                            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/recorder/purge.py", line 596, in _purge_filtered_data
    if entity_filter and not entity_filter(entity_id)
                             ~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entityfilter.py", line 232, in entity_not_excluded
    or split_entity_id(entity_id)[0] in exclude_d
       ~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 209, in split_entity_id
    raise ValueError(f"Invalid entity ID {entity_id}")
ValueError: Invalid entity ID 656362461

How can I identify this bad entity ID, get rid of it and fix database so I can purge etc.?
Searched community guides, Discord group, Google of course :frowning:
Best (and in big trouble), JR

Are you certain it’s not just running out of room? It is likely that is is creating another copy of the database to rebuild it.

Thanks for an idea :slight_smile:
But this is not the case as DB is some GB but free space on SSD is over 80GB.
Downloaded the DB and repaired it with sqlite3 tools. Uploaded and now another problem arose:

Logija: homeassistant.components.recorder.core
Allikas: components/recorder/core.py:907
sidumine: Recorder (dokumentatsioon, probleemi)
Esmalt esines: 09:25:26 (1 esinemist)
Viimati logitud: 09:25:26

Unhandled database error while processing task PurgeTask(purge_before=datetime.datetime(2025, 4, 9, 6, 25, 22, 532233, tzinfo=datetime.timezone.utc), repack=False, apply_filter=False)
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 942, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.IntegrityError: FOREIGN KEY constraint failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/core.py", line 907, in _process_one_task_or_event_or_recover
    task.run(self)
    ~~~~~~~~^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/recorder/tasks.py", line 120, in run
    if purge.purge_old_data(
       ~~~~~~~~~~~~~~~~~~~~^
        instance, self.purge_before, self.repack, self.apply_filter
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 642, in wrapper
    return job(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/recorder/purge.py", line 121, in purge_old_data
    _purge_old_entity_ids(instance, session)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/recorder/purge.py", line 570, in _purge_old_entity_ids
    deleted_rows = session.execute(delete_states_meta_rows(states_metadata_ids))
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2365, in execute
    return self._execute_internal(
           ~~~~~~~~~~~~~~~~~~~~~~^
        statement,
        ^^^^^^^^^^
    ...<4 lines>...
        _add_event=_add_event,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<4 lines>...
        conn,
        ^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/bulk_persistence.py", line 2021, in orm_execute_statement
    return super().orm_execute_statement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        session, statement, params, execution_options, bind_arguments, conn
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
        statement, params or {}, execution_options=execution_options
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
    return meth(
        self,
        distilled_parameters,
        execution_options or NO_OPTIONS,
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/sql/lambdas.py", line 597, in _execute_on_connection
    return connection._execute_clauseelement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self, distilled_params, execution_options
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
    ret = self._execute_context(
        dialect,
    ...<8 lines>...
        cache_hit=cache_hit,
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1843, in _execute_context
    return self._exec_single_context(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        dialect, context, statement, parameters
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
    self._handle_dbapi_exception(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        e, str_statement, effective_parameters, cursor, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2352, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 942, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
[SQL: DELETE FROM states_meta WHERE states_meta.metadata_id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (43046, 42029, 43047, 2054, 2055, 2056, 2057, 2058, 2059, 2060, 2061, 2062, 2063, 2064, 2065, 2066, 2067, 2068, 2069, 2070, 2071, 2072, 2073, 2074, 27, 2075, 2076, 2077, 2078, 2079, 2080, 2081, 2082, 2083, 2084, 2085, 39, 2086, 2087, 2088, 2089, 2090, 2091, 2092, 2093, 2094, 2095, 2096, 2097, 2098 ... 34 parameters truncated ... 215, 43315, 43317, 43327, 43358, 2912, 43398, 43399, 396, 43405, 40847, 40848, 43415, 43416, 42394, 42395, 42396, 42397, 42404, 42407, 42011, 41897, 43433, 43434, 42012, 43439, 944, 43440, 42013, 43441, 43442, 1974, 42014, 42015, 42016, 35271, 42017, 42018, 42447, 42019, 42020, 42021, 42022, 42023, 42024, 33772, 42025, 42026, 42027, 42028)]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

So I guess I backup the DB for further investigation and start with new one :frowning:
Is it possible to merge/concat two databases into one after finding a solution?

Best (and clueless) JR