Help: Event firing 160,000 times

Configuration
1

I’ve been having stability issues lately, and I’ve found the suspect… but I need some assistance.

For starters, Home Assistant has been crashing every few days. Looking through the logfiles, I see a list of these errors, which are curious:
2023-03-25 19:39:33.321 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.506 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.509 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.513 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.517 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.532 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.809 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.966 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.977 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.978 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.985 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:33.991 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.000 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.089 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.091 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.092 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.179 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.215 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.252 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.265 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2023-03-25 19:39:34.348 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart

So I google. I find this thread, NCP entered failed state. · Issue #179 · zigpy/bellows · GitHub And one comment pops out at me: “check that /config/home-assistant_v2.db is under 300MB”

And yes, that DB has gotten very large lately. It was up around 800 MB. So I pruned it down, got it under 300. System stable for a few days. Interesting. I also add a DB filesize monitor sensor.

Today I come back from dinner and everything is crashed again. So I look at my file and it grew. Back up to 400MB.
image

Well that’s odd. WTH? So I start learning about the Events table.

image
image885×425 24.2 KB

Then group by Context:

image
image715×348 23.5 KB

Then let’s find the call that’s going nuts:
image

Here’s the culpret:

image
image875×339 22.4 KB

Ok, so why the hell is my kids bedtime turn-off-the lights service-call firing 160,000 times?

Here’s the script - it’s pretty simple:

alias: Ellie bedtime
description: ""
trigger:
  - platform: time
    at: "19:30:00"
condition: []
action:
  - service: scene.turn_on
    data:
      transition: 60
    target:
      entity_id:
        - scene.ellie_bedtime
  - service: cover.close_cover
    data: {}
    target:
      entity_id:
        - cover.ellie_bedroom_curtains
  - service: cover.close_cover
    data: {}
    target:
      entity_id: cover.ellie_blinds
  - delay:
      hours: 0
      minutes: 0
      seconds: 10
      milliseconds: 0
  - service: homeassistant.update_entity
    data: {}
    target:
      entity_id: cover.ellie_blinds
mode: single

The nuts service call is a part of a scene, it’s a WLED light strip.

- id: '1649851686816'
  name: Ellie - Bedtime
  entities:
    switch.ellie_heart:
      icon: mdi:heart-circle
      friendly_name: Ellie heart
      state: 'off'
    light.framelight:
      effect_list:
      - Android
      - Aurora
      - Blends
      - Blink
      - Blink Rainbow
      - Bouncing Balls
      - Bpm
      - Breathe
      - Candle
      - Candle Multi
      - Candy Cane
      - Chase
      - Chase 2
      - Chase 3
      - Chase Flash
      - Chase Flash Rnd
      - Chase Rainbow
      - Chase Random
      - Chunchun
      - Colorful
      - Colorloop
      - Colortwinkles
      - Colorwaves
      - Dancing Shadows
      - Dissolve
      - Dissolve Rnd
      - Drip
      - Dynamic
      - Dynamic Smooth
      - Fade
      - Fairy
      - Fairytwinkle
      - Fill Noise
      - Fire 2012
      - Fire Flicker
      - Fireworks
      - Fireworks 1D
      - Fireworks Starburst
      - Flow
      - Glitter
      - Gradient
      - Halloween
      - Halloween Eyes
      - Heartbeat
      - ICU
      - Juggle
      - Lake
      - Lighthouse
      - Lightning
      - Loading
      - Meteor
      - Meteor Smooth
      - Multi Comet
      - Noise 1
      - Noise 2
      - Noise 3
      - Noise 4
      - Noise Pal
      - Oscillate
      - Pacifica
      - Palette
      - Percent
      - Phased
      - Phased Noise
      - Plasma
      - Police
      - Popcorn
      - Pride 2015
      - Railway
      - Rain
      - Rainbow
      - Rainbow Runner
      - Random Colors
      - Ripple
      - Ripple Rainbow
      - Running
      - Running Dual
      - Saw
      - Scan
      - Scan Dual
      - Scanner
      - Scanner Dual
      - Sine
      - Sinelon
      - Sinelon Dual
      - Sinelon Rainbow
      - Solid
      - Solid Glitter
      - Solid Pattern
      - Solid Pattern Tri
      - Sparkle
      - Sparkle Dark
      - Sparkle+
      - Spots
      - Spots Fade
      - Stream
      - Stream 2
      - Strobe
      - Strobe Mega
      - Strobe Rainbow
      - Sunrise
      - Sweep
      - Sweep Random
      - TV Simulator
      - Tetrix
      - Theater
      - Theater Rainbow
      - Traffic Light
      - Tri Fade
      - Tri Wipe
      - Twinkle
      - Twinklecat
      - Twinklefox
      - Twinkleup
      - Two Dots
      - Washing Machine
      - Wipe
      - Wipe Random
      supported_color_modes:
      - rgb
      color_mode: rgb
      brightness: 3
      hs_color:
      - 0
      - 100
      rgb_color:
      - 255
      - 0
      - 0
      xy_color:
      - 0.701
      - 0.299
      effect: Twinklefox
      icon: mdi:white-balance-iridescent
      friendly_name: FrameLight
      supported_features: 36
      state: 'on'
    light.ellie_bed_lights:
      effect_list:
      - Android
      - Aurora
      - Blends
      - Blink
      - Blink Rainbow
      - Bouncing Balls
      - Bpm
      - Breathe
      - Candle
      - Candle Multi
      - Candy Cane
      - Chase
      - Chase 2
      - Chase 3
      - Chase Flash
      - Chase Flash Rnd
      - Chase Rainbow
      - Chase Random
      - Chunchun
      - Colorful
      - Colorloop
      - Colortwinkles
      - Colorwaves
      - Dancing Shadows
      - Dissolve
      - Dissolve Rnd
      - Drip
      - Dynamic
      - Dynamic Smooth
      - Fade
      - Fairy
      - Fairytwinkle
      - Fill Noise
      - Fire 2012
      - Fire Flicker
      - Fireworks
      - Fireworks 1D
      - Fireworks Starburst
      - Flow
      - Glitter
      - Gradient
      - Halloween
      - Halloween Eyes
      - Heartbeat
      - ICU
      - Juggle
      - Lake
      - Lighthouse
      - Lightning
      - Loading
      - Meteor
      - Meteor Smooth
      - Multi Comet
      - Noise 1
      - Noise 2
      - Noise 3
      - Noise 4
      - Noise Pal
      - Oscillate
      - Pacifica
      - Palette
      - Percent
      - Phased
      - Phased Noise
      - Plasma
      - Police
      - Popcorn
      - Pride 2015
      - Railway
      - Rain
      - Rainbow
      - Rainbow Runner
      - Random Colors
      - Ripple
      - Ripple Rainbow
      - Running
      - Running Dual
      - Saw
      - Scan
      - Scan Dual
      - Scanner
      - Scanner Dual
      - Sine
      - Sinelon
      - Sinelon Dual
      - Sinelon Rainbow
      - Solid
      - Solid Glitter
      - Solid Pattern
      - Solid Pattern Tri
      - Sparkle
      - Sparkle Dark
      - Sparkle+
      - Spots
      - Spots Fade
      - Stream
      - Stream 2
      - Strobe
      - Strobe Mega
      - Strobe Rainbow
      - Sunrise
      - Sweep
      - Sweep Random
      - TV Simulator
      - Tetrix
      - Theater
      - Theater Rainbow
      - Traffic Light
      - Tri Fade
      - Tri Wipe
      - Twinkle
      - Twinklecat
      - Twinklefox
      - Twinkleup
      - Two Dots
      - Washing Machine
      - Wipe
      - Wipe Random
      supported_color_modes:
      - onoff
      - rgb
      color_mode: rgb
      brightness: 3
      hs_color:
      - 0
      - 100
      rgb_color:
      - 255
      - 0
      - 0
      xy_color:
      - 0.701
      - 0.299
      effect: Twinklefox
      entity_id:
      - light.ellie_bed_lights
      - light.ellie_lower_bed_lights
      - light.ellie_desk_light
      icon: mdi:lightbulb-group
      friendly_name: Ellie Bed Lights
      supported_features: 36
      state: 'on'
    light.ellie_lower_bed_lights:
      effect_list:
      - Android
      - Aurora
      - Blends
      - Blink
      - Blink Rainbow
      - Bouncing Balls
      - Bpm
      - Breathe
      - Candle
      - Candle Multi
      - Candy Cane
      - Chase
      - Chase 2
      - Chase 3
      - Chase Flash
      - Chase Flash Rnd
      - Chase Rainbow
      - Chase Random
      - Chunchun
      - Colorful
      - Colorloop
      - Colortwinkles
      - Colorwaves
      - Dancing Shadows
      - Dissolve
      - Dissolve Rnd
      - Drip
      - Dynamic
      - Dynamic Smooth
      - Fade
      - Fairy
      - Fairytwinkle
      - Fill Noise
      - Fire 2012
      - Fire Flicker
      - Fireworks
      - Fireworks 1D
      - Fireworks Starburst
      - Flow
      - Glitter
      - Gradient
      - Halloween
      - Halloween Eyes
      - Heartbeat
      - ICU
      - Juggle
      - Lake
      - Lighthouse
      - Lightning
      - Loading
      - Meteor
      - Meteor Smooth
      - Multi Comet
      - Noise 1
      - Noise 2
      - Noise 3
      - Noise 4
      - Noise Pal
      - Oscillate
      - Pacifica
      - Palette
      - Percent
      - Phased
      - Phased Noise
      - Plasma
      - Police
      - Popcorn
      - Pride 2015
      - Railway
      - Rain
      - Rainbow
      - Rainbow Runner
      - Random Colors
      - Ripple
      - Ripple Rainbow
      - Running
      - Running Dual
      - Saw
      - Scan
      - Scan Dual
      - Scanner
      - Scanner Dual
      - Sine
      - Sinelon
      - Sinelon Dual
      - Sinelon Rainbow
      - Solid
      - Solid Glitter
      - Solid Pattern
      - Solid Pattern Tri
      - Sparkle
      - Sparkle Dark
      - Sparkle+
      - Spots
      - Spots Fade
      - Stream
      - Stream 2
      - Strobe
      - Strobe Mega
      - Strobe Rainbow
      - Sunrise
      - Sweep
      - Sweep Random
      - TV Simulator
      - Tetrix
      - Theater
      - Theater Rainbow
      - Traffic Light
      - Tri Fade
      - Tri Wipe
      - Twinkle
      - Twinklecat
      - Twinklefox
      - Twinkleup
      - Two Dots
      - Washing Machine
      - Wipe
      - Wipe Random
      supported_color_modes:
      - rgb
      icon: mdi:led-strip-variant
      friendly_name: Ellie  Lower Bed Lights
      supported_features: 36
      state: unavailable
    light.ellie_bedroom_main_lights:
      supported_color_modes:
      - brightness
      color_mode: brightness
      brightness: 2
      device_id: '23'
      zone_id: '17'
      icon: mdi:dome-light
      friendly_name: Ellie Bedroom Main Lights
      supported_features: 32
      state: 'on'
  metadata:
    light.framelight:
      entity_only: true
    light.ellie_bed_lights:
      entity_only: true
    light.ellie_lower_bed_lights:
      entity_only: true
    light.ellie_bedroom_main_lights:
      entity_only: true

It’s odd that putting WLED in the scene is repeating a bunch of data.

But WTH is this firing 160,000 times?!?

2

Looks ok to me. Maybe you could throw in a counter and call counter.increment each time it is run to see if it’s actually your automation screwing up or something else odd going on. It looks more like it’s retries in backend code rather than anything you have direct control over. If HA is sending thousands of requests it’s perhaps no suprise “bellows” in unable to cope.

I think the comment about being <300MB is a red herring. Databases in excess of 1GB are pretty normal. The issue you linked seems to be referring to one instance of the error rather than the many you’re getting. Since the automation runs at 19:30 and the logs show 19:39 it looks particularly troubled. It might be worthwhile taking individual zigbee entities out of the scene to see if you can narrow it down. Also, what zigbee integration are you using and what hardware coordinator(s)?

And scenes work that way - they take a complete snaphot of each entity’s state including attributes and you have three of the same type, so many of their attributes will be the same.

3

It’s ZHA, with a SiliconLabs Hubz via USB. Overall hardware is a Pi4. There is no zigbee devices in that scene.

Agree the automation isn’t repeating, there’s only one automation_triggered entry in events. It’s just the service_call repeating.

4

I logged a bug with WLED, which they promptly closed as it’s a HomeAssistant issue.

So I’ve logged a HomeAssistant bug: Activating a scene that includes WLED with multiple segments causes unrecoverable looping of call_service · Issue #90304 · home-assistant/core · GitHub

5

WLED isn’t very responsive with any bugs for their JSON api, which cannot handle a large amount of asynchronous calls. It’s most definitely an issue with how that API is written in WLED. You can probably avoid this issue by switching to MQTT. The developers over at WLED blame HA for all the problems, however when I simply make calls to their JSON api without HA, the same problems arise. It’s 100% a WLED issue and they are unwilling to fix it.

6

There’s nothing about my scripts which would cause “a large number of asynchronous calls”. Activating one scene which 1. Turns them on, 2. sets a color, and 3. activates an effect. And HomeAssistant is falling into an endless loop that consumes all resources until it brings down the environment.

I logged a defect that’s assigned to frenk, so we’ll see what happens. Bugs in core supported API shouldn’t bring down HomeAssistant.

7

It looks like you created that scene in the UI as it has stored a large amount of redundant information. While you wait for a fix you could try creating the scene in YAML instead and only specifying the states you need. It would be interesting to see if that makes a difference.

- id: '1649851686816'
  name: Ellie - Bedtime
  entities:
    switch.ellie_heart:
      state: 'off'
    light.framelight:
      brightness: 3
      rgb_color:
      - 255
      - 0
      - 0
      effect: Twinklefox
      state: 'on'
    light.ellie_bed_lights:
      brightness: 3
      rgb_color:
      - 255
      - 0
      - 0
      effect: Twinklefox
      state: 'on'
    light.ellie_lower_bed_lights:
      state: unavailable
    light.ellie_bedroom_main_lights:
      brightness: 2
      state: 'on'

Also you might want to check the state you have specified for light.ellie_lower_bed_lights.

8

A single service call from home assistant does not mean that there’s a single JSON call to WLED.

9

I can accept that. However, still something isn’t right, HomeAssistant shouldn’t be writing 700MB to the events table because I tried to turn on a scene. (if you’re interested, I’d draw your attention to the screenshots of my Events Table in my first post).

In the meantime I’ve moved on to using a WLED preset instead of the scene, that does seem to be working properly.

10

I never said it wasn’t. I’m talking about you and that particular dev of WLED dismissing that WLED could be the problem. Home assistant frequently puts WLED devices into boot loops because WLED’s JSON api can’t handle it. When that happens, odd things occur across the board. That particular dev that closed your issue has even admitted that the JSON api cannot handle a high volume of calls.