Hi all,
I’ve been running this for a few weeks and it seems stable enough to share — but treat it as early beta. There are likely rough edges, and I’d appreciate feedback before calling it production-ready.
A couple of honest caveats upfront: this app was predominantly written with AI assistance (Claude), and development time is very limited. I won’t be monitoring this thread consistently — if you have bug reports, feature requests, or extended feedback, please use GitHub Discussions rather than this thread, as that’s where I’m more likely to see it. Issues and feature requests will be addressed as time permits.
What it does
Trains a LightGBM model on your historical grid-import energy data and local weather, then publishes a 48-hour hourly forecast as native HA sensor entities — updated every hour, retrained weekly.
Following up on the original post — v0.3.0 is out. Same caveats apply: early beta, AI-assisted development, limited support bandwidth. GitHub Discussions for bug reports and feedback.
Honest note on the bug fixes
A few of these are significant enough to call out directly, because they affected accuracy from day one:
Sunshine duration was hardcoded to zero on the Open-Meteo path — anyone without SRG-SSR credentials has been training and predicting without sunshine data. Fixed; Open-Meteo now returns real values.
Forecast timestamps were shifted +1 hour inside _update_sensors, causing the weather merge to find zero matches on every sensor update. All weather features (temperature, sunshine, precipitation) were silently imputed from training-set medians instead of the actual forecast. Fixed.
Rolling features (24h/7d mean, 24h std) were broadcast as a single scalar across all 48 prediction hours rather than sliding per-hour. This created a systematic train/predict mismatch. Fixed.
SRG-SSR API v1 was decommissioned — anyone using SRG credentials would have been falling back to Open-Meteo silently (or crashing under pandas 3.x). Migrated to v2.
If you’ve been running a previous version and your MAE sensor looked suspiciously flat or high, these are likely why.
sensor.energy_forecast_today and today_HH_HH block sensors now blend actuals for elapsed hours — past hours show measured consumption, future hours show the forecast
Model improvements
Log-transform on the training target — reduces the influence of high-consumption outliers
LightGBM early stopping per CV fold — avoids over-fitting on longer histories
lag_72h autoregressive feature (same time 3 days ago)
Cloud cover percentage and direct radiation (W/m²) added as features; available to all users via Open-Meteo
72h of measured Open-Meteo history now anchors the temp_rolling_3d feature at predict time
Bridge-day proximity: days_to_next_holiday / days_since_last_holiday features
likely_ev_hour binary feature — marks hour-of-week slots with a historical EV charging pattern
Cantonal holiday support: set holiday_canton: "ZH" (or any Swiss canton code) in apps.yaml
Operational
Adaptive retraining: if live day-ahead MAE exceeds adaptive_retrain_threshold × cv_MAE (default 2×) with sufficient matched pairs, an early retrain is triggered automatically
Upgrading
No schema changes. Drop in the new files and restart AppDaemon. Existing pickle files will be used as-is until the next scheduled retrain; the log-transform and new features activate automatically after that.
SRG OAuth token cached for 55 min — reduces API calls ~24×/day → ~1/day (M1)
Missing cloud/radiation keys fall back to NaN instead of 0 (M2)
Holiday vectorisation via np.searchsorted (#32)
Sensor UX (v0.5.2)
MDI icons on all published sensors (mdi:lightning-bolt, mdi:car-electric, etc.)
unique_id attribute on every sensor (groundwork for future MQTT Discovery)
Bugfix: pandas 3.x ValueError on date-only midnight entries in energy_history.csv — without this fix all lag/rolling features silently degraded to medians for the rest of the affected day
Upgrade notes
No breaking changes. All new config keys are optional with safe defaults.
Existing meta.pkl / energy_model.pkl files are backward-compatible.
After restart, AppDaemon will publish the setup status sensor on first initialise.
404 DELETE spam on startup: _cleanup_legacy_states now guards every remove_entity call with entity_exists, eliminating ~30 [404] HTTP DELETE: Not Found log errors on fresh installs where legacy entities were never created (fixes #47).
Anomaly binary sensor attributes missing in MQTT mode: _publish now routes the four anomaly attributes (residual_kwh, residual_std_kwh, sigma_threshold, n_pairs) to a dedicated binary_sensor/.../attributes MQTT topic. Discovery payload for energy_forecast_unusual_consumption now includes json_attributes_topic. State topic path corrected from sensor/ to binary_sensor/ (fixes #48).
Added
Dashboard card — anomaly detection (dashboard/anomaly-detection.yaml): vertical-stack with mushroom state card + conditional attribute detail that expands when the sensor is ON.
Dashboard card — SHAP feature importance (dashboard/shap-importance.yaml): native Lovelace markdown card using a Jinja2 template — no custom card dependency.
dashboard/dashboard.yaml updated: anomaly mushroom card inserted after the MAE mini-graph card.
v0.7.0 — 2026-03-23
Added
Quantile interval calibration (CQR): prediction intervals (10th–90th percentile) are calibrated via split conformal prediction. Gives ≥ 80% marginal coverage on held-out data. q_hat correction is persisted to energy_model_interval_correction.json.
SHAP feature importance (#42): top-N driving features exposed as shap_top_features attribute on sensor.energy_forecast_today. LightGBM uses native TreeSHAP; sklearn GBR falls back to global feature_importances_. New config key: shap_top_n (default 5; set to 0 to disable).
Anomaly detection sensor (#39): binary_sensor.energy_forecast_unusual_consumption fires when actual consumption deviates more than anomaly_sigma_threshold σ from the day-ahead prediction. Cold-start state is off until 10 matched pairs accumulate. Attributes: residual_kwh, residual_std_kwh, sigma_threshold, n_pairs.
Rolling MAE sensors (#41): sensor.energy_forecast_mae_7d and sensor.energy_forecast_mae_30d track live forecast accuracy using stored prediction-vs-actual pairs. Published in both set_state and MQTT Discovery modes.
Vacation / away flag (#25): new is_away binary feature lets the model learn reduced consumption during vacations. Optional config keys: away_mode_entity, away_return_entity. Fully backward-compatible.
ApexCharts dashboard (dashboard/energy-today.yaml, dashboard/dashboard.yaml): copy-paste Lovelace YAML showing 48-hour forecast vs actuals with prediction-interval shading.
Fixed
_load_interval_correction stale-value bug: _interval_correction is reset to 0.0 before parsing the JSON file, preventing a stale value from persisting across corrupt-file events.
SHAP summary early-day fallback: shap_summary now falls back to all 48 prediction rows when fewer than 3 rows match today’s date slice.
Upgrade notes
No breaking changes. Existing apps.yaml and model pickles continue to work.
New optional config keys: shap_top_n, anomaly_sigma_threshold, away_mode_entity, away_return_entity.
MQTT users with the anomaly sensor: json_attributes_topic is now included in the discovery payload — HA will pick up attributes automatically on the next restart.
Dashboard YAMLs are in dashboard/ — copy the cards you want into your Lovelace config.
Test suite
243 tests, all passing (python -m pytest tests/ -v).
v0.6.0 — MQTT Discovery + next_1h sensor
Added
MQTT Discovery (#37) (energy_forecast.py): opt-in entity registration via MQTT Discovery. Set mqtt_discovery: true in apps.yaml to register all ~29 sensors in the HA entity registry, enabling area assignment and labels. Requires the AppDaemon MQTT plugin and a running MQTT broker. Config keys: mqtt_discovery (default false), mqtt_namespace (default mqtt), mqtt_discovery_prefix (default homeassistant). All sensors grouped under a single HA Energy Forecast device. Prediction interval sensors (*_low/*_high) are registered lazily on the first update cycle where quantile models exist. Availability topic publishes "online" at startup and "offline" on AppDaemon shutdown. Existing set_state() behaviour is unchanged when mqtt_discovery: false.
sensor.energy_forecast_next_1h: 1-hour-ahead point forecast sensor.
README: added MQTT Discovery section (prerequisites, appdaemon.yaml snippet, apps.yaml example, sensor count table, availability behaviour, revert instructions); added mqtt_discovery / mqtt_namespace / mqtt_discovery_prefix to parameter reference.
Fixed
Doubled “Energy Forecast” prefix in MQTT Discovery sensor names: Discovery name values are now short labels ("Model MAE", "Today", "Setup Status", etc.); set_state() paths are unchanged.
Doubled sensors after enabling MQTT Discovery: _cleanup_legacy_states() removes ghost set_state entities on startup when mqtt_discovery=True, without requiring an HA restart.
MQTT publish broken on HASS apps: replaced self.mqtt_publish() with self.call_service("mqtt/publish", ...), which works from any AppDaemon HASS app.
numpy 2.x retraining error: df["gross_kwh"].to_numpy(dtype=float) forces float64 before np.log1p, fixing "loop of ufunc does not support argument 0 of type float" on numpy 2.x.
Align hourly sensor updates to XX:01:00 wall-clock time using run_hourly; eliminates startup-time drift.
Downgrade prediction-time sub-sensor NaN log from WARNING to DEBUG; training-time WARNING (weekly) is sufficient.
Your math may benefit a an integration i been working on, alternative energy monitoring and power consumption protection/enforcement. Been running it for few months working out the bugs needed just what you made among a few other things before i start releasing this
Great to hear that you can use part of the work. Your UI screenshots look interesting and it seems like you already have a good data foundation for the predictions. Let me know how it works once it’s running!
Actually, this is also part of a bigger project on my end. I’m working on a custom energy management system for solar energy usage optimization.
And I will definitely have a read through your summary.
Strictly because I see your using mqtt to communicate from Addon/App to ha and had duplicate sensor trouble tho you fixed it. I was going down same path when I was eager to get away from opwer integration and decided to use same underlying api library from the GitHub gist to make an Addon/App that does why power does + allot more using advance scrapping, python logic, and make ha service calls automatically when it detects an event on my Con Edison account.
One of those special things I wanted wanted the app to do were to send actionable notification which is easy until you want it to be able to know what the user selected from the notification choices (mqtt can’t do this and using ha api alone cannot either, as an Addon/App has no access to ha event loop ) so my solution is always to break apart other well known apps and integration and the solution was in plain sight. You can ditch mqtt altogether and make your App/Addon install an integration so apart of your project runs inside ha full and your Addon does the other stuff which an integration running in ha cannot
Soon I’ll release them so a more detailed post but I just feel overall we’ve been holding back on what intergration/apps can be rather than just a data display I see and build them as an integral part of the home automation system with built in logic to run its own automation and make service calls and being interactive with the user via things like embedded tts or actionable notifications
The duplicate sensors were coming from the migration of direct appdaemon publication to MQTT. The deletion logic simply removes the sensors originating from the previous implementation.
From what I’ve seen so far, I’m pretty happy with the decision to migrate to MQTT. Sensors and controls show up grouped under one device and are manageable through the HA UI.
In another module I’m working on, I’m now also able to send commands back over the MQTT integrated device. However, I haven’t tried actionable notifications yet.
service calls will work via the api but actionable notification wont as you will be only able to send the initial notification but not receive the response regardless keep me updated
SRG-SSR API quota over-consumption (weather.py): geolocation lookups now cached at module scope, reducing daily API quota from ~48 calls to ~24 calls. Fixes repeated 429 rate-limit errors and aligns with Freemium tier cap (50 calls/day).
Tests
test_srg_geolocation_caching: verifies cached geolocation lookups prevent duplicate API requests.
[0.8.0] — 2026-03-31
Fixed
Temperature sensor blending bias-fade (model.py): 6-hour outdoor sensor blend now uses bias-fade semantics instead of linear interpolation, preserving forecast trajectory while smoothly fading sensor offset.
Added
Model versioning: automatic archive/rollback of model snapshots on weekly retrain to models/archive/<timestamp>/. Configurable via model_archive_count (default 3). Roll back via HA event energy_forecast_rollback_model.
CSV health checks: validate_energy_cache() validates timestamps, gaps, and value ranges with detailed logging.
Solar PV + battery target correction: optional config keys for household consumption correction from solar/battery sensors.
Tests
30+ new tests for temperature blending, model versioning, cache validation, and solar/battery correction logic.
v0.10.0 transforms the forecaster from a weather-correlated statistical model into a physics-aware, scenario-capable energy planning engine. Four new stages add a passive-house baseline mode, intent-driven thermal and DHW modelling from climate entity setpoints, automated appliance load signature discovery, and a what-if scenario API — so the Energy Manager can ask “what happens if the dishwasher runs at 14:00?” and get a delta-annotated 48-hour forecast in a single service call. A physics feature pack rounds out the release with solar-compensated thermal pressure, wind-driven infiltration load, heat-pump defrost risk, and building thermal time-constant calibration — expressed in physical units for climate-agnostic generalisation. 474 tests; confirmed stable on Home Assistant 2026-04-17.
Added
Stage 1: Passive House Baseline mode — new baseline_mode config flag (default false). When enabled, all controllable sub_energy_sensors are subtracted from the training target, keeping appliance noise out of the baseline model so predict_scenario() deltas are meaningful. Also wires presence_sensors config end-to-end into the hourly update and retrain cycle.
Stage 2: Intent-Driven Thermal & DHW Modeling — fetch_climate_history() reads setpoint and current temperature from HA climate entities; fetch_generic_sensor_history() reads a DHW buffer sensor. Two new model features: thermal_pressure (mean setpoint − current-temp across configured rooms) and dhw_pressure (heat-loss urgency score). Config keys: climate_entities, dhw_buffer_sensor, heating_system_active_entity.
Stage 3: Automated Load Signature Discovery — _learn_appliance_signatures() scans sub-sensor histories for run cycles, computes per-appliance average hourly energy profiles, and persists them to models/appliance_signatures.json. Supports adaptive cycle windows, demand-surge detection for always-on devices, outlier rejection, duration clustering (short/long), CoV-based reliability labels, and program-type sensor grouping (per-program profiles stored under sig["programs"][<label>]).
Stage 4: Scenario Modeling & What-If API — _composite_forecast() overlays learned appliance profiles onto a 48-hour baseline. predict_scenario() returns [timestamp, predicted_kwh, delta_kwh]. AppDaemon service energy_forecast/get_scenario fires energy_forecast_scenario_result with the full composite; optional publish=True writes sensor.energy_forecast_scenario_today/tomorrow/delta_today and 8 block sensors.
humidity — fetched from Open-Meteo relativehumidity_2m; defaults to 70 % when absent (#55).
thermal_pressure_net — thermal pressure reduced by weighted_solar_gain; captures solar gain offsetting heat deficit before the heat pump acts (#56).
infiltration_pressure — wind × thermal gradient interaction term; infiltration load driven by both wind speed and indoor–outdoor delta-T (#57).
defrost_risk — humidity-scaled Gaussian centred at +2 °C; proxy for heat-pump defrost cycles that spike power draw (#58).
SHAP labels added for all four physics features.
Thermal time-constant calibration (τ) — _calibrate_tau() fits log-linear OLS on passive-cooling windows (confirmed heating-off periods) to estimate the building time constant. Safeguards: daytime exclusion (09:00–15:00), solar radiation mask (>150 W/m²), EMA smoothing when estimate changes >50 %. τ persisted in meta.pkl; skipped gracefully when heating_system_active_entity is not configured.
RC-ODE indoor temperature projection — _project_indoor_temps() integrates the RC heat-balance ODE (Euler forward) to project indoor temperatures for all 48 forecast hours, eliminating the zero-fill problem where thermal_pressure defaulted to 0 beyond hour 2.
Area-weighted thermal pressure — rooms weighted by floor area via climate_room_areas config (m² per entity); defaults to 15 m². Secondary features: thermal_pressure_max, thermal_pressure_std. thermal_pressure_cop divides heat debt by COP estimate to express urgency in electrical terms. weighted_solar_gain scales direct radiation by a half-cosine window (09:00–17:00, peak 13:00).
Program-type sensor support — sub-sensor CSVs persist a program column (LVFC label). _resolve_programs_for_series() includes a 1-hour forward-lookup pass for late-firing sensors. Forward-lookup tolerance widened to 2 hours. program_type_sensor config key per sub-sensor.
baseline_included_sensors — list of entity IDs to keep in the baseline model target when baseline_mode: true; heating/DHW can remain in the model while schedulable appliances are subtracted.
Timezone generalization — timezone apps.yaml key fully wired (was silently ignored); all hardcoded "Europe/Zurich" strings replaced with self._timezone.
holiday_country config key (ISO 3166-1 alpha-2, default CH); propagated through train() → _engineer_features() → _add_holiday_feature().
DEFAULT_TAU = 12.0 h in const.py — better residential prior for the RC-ODE before calibration data is available (was 24 h).
4 new dashboard cards in dashboard/: MAE_minigraph.yaml, forecast-over-time_minigraph.yaml, overview_today-tomorrow-3h.yaml, shap-narrative.yaml.
Fixed
Logging consistency — all modules (energy_forecast.py, energy_history_backfill.py, ha_data.py, model.py, weather.py) now route through AppDaemon’s per-app logger by wiring self.logger into the module-level _LOGGER in initialize() and patching the same logger into sub-module globals. Ensures all entries appear under the energy_forecast AppDaemon app category.
pandas 3.x dtype coercion — _merge_frames() coerces the value column back to float64 after pd.concat(); pandas 3.0.2 promoted to object dtype even on empty DataFrames, causing LightGBM to reject the feature matrix.
Sub-sensor dtype error — fetch_recent_sub_sensor / fetch_sub_sensor_history: fixed merge_asofIncompatible merge dtype error when raw HA fetch returns empty data and program_entity_id is configured; empty fallback DataFrames now use explicit dtype= per column.
τ calibration data pipeline — _fetch_history() now maps "on" → 1.0 / "off" → 0.0 before float(), fixing silent empty returns for input_boolean entities. _update_sensors() incrementally caches heating_active each prediction cycle. Passive-cooling windows trimmed at first ΔT ≤ 0 or rising delta; prefix-trim replaced with full sub-sequence scan to capture evening cooling windows.
Concurrency — _update_cb no longer acquires the training lock; hourly sensor updates always run against the last-good model during a background retrain, eliminating the ~60-second sensor-silent window.
CSV tail read — fetch_recent_energy() uses deque(maxlen=400): O(400) memory instead of O(all rows) per hourly call.
Rolling MAE persistence — pred_history.json is loaded at startup with 30-day pruning, eliminating the ~24-hour recovery period of high-volatility relative MAE after AppDaemon restart.
MQTT NaN guard — relative MAE published safely when mean consumption is zero or undefined.
UID slicing robustness — topic splitting validates segment count before slicing; malformed UIDs logged at DEBUG and dropped cleanly.
Replaced deprecated datetime.utcnow() with datetime.now(timezone.utc) throughout.
Eliminated pandas PerformanceWarning in lag column accumulation loops.
Retrain exception handler logs full traceback for diagnosability.
Replaced three broad except Exception clauses with narrower exception types.
Tests
474 passing (up from 325 in v0.9.0; +149 new tests covering all new features, regression cases, and edge conditions).
[0.9.1-alpha] — 2026-04-07
Added
apps/energy_forecast/energy_forecast.py — relative MAE sensors (mae_7d_pct, mae_30d_pct) express rolling MAE as a percentage of mean consumption, providing a normalized accuracy metric independent of consumption scale. Useful for comparing forecast accuracy across seasons (heating/cooling) and across households. Implements same persistence logic as absolute MAE sensors.
apps/energy_forecast/shap_analysis.py — SHAP narrative attribute (explanation) on sensor.energy_forecast_shap provides a human-readable “Why today?” summary via SHAP force_plot interpretation. Formats feature contributions (base value + top N positive/negative pushes) in a single-line narrative, published as an entity attribute for display in automations/notifications.
Fixed
apps/energy_forecast/energy_forecast.py — rolling MAE sensors (mae_7d, mae_30d) now remain stable across AppDaemon restarts. Root cause was loss of _pred_history and _actuals_history dicts on restart, causing ~24h recovery period with very low n_pairs and high volatility. Implemented JSON persistence layer: _load_pred_history() reads pred_history.json at startup (with 30-day pruning), _save_pred_history() atomically writes JSON after each forecast cycle. Includes 7 new tests covering roundtrip, pruning, keep-first semantics, error handling, and atomic writes.
apps/energy_forecast/mqtt_mixin.py — relative MAE percentages are now safely published even when mean consumption is zero or undefined (e.g., first day of month). Prevents NaN/inf from propagating to MQTT and breaking Lovelace graphs. Explicitly sets value to 0.0 and logs WARNING.
apps/energy_forecast/mqtt_mixin.py — topic splitting for sub-entity UID extraction was fragile to extra colons or missing segments. Replaced naive split(':')[X] with robust parsing that validates segment count before slicing and logs DEBUG for dropped malformed UIDs.
v0.11.0 introduces Daily Regime Clustering, an optional ML subsystem that identifies the household’s recurring 24-hour consumption patterns from historical data and uses a weather- and calendar-aware Random Forest classifier to predict which regime to expect each day. The predicted regime’s centroid profile is injected as a regime_kwh prior into the main LightGBM model, giving hourly forecasts a stable, physics-informed baseline anchored to real behavioral patterns. A 16-alpha hardening cycle refined auto-K elbow selection, added OOB tie-breaking, hardened centroid fitting with outlier guards and inertia normalisation, synced cluster weights to training-data decay, and excluded EV-charging days so centroids encode genuine thermal and occupancy patterns rather than EV session timing.
Added
Daily Regime Clustering (clustering.py, model.py) — K-Means on historical 24h profiles + Random Forest RegimePredictor; regime_kwh injected as prior into the main LightGBM model. Config: enable_regimes, regime_count. (#60)
Adaptive Regime Selection (Auto-K) — inertia elbow detection with OOB tie-breaking; auto-selects K ∈ [2, 8] at each training run when regime_count: 0. (#62)
__version__ in __init__.py as single source of truth. (#73)
Fixed
EV day exclusion from centroid fitting — 15% of history days with EV charging no longer distort regime centroids. (#82)
Auto-K K=2 silhouette bias replaced by inertia elbow method. (#80)
OOB tie-breaking for multi-K elbow candidates. (#82)
CQR calibration: reproducible random holdout for valid ≥80% marginal coverage. (#64)
EWMA temperature reset at weather data gaps > 2h. (#69)
Regime label forward-fill in prediction path. (#67)
Thermal pressure discontinuities at heating on/off (setpoint hysteresis projection).
fillna(method=) crash on pandas 3.x.
Series.date AttributeError in train/predict with regimes enabled.
RegimePredictor overfitting; OOB score now logged.
get_scenario schedule key validation. (#72)
Tests: 539 passing (up from 474 in v0.10.0; +65 new tests).
) so my solution is always to break apart other well known apps and integration and the solution was in plain sight. You can ditch mqtt altogether and make your App/Addon install an integration so apart of your project runs inside ha full and your Addon does the other stuff which an integration running in ha cannot
