This is the Problem with Home assistant. You keep thinking about more stuff :).
I have actually scraped this entire idea and now have everything on Adaptive brightness from GitHub - basnijholt/adaptive-lighting: Adaptive Lighting custom component for Home Assistant.
But I am learning with every solution going through some tutorials for YAML Now 
Its unreal. This way I don’t need to touch anything anyway. And its working perfectly.