Custom component: EdgeOS

bar · September 29, 2022, 9:41pm

This integration is working with the API / WebSockets, if the functionality is available I can add it

alex3m3ti8 · September 30, 2022, 10:38am

That would be awesome. The ability to turn off and back on any of the interfaces would be terrific to have.

bar · October 2, 2022, 5:30am

found the API calls that can do that, but before adding that feature,
I was thinking of restructure the entities of the component, change the configuration also to be another set of HA entities (now that component can have entities of configuration), it will look like:
Configuration UI will hold:

Username
Password

Device tracker

Per device

Binary Sensor

Status

Sensor:

Unknown devices
Per device - Rate Incoming (Measurement)
Per device - Rate Outgoing (Measurement)
Per device - Total Incoming (Total)
Per device - Total Outgoing (Total)
Per interface - Rate Incoming (Measurement)
Per interface - Rate Outgoing (Measurement)
Per interface - Total Incoming (Total)
Per interface - Total Outgoing (Total)

Switch

Per interface
Store incoming messages

Select

Unit - Mb, Kb, b

wdyt?

alex3m3ti8 · October 2, 2022, 12:02pm

Yeah. That makes sense to me. I know it might be breaking, but I think it would also make sense if each interface were a HA device, and the Binary Status, Sensors (Rate and Total), and Switch were grouped together, So that I could look at one device, say ETH0 and see here’s the interface Rate Incoming, Rate Outgoing, Total Incoming, Total Outgoing, and here the switch for it.

I have nothing really doing on the existing sensors, so I am happy to alpha test for you with no impact to my configuration. I’m sure we would probably want someone who is using the existing sensors in a meaningful way to take a look at what a restructure like that might do for existing users.

bar · October 11, 2022, 10:36am

v2.0.0 released

Component refactored to allow faster future integration for additional features.

New features:

Enable / Disable interface (Ethernet / Bridge) using a new switch per interface
Enable / Disable interface monitoring for received and sent data / rate / errors / packets and dropped packets using a switch per interface
Enable / Disable device monitoring for received and sent data and rate (including device tracker) using a switch per interface
Enable / Disable store debug data to ./storage directory of HA for API (edgeos.debug.api.json) and WS (edgeos.debug.ws.json) data for faster debugging or just to get more ideas for additional features
Firmware Update binary sensor including link to the new firmware
Warning when prerequisites of traffic analysis (DPI and Export) are not turned on
Asynchronous data updates of API and WebSocket to handle disconnection better
New service: Update configuration allows to edit configuration of unit, store debug data, log incoming messages and consider away interval

Breaking Changes!

Most of the configurations moved to be regular components of HA (Log incoming messages, Unit of measurement, Store debug data)
Configuration UI will hold EdgeOS URL and credentials only:
- Hostname
- Port
- Username
- Password

System

Entity Name	Type	Description	Additional information
{Router Name} Unit	Select	Sets whether to monitor device and create all the components below or not
{Router Name} Unknown devices	Sensor	Represents number of devices leased by the DHCP server	Attributes holds the leased hostname and IPs
{Router Name} CPU	Sensor	Represents CPU usage	Attributes holds the leased hostname and IPs
{Router Name} RAM	Sensor	Represents RAM usage	Attributes holds the leased hostname and IPs
{Router Name} Last Restart	Sensor	Represents last time the EdgeOS was restarted	Attributes holds the leased hostname and IPs
{Router Name} Firmware Updates	Binary Sensor	New firmware available indication	Attributes holds the url and new release name
{Router Name} Log incoming messages	Switch	Sets whether to log WebSocket incoming messages for debugging
{Router Name} Store Debug Data	Switch	Sets whether to store API and WebSocket latest data for debugging

Per device

Entity Name	Type	Description	Additional information
{Router Name} {Device Name} Monitored	Sensor	Sets whether to monitor device and create all the components below or not
{Router Name} {Device Name} Received Rate	Sensor	Received Rate per second	Statistics: Measurement
{Router Name} {Device Name} Received Traffic	Sensor	Received total traffic	Statistics: Total Increment
{Router Name} {Device Name} Sent Rate	Sensor	Sent Rate per second	Statistics: Measurement
{Router Name} {Device Name} Sent Traffic	Sensor	Sent total traffic	Statistics: Total Increment
{Router Name} {Device Name}	Device Tracker	Indication whether the device is or was connected over the configured timeframe

Per interface

Entity Name	Type	Description	Additional information
{Router Name} {Interface Name} Status	Switch	Sets whether to interface is active or not
{Router Name} {Interface Name} Monitored	Switch	Sets whether to monitor interface and create all the components below or not
{Router Name} {Interface Name} Received Rate	Sensor	Received Rate per second	Statistics: Measurement
{Router Name} {Interface Name} Received Traffic	Sensor	Received total traffic	Statistics: Total Increment
{Router Name} {Interface Name} Received Dropped Packets	Sensor	Received packets lost	Statistics: Total Increment
{Router Name} {Interface Name} Received Errors	Sensor	Received errors	Statistics: Total Increment
{Router Name} {Interface Name} Received Packets	Sensor	Received packets	Statistics: Total Increment
{Router Name} {Interface Name} Sent Rate	Sensor	Sent Rate per second	Statistics: Measurement
{Router Name} {Interface Name} Sent Traffic	Sensor	Sent total traffic	Statistics: Total Increment
{Router Name} {Interface Name} Sent Dropped Packets	Sensor	Sent packets lost	Statistics: Total Increment
{Router Name} {Interface Name} Sent Errors	Sensor	Sent errors	Statistics: Total Increment
{Router Name} {Interface Name} Sent Packets	Sensor	Sent packets	Statistics: Total Increment

Service: Update configuration
Allows to set:

Consider away interval - Time to consider a device without activity as AWAY (any value between 10 and 1800 in seconds)
Log incoming messages - Enable / Disable logging of incoming WebSocket messages for debug
Store debug data - Enable / Disable store debug data to ‘./storage’ directory of HA for API (edgeos.debug.api.json) and WS (edgeos.debug.ws.json) data for faster debugging or just to get more ideas for additional features
Unit of measurement

More details available in Developer tools → Services → edgeos.update_configuration

service: edgeos.update_configuration
data:
  device_id: {Main device ID}
  unit: Bytes
  store_debug_data: true
  log_incoming_messages: true
  consider_away_interval: 180

Changing the unit will reload the integration

alex3m3ti8 · October 11, 2022, 1:30pm

Fantastic! Great job! Hate to be that guy, but any possibility that we could also get Interfaces like vtun. When ssh-ed to the EdgeOS device and do a

configure
show interfaces

I get all the eth0-7 which have corresponding switches (Very cool! Great way for me to reboot my Internet interface). I also get loopback (probably can be dropped and not given a Monitor Switch or Status Switch). But I do also get vtun0 interface (which is my sometimes problematic OpenVPN tunnel interface that I would also LOVE a status switch on to reboot when I need to).

Took a very rudimentary look at your code. I believe the way to enable this is under

edgeos/component/helpers/const.py

Original Line 348

INTERFACE_TYPE_BRIDGE = "bridge"
INTERFACE_TYPE_ETHERNET = "ethernet"

MONITORED_INTERFACE_TYPES = [
    INTERFACE_TYPE_BRIDGE,
    INTERFACE_TYPE_ETHERNET
]

to

INTERFACE_TYPE_BRIDGE = "bridge"
INTERFACE_TYPE_ETHERNET = "ethernet"
INTERFACE_TYPE_OPENVPN = "openvpn"

MONITORED_INTERFACE_TYPES = [
    INTERFACE_TYPE_BRIDGE,
    INTERFACE_TYPE_ETHERNET,
    INTERFACE_TYPE_OPENVPN
]

where OPENVPN Interface Type is the type in config.

Unfortunately, I’m very new to this, and not very sure of myself or if I missed anything else. Of course I get that not everyone will have an OpenVPN tunnel or other supportable Installed Interface. Does it make sense to maybe instead of specifying OpenVPN only to wait on the next request for whatever other Interfaces people are using to instead simply drop the loopback and add all other interfaces? Wasn’t sure it that make more sense or not.

bar · October 11, 2022, 2:25pm

Would it be ok if I sent a pre-release? since i don’t have openvpn / virtual tunnel setup

alex3m3ti8 · October 11, 2022, 2:50pm

Absolutely. Let me know when you release it, and I’ll install the pre-release and let you know if it is working.

iDontWantAUsername · October 12, 2022, 2:01am

Hi
Thanks for the latest update and continued work with the integration.

I think I have found a small bug where Interface Received Rate is also being mapped to Interface Received Traffic. Looking in History the graphs are the same. I would expect Received Traffic to gradually increase - the same as Sent Traffic does.

Thanks,
Richard

Update: I’ve also noticed that this is flogging the DB quite heavily with updates increasing the CPU of both Home Assistant and NAS running the DB. Is there a way to reduce the polling in version 2?

bar · October 12, 2022, 3:06pm

thanks for the feedback,
found the issue you have decribed and will release a new version later today with a fix.

as for the updates to DB, in v1.x you could’ve control the entities update interval, i’m planning to re-introduce that feature soon, but it requires time to implement.

alex3m3ti8 · October 12, 2022, 6:16pm

I got a chance to try the switches and while they do display the current state of the interface, toggling the switch does not seem to disable or enable the interface. Didn’t get to try this until today, been in the middle of work stuff so couldn’t bring down major sections of my network…

I did another look, and I THINK (again, sorry but very new, eager to learn, but not sure of myself) that the

ha-edgeos/custom_components/edgeos/core/components/switch.py

There are these functions:

    async def async_turn_off(self, **kwargs: Any) -> None:
        await self.ha.async_core_entity_turn_off(self.entity)

    async def async_turn_on(self, **kwargs: Any) -> None:
        await self.ha.async_core_entity_turn_on(self.entity)

And I don’t think they actually submit the configuration change

set interfaces ethernet ethx disable

or

set interfaces ethernet ethx enable

which would actually set them to disabled/enabled. So while the switches do represent the current state, their action doesn’t do anything at this time.

bar · October 12, 2022, 7:05pm

you don’t need to be sorry, thanks for letting me know that there is an issue

it’s not supposed to submit ssh command but an API, can you please set the logs to debug and post the log of the action?

thanks

iDontWantAUsername · October 12, 2022, 7:37pm

Hi,

have you checked the role of the user that is using the home assistant integration? Mine is and always has been an ‘operator’ they shouldn’t have the option to make updates just view stats.

I haven’t tried this myself but my user’s role is operator and I cannot change the interface states.

bar · October 12, 2022, 7:47pm

that’s most probably the difference, the user i’m using is an admin,
otherwise you cannot perform update operations and this is the first time we are using such an operation, i will add verification for that during the weekend.

only admin user will see the switch, an info message will be added to logs saying the user is limited for readonly operations, wdyt?

bar · October 12, 2022, 7:48pm

v2.0.4 released with the following features / fixes:

Add IP address to status switch of its interface #71
Constants clean up
Add ability to set the interval to update data and entities separately, to update use the edgeos.update_configuration service and set the number of seconds per update_api_interval and/or update_entities_interval parameters, defaults are API: 30 seconds, Entities: 1 second
Improved logic of service edgeos.update_configuration
Add to store debug data HA data that is being used to generate HA components
Add openvpn as supported interface type

bar · October 12, 2022, 7:49pm

in case the user is operator instead of switch, it will create a binary sensor of status

bar · October 12, 2022, 8:11pm

v2.0.5 released
Non-admin user will have binary sensor for status of interface instead of switch

alex3m3ti8 · October 12, 2022, 8:41pm

Correct on the Admin vs Operator setup! Because I was in monitor-only mode with previous versions of your add-on, I granted the account that Home Assistant uses operator account only (least-privilege etc). Changed the account to Admin, and Ethernet Interfaces are controllable! Awesome work!

Small issue (I know, I’m such a pain). The VPN tunnel is now showing up. It has all the same functionality that the Ethernet Interfaces do.

Awesome!

If I disable the switch (now that I have admin), it will disable the VPN!

Under Actions the option changes to Enable and the ssh configuration shows Disabled

But after a few seconds, the switch reverts to On. BUT the configuration is still disabled! That also means I can’t toggle the switch back to ON myself and reenable the configuration. SO CLOSE!!!
The only thing I can think of is while the Configuration is Disabled, the Status still shows Connected.
When I tried this on other Interfaces, like Ethernet3, once it is disabled through toggling the switch the configuration changes to Disabled and the status also changes to Disconnected.

So looks like the way that EdgeOS reports VPN Configuration as Disabled but Status still Connected may be why the switch thinks that it is Connected/Enabled. I think EdgeOS is doing this wrong, and your code is interpreting status correctly. As such, I don’t think there’s anything for you to change since that would mean coding an exception just for VPN which seems stupid. I can try and see if Ubiquiti would consider changing this behavior so that Configuration Disabled will make Status Disconnected.

Only thing I’ll ask, is it POSSIBLE that the switch and/or binary_sensor that you’ve made could rely upon disable/enable vs connected/disconnected? Seems like that would also fix it. Hope that makes sense. Great work btw!

bar · October 12, 2022, 9:24pm

Can you enable store debug data,
Once you will enable it, you will have in ha config/.storage a file that begins with edgeos. And ends with debug.api.json,
At that file a property of interfaces, can you pls post here the section of the openvpn?

Thanks

iDontWantAUsername · October 13, 2022, 12:18am

Thanks for the updates, I tried changing the update intervals to 60 seconds using the Developer Tools → Services however this didn’t seem to have an impact. The sensors on my Monitored interface are still updating/incrementing every second.

I tried updating other settings such as debug logging but this didn’t change either, it doesn’t seem to make any changes to the config file in .storage.

This error originated from a custom integration.

Logger: homeassistant
Source: custom_components/edgeos/component/managers/home_assistant.py:1462
Integration: Ubiquiti EdgeOS Routers (documentation, issues)
First occurred: 11:08:53 (5 occurrences)
Last logged: 11:11:01

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/edgeos/component/managers/home_assistant.py", line 1462, in _async_update_configuration
    can_handle_device = self.entry_id in device.config_entries
AttributeError: 'NoneType' object has no attribute 'config_entries'