FortiOS device_tracker

Development
1

Hey,

I’ve been working on a device tracker for my Fortigate router. I have managed to get it to shell in and get a list of devices that the router has detected.

One of the values it provides is the age of when it was last seen (based on packet data). However, how do I feed this data back to the device tracker so it can be marked as Home or Away? I’ve looked at several of the existing device trackers and I haven’t discovered how this works.

For example, in this screenshot, two of the three devices should be away as they haven’t sent traffic in days.
21 PM

Any help or direction would be greatly appreciated.

Thanks,

Chris

2

Found this thread because I just migrated behind a Fortigate myself and was hoping to use if for device tracking like I did with my Netgear.

I don’t have a direct answer for your question, but I know some of the device trackers use the ‘consider_home’ parameter to determine if a device is home or away.

Description of consider_home:

Seconds to wait till marking someone as not home after not being seen. This parameter is most useful for households with Apple iOS devices that go into sleep mode while still at home to conserve battery life. iPhones will occasionally drop off the network and then re-appear. consider_home helps prevent false alarms in presence detection when using IP scanners such as Nmap. consider_home accepts various time representations, (E.g. the following all represents 3 minutes: 180, 0:03, 0:03:00)

Not sure if you could direct it to use the “last seen” field or not.

Anyway. I suck at coding so I won’t be much help there, but I am available to help do some testing on this if you ever need an additional test bed since it looks like the number of folks using a FortiGate is probably low.

3

Hmm. I’ll have to look into that and see what I can find.

Once I get this last part done, I’ll make a PR for it and the real testing can start. :wink:

Thanks for the extra eyes.

Chris

1 Like
4

Hi,

Just new here, using Home Assistant for a month or so. Hope I can help though !

I’m using a Fortigate 110C and also Cisco AP for the WiFi. I used to use the cisco_ios device tracker but for some reason I had to change the ARP request command. This was my first attempt at coding in python. This works and I was happy for a while with this except that for some reason, devices are randomly not ‘disconnected’ from the AP.
Seeing your post made me think that I could use the Forti instead and hoping it’s more reliable.

I based the code on the cisco_ios.py component so all merit goes to the original dev, this is just a lame copy. But it should work.
The max age in the ARP table is set to 10mn (see line 92) otherwise it’s not taken, though I guess a present device shouldn’t stay more than 5mn inactive ?
Note that I only use the MAC address and the age of the entry, there are also IP address and interface information. I don’t know if this could be used in the standard device tracker but could be useful.

Here it is : https://paste.fedoraproject.org/paste/IMy2MGBu5gVnDoOvaSRnOg

Put it as fortiOS.py file in <conf dir>/custom_components/device_tracker and configure a device tracker like this :

  • platform: fortiOS
    host: !secret fg_host
    username: !secret fg_user
    password: !secret fg_pass
    port: !secret fg_port
    track_new_devices: True
    consider_home: 300

Feel free to modify, enhance, debug, distribute as you like.

2 Likes
5

Awesome! I’ll have to give this a shot. You should probably also consider submitting it to the official git as well.

Edit: Quick question. I assume that for the port it is using SSH? Does it default to 22?

6

Yes port is for SSH connection (you should consider using something different from the standard if it’s open outside). Be careful that the IP also is the one where you have enabled the admin access.

I’ve found that there is a limitation in the code : the interface name shouldn’t have a space in it because it’s the separator to split the fields. If needed, I can upload a modified version.

I have absolutely no idea how to submit to official though…

7

Cool. I expected as much. Just wanted to make sure. I already have admin access locked down to only a couple trusted hosts; but once I have it working I’ll probably change the SSH port just as an added precaution.

I haven’t added anything myself since I’m not a coder by any strech of the imagination. But the process is briefly outlined here:

&

Thanks for the good work! Maybe I can stop using nmap now.

8

Thanks, I’ll read that and try to understand. :sweat:

1 Like
9

I’m afraid you’ll need some other detection along with this. A few tests later, I found out that the FortiOS ARP table is not that nice. Timeout is 5mn but, as stated in the docs, device can remain in the ARP table “longer depending on several condition”. Nice… :roll_eyes: And timeout cannot be changed.

So it happens that a device can remain over 30mn in the ARP table. VMWare guests for instance do that a lot.
And it also happens that the table removes a device that didn’t communicate though it is still here. I’ve seen that on my old Foscam Camera on which I have an automation to restart it if away…
Good thing is that I didn’t find a case where a device remained in the ARP table after disconnected. (what I had in the Cisco AP)

I could change the max age to 30 minutes, but I found more elegant to add a “timeout” option (optional), in minutes. Default is 5 of course.
I also changed code so that a space in the interface name is no more a problem.

Here is the new version : https://paste.fedoraproject.org/paste/XqhogPbazvDcThCZ5F0b5w

All tests are on a 110C version 5.2b754, if someone has use of it, tell me how it works and your firmware.

10

Very cool! I used the device tracking added to FortiOS a number of years back in my PR.

This looks like a much simpler approach.

11

Problem with this solution is that some devices disappear from the ARP table immediately. This happens on Foscam cameras I have, they are here, they show in the WiFi access point list, they respond to ping, making them appear in the ARP table, then go away right after that.

So I had to add a “wakeup cam” automation, so that I can check a camera isn’t away for too long. Before the automation "reboot cam"is triggered that shuts the plug (WiFi) to cold restart it…

12

That’s weird. But, if you’ve used Fortinet products long enough, that’s in your standard vocabulary. :wink:

I’ll try it in my test env against FortiOS 5.4.6 and see if it has the same issue. I have three Foscam cameras to test with, too.

13

Yes I have a lot, mostly at work, but I recycle the old ones to bug my friends at home… :wink:
(From 60A to 100D…)

As long as “get system arp” works and returns the same columns (at least the 3 first), this code should work as expected, whatever OS version.

14

Hi,

Just curious if this is still valid? I understand about creating the fortiOS.py and then adding the device tracker in the config, but the fedoraproject doesn’t show up anymore. Wonder what the fortiOS.py file should look like.

Thanks.

15

Alive for more than 1 year, really reliable (well, as much as a fortigate is :yum: )

Create a <config>/custom_components/device_tracker/fortiOS.py with this inside and configure as stated above :

"""
Support for FortiOS ARP table

"""
import logging

import voluptuous as vol

import homeassistant.helpers.config_validation as cv
from homeassistant.components.device_tracker import (
    DOMAIN, PLATFORM_SCHEMA, DeviceScanner)
from homeassistant.const import CONF_HOST, CONF_PASSWORD, CONF_USERNAME, \
    CONF_PORT, CONF_TIMEOUT

_LOGGER = logging.getLogger(__name__)

REQUIREMENTS = ['pexpect==4.0.1']

PLATFORM_SCHEMA = vol.All(
    PLATFORM_SCHEMA.extend({
        vol.Required(CONF_HOST): cv.string,
        vol.Required(CONF_USERNAME): cv.string,
        vol.Optional(CONF_PASSWORD, default=''): cv.string,
        vol.Optional(CONF_PORT): cv.port,
        vol.Optional(CONF_TIMEOUT, default=5): cv.positive_int,
    })
)


def get_scanner(hass, config):
    """Validate the configuration and return a FortiOS scanner."""
    scanner = FortiDeviceScanner(config[DOMAIN])

    return scanner if scanner.success_init else None


class FortiDeviceScanner(DeviceScanner):
    """This class queries a fortiGate firewall."""

    def __init__(self, config):
        """Initialize the scanner."""
        self.host = config[CONF_HOST]
        self.username = config[CONF_USERNAME]
        self.port = config.get(CONF_PORT)
        self.password = config.get(CONF_PASSWORD)
        self.timeout = config.get(CONF_TIMEOUT)

        self.last_results = {}

        self.success_init = self._update_info()
        _LOGGER.info('fortiOS scanner initialized')

    # pylint: disable=no-self-use
    def get_device_name(self, device):
        """Get the firmware doesn't save the name of the wireless device."""
        return None

    def scan_devices(self):
        """Scan for new devices and return a list with found device IDs."""
        self._update_info()

        return self.last_results

    def _update_info(self):
        """
        Ensure the information from the Fortigate firewall is up to date.

        Returns boolean if scanning successful.
        """
        string_result = self._get_arp_data()

        if string_result:
            self.last_results = []
            last_results = []

            lines_result = string_result.splitlines()

            # Remove the first two lines, as they contains the arp command
            # and the arp table titles e.g.
            # get system arp
            # Address           Age(min)   Hardware Addr      Interface
            lines_result = lines_result[3:]

            for line in lines_result:
                parts = line.split()
                if len(parts) < 4:
                    continue

                # ['192.168.1.1','0','XX:XX:XX:XX:XX:XX','wan1']
                age = parts[1]
                hw_addr = parts[2]

                age = int(age)
                if age < self.timeout:
                    last_results.append(hw_addr)

            self.last_results = last_results
            return True

        return False

    def _get_arp_data(self):
        """Open connection to the router and get arp entries."""
        from pexpect import pxssh
        import re

        try:
            fortiOS_ssh = pxssh.pxssh()
            fortiOS_ssh.login(self.host, self.username, self.password,
                            port=self.port, auto_prompt_reset=False)

            # Find the hostname
            initial_line = fortiOS_ssh.before.decode('utf-8').splitlines()
            router_hostname = initial_line[len(initial_line) - 1]
            router_hostname += "#"
            # Set the discovered hostname as prompt
            regex_expression = ('(?i)^%s' % router_hostname).encode()
            fortiOS_ssh.PROMPT = re.compile(regex_expression, re.MULTILINE)
            fortiOS_ssh.prompt(1)

            fortiOS_ssh.sendline("get system arp")
            fortiOS_ssh.prompt(1)

            devices_result = fortiOS_ssh.before

            return devices_result.decode('utf-8')
        except pxssh.ExceptionPxssh as px_e:
            _LOGGER.error("pxssh failed on login")
            _LOGGER.error(px_e)

        return None
16

thanks, I’ll try later today.

Interesting to hear your comments about Fortigate. I have used them exclusively for at least 10 years, and have many that have been up for 2 years plus with no reboots, with the only reason for a reboot to update the firmware. I mostly have them in businesses. At my house I am running one from a company that upgraded, and mine has been up for 1.5 years with no reboot, until about 3 months ago when I updated the firmware. I am running an 80CM on firmware 5.6.2. I consider them tanks.

17

I had logged a support call with Fortinet a few years back (on previous firmware) as their device inventory returned the incorrect status. Fortinet’s final response was as below

"I was not able to reproduce the issue in lab.This issue seems expected per current design.

The only reliable mechanism is to have an agent, i.e. FortiClient, installed on the end user device.

All other methods can give incorrect results and should only be considered “as-is”.
There are several different methods to identify a device and the the operating system but none of them is reliable.
I cannot give much details about the techniques used as they are considered internal information.

Just as an example:
One common techniques is for example TCP Fingerprinting. There are many details i.e. on https://nmap.org/book/osdetect-methods.html which explain how this works.
This method examines packets and gives a conclusion about the used TCP/IP stack. It can for example detect if a device is using Linux or Windows, but can not differentiate well between versions.
Some systems might share the same stack like Apple Mac and Apple iPhones/Tablets.
The same applies for a FortiGate which uses part of the Linux kernel and therefore can be detected as Linux.
Packets of course can be modified by several network devices and cause false results.

Another example is looking at the http user agent which requires the client to actively browse to certain web sites.
This method is quite reliable but as user-agents can be spoofed this is not a fool proof method either.
If HTTP is used the browser agent detection can be used, similar for SMTP or other protocols where the banner can be extracted. Banners and user agents can be modifed of course.
This is pretty trustworthy but requires HTTP traffic.

There are several other ways, like looking at protocol specific fields: SIP user agents, Cisco Discovery Protocol, DHCP options, but again, none of them is absolutely reliable.
If DHCP is used then certain fields in the request can be examined and are unique for certain platforms but not for others.
You can also check the MAC address but this not very reliable and will not help differentiating an iPad from a Mac and it can be spoofed.

This issue is nearly impossible to debug as we would need network captures while the device in question connected the first time and subsequent captures to track the active scanning.
The output of “diagnose user device list” would also give more details plus a debug log of the scanning processes during the whole time of the scanning.

This all combined with a few other techniques looking for certain protocols (i.e. to detect the difference between iPhones and Mac which have same fingerprints) helps improving the detection.

The OS detection will only work reliably if the client is directly connected to the Fortigate and even then might not always be accurate.
The only really reliable way is using a FortiClient with end point control and device detection.

In your case the traffic was maybe modified by a FortiGate or the Cisco AP340 as it passed this device, so it appears to be coming from this operating system.

The command “diagnose user device list” would show a bit more information, i.e. if the detection is completed (reliable method was used) or still in progress (not enough information to be reliable).

To summarize, unless you have a FortiClient installed on the device the OS detection is not reliable and device detection can have false results per design.

Also I found few known issues with Fortios 5.4.1 but they are not related directly to your scenario. I would advise if you can wait for FortiOS 5.4.2/5.4.3."

I have not checked if the reliability and the device inventory detection has been improved in the latest releases of FortiOS, but shall have another glance to see if this is now more reliable.

18

Don’t get me wrong, I love those boxes ! But it’s kind of a love/hate relationship !
As far as hardware goes, they are really good, well most of the time. I have hundreds of them (literally) at work, now mostly 100D and those died by dozen because of the internal drive. In over 15 years of heavy use, that’s the only hard failure I can remember though. That’s why I’ve kept a 110C at home even if it only has 2 gigabits ports. I still have 100A working !

I’m not as enthusiastic about the software. They have their own logic and names, and they change it very often (sd-wan now, was wan load balance, and several other things before, HA also changed a lot…). I could live with that, always fun to test an update. :wink:
The main problem, as stated above, is that some things just don’t behave as expected. ARP table isn’t stable (loses some devices, still don’t know why), some firmware became unstable after a WAN router was restarted (I had to program a daily reboot at home for a while to get my VPN access up again… seems corrected now).
But in the end, once understood, it’s easy to configure and monitor, they do the job, they’re not that expensive. And I’m not going back to iptables !!!

19

Ahh, totally understand.

The SD-Wan vs wan load balance was a crazy switch! And I hate how sometimes they randomly choose to remove things from the GUI.

20

Oh, I wanted to say the device tracker using the fortiOS.py is working great! Thanks.