Troubleshooting Discovery

Discovery Settings

This section describes the Global Settings. You can also make adjustments to specific snapshots. Changes made to the Global Settings do not apply to previously run snapshots. Here are two methods for testing snapshot settings:

Make your changes in the snapshot Settings and then try to Add Devices (or Rediscover timed out devices). If these changes fixed your issue, then apply them globally.
Make your changes globally and then run a new snapshot. In Settings → Discovery & Snapshots → Discovery Settings → Discovery → IP Scope → IP networks to include in discovery and analysis, you can also filter the allowed discovery scope to a subset of /32 addresses to lower the discovery time. In the example below, we allow only one subnet to be part of the discovery:

Discovery Seeds

Location: Settings → Discovery & Snapshots → Discovery Settings → Discovery Seeds

These are user-defined IP addresses that IP Fabric will try to connect to and discover. If a device is successfully discovered, it will check the settings in the Discovery tasks settings to see what technologies to use to discover neighbors.

If you have remote locations possibly accessed through a WAN that is not owned by you, this would be the place to put the IP addresses of some of your devices, so IP Fabric can discover and crawl those.

Important

Adding new devices here will only discover them if Limit discovery to already discovered devices is not enabled in the Discovery tasks settings.

Discovery History

Location: Management → Discovery History

The Discovery History table is also used by IP Fabric as seed addresses. Once a device is discovered in a snapshot, it will be added to this table for use in other snapshots.

The username last successfully used for a device will be recorded here and be the first one tried in a future snapshot. If unsuccessful, IP Fabric will check the list in Settings → Discovery & Snapshots → Discovery Settings → Device Credentials and go back through the decision tree. This may cause the next snapshot to take longer due to authentication failures, but once this table is updated with the new username, the discovery time will return to normal.

If you have removed devices from your network, IP Fabric will still try to connect to them, because the IP addresses are defined as seeds in this Discovery History table. To speed up discovery, you could delete stale records. You can also filter on Last discovery time and delete records older than X months.

IP Scope

Location: Settings → Discovery & Snapshots → Discovery Settings → Discovery → IP Scope

IP Scope tells IP Fabric which networks to include or exclude in the discovery process. The default is set to include everything (0.0.0.0/0). When IP Fabric discovers a neighbor in this network, it will try to log in to and discover it. If the IP address is not in the include list, or is in the exclude list, then IP Fabric will not try to connect to and discover the device.

The exclude list is a great way to exclude networks managed by a different department or perhaps vendors or devices that IP Fabric does not currently support. For instance, a customer had discovery hanging due to it trying to log in to a terminal server, which IP Fabric could not understand; once added to the exclude list, the discovery completed successfully.

Important

Do not confuse the include list with the seed list. The include list will only try to discover an IP address if it is found through the Discovery task or the IP address is in the seeds. Also note that IP Fabric does not do any ICMP pings to find hosts, so having a /16 here will not send massive amounts of pings.

Discovery tasks settings

Location: Settings → Discovery & Snapshots → Discovery Settings → Discovery → Discovery tasks settings

Discovery tasks settings are where the magic happens for automated discovery of new devices on your network.

If you are not finding new devices in your snapshot, check if Limit discovery to already discovered devices is not enabled. If this is enabled, then only devices in the Discovery History table will be added to the snapshot, but not new devices (even if those devices are manually added to the global discovery seeds in Settings → Discovery & Snapshots → Discovery Settings → Discovery Seeds).

If you want to limit to only discovered devices but want to add new devices, this is still possible by going to Discovery Snapshot and manually adding devices into a snapshot. Those will then be added to the Discovery History table and picked up in future snapshots.

xDP (neighbors) signifies using CDP or LLDP information to discover devices in your network.

ARP uses the ARP and MAC address OUI information to find devices. If the OUI is set to Enabled for discovery in the table in Settings → Discovery & Snapshots → Global Configuration → OUI, then IP Fabric will attempt to connect to and discover the device.

Routing Table will try to connect to next-hop devices.

Trace signifies using traceroute to RFC1918 addresses to help discover your internal network. This is helpful for the VM as IP Fabric might not be able to log in to the default gateway (vRouter), but using traceroute, it can find some other physical devices in the network to use as starting points.

Advanced CLI

Location: Settings → Discovery & Snapshots → Discovery Settings → Advanced CLI

See Advanced CLI for the explanation of these settings.

Vendors API

Location: Settings → Discovery & Snapshots → Discovery Settings → Vendors API

See Vendors API as each vendor has different requirements.

Device Credentials

Location: Settings → Discovery & Snapshots → Discovery Settings → Device Credentials

Authentication is where you define the username and password IP Fabric uses to connect to a physical device (devices discovered through the API are managed through Advanced → Vendors API). Ensure that you have a username configured for all scopes of the network you wish to discover or set to the default of 0.0.0.0/0.

Tip

If you are having issues with Configuration Backup not pulling data, ensure that you have Use for configuration management set on the proper usernames.

Further information can be located in Device Credentials.

Summary of Issues

Each snapshot under the Discovery Snapshot menu has a Summary of Issues tab:

Error Type	Description
Result mapping failures (per command)	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Parsing failures (per command)	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Result mapping failures (per task)	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Ambiguous or Incomplete command	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Command authorization failure	Can be resolved by you. Click the item – you should see more details in the `Error` column.
Invalid command	IP Fabric parsing issues, most likely wrong command argument. Can be fixed by us based on a provided techsupport file.
Command timeout	Needs to be investigated separately – as it can be caused by various factors (IP Fabric SSH client issue, device issue, etc.).
Fatal error	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Unspecified command	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Unknown device version	Incorrect device version parsing / missing devices. Can be fixed by the IP Fabric team.
Possible device issue (during parsing)	IP Fabric parsing issues, can be fixed by us based on a provided techsupport file.
Request failed	Related to Vendor API tasks.
Possible device issue (during task mapping)	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.
Unsupported device	Click the item. Compare with https://matrix.ipfabric.io to confirm if the devices are indeed not supported by IP Fabric.
Authentication error	Can be resolved by you. Click the item – you should see more details in the `Error` column.
TypeError	IP Fabric parsing issues. Can be fixed by us based on a provided techsupport file.

Troubleshooting Missing Devices

There can be numerous reasons why IP Fabric did not discover a device from Day 0: incorrectly configured devices, AAA outage, insufficient authorization privileges, just to name a few. First, we will look at some tables to help you find these and then show how to use the Connectivity Report to debug them.

Vendor-Specific Troubleshooting

You may notice that some devices of a certain vendor are not being discovered. Please read through the Known Issues to ensure a bug has not been already raised or a fix suggested. For instance, some firewall vendors require elevated or admin profiles to pull all data.

Technology Tables

CDP/LLDP → Unmanaged neighbors

This is the best table to start with because it not only shows the local host and interface names but also provides the remote hostname, interface name, and remote IP address. Since devices can connect to multiple other devices, there can be duplicate remote hostnames in this list.

You can export this view to CSV and remove duplicate hostnames to obtain a unique list of devices. For instance, in this demo, we observe a total of 59 unmanaged neighbors, but only 40 unique remote hostnames.

Interfaces → Connectivity matrix → Unmanaged Neighbors Detail

This table displays unmanaged neighbors based on various protocols and includes a built-in default intent check for you.

RED: Provides a list of unmanaged CDP/LLDP neighbors (cdp|lldp|mndp). However, it does not provide the remote hostname, which is why it is recommended to look at this table second.

AMBER: Is a list of unmanaged Interior Gateway Protocols (eigrp|ospf|rip|isis|pim|vxlan|cef). It is specified as such because if you have a neighbor adjacency with an iGP, it is assumed to be under the control of your network.

Tip

Check this table for assistance with locating rogue devices in your network.

BLUE: Is the default rule if the protocol does not match any other regex.

GREEN: Is defined for the Exterior Gateway Protocol BGP.

Auditing Using an External Network Management System

Perhaps you would like to audit IP Fabric with an NMS (or vice versa). This can be done, as all the tables can return the data in CSV format via the UI or JSON using the API. This would perhaps help you find other missing devices IP Fabric could not discover.

Tip

Don’t forget to audit your NMS or alarming system’s inventory with what IP Fabric discovered on the network. Many customers have found devices that were not being monitored.

Note on Management IP

In IP Fabric 4.3.5 or lower, the UI in many places has a Management IP column; since 4.4, this has been renamed to Login IP to avoid confusion and to match the API column name loginIp.

Since IP Fabric is an automated discovery process, it will try to log in to any IP address found during the discovery tasks processes. This value could be a loopback, management, physical, or virtual interface. Auditing your external NMS will need extra care as in many cases, the IP addresses will not match.

One solution is to export your NMS inventory to Excel and the Technology → Addressing → Managed IP table to CSV and then perform vlookups between the two datasets. Since the Managed IP table contains all the IP addresses found on devices, the NMS data should find a match.

Another option is to use the device’s Serial Number. Serial numbers are not always unique, and there is a chance of overlap, so please consider this.

Note on Serial Numbers

IP Fabric records two serial numbers for a device. The column named Serial Number is the actual hardware serial number of the device, which is why it is labeled as snHw in the API.

The column named Unique serial number is an IP Fabric-unique identifier and primary key for the device. This column is labeled as sn in the API, so it is easy to confuse the two if you are not aware. The main occasion there are differences between the two is seen in firewalls with virtual contexts, vdoms, etc.

Connectivity Report

The Connectivity Report is a per-snapshot report stating successes or errors for IP addresses that IP Fabric tried to discover. Once you have a list of IP addresses from the unmanaged neighbor tables above or perhaps an external Network Management System, this is where you would start your troubleshooting process.

The most common reason why a new installation of IP Fabric will not connect is due to the connect ECONNREFUSED XX.XX.XX.XX:22 error, which signifies that the traffic is being blocked by an ACL or firewall. This is why it is recommended to have IP Fabric in your management subnet, which is already allowed for remote access to devices, so you do not have to reconfigure all your ACLs. Another option is to use a jumphost as described in Jumphost.

Another common error is due to authentication or authorization errors. Please ensure that the configured username has the correct permissions. There is also a list of CLI commands used, which can help ensure authorization is correct.

If you encounter other errors, you can refer to Connectivity report - Type of Error and the CLI output of the device is usually helpful as well.

If you need further assistance, please feel free to reach out to your IP Fabric Solution Architect or open a ticket following the Technical Support instructions.

Finally, another option for testing devices is to log in to the IP Fabric CLI using the osadmin account and manually trying to SSH into your devices, preferably using the username and password the system was configured to use. If you cannot open a connection, this is due to an external reason.

Further information can be found in No Devices Discovered.

Troubleshooting Configuration Management

Configuration Management is a separate process from discovery. First, a device must be discovered in a snapshot and placed in the Management → Discovery History table. Once in this table, next time IP Fabric is scheduled to pull configs, the device should be populated in the Configuration Management list.

Example

Settings → Discovery & Snapshots → Snapshot Retention → Create Snapshots Periodically is set to create snapshots every day at 6 AM.

Settings → Configuration Management → Schedule is set to create device configuration backups every day at 11 PM.

The new device will be found at 6 AM, but since its configuration backup is scheduled for 11 PM, you will need to wait until the next day to be sure that its configuration is pulled.

Tip

Please ensure that in Settings → Discovery & Snapshots → Discovery Settings → Device Credentials, you have enabled the correct usernames to Use for configuration management. If no usernames have been enabled for configuration management, then configurations will not be backed up.