Understanding System Logs

System logs are important because they provide a record of all activities that have occurred on the system. They can be used to track core system or snapshot activity and help with troubleshooting issues.

System Logs

System logs are stored in the /var/log directory. The osadmin user has access to these sub-directories:

Files	Description
/var/log/syslog*	All service and system logs, contains additional logs from components called by the worker and tasker services
/var/log/arangodb3/*	ArangoDB-related logs
/var/log/nginx/*	nginx-related logs
/var/log/redis/*	Redis-related logs
/var/log/rabbitmq/*	RabbitMQ-related logs

Service Logs

Service logs for IP Fabric services are stored in the /var/log/nimpee directory:

File(s)	Description
/var/log/nimpee/api-errors.log	API error logs
/var/log/nimpee/api.log	All API logs, including error logs
/var/log/nimpee/migrate/	Database migration logs
/var/log/ipf/ipf-cli-config/ipf-cli-config.log	IPF CLI Config logs
/var/log/ipf/ipf-jumphost/*.log	Logs related to a specific jumphost service
/var/log/nimpee/sys-lvm-resize.log	Logs for automatic hard disk resize
/var/log/nimpee/net-shaping-newshape.log	Discovery bandwidth control logs
/var/log/nimpee/support-vpn.log	Support VPN-related logs
/var/log/nimpee/sys-arangodb-dump.log	ArangoDB dump logs
/var/log/nimpee/sys-backup-duplicity.log	Logs for backup services
/var/log/nimpee/duplicity/sys-duplicity-*.log	Detailed logs for each backup session
/var/log/nimpee/sys-certificate.log	SSL certificate-related logs
/var/log/nimpee/sys-install.log	Logs related to IP Fabric installation
/var/log/nimpee/sys-service-autorestart.log	Logs related to service auto restart
/var/log/nimpee/sys-techsupport.log	Techsupport-related logs
/var/log/nimpee/sys-update.log	IP Fabric new version update logs
/var/log/nimpee/webhook-worker-errors.log	Webhook error logs
/var/log/nimpee/webhook-worker.log	Webhook worker logs
/var/log/nimpee/frontend.log	Web console errors (received by API)
/var/log/nimpee/discovery/syslogWorker/*	Configuration management logs

Snapshot Logs

Snapshots are stored in the /home/autoboss/snapshots directory. Each sub-directory represents one snapshot. Snapshot-related logs are in /home/autoboss/snapshots/<id>/*:

Files	Description
/home/autoboss/snapshots/<id>/*	Snapshot-related logs
/home/autoboss/snapshots/<id>/cli/*	CLI logs collected during discovery in JSON Lines format
/home/autoboss/snapshots/<id>/devices/*	Information about devices processed by IP Fabric from the CLI logs

The CLI logs under /home/autoboss/snapshots/<id>/cli/* are differentiated by the messageSubType key as follows: - cliUi: Messages containing the raw text received from the device and some error reports. Available for download in the text form from the UI. - cliSession: Messages with a rigid structure describing the commands sent and the responses received. Usable for scripting on top of the logs. - cliInternal: Messages with no formal structure used purely for debugging.

The following 4 services – Networker, Tasker, Updater, and Worker – all log in two formats:

File	Description
/home/autoboss/snapshots/<id>/services/<service>/all.txt	Simple text format, basic information
/home/autoboss/snapshots/<id>/services/<service>/structured.json	More detailed information in JSON Lines format

Service directories and their descriptions:

Directory	Description
/home/autoboss/snapshots/<id>/services/networker/	Networker via traceroute looks for other possible next tasks for the worker service.
/home/autoboss/snapshots/<id>/services/tasker/	Tasker prepares vTask records – connecting into (network) devices.
/home/autoboss/snapshots/<id>/services/updater/	Updater is transforming device JSONs into the database.
/home/autoboss/snapshots/<id>/services/worker/	Worker does the parsing.

Remote Syslog

In IP Fabric versions greater than 5.0.0, it is possible to send logs to a remote collector. Here, we will show a basic example using the default UDP port 514. For more advanced examples (such as using a TCP connection), please consult the syslog-ng Administration Guide

Changes to /etc/*.conf Files

It is important not to modify any *.conf files in the /etc/ directory, as this can cause issues during upgrades.

The recommended approach is to create a new file under the service’s conf.d directory.

Warning

Any action on the Command-Line Interface (CLI) using the root, osadmin, or autoboss account may cause irreversible, detrimental changes to the product. Actions taken without direct communication with the IP Fabric Support or Solution Architect teams can render the system unusable.

Forwarding Syslog Messages

1. Log in to the IP Fabric CLI as the osadmin user.

2. Switch to root (you will be asked to enter the osadmin password):

   sudo su -
   

3. Create a new configuration file in the /etc/syslog-ng/conf.d/ directory:

   nano /etc/syslog-ng/conf.d/custom-remote-syslog.conf`
   

   The file name should be unique and must not conflict with other files in the directory:

   ls /etc/syslog-ng/conf.d/ -l
   

   root@ipfabric-server:~# ls /etc/syslog-ng/conf.d/ -l
   total 8
   -rw-r--r-- 1 root root  580 Nov  5 11:10 ipf-api-syslog.conf
   -rw-r--r-- 1 root root 1414 Oct 11 13:31 ipfabric-log.conf
   

4. Add the configuration options (replace <YOUR_IP> with the IP address of your syslog server):

   1. Forwarding all syslog messages (including system messages):

      destination remote { network("<YOUR_IP>" transport("udp") port(514)); };
      log { source(s_src); destination(remote); };
      

   2. Forwarding only IP Fabric syslog messages:

      destination remote { network("<YOUR_IP>" transport("udp") port(514)); };
      
      log {
        source(s_src);
        filter(f_ipf_api);
        parser(p_json);
        destination(remote);
      };
      

5. Save the file and exit.

6. Restart the syslog-ng service:

   systemctl restart syslog-ng
   

7. Confirm the syslog-ng service status:

   systemctl status syslog-ng
   

   root@ipfabric-server:~# systemctl status syslog-ng
   ● syslog-ng.service - System Logger Daemon
        Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
        Active: active (running) since Fri 2022-12-09 20:16:51 UTC; 5 days ago
          Docs: man:syslog-ng(8)
       Process: 741235 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
      Main PID: 786 (syslog-ng)
         Tasks: 4 (limit: 18710)
        Memory: 79.0M
           CPU: 1min 5.538s
        CGroup: /system.slice/syslog-ng.service
                └─786 /usr/sbin/syslog-ng -F