Understanding System Logs
System logs are important because they provide a record of all activities that have occurred on a system. They can be used to track core system or snapshot activity and help with troubleshooting issues.
System Logs
System logs are stored in the /var/log
directory. The osadmin
user has access to these sub-directories:
/var/log/syslog* - all service and system logs
/var/log/arangodb3/* - ArangoDB related logs
/var/log/nginx/* - NGINX related logs
/var/log/redis/* - REDIS related logs
/var/log/rabbitmq/* - RabbitMQ related logs
Service Logs
Service logs for IP Fabric services are stored in the /var/log/nimpee
directory:
/var/log/nimpee/api-errors.log - API error logs
/var/log/nimpee/api.log - all API logs including error logs
/var/log/nimpee/migrate/ - DB migrate logs
/var/log/nimpee/net-config.log - Boot wizard related logs
/var/log/nimpee/net-jumphost-*.log - Logs related to a specific jumphost service
/var/log/nimpee/sys-lvm-resize.log - Logs for automatic HDD resize
/var/log/nimpee/net-shaping-newshape.log - Discovery bandwidth control logs
/var/log/nimpee/support-vpn.log - Support VPN related logs
/var/log/nimpee/sys-arangodb-dump.log - ArangoDB dump logs
/var/log/nimpee/sys-backup-duplicity.log - Logs for backup services
/var/log/nimpee/duplicity/sys-duplicity-*.log - Detailed logs for each backup session
/var/log/nimpee/sys-certificate.log - SSL certificates related logs
/var/log/nimpee/sys-install.log - Logs related to IP Fabric installation
/var/log/nimpee/sys-service-autorestart.log - Logs related to service auto restart
/var/log/nimpee/sys-techsupport.log - Techsupport related logs
/var/log/nimpee/sys-update.log - IP Fabric new version update logs
/var/log/nimpee/webhook-worker-errors.log - webhook errors logs
/var/log/nimpee/webhook-worker.log - webhook worker logs
/var/log/nimpee/frontend.log - web console errors (received by API)
/var/log/nimpee/discovery/syslogWorker/* - Configuration management logs
Snapshot Logs
Snapshots are available in the /home/autoboss/snapshots
directory. Each sub-directory represents one snapshot. Snapshot-related logs are located in /home/autoboss/snapshots/<id>/*
:
/home/autoboss/snapshots/<id>/* - snapshot related logs
/home/autoboss/snapshots/<id>/cli/* - CLI logs collected during the disocvery
/home/autoboss/snapshots/<id>/devices/* - information about devices processed by IP Fabric from the CLI logs
Following 4 services - Networker, Tasker, Updater, Worker all log in two formats:
/home/autoboss/snapshots/<id>/services/<service>/all.txt - simple text format, basic information
/home/autoboss/snapshots/<id>/services/<service>/structured.json - more detailed information in JSON Lines format
Service folders and their descriptions:
/home/autoboss/snapshots/<id>/services/networker/ - networker via traceroute looks for other possible next tasks for worker service
/home/autoboss/snapshots/<id>/services/tasker/ - tasker prepares `vTask` records - connecting into (network) devices
/home/autoboss/snapshots/<id>/services/updater/ - updater is transforming device JSON into DB
/home/autoboss/snapshots/<id>/services/worker/ - worker does parsing
Remote Syslog
In IP Fabric version greater than 5.0.0
, it is possible to send logs to a remote collector. Here we will show a basic example using the default UDP port 514
. For more advanced examples (such as using TCP connection), please consult the syslog-ng documentation
Changes to /etc/*.conf
files
It is important not to modify any *.conf
files in the /etc/
directory as this can cause issues during upgrades.
The recommended approach is to create a new file under the service’s conf.d
directory.
Warning
Any action on the command-line interface (CLI) using the root
, osadmin
, or autoboss
account may cause irreversible detrimental changes to the product. Actions taken without direct communication with the IP Fabric Support or Solution Architect teams can render the system unusable.
Forwarding Syslog Messages
- Log in to the IP Fabric CLI with the
osadmin
user. - Switch to root account using
sudo su
and enter theosadmin
password. -
Create a new configuration file in the
/etc/syslog-ng/conf.d/
foldernano /etc/syslog-ng/conf.d/custom-remote-syslog.conf`
The file name should be unique and must not conflict with other files in folder.
ls /etc/syslog-ng/conf.d/ -l
root@ipfabric-server:~# ls /etc/syslog-ng/conf.d/ -l total 8 -rw-r--r-- 1 root root 580 Nov 5 11:10 ipf-api-syslog.conf -rw-r--r-- 1 root root 1414 Oct 11 13:31 ipfabric-log.conf
-
Add the configuration options (replacing
<YOUR_IP>
with the IP of your syslog server)-
Forwarding All Syslog Messages (including system messages):
destination remote { network("<YOUR_IP>" transport("udp") port(514)); }; log { source(s_src); destination(remote); };
-
Forwarding Only IP Fabric Syslog Messages:
destination remote { network("<YOUR_IP>" transport("udp") port(514)); }; log { source(s_src); filter(f_ipf_api); parser(p_json); destination(remote); };
-
-
Save the file and exit.
-
Restart syslog-ng.
systemctl restart syslog-ng
-
Confirm the
syslog-ng
service status.systemctl status syslog-ng
root@ipfabric-server:~# systemctl status syslog-ng ● syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-12-09 20:16:51 UTC; 5 days ago Docs: man:syslog-ng(8) Process: 741235 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 786 (syslog-ng) Tasks: 4 (limit: 18710) Memory: 79.0M CPU: 1min 5.538s CGroup: /system.slice/syslog-ng.service └─786 /usr/sbin/syslog-ng -F