Understanding System Logs
System logs are important because they provide a record of all activities that have occurred on the system. They can be used to track core system or snapshot activity and help with troubleshooting issues.
System Logs
System logs are stored in the /var/log
directory. The osadmin
user has
access to these sub-directories:
Files | Description |
---|---|
/var/log/syslog* |
All service and system logs, contains additional logs from components called by the worker and tasker services |
/var/log/arangodb3/* |
ArangoDB-related logs |
/var/log/nginx/* |
nginx-related logs |
/var/log/redis/* |
Redis-related logs |
/var/log/rabbitmq/* |
RabbitMQ-related logs |
Service Logs
Service logs for IP Fabric services are stored in the /var/log/nimpee
directory:
File(s) | Description |
---|---|
/var/log/nimpee/api-errors.log |
API error logs |
/var/log/nimpee/api.log |
All API logs, including error logs |
/var/log/nimpee/migrate/ |
Database migration logs |
/var/log/ipf/ipf-cli-config/ipf-cli-config.log |
IPF CLI Config logs |
/var/log/ipf/ipf-jumphost/*.log |
Logs related to a specific jumphost service |
/var/log/nimpee/sys-lvm-resize.log |
Logs for automatic hard disk resize |
/var/log/nimpee/net-shaping-newshape.log |
Discovery bandwidth control logs |
/var/log/nimpee/support-vpn.log |
Support VPN-related logs |
/var/log/nimpee/sys-arangodb-dump.log |
ArangoDB dump logs |
/var/log/nimpee/sys-backup-duplicity.log |
Logs for backup services |
/var/log/nimpee/duplicity/sys-duplicity-*.log |
Detailed logs for each backup session |
/var/log/nimpee/sys-certificate.log |
SSL certificate-related logs |
/var/log/nimpee/sys-install.log |
Logs related to IP Fabric installation |
/var/log/nimpee/sys-service-autorestart.log |
Logs related to service auto restart |
/var/log/nimpee/sys-techsupport.log |
Techsupport-related logs |
/var/log/nimpee/sys-update.log |
IP Fabric new version update logs |
/var/log/nimpee/webhook-worker-errors.log |
Webhook error logs |
/var/log/nimpee/webhook-worker.log |
Webhook worker logs |
/var/log/nimpee/frontend.log |
Web console errors (received by API) |
/var/log/nimpee/discovery/syslogWorker/* |
Configuration management logs |
Snapshot Logs
Snapshots are stored in the /home/autoboss/snapshots
directory. Each
sub-directory represents one snapshot. Snapshot-related logs are in
/home/autoboss/snapshots/<id>/*
:
Files | Description |
---|---|
/home/autoboss/snapshots/<id>/* |
Snapshot-related logs |
/home/autoboss/snapshots/<id>/cli/* |
CLI logs collected during discovery in JSON Lines format |
/home/autoboss/snapshots/<id>/devices/* |
Information about devices processed by IP Fabric from the CLI logs |
The CLI logs under /home/autoboss/snapshots/<id>/cli/*
are differentiated by
the messageSubType
key as follows:
- cliUi
: Messages containing the raw text received from the device and some
error reports. Available for download in the text form from the UI.
- cliSession
: Messages with a rigid structure describing the commands sent
and the responses received. Usable for scripting on top of the logs.
- cliInternal
: Messages with no formal structure used purely for debugging.
The following 4 services – Networker, Tasker, Updater, and Worker – all log in two formats:
File | Description |
---|---|
/home/autoboss/snapshots/<id>/services/<service>/all.txt |
Simple text format, basic information |
/home/autoboss/snapshots/<id>/services/<service>/structured.json |
More detailed information in JSON Lines format |
Service directories and their descriptions:
Directory | Description |
---|---|
/home/autoboss/snapshots/<id>/services/networker/ |
Networker via traceroute looks for other possible next tasks for the worker service. |
/home/autoboss/snapshots/<id>/services/tasker/ |
Tasker prepares vTask records – connecting into (network) devices. |
/home/autoboss/snapshots/<id>/services/updater/ |
Updater is transforming device JSONs into the database. |
/home/autoboss/snapshots/<id>/services/worker/ |
Worker does the parsing. |
Remote Syslog
In IP Fabric versions greater than 5.0.0
, it is possible to send logs to a
remote collector. Here, we will show a basic example using the default UDP port
514
. For more advanced examples (such as using a TCP connection), please
consult the
syslog-ng Administration Guide
Changes to /etc/*.conf
Files
It is important not to modify any *.conf
files in the /etc/
directory,
as this can cause issues during upgrades.
The recommended approach is to create a new file under the service’s
conf.d
directory.
Warning
Any action on the Command-Line Interface (CLI) using the root
, osadmin
,
or autoboss
account may cause irreversible, detrimental changes to the
product. Actions taken without direct communication with the IP Fabric
Support or Solution Architect teams can render the system unusable.
Forwarding Syslog Messages
-
Log in to the IP Fabric CLI as the
osadmin
user. -
Switch to
root
(you will be asked to enter theosadmin
password):sudo su -
-
Create a new configuration file in the
/etc/syslog-ng/conf.d/
directory:nano /etc/syslog-ng/conf.d/custom-remote-syslog.conf`
The file name should be unique and must not conflict with other files in the directory:
ls /etc/syslog-ng/conf.d/ -l
root@ipfabric-server:~# ls /etc/syslog-ng/conf.d/ -l total 8 -rw-r--r-- 1 root root 580 Nov 5 11:10 ipf-api-syslog.conf -rw-r--r-- 1 root root 1414 Oct 11 13:31 ipfabric-log.conf
-
Add the configuration options (replace
<YOUR_IP>
with the IP address of your syslog server):-
Forwarding all syslog messages (including system messages):
destination remote { network("<YOUR_IP>" transport("udp") port(514)); }; log { source(s_src); destination(remote); };
-
Forwarding only IP Fabric syslog messages:
destination remote { network("<YOUR_IP>" transport("udp") port(514)); }; log { source(s_src); filter(f_ipf_api); parser(p_json); destination(remote); };
-
-
Save the file and exit.
-
Restart the
syslog-ng
service:systemctl restart syslog-ng
-
Confirm the
syslog-ng
service status:systemctl status syslog-ng
root@ipfabric-server:~# systemctl status syslog-ng ● syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-12-09 20:16:51 UTC; 5 days ago Docs: man:syslog-ng(8) Process: 741235 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 786 (syslog-ng) Tasks: 4 (limit: 18710) Memory: 79.0M CPU: 1min 5.538s CGroup: /system.slice/syslog-ng.service └─786 /usr/sbin/syslog-ng -F