Skip to content

Sync Process Overview

This page provides a high-level overview of how data flows from IP Fabric into NetBox through the plugin’s synchronization process. Understanding this workflow will help you configure and troubleshoot your synchronizations effectively.

The Big Picture

The IP Fabric NetBox plugin acts as a bridge between IP Fabric’s network discovery platform and NetBox’s infrastructure management database. Rather than directly modifying your NetBox database, the plugin uses a staging approach through branches to propose changes that you can review before applying them to production.

graph LR
    A[IP Fabric] --> B[Plugin Sync]
    B --> C[NetBox Branch]
    C --> D{Review Changes}
    D -->|Approve| E[NetBox Database]
    D -->|Reject| F[Discard Branch]

Key Concepts

Before diving into the synchronization workflow, it’s important to understand the following core concepts.

Sources

Sources represent your IP Fabric instances. They contain the connection details (URL, API token) needed to communicate with IP Fabric. You can configure multiple sources if you have multiple IP Fabric instances (e.g., production and development).

Snapshots

Snapshots are point-in-time captures of your network state in IP Fabric. Each sync pulls data from a specific snapshot, ensuring consistency. You can use the special $last snapshot to always sync from the most recent discovery.

Endpoints

Endpoints are specific IP Fabric API paths that the plugin queries to retrieve data. Examples include /inventory/devices for device information or /technology/vlans/site-summary for VLAN data. Each endpoint corresponds to a specific data type in IP Fabric.

Filters

Filters allow you to control which data is retrieved from IP Fabric. You can filter by hostname, site, vendor, or any other criteria to limit the scope of synchronization. Filters are applied to specific endpoints and can be combined using AND/OR logic.

Transform Maps

Transform Maps define how data from IP Fabric should be mapped to NetBox objects. They specify field mappings, transformations, and business logic for converting IP Fabric data into NetBox format. For example, they determine how an IP Fabric device becomes a NetBox device with the correct site, role, and attributes.

Syncs

Syncs are the main configuration objects that tie everything together. A sync defines:

  • Which source to pull data from
  • Which snapshot to use
  • What filters to apply
  • Which transform maps to use
  • What data types to synchronize
  • Whether to automatically merge changes

Ingestions

Ingestions are the actual execution instances of a sync. Each time you run a sync, it creates a new ingestion that:

  1. Creates an isolated branch in NetBox.
  2. Queries IP Fabric for data.
  3. Transforms and stages changes in the branch.
  4. Presents the changes for review or automatic merge.

Think of syncs as templates and ingestions as the actual runs of those templates.

Branches

Branches (from the NetBox Branching plugin) provide isolated workspaces where changes are staged before being applied to the live NetBox database. This is similar to Git branches - you can review, modify, or discard changes before merging them into production.

The Synchronization Workflow

Here’s how the complete synchronization process works, step by step:

Phase 1: Configuration

  1. Configure a Source: Connect to your IP Fabric instance by providing the URL and API token.
  2. Sync Snapshots: Pull the list of available snapshots from IP Fabric.
  3. Create Filters (optional): Define which data you want to retrieve by specifying filter criteria.
  4. Configure Transform Maps (optional): Define or customize how IP Fabric data maps to NetBox objects.
  5. Create a Sync: Combine all the components above into a sync configuration.

Pre-configured Defaults

The plugin comes with default Filters and Transform Maps pre-configured during installation. You can use these out-of-the-box configurations to get started quickly, or customize them to match your specific requirements.

Phase 2: Data Retrieval

When you execute a sync (creating an ingestion):

  1. Sync Snapshots: Before retrieving any data, the plugin syncs all available snapshots from the IP Fabric source. This ensures that if you’re using the $last snapshot, it always references the most recent discovery data.
  2. Identify Endpoints: Based on the enabled data types in your sync configuration, the plugin determines which IP Fabric endpoints to query.
  3. Apply Filters: All filters associated with the sync are combined and applied to each relevant endpoint. Filters for the same endpoint are merged according to their filter type (AND/OR).
  4. Apply Filters: All filters associated with the sync are combined and applied to each relevant endpoint. Filters for the same endpoint are merged according to their filter type (AND/OR).
  5. Query IP Fabric: The plugin makes API calls to IP Fabric, retrieving data from the specified snapshot with the applied filters.
  6. Receive Raw Data: IP Fabric returns the requested data in JSON format.

Phase 3: Transformation

  1. Match Transform Maps: For each data type, the plugin identifies the appropriate transform map based on the transform map groups configured in your sync.
  2. Apply Field Mappings: The plugin uses the transform map’s field mappings to convert IP Fabric data structures to NetBox data structures.
  3. Execute Transformations: Any custom transformation logic defined in the transform maps is executed (e.g., hostname modifications, conditional field assignments).
  4. Resolve Dependencies: The plugin ensures objects are created in the correct order (e.g., manufacturers before device types, device types before devices).
  5. Handle Relationships: Foreign key relationships are resolved to ensure objects reference the correct NetBox IDs.

Phase 4: Staging (Branch Creation)

  1. Create Branch: A new NetBox branch is created for this ingestion, providing an isolated workspace.
  2. Stage Changes: All proposed changes (creates, updates, deletes) are written to the branch, not the production database.
  3. Track Issues: Any problems encountered during transformation (missing required fields, validation errors) are logged as ingestion issues.
  4. Calculate Diffs: For each object, the plugin calculates the differences between the proposed state and current state.

Phase 5: Review and Merge

  1. Review Changes: Navigate to the ingestion detail page to see all proposed changes, organized by object type and action (create/update/delete).
  2. Inspect Diffs: Click individual objects to see detailed diffs showing what will change.
  3. Handle Issues: Review any ingestion issues and decide how to address them.
  4. Merge or Discard:
    • If auto-merge is enabled, changes are automatically applied once the ingestion completes successfully.
    • If auto-merge is disabled, you manually review and merge changes using the Merge button.
    • You can also choose to discard the branch if the changes aren’t suitable.

Phase 6: Completion

  1. Apply Changes: When merged, all changes from the branch are committed to the NetBox production database.
  2. Update Objects: NetBox objects are created, updated, or deleted according to the staged changes.
  3. Clean Up: The branch can be optionally deleted after successful merge.
  4. Record History: The ingestion remains in the system as a historical record of what was synchronized and when.

Data Flow Example

Let’s walk through a practical example of synchronizing devices from IP Fabric to NetBox:

  1. You create a sync configured to synchronize devices, filtering for only devices at the “HQ” site.
  2. You execute the sync, creating an ingestion.
  3. The plugin queries IP Fabric’s /inventory/devices endpoint with a filter for siteName = "HQ".
  4. IP Fabric returns 50 devices from the HQ site.
  5. Transform maps process each device, creating prerequisite objects first:
    • Site: Maps siteName → Site name and slug
    • Manufacturer: Maps vendor → Manufacturer name and slug
    • Platform: Maps vendor and family → Platform name and slug, links to Manufacturer
    • Device Type: Maps model → DeviceType model and slug, links to Manufacturer
    • Device Role: Maps devType → DeviceRole name and slug
    • Virtual Chassis (if applicable): Maps master hostname, links to master Device
    • Device:
      • Field mappings:
        • hostname → name
        • sn → serial
        • virtual_chassis.member → vc_position (if part of stack)
      • Relationship mappings:
        • Links to Virtual Chassis (if part of stack)
        • Links to Platform (based on vendor and family)
        • Links to Site (based on siteName)
        • Links to Device Type (based on model or vendor/family/platform)
        • Links to Device Role (based on devType)
  6. Changes are staged in a branch:
    • 5 new devices to create (along with any new Sites, Manufacturers, Platforms, Device Types, and Device Roles)
    • 40 existing devices to update
    • 5 devices to skip (no changes)
  7. You review the changes in the ingestion detail page and see the diffs.
  8. You click Merge, and the 5 new devices are created and 40 devices are updated in NetBox.

Scheduling and Automation

Syncs can be executed on-demand or scheduled to run automatically:

  • Manual Execution: Run a sync whenever you need updated data by clicking the Sync button.
  • One-Time Schedule: Schedule a sync to run at a specific date and time in the future.
  • Recurring Schedule: Configure syncs to run automatically at regular intervals (e.g., daily, weekly).

Best Practice for Recurring Syncs

When using recurring syncs, configure them to use the $last snapshot. This ensures each sync run uses the most recent IP Fabric discovery data automatically.

Error Handling and Troubleshooting

The sync process includes robust error handling:

  • Ingestion Issues: Validation errors, missing dependencies, and transformation problems are logged as issues attached to the ingestion. These don’t stop the sync but allow you to identify and fix problems.
  • Job Failures: If a critical error occurs, the ingestion job fails and the branch is preserved for investigation.
  • Logs: Detailed logs are available for each ingestion, showing exactly what happened during the sync process.
  • Status Tracking: Ingestions progress through status states (New → Queued → Syncing → Merging → Completed/Failed), helping you understand where the process is.

For more detailed troubleshooting guidance, see the Troubleshooting page.

Next Steps

Now that you understand the high-level sync process, you can:

  1. Configure Your First Source: Connect to your IP Fabric instance.
  2. Understand Data Mapping: Learn how IP Fabric data maps to NetBox models.
  3. Create Filters: Control which data to synchronize.
  4. Customize Transform Maps: Adjust how data is transformed.
  5. Set Up Syncs: Configure and execute your first synchronization.
  6. Review Ingestions: Monitor and merge your synchronized data.