Skip to content

Transform Maps

By default, we provide users with the option to create their own transform map. This allows users to customize the transform map to their specific needs. However, we also offer a default transform map that can serve as a starting point. These come pre-installed using Django migrations, executed during installation process. They cover common usage scenarios and can be restored at any time. We strongly suggest to not change these and instead use transform map groups for any custom maps.

Restoring Transform Maps without Group

Within the transform map page, you can easily restore the default transform maps. This will overwrite any existing transform maps without a group. You can do this by selecting restore in the top-right corner of the transform map page.

Transform Maps

In this scenario, envision a transform map as akin to a carefully crafted recipe. This recipe serves as a guide, orchestrating the process of extracting data from IP Fabric and delicately depositing it into the appropriate “containers” represented by Django models within NetBox. Its primary objective is to guarantee precise and efficient organization of the data, meticulously tailored to meet your network management requirements. In essence, it acts as a bridge, harmonizing disparate data formats and structures, ensuring a seamless integration of your network data with your NetBox database.

The concept of transform maps is not novel; it has been widely employed in various tools for transforming data from one format to another, a practice commonly referred to as ETL (Extract, Transform, Load). This concept draws substantial inspiration from ServiceNow and their implementation of Transform Maps.

The IP Fabric plugin has four key transform map areas:

  • Transform Map Groups: This is a grouping of transform maps that can be used to organize and apply them to specific sync jobs.
  • Transform Map: This is the parent object that contains information about the source and destination of the data.
  • Transform Map Fields: This is the mapping between the source fields and destination fields.
  • Transform Map Relationship Fields: This is the mapping between the source fields and destination fields for relationships.

Transform Map Group

Transform Map Groups are used to group transform maps together. This is useful when you want to apply the same transform map to multiple sync jobs. For example, you can create a group for all device-related transform maps and apply it to all device sync jobs.

Transform Map with group assigned takes precedence over transform maps without a group assigned. This means that if you have a transform map with a group assigned, it will be used instead of the transform map without a group assigned when this group is added to Sync settings.

Parameters

Field Description Type
Name Name of the transform map group CharField
Description Text describing this group CharField

Transform Map

This model serves as the parent for both Transform Map Fields and Transform Map Relationship Fields. It empowers users to select a supported IP Fabric endpoint and a supported target NetBox model.

The combination of group and target_model is unique, meaning that you can only have one transform map for a specific group and target model. If you attempt to create a new transform map with the same group and target model, it will raise an error.

Parameters

Field Description Type
Name Name of the transform map (e.g., Device Transform Map) CharField
Group (optional) IPFabric Transform Map Group ChoiceField
Source Endpoint IP Fabric Endpoint ForeignKey
Target Model NetBox Model (Content Type format, e.g., DCIM | Device) ChoiceField
Parent (optional) Parent transform map for hierarchical ordering ForeignKey
Source Endpoint

Source Endpoint is a foreign key relationship to the IPFabricEndpoint model, which defines the IP Fabric API endpoint to fetch data from. Each endpoint represents an IP Fabric table with its associated API path.

Available endpoints include:

  • Site (/inventory/sites/overview)
  • Inventory (/inventory/part-numbers)
  • Device (/inventory/devices)
  • Virtual Chassis (/technology/platforms/stack/members)
  • Interface (/inventory/interfaces)
  • VLAN (/technology/vlans/site-summary)
  • VRF (/technology/routing/vrf/detail)
  • Prefix (/technology/networks/managed-networks)
  • IP Address (/technology/addressing/managed-ip)

Note

IP Fabric Endpoints are currently read-only and are pre-configured during installation. They cannot be created, modified, or deleted through the user interface.

Target Model Choices

Target models are displayed in Content Type format: APP | Model Name

Available target models include:

  • DCIM | Site
  • DCIM | Manufacturer
  • DCIM | Platform
  • DCIM | Device Role
  • DCIM | Device Type
  • DCIM | Device
  • DCIM | Virtual Chassis
  • DCIM | Interface
  • DCIM | MAC Address
  • IPAM | VLAN
  • IPAM | VRF
  • IPAM | Prefix
  • IPAM | IP Address
  • Content Types | Content Type

Parent Relationship

The Parent field establishes hierarchical relationships between transform maps, controlling the order in which objects are created during synchronization. A transform map can have multiple parents, ensuring that all prerequisite objects exist before dependent objects are created.

How it works:

  • Transform maps without parents are processed first (root level).
  • Transform maps with parents are processed only after ALL their parents have been processed.
  • This creates a dependency chain (Directed Acyclic Graph/DAG) ensuring proper object creation order.
  • The system automatically detects and prevents circular dependencies.

Real-world example - IP Address with multiple parents:

An IP Address in NetBox has two key dependencies:

  1. Interface - where the IP is assigned (assigned_object).
  2. VRF - which routing table it belongs to (vrf field).

Therefore, the IP Address Transform Map has TWO parents: ["dcim.interface", "ipam.vrf"].

This ensures that:

  • Interfaces are created before IP addresses (so we have something to assign to).
  • VRFs are created before IP addresses (so the routing context exists).
  • IP Address creation only happens when BOTH dependencies are satisfied.

Example hierarchy from default transform maps:

Site (no parent)
├── Manufacturer (parent: site)
│   ├── Platform (parent: manufacturer)
│   ├── Device Type (parent: manufacturer)
│   │   └── Device (parent: devicetype)
│   │       ├── Inventory Item (parent: device)
│   │       └── Interface (parent: device)
│   │           ├── MAC Address (parent: interface)
│   │           └── IP Address (parents: interface, vrf) ← Multiple parents!
│   └── Virtual Chassis (parent: manufacturer)
├── Device Role (parent: site)
└── VLAN (parent: site)
    └── VRF (parent: vlan)
        ├── Prefix (parent: vrf)
        └── IP Address (parents: interface, vrf) ← Multiple parents!

Note: IP Address has two parent branches - it requires Interface (from the Device hierarchy) AND VRF (from the VLAN hierarchy). The topological sort algorithm ensures it’s only processed once, after BOTH parents complete.

Processing algorithm:

The plugin uses topological sort (Kahn’s algorithm) to:

  1. Identify all root nodes (transform maps with no parents).
  2. Process nodes level-by-level in breadth-first order.
  3. Only process a node when ALL its parents are complete.
  4. Detect circular dependencies automatically.

Circular dependency prevention:

The system prevents invalid configurations like:

  • Transform Map A has parent B
  • Transform Map B has parent C
  • Transform Map C has parent A ❌ (Creates a cycle - validation error)

Circular Dependencies

When adding parents to a transform map, the system validates that no circular dependency is created. If you attempt to create a cycle, you’ll receive a validation error before the change is saved.

Customizing Hierarchies

When creating custom transform maps in a Transform Map Group:

  • Set ALL parent relationships that reflect the NetBox foreign key dependencies.
  • A transform map can have 0, 1, or many parents.
  • Review the relationship_maps to identify which models are referenced.
  • For IP Address: has relationships to Interface AND VRF → needs both as parents.
  • Order among parents doesn’t matter - all must complete first.
  • Incorrect parent relationships may cause sync failures due to missing foreign key references.

Cloning Transform Map

You can clone a transform map by clicking the Clone button in the top-right corner of the transform map detail page. This will create a new transform map with the same source endpoint and target model. Cloning of related Fields and Relationships is enabled by default.

Clone Transform Map

Transform Field

Transform fields belong to a transform map and are used to map the source fields to the destination fields. The source fields are the fields from the IP Fabric table and the destination fields are the fields from the NetBox model.

Parameters

Field Description Type
Transform Map Transform map the field is associated with ForeignKey (IPFabricTransformMap)
Source Field IP Fabric Table CharField
Target Field NetBox Model CharField
Coalesce Determine if field should be used to match existing objects BooleanField
Template Jinja Template to transform the data during ingestion TextField

Source Field

While the source field is defined as a CharField, it’s important to note that the form restricts your selection to fields that are supported by the IP Fabric API. The list of available choices can be found in the table description within your IP Fabric instance, and it’s limited to the models mentioned above.

Target Field

The target field is a CharField and the form will only allow you to select fields that are supported by the NetBox Model and are dynamically generated based on the target model selected in the transform map.

Coalesce

This term, also employed in ServiceNow, refers to a field that determines whether it should be utilized for identifying and matching existing objects. For instance, when importing a device and aiming to match it based on the hostname, you would set the “coalesce” parameter to true. This would enable the matching process based on the hostname, and, if a device with the same hostname already exists, it would be updated accordingly.

Template

This field provides the capability to employ a Jinja Template for data transformation during the ingestion process. It proves valuable when you wish to modify the data before inserting it into NetBox. For instance, you can use it to alter the data format or combine multiple fields into a single field, depending on your specific requirements.

Info

If template is used the source field is ignored.

The Jinja template has access to the following variables via context data:

  • <fieldname> – The value of the source field (e.g., {{ siteName }})
  • object – The full raw source dictionary from IP Fabric
    • Example: Raw data from IP Fabric for a site 
      {
        "siteName": "Site 1",
        "devicesCount": 173,
        "usersCount": 7075,
        "stpDCount": 10,
        "switchesCount": 165,
        "vlanCount": 113,
        "rDCount": 2,
        "routersCount": 12,
        "networksCount": 77
}
    • You could select items out of this object like so: {{ object.siteName }}
  • The template can access netBox model classes. For example:
    • {{ dcim.Site.objects.filter(sn=object.sn) }}
  • When None is returned from the template on coalesce field/relationship field, field=None will be replaced with field__isnull=True instead to be able to query for fields allowing null value. If you need an empty value, use "" instead of None.

Transform Relationship Field

Transform Relationship fields are an integral part of a transform map, designed to establish connections between various objects within NetBox. These fields encompass source fields, which originate from the IP Fabric table, and destination fields, which correspond to the fields within the NetBox model. For instance, in the context of importing a device and desiring to establish a relationship with a specific site, a transform relationship field becomes essential to achieve this linkage.

Parameters

Field Description Type
Transform Map Transform map the field is associated with ForeignKey (IPFabricTransformMap)
Source Model NetBox model that will be mapped ForeignKey
Target Field Field where the NetBox model will be placed. CharField
Coalesce Determine if field should be used to match existing objects BooleanField
Template Jinja Template to transform the data during ingestion TextField

Source Model

The source model only allows you to select a value that is supported. When objects are imported using this plugin, there is a specific order needed based on the parent relationship hierarchy defined in the transform maps. For example, when importing a device, the order is:

  • Create site
  • Create manufacturer
  • Create device type
  • Create platform
  • Create device role
  • Create device

This order is automatically determined by the parent relationships configured in transform maps. When we create or query for each of these objects, we store them in a cache to facilitate their use by the relationship field for creating connections. For instance, when importing a device and aiming to establish a relationship with a site, you would employ a transform relationship field. In this scenario, the source model would be ‘site,’ and the target field would also be ‘site.’

Target Field

This field is a CharField and the form will only allow you to select fields that are relationship fields on the NetBox model. This is dynamically generated based on the target model selected in the transform map.

Coalesce

This field, also referred to in ServiceNow, signifies whether the field should be utilized to identify and match existing objects. For instance, if you’re importing a device and wish to match it based on the hostname, you would set the “coalesce” parameter to true. This proves valuable when you need to retrieve or create an object in NetBox, especially when a standard field lacks the uniqueness required for matching. For example, an interface like GE-1/0/1 may not be unique to just one device; it could exist on multiple devices. To uniquely identify and query, you must specify that you are searching for GE-1/0/1 specifically on device1.

Template

Much like the transform field, this field enables the utilization of a Jinja Template to manipulate data during the ingestion process. This proves valuable when you prefer not to rely on cached objects related to devices and need to execute a more intricate query. The template query can return an integer or values such as True, False and None. If an integer is returned it is combined with the source_model, to facilitate the establishment of a relationship between the resulting object and the target model. If a value such as True, False or None is returned, it will insert these values into the target field. If the template returns an empty string, it will not create a relationship.

Info

If template is used, the source model cache is ignored.

The Jinja template has access to the following variables via context data:

  • object – The full raw source dictionary from IP Fabric
    • Example: Raw data from IP Fabric for a site 
      {
        "siteName": "Site 1",
        "devicesCount": 173,
        "usersCount": 7075,
        "stpDCount": 10,
        "switchesCount": 165,
        "vlanCount": 113,
        "rDCount": 2,
        "routersCount": 12,
        "networksCount": 77
}
    • You could select items out of this object like so: {{ object.siteName }}
  • NetBox model classes can be accessed by the template. For example:
    • {{ dcim.Device.objects.filter(site=object.siteName).first().pk }}

Flow Chart

flowchart TD
    A[Sync] -->B(Fetch IP Fabric Data)
    subgraph TM[Transform Map]
      direction TB
      TF[Transform Field] --> RESULT[Result]
      RF[Relationship Field] --> RESULT[Result]
    end
    B --> TM
    TM --> TF
    TM --> RF