NVIDIA NIC Configuration Operator provides Kubernetes API(Custom Resource Definition) to allow FW configuration on Nvidia NICs in a coordinated manner. It deploys a configuration daemon on each of the desired nodes to configure Nvidia NICs there. NVIDIA NIC Configuration Operator uses the Maintenance Operator to prepare a node for maintenance before the actual configuration.

• Kubernetes cluster
• NVIDIA Network Operator deployed. It is recommended to deploy the DOCA-OFED driver
• Maintenance Operator deployed

NVIDIA NIC Configuration Operator can be deployed as part of the NIC Cluster Policy CRD.

The NICConfigurationTemplate CRD is used to request FW configuration for a subset of devices.

NIC Configuration Operator will select NIC devices in the cluster that match the template's selectors and apply the configuration spec to them.

If more than one template matches a single device, none will be applied and the error will be reported in all of their statuses.

For more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
   name: connectx6-config
   namespace: network-operator
spec:
   nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
   nicSelector:
      # nicType selector is mandatory the rest are optional. Only a single type can be specified.
      nicType: 101b
      pciAddresses:
         - "0000:03:00.0"
         - “0000:04:00.0”
      serialNumbers:
         - "MT2116X09299"
      partNumbers:
         - "MCX713106AEHEA_QP1"
   resetToDefault: false # if set, template is ignored, device configuration should reset
   template:
      numVfs: 2
      linkType: Ethernet
      pciPerformanceOptimized:
         enabled: true
         maxReadRequest: 4096
      roceOptimized:
         enabled: true
         qos:
            trust: dscp
            pfc: "0,0,0,1,0,0,0,0"
            tos: 0
      gpuDirectOptimized:
         enabled: true
         env: Baremetal

• numVFs: if provided, configure SR-IOV VFs via nvconfig.
  • This is a mandatory parameter.
  • E.g: if numVFs=2 then SRIOV_EN=1 and SRIOV_NUM_OF_VFS=2.
  • If numVFs=0 then SRIOV_EN=0 and SRIOV_NUM_OF_VFS=0.
• linkType: if provided configure linkType for the NIC for all NIC ports.
  • This is a mandatory parameter.
  • E.g linkType = Infiniband then set LINK_TYPE_P1=IB and LINK_TYPE_P2=IB if second PCI function is present
• pciPerformanceOptimized: performs PCI performance optimizations. If enabled then by default the following will happen:
  • Set PCI max read request size for each PF to 4096 (note: this is a runtime config and is not persistent)
  • Users can override the runtime value via maxReadRequest
  • maxAccOutRead is deprecated and ignored; use rawNvConfig for explicit MAX_ACC_OUT_READ management if needed.
• roceOptimized: performs RoCE related optimizations. If enabled performs the following by default:
  • Nvconfig set for both ports (can be applied from PF0)
    • Conditionally applied for second port if present
      • ROCE_CC_PRIO_MASK_P1=255, ROCE_CC_PRIO_MASK_P2=255
      • CNP_DSCP_P1=4, CNP_DSCP_P2=4
      • CNP_802P_PRIO_P1=6, CNP_802P_PRIO_P2=6
  • Configure pfc (Priority Flow Control) for priority 3, set trust to dscp on each PF, set ToS (Type of Service) to 0.
    • Non-persistent (need to be applied after each boot)
    • Users can override values via trust, pfc and tos parameters
  • Can only be enabled with linkType=Ethernet
• gpuDirectOptimized: performs gpu direct optimizations. ATM only optimizations for Baremetal environment are supported. If enabled perform the following:
  • Set nvconfig ATS_ENABLED=0
  • Can only be enabled when pciPerformanceOptimized is enabled
  • Both the numeric values and their string aliases, supported by NVConfig, are allowed (e.g. REAL_TIME_CLOCK_ENABLE=False, REAL_TIME_CLOCK_ENABLE=0).
  • For per port parameters (suffix _P1, _P2) parameters with _P2 suffix are ignored if the device is single port.
• spectrumXOptimized: enables Spectrum-X specific NIC optimizations. When enabled:
  • Requires linkType=Ethernet and numVfs=1
  • Cannot be combined with roceOptimized (RoCE settings are included automatically)
  • Can be combined with rawNvConfig — raw params are merged as overrides on top of Spectrum-X calculated params
  • Only supported on ConnectX-8 (nicType: 1023), ConnectX-9 (nicType: 1025) and BlueField-3 SuperNIC (nicType: a2dc)
  • version: Required. Must match the name of a Spectrum-X profile ConfigMap
  • overlay: Optional, default none. Set to l3 for L3 EVPN overlay
  • multiplaneMode: Optional, default none. Options: none, swplb, hwplb, uniplane
  • numberOfPlanes: Optional, default 1. Options: 1, 2, or 4
• If a configuration is not set in spec, its non-volatile configuration parameters (if any) should be set to device default.

The NIC Configuration Operator supports Spectrum-X-specific NIC configuration through Spectrum-X profile ConfigMaps. A profile ConfigMap contains the Spectrum-X YAML profile consumed by the daemon at runtime.

To create a Spectrum-X profile ConfigMap:

1. Create one ConfigMap per profile.
2. Set the ConfigMap name to the value that will be used in template.spectrumXOptimized.version.
3. Add the label network.nvidia.com/operator.nic-configuration.spectrum-x-profile. The label value is ignored; only the label key must be present.
4. Put the complete Spectrum-X profile YAML under data.profile.

The profile ConfigMap can be created in any namespace watched by the operator.

Warning: If two labeled ConfigMaps in different namespaces share the same name, they define the same Spectrum-X version key and the latest-reconciled ConfigMap silently wins with no error. To avoid unpredictable behavior, use unique ConfigMap names across all watched namespaces.

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-spectrum-x-profile
  namespace: nvidia-network-operator
  labels:
    network.nvidia.com/operator.nic-configuration.spectrum-x-profile: ""
data:
  profile: |
    useSoftwareCCAlgorithm: true
    docaCCVersion: "example-version"
    mlxConfig:
      none:
        "1023":
          postBreakout:
            EXAMPLE_NVCONFIG_PARAMETER: "example-value"
    runtimeConfig:
      roce:
        - name: Example RoCE runtime parameter
          value: "example-value"
          valueType: string
          dmsPath: "<dms-path-for-runtime-parameter>"
      adaptiveRouting:
        - name: Example mlxreg runtime parameter
          value: "0x00000001"
          mlxreg:
            register: ROCE_ACCL
            field: "<mlxreg-field-to-check>"
            setFields:
              - name: "<mlxreg-field-to-set>"
                value: "0x1"
              - name: "<mlxreg-field-select-to-set>"
                value: "0x1"

Reference the profile from a NicConfigurationTemplate by using the ConfigMap name as the Spectrum-X version:

spectrumXOptimized:
  enabled: true
  version: "example-spectrum-x-profile"
  overlay: "none"
  multiplaneMode: "none"
  numberOfPlanes: 1

Supported NIC types for Spectrum-X:

• ConnectX-8 (device ID 1023) -- supports all multiplane modes
• ConnectX-9 (device ID 1025) -- supports all multiplane modes (same configuration as ConnectX-8)
• BlueField-3 SuperNIC (device ID a2dc) -- supports all multiplane modes except hwplb

Spectrum-X profiles can configure NICs with multiple data planes. Available modes:

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-multiplane-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
      nicType: "1023" # ConnectX-8. Use "1025" for ConnectX-9, or "a2dc" for BlueField-3 SuperNIC (hwplb not supported on BF3)
      # partNumbers:
      #   - "MCX713106AEHEA_QP1"
  template:
      numVfs: 1
      linkType: Ethernet
      spectrumXOptimized:
          enabled: true
          version: "RA2.1"
          overlay: "none"
          multiplaneMode: "hwplb" # Hardware Packet Load Balancing, ConnectX-8 only
          numberOfPlanes: 4

The NICFirmwareSource CR represents a list of url sources with NIC FW binaries archives.

To enable the NIC FW upgrade feature, NicFirmwareStorage section should be provided in the NIC Cluster Policy. There is an option to create a new PVC or use the existing one. Firmware binaries will be provisioned by a provisioner controller which will watch for NICFirmwareSource obj and provision the binaries in a shared volume enabled by the given storage class. Node agents will need to make sure that the reference NICFirmwareSource object is fully reconciled (status.state == Success) before proceeding with firmware update.

To set up a persistent NFS storage in the cluster, the example from the CSI NFS Driver repository might be used. After deploying the NFS server and NFS CSI driver, the storage class should be deployed. The name of the storage class can then be passed to the NIC Cluster Policy.

# Change this template first as it only provides an example of configuration
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: connectx-6dx-firmware-22-44-1036
  namespace: network-operator
  finalizers:
    - configuration.net.nvidia.com/nic-configuration-operator
spec:
  # a list of firmware binaries from mlnx website if they are zipped try to unzip before placing
  binUrlSources:
    - https://www.mellanox.com/downloads/firmware/fw-ConnectX6Dx-rel-22_44_1036-MCX623106AC-CDA_Ax-UEFI-14.37.14-FlexBoot-3.7.500.signed.bin.zip
  bfbUrlSource: https://example.com/bluefield3-31.41.0.bfb
status:
  state: Success
  binaryVersions:
    22.44.1036:
    - mt_0000000436
  bfbVersions:
    a2dc: 34.41.0 # BF3 NIC FW
    a2d6: 25.21.0 # BF2 NIC FW

The NICFirmwareTemplate CRD is used to request FW validation or update from a referenced NICFirmwareSource for a subset of devices.

NIC Configuration Operator will select NIC devices in the cluster that match the template's selectors and apply the configuration spec to them.

If more than one template matches a single device, none will be applied and the error will be reported in all of their statuses.

For more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareTemplate
metadata:
  name: connectx6dx-config
  namespace: network-operator
spec:
  nodeSelector:
    kubernetes.io/hostname: cloud-dev-41
  nicSelector:
    nicType: "101d"
    # partNumbers:
    #   - "MCX713106AEHEA_QP1"
  template:
    nicFirmwareSourceRef: connectx6dx-firmware-22-44-1036
    updatePolicy: Update

The NICDevice CRD is created automatically by the configuration daemon and represents a specific NVIDIA NIC on a specific K8s node. The name of the device combines the node name, device type and its PCI device address (Domain:Bus:Device, with : and . replaced by -). PCI device address is used for uniqueness because serial numbers are not guaranteed unique on systems with embedded NICs that share a flashed VPD image (e.g. HGX B300).

FirmwareUpdateInProgress status condition can be used for tracking the state of the FW validation/update on a specific device. If an error occurs during FW update, it will be reflected in this field. ConfigUpdateInProgress status condition can be used for tracking the state of the FW configuration update on a specific device. If an error occurs during FW configuration update, it will be reflected in this field.

for more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicDevice
metadata:
   name: co-node-25-101b-0000-04-00
   namespace: nic-configuration-operator
spec:
   firmware:
     nicFirmwareSourceRef: connectx6dx-firmware-22-44-1036
     updatePolicy: Update
   configuration:
      template:
         linkType: Ethernet
         numVfs: 8
         pciPerformanceOptimized:
            enabled: true
status:
   conditions:
      - reason: DeviceFirmwareConfigMatch
        type: FirmwareUpdateInProgress
        status: "False"
        message: Firmware matches the requested version
      - reason: UpdateSuccessful
        type: ConfigUpdateInProgress
        status: "False"
   firmwareVersion: 20.42.1000
   node: co-node-25
   partNumber: mcx632312a-hdat
   ports:
      - networkInterface: enp4s0f0np0
        fwctlDevice: /dev/fwctl/fwctl0
        pci: "0000:04:00.0"
        rdmaInterface: mlx5_0
      - networkInterface: enp4s0f1np1
        fwctlDevice: /dev/fwctl/fwctl1
        pci: "0000:04:00.1"
        rdmaInterface: mlx5_1
   psid: mt_0000000225
   serialNumber: mt2232t13210
   type: 101b

The NicDevice CRD is created and reconciled by the configuration daemon. The reconciliation logic scheme can be found here.

The NicInterfaceNameTemplate CRD allows you to define custom naming patterns for RDMA and network device interfaces on Spectrum-X NICs. This is useful in multiplane and multi-rail deployments where predictable interface naming is required.

The operator deploys udev rules to the host to rename network and RDMA interfaces according to the specified naming template.

The template uses the following placeholders for device name construction:

• %nic_id%: The index of the NIC in the flattened list of NICs
• %plane_id%: The index of the plane of the specific NIC
• %rail_id%: The index of the rail where the given NIC belongs to

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  pfsPerNic: 2
  rdmaDevicePrefix: "rdma_%nic_id%_%plane_id%_%rail_id%"
  netDevicePrefix: "net_%nic_id%_%plane_id%_%rail_id%"
  railPciAddresses:
    - ["0000:1a:00.0", "0000:2a:00.0"]
    - ["0000:3a:00.0", "0000:4a:00.0"]

The railPciAddresses field defines the PCI address to rail mapping. The first dimension is the rail index and the second dimension is the list of PCI addresses of the NICs in that rail.

The operator generates udev rules based on the template and writes them to the host. The rules are written to two separate files:

Example generated udev rules for net devices (/etc/udev/rules.d/10-nic-net-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"

Example generated udev rules for RDMA devices (/etc/udev/rules.d/10-nic-rdma-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

ACTION=="add", KERNELS=="0000:1a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_0_0"
ACTION=="add", KERNELS=="0000:1a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_1_0"
ACTION=="add", KERNELS=="0000:3a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_0_1"
ACTION=="add", KERNELS=="0000:3a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_1_1"

To include the NIC Configuration Operator as part of network configuration workflows, strict order of operations might need to be enforced. For example, SR-IOV Network Configuration Daemon pod should start AFTER the NIC Configuration Daemon has finished. To indicate when NIC configuration is in progress to the pods that depend on it, the operator manages the nvidia.com/operator.nic-configuration.wait label, which has the value false when the requested NIC configuration has successfuly been applied, and the value true when the NIC configuration is in progress. To use this mechanism, the next pods in the pipeline can add nvidia.com/operator.nic-configuration.wait=false to their node label selectors. That way, they will automatically be evicted from the node when the NICs are being configured.

The NIC Configuration Daemon itself relies on the network.nvidia.com/operator.mofed.wait=false label to be present on the node as it requires the DOCA-OFED driver to be running for some of the configurations.

Feature flags can be enabled via environment variables in the helm chart or NVIDIA Network Operator's NicClusterPolicy. Supported flags:

• FW_RESET_AFTER_CONFIG_UPDATE=true: explicitely reset the NIC's Firmware before the reboot and after updating its non-volatile configuration. Might be required on DGX servers where configuration update is not successfully applied after the warm reboot.

• SKIP_VM_CHECK=true: pass --skip_vm_check to mlxfwreset resetting NIC firmware while running on VMs.

• SKIP_DEVICE_ON_DISCOVERY_ERROR=true: skip a physical NIC during device discovery when firmware/PSID lookup or BlueField operation-mode lookup fails. When unset, these errors fail discovery so the daemon retries. Existing NicDevice CRs are preserved for skipped devices to avoid delete/recreate churn during transient per-device discovery failures.