Skip to content

Guard VPD skip reporting behind discovery flag - #410

Merged
almaslennikov merged 1 commit into
Mellanox:mainfrom
almaslennikov:fix-vpd-skip-reporting
Jul 13, 2026
Merged

Guard VPD skip reporting behind discovery flag#410
almaslennikov merged 1 commit into
Mellanox:mainfrom
almaslennikov:fix-vpd-skip-reporting

Conversation

@almaslennikov

Copy link
Copy Markdown
Collaborator

Summary

  • keep VPD discovery failures out of SkippedDevices() unless SKIP_DEVICE_ON_DISCOVERY_ERROR=true
  • preserve existing enabled behavior where VPD failures are reported as skipped devices when the flag is set
  • add regression coverage for both default and enabled VPD failure behavior

Testing

  • go test ./pkg/devicediscovery/... -v
  • go build ./...
  • git diff --check

Note: this follows up on the release backport review finding so main and the backport keep the same opt-in semantics.

@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates VPD discovery failure handling for skipped-device reporting. The main changes are:

  • Adds a separate incomplete-device report for discovery failures that should preserve existing CRs.
  • Keeps VPD failures out of SkippedDevices() unless SKIP_DEVICE_ON_DISCOVERY_ERROR=true.
  • Updates reconciliation to preserve CRs for both skipped and incomplete devices.
  • Adds tests for default and enabled VPD failure behavior.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
pkg/devicediscovery/devicediscovery.go Adds incomplete-device tracking and keeps default VPD failures separate from skipped-device reporting.
internal/controller/devicediscovery_controller.go Uses incomplete and skipped device reports to preserve existing CRs when discovery omits a physical device.
pkg/devicediscovery/devicediscovery_test.go Adds tests for VPD failures with default and enabled skipped-device reporting.
internal/controller/devicediscovery_controller_test.go Adds controller coverage for preserving CRs when discovery reports an incomplete device.

Reviews (3): Last reviewed commit: "Guard VPD skip reporting behind discover..." | Re-trigger Greptile

Comment on lines +101 to +103
if skipDeviceOnDiscoveryError {
d.skipPhysicalDevice(statuses, pciKey, err)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Partial Device Survives VPD Failure

When one port of a multi-port NIC has already populated statuses[pciKey] and a later port fails GetVPD, this branch no longer removes the partially built physical device unless reporting is enabled. Discovery can return a NicDevice with only the earlier ports, so the controller reconciles a truncated device instead of treating the physical NIC as incompletely discovered.

Comment on lines +101 to +103
if skipDeviceOnDiscoveryError {
d.skipPhysicalDevice(statuses, pciKey, err)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Skipped Device Becomes Deletable

For a single-port NIC with an existing NicDevice CR, a GetVPD failure now returns no observed device and leaves SkippedDevices() empty when the flag is unset. The controller path that preserves skipped devices will not run, so the next reconcile can delete the CR for a physically present device that was only omitted because VPD discovery failed.

@almaslennikov
almaslennikov force-pushed the fix-vpd-skip-reporting branch from 431b612 to b6154fd Compare July 13, 2026 09:38
@almaslennikov

Copy link
Copy Markdown
Collaborator Author

Addressed Greptile's VPD cleanup finding and force-pushed amended commit b6154fd:

  1. Partial Device Survives VPD Failure: split partial-status cleanup from skip reporting. A VPD failure now always removes any partially assembled physical device from statuses, so discovery cannot return a truncated multi-port NicDevice.
  2. Skipped Device Becomes Deletable: kept CR preservation opt-in. SkippedDevices() is still populated for VPD failures only when SKIP_DEVICE_ON_DISCOVERY_ERROR=true, matching the flag semantics while preserving the cleanup behavior above.

Validation: go test ./pkg/devicediscovery/... -v, go build ./..., and git diff --check are clean locally.

Comment on lines +104 to 106
d.skippedDevices[pciKey] = err
}
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 CR deletion remains

When GetVPD fails with SKIP_DEVICE_ON_DISCOVERY_ERROR unset, this branch removes the device from statuses but leaves skippedDevices empty. The controller then receives no observed device and no skipped marker for the PCI key, so an existing NicDevice CR for a physically present NIC can be treated as removed hardware and deleted. This is reachable for a single-port NIC whose VPD read fails, and for a multi-port NIC when a later port fails after an earlier port populated statuses. The discovery result can stay hidden from SkippedDevices() for reporting, but reconciliation still needs a preservation signal for devices omitted because discovery was incomplete.

Signed-off-by: Alexander Maslennikov <amaslennikov@nvidia.com>
@almaslennikov
almaslennikov force-pushed the fix-vpd-skip-reporting branch from b6154fd to 455acc6 Compare July 13, 2026 09:45
@almaslennikov

Copy link
Copy Markdown
Collaborator Author

Addressed the new Greptile P1 and force-pushed amended commit 455acc6:

  1. CR deletion remains: split discovery omission reporting from reconciliation preservation. SkippedDevices() stays controlled by SKIP_DEVICE_ON_DISCOVERY_ERROR, while the new IncompleteDevices() signal tells the controller to preserve existing CRs for devices omitted because discovery could not build complete status.
  2. Partial/truncated devices: VPD failures now always mark the physical PCI key incomplete and drop any partially assembled status, so multi-port NICs cannot be reconciled with only the successfully discovered ports.
  3. Controller behavior: the device discovery controller now preserves CRs for IncompleteDevices() and still supports the previous SkippedDevices() signal for compatibility.

Validation: go test ./pkg/devicediscovery/... -v, go test -c ./internal/controller -o /tmp/nco-controller-tests-main-followup.test, go build ./..., and git diff --check are clean locally.

@almaslennikov
almaslennikov merged commit c7cb556 into Mellanox:main Jul 13, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant