Skip device on discovery errors when enabled

[almaslennikov]

Summary

• add SKIP_DEVICE_ON_DISCOVERY_ERROR support for per-device discovery failures
• skip the affected physical NIC when firmware/PSID lookup or BlueField operation-mode lookup fails and the env var is true
• preserve existing NicDevice CRs for skipped physical PCI keys to avoid transient delete/recreate churn
• document the feature flag in README and chart values

Testing

• GOCACHE=/private/tmp/nco-skip-discovery-go-cache go test ./pkg/devicediscovery/... -v
• GOCACHE=/private/tmp/nco-skip-discovery-go-cache go test ./internal/controller -v --ginkgo.focus='DeviceDiscoveryController'
• GOCACHE=/private/tmp/nco-skip-discovery-go-cache go build ./...
• GOCACHE=/private/tmp/nco-skip-discovery-go-cache GOLANGCI_LINT_CACHE=/private/tmp/nco-skip-discovery-golangci-cache make lint

[greptile-apps]

Greptile Summary

This PR adds an opt-in path for skipping devices with discovery read errors. The main changes are:

• A new SKIP_DEVICE_ON_DISCOVERY_ERROR feature flag.
• Per-device skip tracking during NIC discovery.
• Whole-physical-device skipping for firmware, PSID, VPD, and BlueField mode lookup failures.
• Controller preservation of existing NicDevice objects for skipped devices.
• README, chart values, and tests for the new behavior.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
pkg/devicediscovery/devicediscovery.go	Adds skipped-device tracking and removes any partially built status for a physical NIC when a covered discovery read fails.
internal/controller/devicediscovery_controller.go	Preserves existing NicDevice objects when the matching physical PCI key was skipped in the current discovery pass.
pkg/devicediscovery/devicediscovery_test.go	Adds coverage for enabled skip behavior across single-device, multi-device, multi-port, and BlueField discovery paths.
internal/controller/devicediscovery_controller_test.go	Adds coverage for preserving an existing NicDevice object when discovery reports that device as skipped.

_{Reviews (2): Last reviewed commit: "Skip device on discovery errors when ena..." | Re-trigger Greptile}

[greptile-apps]

Physical NIC Partially Discovered

When firmware or PSID lookup fails for the first PCI function of a multi-function NIC, this continue skips only that function. A later function with the same collapsed PCI key can still initialize the device, so discovery can return a NicDevice with only the later port instead of skipping the whole physical NIC.

[greptile-apps]

BlueField Ports Can Disappear

When operation-mode lookup fails for one BlueField PCI function but succeeds for a later function with the same collapsed PCI key, this branch skips only the failed function. Discovery can still create the device from the later function, leaving the returned NicDevice with a truncated Ports list and DPU metadata from whichever function first succeeds.

[greptile-apps]

Transient Failure Deletes CRs

With the flag enabled, a transient firmware or PSID lookup failure omits the device from the returned discovery map while returning no error. The controller treats missing discovered devices as removed hardware and deletes existing NicDevice CRs, so a flaky per-device lookup can cause delete/recreate churn instead of just skipping updates for that device.

[almaslennikov]

Addressed the Greptile findings and force-pushed amended commit 55b11b3:

1. Physical NIC partially discovered: discovery now tracks skipped devices by collapsed physical PCI key and removes any already-assembled status for that key, so a later PCI function cannot create a partial NicDevice.
2. BlueField ports can disappear: the same physical-key skip handling is used for BlueField operation-mode lookup failures, preventing truncated port lists and mixed metadata from later functions.
3. Transient failure deletes CRs: DeviceDiscoveryController now consumes skipped-device reports when available and preserves existing NicDevice CRs for skipped physical PCI keys instead of treating them as removed hardware.

Validation: go test ./pkg/devicediscovery/... -v, go test ./internal/controller -v --ginkgo.focus='DeviceDiscoveryController', go build ./..., and make lint are clean locally.