Is this a new feature, an enhancement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
High
Please provide a clear description of problem this feature solves
Global admission control bounds API work, but one noisy client can still monopolize the available capacity by:
- Occupying all execution slots.
- Filling the pending queue.
- Starving the UI, controllers, monitoring, and unrelated clients.
This occurred during the 4,500-host GB300 simulation, where Machine-a-Tron generated approximately 9,000 DHCP actors.
The goal is to prevent one noisy or malfunctioning client from denying API service to other clients.
Feature Description
When multiple clients have pending requests, Carbide API shall distribute admission capacity fairly between them.
A single client shall not be able to consume all:
- Work-in-flight capacity.
- Pending-request capacity.
- Newly available admission opportunities.
Client identity must come from trusted authentication context - not the request path, TCP connection, HTTP/2 stream, or a client-provided header.
The feature must cover both native gRPC and admin HTTP traffic while preserving the existing global admission limits, timeouts, cancellation behavior, overload responses, and disable option.
Describe your ideal solution
Extend API admission control with:
- Fair scheduling between authenticated clients.
- Bounded pending capacity per client.
- A per-client work-in-flight limit.
- Reasonable ordering within each client.
- Cleanup of inactive client state.
- Shared capacity across gRPC and admin HTTP.
Reuse the bounded queue and scheduling primitives provided by nv-redfish-dispatcher. Do not implement another generic queue or round-robin scheduler in Carbide.
Client identity
Possible trusted identity sources include:
- SPIFFE or mTLS workload identity.
- Authenticated admin web identity.
- A fixed fallback identity for anonymous or permissive-mode traffic.
A client must not receive additional scheduling shares by opening more connections or HTTP/2 streams.
Document how Machine-a-Tron, controllers, monitoring, admin users, and anonymous traffic map to admission clients. Scale validation must confirm whether multiple Machine-a-Tron pods share one logical identity or appear as separate clients.
Overload behavior
When a client exhausts its pending allowance, reject the arriving request without invoking its handler.
Preserve the existing protocol behavior:
- gRPC: ResourceExhausted
- HTTP: 503 Service Unavailable
Use reject-new-work behavior for API queues. Do not evict an older request whose caller is already waiting.
Observability
Provide enough visibility to determine:
- Whether clients are being limited.
- Global and per-client admission pressure.
- Rejections caused by client versus global limits.
- Admission wait duration.
- Active client queue count.
- Whether inactive client state is being cleaned up.
Do not use raw client identities as unrestricted metric labels.
Dependencies
Describe any alternatives you have considered
No response
Additional context
This feature mitigates denial of service caused by one noisy authenticated client after its identity is known. It is not complete protection against distributed attacks using many identities, TLS/authentication exhaustion, or network-level denial of service.
Code of Conduct
Is this a new feature, an enhancement, or a change to existing functionality?
New Feature
How would you describe the priority of this feature request
High
Please provide a clear description of problem this feature solves
Global admission control bounds API work, but one noisy client can still monopolize the available capacity by:
This occurred during the 4,500-host GB300 simulation, where Machine-a-Tron generated approximately 9,000 DHCP actors.
The goal is to prevent one noisy or malfunctioning client from denying API service to other clients.
Feature Description
When multiple clients have pending requests, Carbide API shall distribute admission capacity fairly between them.
A single client shall not be able to consume all:
Client identity must come from trusted authentication context - not the request path, TCP connection, HTTP/2 stream, or a client-provided header.
The feature must cover both native gRPC and admin HTTP traffic while preserving the existing global admission limits, timeouts, cancellation behavior, overload responses, and disable option.
Describe your ideal solution
Extend API admission control with:
Reuse the bounded queue and scheduling primitives provided by nv-redfish-dispatcher. Do not implement another generic queue or round-robin scheduler in Carbide.
Client identity
Possible trusted identity sources include:
A client must not receive additional scheduling shares by opening more connections or HTTP/2 streams.
Document how Machine-a-Tron, controllers, monitoring, admin users, and anonymous traffic map to admission clients. Scale validation must confirm whether multiple Machine-a-Tron pods share one logical identity or appear as separate clients.
Overload behavior
When a client exhausts its pending allowance, reject the arriving request without invoking its handler.
Preserve the existing protocol behavior:
Use reject-new-work behavior for API queues. Do not evict an older request whose caller is already waiting.
Observability
Provide enough visibility to determine:
Do not use raw client identities as unrestricted metric labels.
Dependencies
Describe any alternatives you have considered
No response
Additional context
This feature mitigates denial of service caused by one noisy authenticated client after its identity is known. It is not complete protection against distributed attacks using many identities, TLS/authentication exhaustion, or network-level denial of service.
Code of Conduct