Enhance Metrics Collection System

Summary

Expand the metrics collection system to provide comprehensive observability for Runner-Q, taking inspiration from Temporal's metrics approach. Currently, only 4 basic counters are tracked. This enhancement will add duration metrics, gauges, histograms, and labeled metrics for better monitoring and debugging.

Current State

The system currently tracks only these metrics:

• activity_completed - Activities completed successfully
• activity_retry - Activities requesting retry
• activity_failed_non_retry - Permanent failures
• activity_timeout - Activities that timed out

Proposed Metrics

1. Activity Lifecycle Metrics (Counters)

• activity_enqueued - Total activities enqueued
• activity_dequeued - Activities dequeued from queue
• activity_started - Activities started execution
• activity_scheduled - Activities scheduled for future execution
• activity_requeued - Activities requeued (lease expiry, etc.)
• activity_dead_letter - Activities moved to dead letter queue
• activity_lease_extended - Lease extensions

2. Activity Execution Metrics (Durations/Histograms)

• activity_execution_duration - Handler execution time (histogram)
• activity_queue_wait_time - Time from enqueue to start (histogram)
• activity_total_time - End-to-end time from enqueue to completion (histogram)
• activity_retry_delay - Backoff delay before retry (histogram)
• activity_schedule_delay - Time until scheduled execution (histogram)

3. Queue Depth and Health (Gauges)

• queue_pending_count - Current pending activities
• queue_processing_count - Currently processing activities
• queue_scheduled_count - Scheduled activities count
• queue_dead_letter_count - Dead letter queue size
• queue_by_priority_{critical,high,normal,low} - Counts by priority level

4. Worker Metrics

• worker_active_count - Number of active workers
• worker_idle_count - Number of idle workers
• worker_utilization - Percentage of workers busy
• worker_dequeue_attempts - Dequeue attempts (with success/failure)
• worker_dequeue_empty - Empty queue polls

5. Retry and Failure Metrics

• activity_retry_count - Total retries (histogram by attempt number)
• activity_failure_rate - Failure rate by activity type
• activity_retry_rate - Retry rate by activity type
• activity_max_retries_exceeded - Activities that exceeded max retries

6. Redis/Queue Operations (Durations)

• redis_enqueue_duration - Enqueue operation time
• redis_dequeue_duration - Dequeue operation time
• redis_snapshot_write_duration - Snapshot write time
• redis_snapshot_read_duration - Snapshot read time
• redis_connection_pool_wait - Connection pool wait time

7. Scheduled Activities Processor

• scheduled_activities_processed - Scheduled activities moved to queue
• scheduled_activities_processor_duration - Processing cycle time
• scheduled_activities_processor_errors - Errors during processing

8. Reaper Metrics

• reaper_items_requeued - Items requeued from expired leases
• reaper_cycle_duration - Reaper cycle time
• reaper_errors - Reaper errors

9. Activity Type Metrics (Labeled/Tagged)

• activity_type_{type}_completed - Completed by type
• activity_type_{type}_failed - Failed by type
• activity_type_{type}_duration - Execution duration by type
• activity_type_{type}_retry_count - Retries by type

10. Idempotency Metrics

• idempotency_key_conflicts - Idempotency conflicts
• idempotency_key_reused - Key reuse events
• idempotency_evaluation_duration - Idempotency check time

11. Error Classification

• error_type_retryable - Retryable errors
• error_type_non_retryable - Non-retryable errors
• error_type_timeout - Timeout errors
• error_type_handler_not_found - Missing handler errors

12. Throughput and Rate Metrics

• activities_per_second - Throughput rate
• activities_completed_per_second - Completion rate
• activities_failed_per_second - Failure rate

Implementation Requirements

Phase 1: Core Infrastructure (High Priority)

1. Extend MetricsSink trait to support:

   • Gauges (for current values like queue depth)
   • Histograms (for duration distributions)
   • Labels/tags (for activity_type, priority, etc.)

2. Add duration tracking at key points:

   • Activity execution start/end
   • Queue wait time (enqueue → start)
   • Redis operation timings

3. Add gauge updates for:

   • Queue depths (pending, processing, scheduled, dead_letter)
   • Worker counts (active, idle)
   • Priority breakdowns

Phase 2: Enhanced Metrics (Medium Priority)

1. Add labeled metrics for activity type breakdown
2. Add histogram metrics for latency distributions
3. Add rate calculations for throughput metrics

Phase 3: Advanced Metrics (Low Priority)

1. Add Redis operation metrics
2. Add processor-specific metrics (scheduled, reaper)
3. Add idempotency metrics

API Design Considerations

Option 1: Extend Current Trait

pub trait MetricsSink: Send + Sync + 'static {
    fn inc_counter(&self, name: &str, value: u64);
    fn observe_duration(&self, name: &str, duration: Duration);
    
    // New methods
    fn set_gauge(&self, name: &str, value: f64);
    fn observe_histogram(&self, name: &str, value: f64);
    fn inc_counter_with_labels(&self, name: &str, value: u64, labels: &[(&str, &str)]);
}

Option 2: Separate Traits

pub trait CounterSink: Send + Sync + 'static { ... }
pub trait GaugeSink: Send + Sync + 'static { ... }
pub trait HistogramSink: Send + Sync + 'static { ... }

Example Metric Names (Prometheus-style)

runnerq_activity_enqueued_total{activity_type="send_email",priority="high"}
runnerq_activity_completed_total{activity_type="send_email",status="success"}
runnerq_activity_duration_seconds{activity_type="send_email",quantile="0.95"}
runnerq_queue_depth{queue="pending",priority="critical"}
runnerq_worker_active_count
runnerq_activity_retry_count{activity_type="send_email",attempt="1"}
runnerq_redis_operation_duration_seconds{operation="enqueue"}

Priority Metrics to Implement First

1. ✅ Activity execution duration (histogram)
2. ✅ Queue wait time (histogram)
3. ✅ Queue depth gauges (by status and priority)
4. ✅ Activity type breakdown (labeled counters)
5. ✅ Worker utilization (gauge)
6. ✅ Retry attempt distribution (histogram)

Testing Considerations

• Ensure metrics don't impact performance significantly
• Test with NoopMetrics to verify zero overhead
• Add integration tests for metrics collection
• Verify metrics are correctly labeled and aggregated

Documentation

• Update README with new metrics
• Add examples for Prometheus integration
• Document metric naming conventions
• Provide guidance on metric interpretation

Related Issues

• Consider if this relates to any existing observability work
• May need to coordinate with UI dashboard updates

Acceptance Criteria

• All Phase 1 metrics are implemented and tested
• MetricsSink trait supports gauges and histograms
• Duration tracking is added at all key points
• Queue depth gauges update in real-time
• Activity type labels work correctly
• Documentation is updated
• No performance regression from metrics collection
• Example Prometheus implementation is provided