Frame Processor Development

Relevant source files

This document provides technical guidance for developers creating new frame processors for the Deep-Live-Cam system. Frame processors are modular components that transform video frames, implementing operations such as face swapping, face enhancement, or other image processing tasks.

For information about the overall processing pipeline architecture, see Processing Pipeline Architecture. For details on face-specific processing implementations, see Face Swapping Implementation and Face Enhancement System.

Frame Processor Interface Specification

All frame processors must implement the FRAME_PROCESSORS_INTERFACE contract defined in modules/processors/frame/core.py17-23 The interface consists of five required methods:

Method	Purpose	Required
`pre_check()`	Verify prerequisites and download models	Yes
`pre_start()`	Validate readiness before processing	Yes
`process_frame(source_face, temp_frame)`	Process a single frame (legacy/internal)	Yes
`process_image(source_path, target_path, output_path)`	Process a static image	Yes
`process_video(source_path, temp_frame_paths)`	Process video frames	Yes

Each processor must also define a NAME module-level constant for identification in status messages modules/processors/frame/face_swapper.py26

Validation and Security

The core system enforces strict validation through the ALLOWED_PROCESSORS whitelist modules/processors/frame/core.py25-30:

Any processor not in this whitelist will be rejected during loading modules/processors/frame/core.py32-35

Sources: modules/processors/frame/core.py16-44 modules/processors/frame/face_swapper.py26

Frame Processor Lifecycle

Dynamic Loading and Unloading

The set_frame_processors_modules_from_ui() function modules/processors/frame/core.py57-81 enables runtime modification of active processors based on the modules.globals.fp_ui dictionary:

Loading: When fp_ui[processor_name] == True, the processor is loaded via load_frame_processor_module() and added to FRAME_PROCESSORS_MODULES modules/processors/frame/core.py61-67
Unloading: When fp_ui[processor_name] == False, the processor is removed from the active list modules/processors/frame/core.py73-79
Synchronization: Changes to fp_ui trigger immediate updates to modules.globals.frame_processors modules/processors/frame/core.py66-67

Sources: modules/processors/frame/core.py32-81 modules/processors/frame/face_swapper.py47-81

Creating a New Frame Processor

Step 1: File Structure and Naming

Create a new Python module in modules/processors/frame/ with a descriptive name (e.g., my_processor.py). The module name must be added to ALLOWED_PROCESSORS in modules/processors/frame/core.py25-30

Step 2: Implement Required Interface Methods

Sources: modules/processors/frame/face_swapper.py47-81 modules/processors/frame/core.py17-23

Step 3: Implement `pre_check()`

The pre_check() method downloads required models and creates necessary directories. Example pattern from face_swapper modules/processors/frame/face_swapper.py47-65:

Key Responsibilities:

Create model storage directories with proper error handling modules/processors/frame/face_swapper.py51-56
Download models using conditional_download() utility modules/processors/frame/face_swapper.py59-64
Return False if any prerequisite fails.

Sources: modules/processors/frame/face_swapper.py47-65

Step 4: Implement `pre_start()`

The pre_start() method validates readiness before processing begins. Example from face_enhancer modules/processors/frame/face_enhancer.py57-63:

Key Responsibilities:

Verify model files exist at expected paths modules/processors/frame/face_swapper.py70-74
Initialize models or processing resources via functions like get_face_swapper() modules/processors/frame/face_swapper.py77-79
Use update_status() from modules.core for user feedback modules/processors/frame/face_swapper.py10
Return False to halt processing if requirements not met.

Sources: modules/processors/frame/face_swapper.py68-81 modules/processors/frame/face_enhancer.py57-63

Step 5: Implement Processing Functions

Processors must handle frame extraction and transformation logic. For face enhancement, this involves aligning the face modules/processors/frame/face_enhancer.py120-150 running inference modules/processors/frame/face_enhancer.py66-117 and pasting it back modules/processors/frame/face_enhancer.py165-214

process_video(source_path, temp_frame_paths)

Delegates to core video processing infrastructure modules/processors/frame/core.py109-114:

Sources: modules/processors/frame/core.py109-114 modules/processors/frame/face_enhancer.py120-214

Multi-Processing Architecture

The core system provides parallel processing infrastructure through multi_process_frame() modules/processors/frame/core.py83-107:

Batch Processing Strategy

The system processes frames in batches to manage memory efficiently modules/processors/frame/core.py87-107:

Batch Size Calculation: batch_size = max(1, min(32, len(temp_frame_paths) // max(1, max_workers))) modules/processors/frame/core.py89
Sequential Batch Execution: Each batch completes before the next begins, preventing memory overflow modules/processors/frame/core.py101-102
Parallel Frame Processing: Within each batch, frames process in parallel using ThreadPoolExecutor modules/processors/frame/core.py91
Error Isolation: Individual frame failures don't halt overall processing modules/processors/frame/core.py104-106

Sources: modules/processors/frame/core.py83-107

Hardware Acceleration Integration

Frame processors should utilize hardware acceleration when available. The get_face_swapper function demonstrates provider-specific configuration modules/processors/frame/face_swapper.py84-147:

Platform-Specific Optimizations

Apple Silicon: Uses optimize_for_coreml to rewrite ONNX nodes for the Neural Engine modules/processors/frame/face_swapper.py105-107
NVIDIA (CUDA): Prefers FP16 models for faster inference and lower bandwidth modules/processors/frame/face_swapper.py89-94
GPU Blending: Utilizes functions like gpu_gaussian_blur() and gpu_add_weighted() from modules.gpu_processing for high-performance blending modules/processors/frame/face_swapper.py19

Sources: modules/processors/frame/face_swapper.py84-147 modules/processors/frame/face_enhancer.py66-117

Best Practices

Thread Safety

Thread Locks: Protect global singletons during initialization modules/processors/frame/face_swapper.py87
Semaphores: Use threading.Semaphore() to limit concurrent access to sensitive resources modules/processors/frame/face_enhancer.py22

Performance

Caching: Cache expensive operations like alpha mask generation modules/processors/frame/face_swapper.py165-174 or face detection results modules/processors/frame/face_swapper.py35
In-Memory Processing: For video, prefer process_video_in_memory to eliminate disk I/O bottlenecks modules/processors/frame/core.py117-127
Tight Bounding Boxes: When pasting back results, operate on a tight crop rather than the full frame to save computation modules/processors/frame/face_enhancer.py175-202

Interface Compliance

Before submitting a new frame processor, ensure it handles both image and video target paths as defined in pre_start modules/processors/frame/face_enhancer.py58-63

Sources: modules/processors/frame/face_swapper.py25-40 modules/processors/frame/face_enhancer.py21-42 modules/processors/frame/core.py117-127