Overview

Modular Diffusers brings a compositional approach to building diffusion pipelines by breaking workflows into self-contained blocks—text encoding, image encoding, denoising, and decoding—that can be independently run, swapped, or removed. Rather than rewriting entire pipelines from scratch, developers can now compose tailored workflows by connecting pre-built blocks together.

Key Features

Flexible Block Composition

• Blocks are self-contained units with defined inputs, outputs, and computation logic
• Dynamically recompose blocks: add, remove, or swap components without rewriting code
• Run any block independently as its own pipeline or as part of a larger workflow
• Maintains the familiar DiffusionPipeline API for backwards compatibility

Custom Block Creation Developers can define custom blocks by specifying:

• expected_components: Model dependencies and Hub repository sources
• inputs and intermediate_outputs: Data types flowing in and out
• __call__: Custom computation logic executed during inference

The example demonstrates a DepthProcessorBlock that extracts depth maps using Depth Anything V2, which can be inserted into existing workflows like Qwen's ControlNet.

Memory Management & Lazy Loading

• ComponentsManager enables sophisticated memory management strategies
• .load_components() configures dtype, quantization, and device placement separately from pipeline definition
• Lazy loading prevents model weights from being unnecessarily loaded until needed

Integration & Visual Workflows

The framework integrates with Mellon, a node-based visual interface that allows users to wire Modular Diffusers blocks together without writing code. This enables both programmatic composition and visual workflow design.

Getting Started

The quickstart example shows running inference with FLUX.2 Klein 4B:

pipe = ModularPipeline.from_pretrained("black-forest-labs/FLUX.2-klein-4B")
pipe.load_components(torch_dtype=torch.bfloat16)
image = pipe(prompt="a serene landscape at sunset").images[0]

Documentation and examples are available in the Modular Diffusers docs.